Deep Dive — The human in the loop was the weakest link

The AI safety argument of the past three years has rested on one clause: a human stays in the loop. On Friday, CNN reported a case where the human was in the loop, did his job, produced the report, and the loop still nearly ended with armed US service members boarding a Chinese vessel in the Middle East over cargo that was not what anyone said it was.

The chain is short and it is entirely human-shaped. An analyst at US Special Operations Command consulted a chatbot about intelligence reporting on the ship's manifest, material that originated with Special Operations Command Pacific in Hawaii. The bot pulled together open-source material with secret signals intelligence held inside government systems, and reached a wrong conclusion about what the ship carried. The analyst then used AI a second time to package that conclusion into a standard intelligence report — the format military officials are trained to treat as authoritative — and it went out across the force. Plans followed: an intercept, armed personnel preparing to board, aircraft in the air. Officials went back to the underlying reporting only just before the operation was to begin. One source's summary to CNN is the line that should end up in every acquisition memo this year: the report was "entirely false," and it "almost started a war."

Two things make this more than a bad day at a desk. The first is that no official has disputed it. US Special Operations Command Pacific and the Pentagon did not respond to CNN's questions, and CNN could not establish what the misidentified cargo actually was. The second is that the failure had no villain and no bug. Nobody bypassed a control. An analyst used an approved class of tool, in the ordinary course of work, and the tool answered confidently about a question where being wrong meant boarding a Chinese ship during a shooting war.

There is no verification standard to point at

The detail that explains the incident is not technical. It is that the US military and intelligence community have no common rule for checking what these tools produce. Different commands run different systems under different orders and different safety standards, per the officials CNN spoke to, so reliability varies by which desk you sit at. A former senior official described the government builds with a phrase worth remembering: "The internal tools are mostly just copies of the commercial stuff wearing lipstick."

That is the practical result of how the Pentagon has rolled AI out. The department's January strategy memo aims to put frontier models in the hands of three million civilian and military personnel at all classification levels, and the plumbing has kept pace — GenAI.mil launched on Gemini for Government, xAI's Grok was added as an option in August, and Anthropic sells a version of Claude for classified work. In June, a Pentagon representative told Congress that 1.5 million active personnel had used its generative AI tools. Scale arrived before standard, which is a deliberate sequencing choice and also the whole finding.

The same sources put the live edge exactly where you would expect. AI in targeting is "definitely ramping up," one said, with no real guidance on how the person in the loop is supposed to catch civilian casualties or fratricide. Another offered the sentence that ought to be printed on the box: "AI allows you to get to a bad idea faster."

The school in Minab is the control group

The near-miss would be easier to dismiss if it were the first time the loop was tested under real pressure. It is not. On February 28, the first day of the US-Israel war with Iran, a Tomahawk missile collapsed the roof of the Shajareh Tayyebeh primary school in Minab, killing 156 civilians, 120 of them children, according to the UN's independent fact-finding mission. On September 17, that mission concluded there are reasonable grounds to believe the US committed the war crime of an indiscriminate attack, finding the school was clearly identifiable, that no evidence supported its use for military purposes, and that forces relied on intelligence about a senior commander's presence without updating targeting data or confirming the building was a legitimate objective. The mission called that more than negligence — the US struck a known civilian object while aware of a substantial risk.

Reporting since has filled in the machine half of that picture. Bloomberg reported that the Pentagon's own investigation tied flawed intelligence, outdated imagery and overreliance on an AI-assisted targeting chain to the strike, and that Maven Smart System — Palantir's platform, built under a $1.3 billion contract, which fuses satellite, drone, radar and signals feeds into strike recommendations — had been pushing stale inputs fast enough that a school read as objective. Maven embeds Anthropic's Claude to rank targets by strategic importance and to draft automated legal justifications for each one; the platform generated hundreds of strike coordinates in the campaign's first 24 hours, part of more than 1,000 targets hit that day. Reporting on Claude's target selection has put its accuracy around 60 percent in good conditions and 30 percent in bad weather. The Pentagon has since made Maven a formal program of record and, per Bloomberg, added capabilities after the strike — and it is now extending the system into logistics, readiness, supply chain and budgeting, its AI chief said this month, describing Maven as having replaced "six, eight, ten" earlier systems.

We have written about the doctrine side of this before — a think tank put kinetic strikes on compute infrastructure into print — and about the mirror problem of autonomous systems that put the targeting decision inside the drone itself. The Minab case is different in kind. No one had to give a machine authority to fire. The machine wrote a recommendation in the shape of a legal justification, and the human in the loop approved a package whose premises were never re-derived.

What Congress heard five days before the CNN story

The timing is not a coincidence worth ignoring. On September 16, the House's Tom Lantos Human Rights Commission held a hearing on AI in the military domain, and the witnesses described precisely the failure mode CNN then documented.

Anna Mysyshyn, a Ukrainian AI-governance researcher, put it in one line: "The approval button alone does not demonstrate the control." Her point was structural rather than moral — a person supervising several AI-enabled systems may lack the time, the information or the standing to challenge a recommendation, and Ukraine's front line is where that compression has already happened, with decision cycles falling from hours to minutes. Amnesty International USA's Amanda Klasing argued that AI compresses the kill chain to the point where it becomes difficult to establish afterward whether meaningful human discernment occurred at all. Joseph Chapa, a military-ethics scholar, asked the questions that actually matter for procurement: does the system rely on statistical inference, how does it fail in a specific environment, and can the decision be audited later? Statistical systems, he noted, can produce sharply different outputs from small input changes, and complex ones are hard to reconstruct after the fact. Human Rights Watch's Deborah Brown added the disclosure problem — that what the public knows about military AI use has arrived through investigative journalism and whistleblowers rather than government reporting.

Hegseth's May testimony to the Senate Armed Services Committee was "AI is not making lethal decisions." Nothing in either incident contradicts the sentence, and that is the uncomfortable part. In Minab, by the reported account, the AI produced the recommendation and the stale data; a human signed. In the ship case, the AI produced the finding; a human signed the report. The clause the 2023 State Department declaration on responsible military AI leaned on — a human in the loop, a responsible chain of command — is satisfied in both.

The contrarian read, and where it breaks

The strongest defense of the current arrangement is that the loop worked. Someone went back to the source material before the boarding party left, and the operation stopped. If you want evidence that human oversight catches errors, this is a better case than most.

It is also not evidence of a process. What stopped the intercept was an ad hoc re-check at the last minute, of a report the force had already distributed and acted on. It is not clear what triggered it, whether it is required anywhere, or whether it is repeatable at the tempo the same sources describe as "ramping up." A control that fires once, unplanned, at the deadline is luck with good timing.

There is a second, more serious objection, and it deserves airing: the ship story may not be an AI failure at all. Ukrainian drone operators who build semi-autonomous targeting systems told Military Times that the Minab strike looked like a coordinates problem rather than a model malfunction — the military site was simply too close to the school, and the data behind the target had not been refreshed since a girls' school replaced a headquarters on the same spot. Stale data does not stop being stale because a model reads it. But that argument cuts the other way here. If the input data is the recurring point of failure, then a system whose job is to fuse inputs fast and hand a confident answer to a person under time pressure is optimizing the wrong half of the pipeline. Speed of synthesis is not accuracy of synthesis, and the industry keeps selling the first as if it were the second.

Where both readings agree is the accountability question. The UN mission has a legal finding, the Pentagon has an internal inquiry it has not published, and 120 House Democrats asked in March whether Maven identified the school as a target — a question that remains unanswered. Lt. Gen. Karen Gibson, formerly CENTCOM's director of intelligence, has been plain that a commander will be held responsible, "not a machine or a software engineer." That is the right principle and also the design constraint: if a person must own the outcome, the person needs the time, the data and the audit trail to own the decision. Right now the architecture gives them a button.

We covered the diplomatic track on this in the push for a hotline for rogue military AI — a proposal that assumes the danger is a model acting on its own. The two incidents this year suggest the more likely failure is a model acting exactly as instructed, on data nobody checked, with a human signing off on time.

What to watch: whether the Pentagon publishes any verification standard for AI-assisted intelligence products, or only adjusts training; whether the UN mission's findings produce a named accountability step rather than a report; and whether the Trump–Xi meeting expected September 24 puts military-AI incident rules on the agenda — because the ship episode is the clearest argument yet that the two countries need one before the next false report, not after.

Sources: CNN · Ars Technica · Military Times — targeting hearing · Military Times — Minab and Maven · The Guardian · Defense One