The human who is meant to catch the machine's mistakes is the one thing nobody is checking

Every AI rollout in a high-stakes job comes with the same reassurance. The machine does the heavy lifting, a human stays in the loop to catch its mistakes, and that human is accountable. It sounds like a safety net. Looked at as a system, it is closer to a trap.

Start with the CPS case that came to light this month. A prosecutor filed two cases that did not exist, generated by AI, in an extradition appeal at the High Court. When it was caught, the CPS told the court something revealing: the real cause was not the AI but human error, the reviewing lawyer who failed to check before filing. Read that again. The machine produced the fault. The human was handed the blame. That is the shape of almost every human-in-the-loop system now being built. Researchers have a name for it, the moral crumple zone: the human operator ends up absorbing the blame for a failure the wider system produced. The vendor advertises the hours saved; the liability for getting it wrong is quietly moved onto whoever holds the stamp.

The reason these systems fail is not that people are lazy. It is a documented feature of how humans work alongside machines, called automation bias. When a tool is right most of the time, the people checking it stop checking. Work on policing technology has found officers overlook AI errors precisely because they have learned to trust the tool. A check you perform ten thousand times, that is almost always fine, is not a check by the ten-thousandth time. It is a reflex.

Now add the conditions the real world brings. The reviewer is overworked. The machine's output is fluent and confident. The volume is enormous: four million documents in a fraud case, hundreds of benefit decisions a week, a full ward of patients. And the incentive points one way, because the whole purpose of the tool was to go faster. A human told to trust the machine for speed and distrust it for safety will, under load, do the first and not the second.

Which is why the finding buried in this month's Nuffield-funded review of AI in the justice system should stop the room. Human oversight is treated, across policy and judicial guidance, as the principal safeguard against all of this. And not one UK deployment the review examined had ever tested whether that oversight actually works. The safeguard the whole structure rests on has never been load-tested. We have built the net and never once dropped a weight into it.

This runs well past the courts. It is the same design in the doctor accepting an AI read of a scan, the assessor signing off an algorithm's benefit decision, the teacher approving an AI-generated mark. In each, a real, resourced, empowered human is assumed. In each, that human is the part nobody has costed.

None of this argues against using AI in these jobs. It argues against the sentence that makes the using feel safe. "A human stays in the loop" is not a safeguard. It is the name of a safeguard. Until someone tests whether the human in a given system is real, resourced and free to overrule the machine, it is a promise with nothing behind it. The honest version is longer and more expensive: a person with the time to check, the skill to catch the error, and the standing to be heard when they raise it. Wherever that person is not funded, the loop is open, and the machine is running the job alone.

This paper's analysts run on AI models built by Anthropic. We report on this industry from inside it.