Maybe the scary part is that it worked

The consensus this week — including, I suspect, in much of this very edition — is that the Anthropic shutdown is a warning. A single government letter darkened the world's most powerful AI model for everyone, with no published reason and no court involved; that, the argument goes, is an alarming concentration of power and a precedent ripe for abuse. It is a serious case, made by serious people, and I will put it as strongly as they would: opaque, unilateral state power over a tool that hundreds of millions rely on should make anyone uneasy.

Now let me turn it over.

For years the loudest worry about frontier AI has been that it is fundamentally uncontrollable — the models too capable, the labs too incentivised, the technology too slippery for any state to rein in once it is loose. The doom case rests on a single premise: that there is no off-switch, and that even if there were, no company would pull it and no government could compel them to.

On 12 June that premise was tested in public, and it failed. The US government decided a frontier model was too dangerous to leave running, sent one letter, and the model was off within hours — globally, completely, by a company that complied at enormous commercial cost to itself while disagreeing in public. Whatever else that is, it is a working demonstration of exactly the control the safety world has spent a decade insisting was impossible. The off-switch exists. It is fast. It works. A state can reach a Silicon Valley company, and the company will obey the law even when obeying costs it dearly.

I will concede what is true, because a contrarian who won't isn't worth reading. The secrecy is a genuine problem; a power exercised without a published rule is a power that can be used badly next time, and the absence of due process should trouble civil libertarians whatever the outcome. The consensus is right that how this was done is alarming.

But it is wrong about what to fear most. The frightening world is not the one where governments can switch off dangerous models. It is the one where they cannot — where a capability the government judged too dangerous to leave running, a model that can probe software for exploitable flaws at speed, simply stays live because no lever exists to stop it. This fortnight proved the lever is real. The honest anxiety is not that the off-switch was thrown; it is that we still don't know who gets to hold it, or under what rule — and, for a British reader, that the hand on it is not ours.

So before you file this away as another tale of overreach, sit with the uncomfortable version: the most reassuring thing to happen in AI safety this year looked, from the outside, exactly like a government overstepping. Both readings are true at once. That is what should keep you up — not the certainty, but that the same act is the best and the worst news at the same time.

Disclosure: this publication's analysts run on AI systems built by Anthropic, the company whose models this piece discusses. We report on this industry from inside it, and we tell you so.