Error Budgets, Not Validators

I read a piece on agentic AI governance this weekend and halfway through it I had a small embarrassed moment. The author was making an argument I'd already lived through (validators stacked on validators, the autonomy squeeze, error budgets borrowed from SRE) and the answer was sitting there in my undergraduate notes from telecommunications class.

Stacking probabilistic systems doesn't make them deterministic. It makes them probabilistic in more places. Bit error rates compound through a chain. Seeding a pseudo-random generator from another pseudo-random generator doesn't add entropy. None of this is news. I just forgot to apply it for about six months.

Cairn, and the watchers I deleted

Earlier this spring I had a multi-agent system that almost worked. Cairn was supposed to test PRs in Github. Plan agent. Execute agent. Verify agent. A separate agent watching all three. I was proud of it.

It didn't work, not in the dramatic way. The agents didn't delete production. They just kept doing the wrong thing in slightly different ways. The plan agent would skip steps. The execute agent would invent endpoints that didn't exist. The verify agent would confidently approve nonsense. And the watcher? It mostly agreed with whatever the agents said. So I added more watchers.

Then I deleted them all.

The watcher problem isn't that the watcher might be malicious or might conspire (which is where the article goes, with its hall-of-mirrors framing). The watcher problem is that the watcher is the same kind of thing as the agent. Same training distribution. Same biases. Same hallucinations sound plausible to both. You can pick a different vendor (and you should), but you're still in the same epistemological swamp. There's no firm ground under there.

The article lands on multi-layer adversarial validators as the best available answer. They help, but they don't compose into certainty. The seed problem from telecom class is the same problem here: you can chain estimators all day, you don't get truth.

The actual question is the error budget

This is where the SRE framing in the article does real work, and I think it deserves more weight than the author gives it.

SREs don't ship perfect systems. They ship systems that miss their service level targets in known, bounded ways. The error budget is what's left between the threshold you commit to and 100%. You spend it deliberately. When it's gone, you stop shipping risk and start shipping reliability.

Photo by Getty Images on Unsplash

That framing is the thing that gets the AI governance conversation out of its for-or-against rut.

Some contexts have an error budget close to zero. Medical devices. State secrets. Financial settlement. Anywhere the consequence of a single miss is unrecoverable, or the accountability has to live with a named human. In those contexts, HITL isn't optional and the autonomy squeeze isn't a complaint, it's a requirement. You accept the cost.

Other contexts have a generous budget. My personal assistant is in this category. The worst case for a misclassified email or a bad draft reply is a wasted morning. Not a deleted patient record. I can run an automation that's right 95% of the time and design around the misses. Different budget, different design.

The mistake in the discourse (the one I had to bonk my head on) is treating these as the same problem. "Should we let agents loose or cripple them?" is the wrong question. Cripple them where the consequences require it. Let them run where the budget can absorb the misses. Most of the work is figuring out which one you're in.

Two architectural moves that follow

Once you frame it that way, two design moves drop out.

The first is making the surface smaller. The "autonomy squeeze" the article worries about isn't a future problem for me. It's the only thing that worked. Cairn's pipeline gave up the romantic dream of an agent that figures things out, and I got back a system I can debug. Less impressive demo, more shipped tests. That's a deliberate trade against my error budget, not a failure of imagination. When something goes wrong, I can read the code, find the LLM call that misbehaved, and either fix the prompt or replace the call with deterministic logic. With agents I was reading reasoning traces and guessing.

The second is architectural enforcement instead of written rules. I wrote about this elsewhere (The Model I Trusted Broke First), so I'll keep it short. Instructions get acknowledged and ignored. If a tool is in the model's toolkit, assume it might be used regardless of what your system prompt says. The only restriction that holds is the one architecture enforces. Restricted tool sets, sandboxes, draft-only outputs. The boundary isn't a sentence in a prompt. It's the absence of the API call.

Human-in-the-loop is a design problem, not a binary

The article's harshest claim is that HITL fails because of automation bias. It's right that monitoring fails. Vigilance fatigue is real. Pilots have crashed planes because of it. The Air France 447 report is required reading if you think passive monitoring is a serious mitigation.

But monitoring isn't the only HITL design. My colleague Isabel Evans introduced me to the human factors literature on this, in a webinar we did together. Cummings on automation levels (a spectrum from full human control to full technology control, not a switch). Nemeth on the difference between humans as monitors and humans as deciders. Two different cognitive jobs. Watching a stream of mostly-correct actions hoping to catch the bad one is a job humans are bad at. Reviewing a proposal in your own editor and choosing whether to approve it is a job humans are quite good at.

My PA proposes things. I review them in Emacs, in org-mode, with my keys and my muscle memory. I edit some, reject most, approve a few. The friction is the feature. The interface is the architecture. The human stays the decider, not the monitor.

That's the part that keeps getting missed in the agentic governance conversation. HITL doesn't fail because humans can't review. It fails when the design quietly turns humans into monitors.

What I keep coming back to

So the question I'd push back to the article is this. When someone asks whether AI agents are safe, the answer isn't "yes, with validators" or "no, lock them down." The answer is what's your error budget, and what's the design that respects it.

For some workflows, that means agents with adversarial validators and a 99% threshold and a written-off margin. For some, it means a pipeline with narrow LLM calls and schema enforcement. For others, it means a suggestions system with the human as decider, and a small enough surface that the worst case is wasted time.

None of this is binary. None of it is a question of for or against. It's a question of context.

Photo by Blaz Erzetic on Unsplash

That's the work.