The Ops Inflection, Verified: A Field Check on the Three Patterns Moving the TOIL Wall
A paper at WWW 2025 made a clean empirical case that constraining an LLM-based RCA agent with Standard Operating Procedures improves accuracy. The reasoning is sound: LLMs hallucinate, and SOPs pin them down at the junctures where hallucination hurts most.
Through 2026, nearly every commercial AI SRE product shipped the opposite bet. Cleric, Resolve.ai, Traversal, Datadog's Bits AI, Microsoft's Azure SRE Agent — all of them investigate incidents by generating hypotheses and testing them against evidence. One industry survey puts it flatly: modern AI SREs "are not runbooks that execute predefined steps; they are agents that plan multi-step investigations based on the specific alert."
The academics proved constraints help. The vendors bet on reasoning. Both sides have data, and both can't be right. Unless they're right about different points on a curve — and the curve is the interesting part.
In The Ops Inflection, I described three architectural patterns moving the TOIL wall: principles-based agents, knowledge graphs with complete signal access, and independent validation agents. That was a field report from teams I know. Since then I went checking the industry at large. All three patterns are out there, at very different maturity levels. And underneath all three, the same argument keeps happening.
Layer 1: Control Flow — Principles vs. SOPs
The contradiction above is the control-flow version of the argument.
The SOP camp has real evidence. Flow-of-Action showed that distilling SRE diagnostic steps into explicit procedures and imposing them at key junctures reduces irrational action selection. Microsoft's RCACopilot uses predefined handlers to collect diagnostic data and lets the LLM analyze what the handlers gathered. Constrain the model where it's weakest. Harvest its judgment where it's strongest.
The principles camp shipped anyway. Cleric generates multiple root-cause hypotheses per alert and works through a transparent hypothesis tree. Resolve.ai describes its investigations as forming and testing theories "like an engineer." Google's own framework for AI in reliable operations doesn't prescribe procedures at all. It defines guardrails, autonomy levels, and progressive authorization, with humans moving "up the abstraction ladder" to become architects of the boundaries. Anthropic's guidance on building agents lands in the same place: strong heuristics over brittle hardcoded branching.
Here's the resolution, and it's not a compromise. The two camps are answering different questions. SOPs answer: how do I stop today's model from hallucinating mid-investigation? Principles answer: how do I handle the incident no checklist anticipated? The first question matters exactly as long as model capability makes it matter. The second question is permanent.
That makes SOP constraints scaffolding, not architecture. Useful, load-bearing even — and temporary by design. Every capability jump shrinks the set of junctures where the model needs pinning down. The checklist handles what you've already seen. The framework handles what you haven't. And the ratio between those two workloads moves in one direction only.
There's a deeper version of this. An SOP is frozen learning: someone once investigated incidents, distilled what worked, wrote it down — and the learning stopped. A principles-based agent with a feedback loop is the same distillation, still running. Research frameworks like ACE make the loop explicit: a Generator produces investigation traces, a Reflector distills what worked and what didn't, a Curator folds the lessons into an evolving playbook.
The hard part isn't extraction. It's validation. A bad case memory only pollutes lookalike incidents; a bad principle generalizes to every future one. The teams that get this right will treat their incident history as a regression suite for candidate principles — replay past incidents with the new principle in place, check whether diagnosis improves, admit it only if it does. The principle is the spec. The incident history is the test suite.
So the argument at this layer resolves to a bet on capability trajectory. Building for today's model? Constrain it. Building for the model eighteen months out? Give it the framework, and spend your engineering on the feedback loop.
Layer 2: Data — The Graph Engine vs. the Reasoning Agent
The knowledge-graph pattern looked like the least controversial of the three. It turned out to contain the same argument, one layer down.
Half of the pattern is old. Dynatrace has run topology-driven root cause analysis for a decade: Smartscape models the full dependency graph in real time, and the Davis engine traverses it deterministically — visiting affected entities, evaluating findings, establishing causality along known edges. Auditable, reproducible, and bounded by exactly the causal relationships someone encoded.
The other half is new. By early 2026, the major observability vendors had shipped MCP servers: Datadog, Grafana's full suite covering logs, metrics, traces, and on-call, Red Hat's obs-mcp for OpenShift, IBM Instana. The manual correlation bottleneck — an engineer jumping between four query languages to stitch together a timeline — is becoming an agent pulling any signal it needs at diagnosis time. The "complete signal access" half of the pattern went from custom plumbing to vendor standard in about a year.
The argument reappears where the two halves meet. One school treats the graph as the reasoner: causality lives in the engine, deterministic and explainable, AI as a presentation layer on top. The other school treats the graph as context: the agent reads the topology, pulls the signals, and does the causal reasoning itself. Research is pushing toward the second. UModel proposes modeling observability data as object-centric semantic graphs built for agent consumption — the data substrate itself, redesigned for a reasoning consumer instead of a human dashboard reader.
Same trade as Layer 1. The deterministic engine handles exactly the causal patterns it encodes, and nothing else. The reasoning agent handles novel relationships, and inherits hallucination risk. As models get better at causal reasoning over structured context, the value of hard-coding causality into the engine declines. The trajectory rhymes.
Layer 3: Verification — The Human Is Still the Validator
The third pattern — an independent agent, initialized fresh, checking the diagnosis before mitigation — is the one I expected to find everywhere. The bias problem it solves is now well documented. Anchoring in LLMs has been measured directly: early conclusions in a context window disproportionately steer everything after. The failure mode has a name — context poisoning. And there's a subtler finding, from a verification framework called MARCH: a verifier that can see the first agent's answer degrades into checking internal coherence instead of checking evidence. Blinding isn't a nice-to-have. It's the mechanism that makes verification work at all.
The design pattern even has a name now — the verifier pattern. A dedicated agent that never sees the generator's reasoning, only the claim and the raw evidence, so it can't rationalize away a discrepancy the way the generator might.
What I found in shipped products: almost nothing. New Relic's SRE agent states its position plainly — "Autopilot recommends actions but does not take them. A person reviews and acts." Cleric keeps its agent read-only and shows a transparent hypothesis tree for a human to judge. Across the market, the validation layer exists, and it's made of people.
It's worth being precise about what a validation agent actually does, because "fresh context" gets misread as "knows nothing." The validator doesn't get the first agent's reasoning path. It does get the claim — root cause X, evidence E1 through E3, this timeline — and the same raw data access. Its job is not to redo the investigation. Diagnosis is open-ended search across a huge hypothesis space. Validation is checking one structured claim against evidence. Does E1 actually appear in the logs? Does the timeline hold, or did correlation get promoted to causation? What else was anomalous in the same window that nobody ruled out?
The first agent spends twenty minutes searching. The validator spends two minutes checking. But a refutation carries the same weight as the original diagnosis, because the validator's errors are uncorrelated with the searcher's. It never walked the path that produced the bias.
That's also why the current human-in-the-loop arrangement works: a human reviewer is a fresh-context validator with excellent priors. And it's why the arrangement doesn't survive autonomy. The moment AI triggers mitigation on its own, the human leaves the real-time loop, and something has to replace the checking function the human was quietly performing. The validation agent is that something. It's the role that stays in the loop on our behalf.
The Same Argument, Three Times
Lay the three layers side by side:
| Layer | Deterministic side | Reasoning side | Who's winning |
|---|---|---|---|
| Control flow | SOP-enhanced execution | Principles-based investigation | Reasoning, in products; deterministic, in papers |
| Data | Graph engine traverses encoded causality | Agent reasons over graph-as-context | Deterministic incumbent, reasoning ascendant |
| Verification | Human reviews and acts | Blinded validation agent | Deterministic (humans), by default |
It's one argument, recurring: encode the knowledge into fixed structure, or hand the structure to a reasoner and let it work. Every layer sits at a different point on that spectrum today. Every layer is drifting the same direction, at the speed model capability allows.
The drift isn't ideology. It's the conversion I keep writing about — open problems becoming closed ones. A checklist, a hard-coded causal graph, a human approval gate: each closes a problem by freezing the answer. A reasoning agent with principles, live topology, and a blinded validator closes the same problems without freezing them. That's the only approach that survives contact with incidents nobody froze an answer for.
The Leading Indicator
One cell in that table is lagging the others, and I think it's the one to watch.
Validation is the least productized of the three patterns — not because it's hard, but because humans are still doing the job for free. Every vendor's roadmap says autonomous remediation is coming. None of them can ship it while their verification layer is a person, because removing the person removes the check. The order of operations is forced: production-grade validation agents first, autonomous mitigation second.
The honest caveat: a validation agent is not a free pass. A validator built on the same base model shares its blind spots — fresh context decorrelates the path, not the model. If neither agent understands an obscure protocol, both miss it together. Production-grade here means more than a second prompt. It means a different model, or a different lens per checker — one on causality, one on timelines, one hunting counter-examples. That's a higher bar than it sounds, which is part of why nobody has shipped it yet.
Which gives you a concrete thing to watch for. Ignore the demos of agents restarting pods. Watch for the first vendor that ships a blinded, fresh-context validation agent as a first-class product feature. That's the vendor about to let go of the human gate — and that's the moment the phase table from the last post advances a row.
Diagnosis got good because teams did the knowledge work. Autonomy will arrive when verification does. The overnight shift doesn't end when the agent can act. It ends when something we trust is checking the agent's work — and that something won't be us.