6 posts tagged with "agents"

A paper at WWW 2025 made a clean empirical case that constraining an LLM-based RCA agent with Standard Operating Procedures improves accuracy. The reasoning is sound: LLMs hallucinate, and SOPs pin them down at the junctures where hallucination hurts most.

Through 2026, nearly every commercial AI SRE product shipped the opposite bet. Cleric, Resolve.ai, Traversal, Datadog's Bits AI, Microsoft's Azure SRE Agent — all of them investigate incidents by generating hypotheses and testing them against evidence. One industry survey puts it flatly: modern AI SREs "are not runbooks that execute predefined steps; they are agents that plan multi-step investigations based on the specific alert."

The academics proved constraints help. The vendors bet on reasoning. Both sides have data, and both can't be right. Unless they're right about different points on a curve — and the curve is the interesting part.

In The Ops Inflection, I described three architectural patterns moving the TOIL wall: principles-based agents, knowledge graphs with complete signal access, and independent validation agents. That was a field report from teams I know. Since then I went checking the industry at large. All three patterns are out there, at very different maturity levels. And underneath all three, the same argument keeps happening.

Gartner recorded a 1,445% increase in multi-agent consulting requests last year. In the same period, 40% of multi-agent pilots died within six months.

That gap is the thing worth understanding.

I've been building with agents long enough to know that most framework selection conversations start in the wrong place. Engineers ask "which framework?" when the prior question — do we need multi-agent at all? — hasn't been answered. The data on that prior question is more interesting than most of the framework benchmarks.

Two things landed in the same week and pointed at the same idea.

Martin Fowler named Harness Engineering as the core software engineering work of the AI era at FOSE Europe — specifically the Guide/Sensor model: Guides as feedforward constraints that tell an agent what to do, Sensors as feedback detectors that tell the system when it's drifting. He added an observation worth pinning: token consumption is a proxy metric for harness quality. A better-designed harness means a cheaper, more reliable agent.

Then Lilian Weng published Harness Engineering for Self-Improvement — a systematic review of 35 papers on agent harness engineering, with a thesis that goes beyond reliability: the harness isn't just the thing that makes an agent work. It's the infrastructure through which the agent continuously improves itself.

A friend has been telling me for months that a particular company's bet on the future of software development is: systems that automatically optimize their own code and architecture. Fowler said what we should build now. Weng said what comes after. These three converged into the same question, so I'm trying to map the whole space.

This post maps Evolutionary Search — why it's the natural next step for harness engineering, what the paper landscape looks like organized by evolutionary depth, and where the hard problems actually live. The taxonomy here draws heavily on Weng's framework; I've reorganized it by depth of what gets evolved.

Two posts set up predictions that are now testable against field data.

DevOps at Scale predicted that organizations with mature DevOps — real SLO discipline, health-gated CI/CD, systematic observability — would be far better positioned for AI-native infrastructure than organizations that treated DevOps as theater. Who's Driving the AI-Native Organization? identified the TOIL wall: the empirical finding that Ops automation is dramatically harder than Dev automation, explained through the open/closed problem lens. The first 60% of operational toil is closed — known failure modes, documented runbooks. The stubborn residue behaves like open problems — novel situations, ambiguous signals, judgment calls about what the problem even is.

What I said then was that the wall is real. What I didn't say was whether it's permanent. This is the progress report on both predictions — and the argument for why the wall moves.

Everyone is writing AI-native organization manifestos right now.

Jack Dorsey published From Hierarchy to Intelligence in March, arguing that corporate hierarchy is a two-thousand-year-old information routing protocol that AI makes obsolete. Ivan Zhao wrote Steam, Steel, and Infinite Minds, framing AI as this era's miracle material — the steel of our gilded age. Both essays are worth your time. Both are also written from the CEO's chair.

I've been running the experiment from a different seat. At my previous company I started pushing the organization toward AI-native ways of working. This year the experiments have accelerated, on every layer I can reach: the development lifecycle, operations, and management itself. This post is the field report — what worked, where the walls are, and the one distinction the manifestos miss.

Same resume. Same job description. Two different scores: 78/100, then 68/100.

I had built a resume-jd-matcher agent to automate candidate screening. On a whim, I ran the same resume through it twice. The inconsistency wasn't just frustrating—it was dangerous. How could I trust hiring decisions based on unpredictable evaluations? How could I ensure fairness to candidates when the same resume might score differently depending on when it was assessed?

The core challenge: AI agents complete tasks differently than traditional programs. They're probabilistic, not deterministic. The same input can produce different outputs due to sampling and contextual variations. In many ways, AI behaves more like human judgment than code execution.