36 posts tagged with "ai"

A paper at WWW 2025 made a clean empirical case that constraining an LLM-based RCA agent with Standard Operating Procedures improves accuracy. The reasoning is sound: LLMs hallucinate, and SOPs pin them down at the junctures where hallucination hurts most.

Through 2026, nearly every commercial AI SRE product shipped the opposite bet. Cleric, Resolve.ai, Traversal, Datadog's Bits AI, Microsoft's Azure SRE Agent — all of them investigate incidents by generating hypotheses and testing them against evidence. One industry survey puts it flatly: modern AI SREs "are not runbooks that execute predefined steps; they are agents that plan multi-step investigations based on the specific alert."

The academics proved constraints help. The vendors bet on reasoning. Both sides have data, and both can't be right. Unless they're right about different points on a curve — and the curve is the interesting part.

In The Ops Inflection, I described three architectural patterns moving the TOIL wall: principles-based agents, knowledge graphs with complete signal access, and independent validation agents. That was a field report from teams I know. Since then I went checking the industry at large. All three patterns are out there, at very different maturity levels. And underneath all three, the same argument keeps happening.

Gartner recorded a 1,445% increase in multi-agent consulting requests last year. In the same period, 40% of multi-agent pilots died within six months.

That gap is the thing worth understanding.

I've been building with agents long enough to know that most framework selection conversations start in the wrong place. Engineers ask "which framework?" when the prior question — do we need multi-agent at all? — hasn't been answered. The data on that prior question is more interesting than most of the framework benchmarks.

Yuval Noah Harari recently gave a lecture at Oxford's Tano event — his sharpest articulation yet of why AI poses an existential threat to human civilization. Watch it here.

If you haven't watched it, the short version: AI is not a tool, it's an agent that makes independent decisions and can lie. It's a "native bureaucrat" that lives inside the language-based systems running civilization — law, finance, religion. It's hacking the underlying code of human culture: language itself. And if governments grant AI legal personhood, we lose accountability entirely. Harari calls this the most dangerous psychological experiment in human history.

I've been thinking about this argument since the lecture dropped. My reaction wasn't panic. It was: this skips a step.

I've spent three posts in this series comparing AI to a new kind of colleague — someone to manage with specs and verification loops, not to fear as an invader. Harari's framework, as sharp as it is, misses something engineers see clearly every day.

I've written about AI across engineering roles — No Junior Engineers?, The Senior Engineer's AI Trap, The AI-Augmented Engineering Manager, and How Ops Engineers Can Stay Relevant. I kept skipping product managers.

At work, my team has a PM. I kept thinking I was the wrong person to write about a role that belonged to someone else. Then I started building my own projects on the side. I noticed I was spending more time on product decisions than on implementation.

That was the point.

Two things landed in the same week and pointed at the same idea.

Martin Fowler named Harness Engineering as the core software engineering work of the AI era at FOSE Europe — specifically the Guide/Sensor model: Guides as feedforward constraints that tell an agent what to do, Sensors as feedback detectors that tell the system when it's drifting. He added an observation worth pinning: token consumption is a proxy metric for harness quality. A better-designed harness means a cheaper, more reliable agent.

Then Lilian Weng published Harness Engineering for Self-Improvement — a systematic review of 35 papers on agent harness engineering, with a thesis that goes beyond reliability: the harness isn't just the thing that makes an agent work. It's the infrastructure through which the agent continuously improves itself.

A friend has been telling me for months that a particular company's bet on the future of software development is: systems that automatically optimize their own code and architecture. Fowler said what we should build now. Weng said what comes after. These three converged into the same question, so I'm trying to map the whole space.

This post maps Evolutionary Search — why it's the natural next step for harness engineering, what the paper landscape looks like organized by evolutionary depth, and where the hard problems actually live. The taxonomy here draws heavily on Weng's framework; I've reorganized it by depth of what gets evolved.

I spent three weeks building my retirement plan with AI.

Not a rough sketch. A proper plan — asset allocation across account types, tax-loss harvesting sequences, Roth conversion ladders, withdrawal ordering optimized for bracket management, Monte Carlo scenarios run from first principles. By the time I was done, I had something that would have cost several thousand dollars at a fee-only advisory firm.

Then I hired a human financial advisor anyway.

I've been thinking about why ever since, because the honest answer surprised me. It wasn't that the AI plan was wrong. I couldn't find anything technically wrong with it. It was something else. The plan had no gray zones. Every decision was clean, optimized, defensible. But when I imagined actually executing it — moving real money, locking in real choices — something wouldn't let go. What I eventually realized: if this plan goes sideways in five years, the AI moves on to the next query. The advisor loses a client, maybe a reputation, maybe sleep. That asymmetry matters. I didn't know how much until I was staring at a plan I couldn't quite trust.

I ended last time with: "Treat AI like a person. The confusion mostly goes away."

I still believe that. But "like a person" is more precise than it sounds — and most people are using it imprecisely.

The mistake is treating human capability as a single axis, and AI as a point somewhere on it. Better or worse, faster or slower, ready or not. The actual shape is jagged. Which makes it considerably more useful.

The frame isn't just "AI is like a person." It's more useful than that: human capability is the map. The places where AI still can't match a person are exactly where the interesting work is.

Two posts set up predictions that are now testable against field data.

DevOps at Scale predicted that organizations with mature DevOps — real SLO discipline, health-gated CI/CD, systematic observability — would be far better positioned for AI-native infrastructure than organizations that treated DevOps as theater. Who's Driving the AI-Native Organization? identified the TOIL wall: the empirical finding that Ops automation is dramatically harder than Dev automation, explained through the open/closed problem lens. The first 60% of operational toil is closed — known failure modes, documented runbooks. The stubborn residue behaves like open problems — novel situations, ambiguous signals, judgment calls about what the problem even is.

What I said then was that the wall is real. What I didn't say was whether it's permanent. This is the progress report on both predictions — and the argument for why the wall moves.

Prompt engineering. Context engineering. Spec-driven development. Harness. Loop engineering. Context minimalism. Tokenmaxxing.

A new term lands every week. The smartest people in tech are pouring into AI, and ideas collide fast — CTOs taking IC roles at Anthropic, researchers shipping frameworks before the last one has been absorbed. The vocabulary is exciting. It's also exhausting. And some of it turns out to be wrong.

Tokenmaxxing already showed cracks. Process-heavy agent frameworks are losing ground. Every few months something that felt foundational quietly gets replaced.

I kept looking for the underlying logic. What actually persists? Then a friend said something that stopped me mid-scroll.

"Treat AI as labor. Instead of inventing new concepts for something entirely unprecedented and making it increasingly complex, design your verification and management methods according to how you'd manage people."

That one sentence turned out to explain almost everything.

Everyone is writing AI-native organization manifestos right now.

Jack Dorsey published From Hierarchy to Intelligence in March, arguing that corporate hierarchy is a two-thousand-year-old information routing protocol that AI makes obsolete. Ivan Zhao wrote Steam, Steel, and Infinite Minds, framing AI as this era's miracle material — the steel of our gilded age. Both essays are worth your time. Both are also written from the CEO's chair.

I've been running the experiment from a different seat. At my previous company I started pushing the organization toward AI-native ways of working. This year the experiments have accelerated, on every layer I can reach: the development lifecycle, operations, and management itself. This post is the field report — what worked, where the walls are, and the one distinction the manifestos miss.