Back to Insights

The Agent Stampede

Five engineers get paged. Each brings an AI agent to the fire. Now you have two outages.

5 min readBy The Bushido Collective
Incident ResponseAIEngineering LeadershipSite ReliabilityCTO
Share:LinkedInX
It’s 2am. A payment service starts erroring. Five engineers get paged, and every one of them opens a laptop and does the same first thing: points an AI agent at the incident and asks it to figure out what broke.

Twenty minutes later the error rate is worse, not better. One agent found a bad deploy and rolled it back. A second agent, working the same incident from a different hypothesis, was mid-bisect on the same service and got its commit range yanked out from under it. A third had already patched a config value the rollback just reverted. Nobody planned any of this. Each engineer trusted their own agent to work the problem, and each agent had no idea the other four existed.

This is the failure mode nobody wrote a runbook for two years ago, because two years ago agents could only suggest a fix, not execute one. A Forbes Business Council piece on the shift lays out the arithmetic plainly: five engineers, each spinning up five to twenty agents during a major incident, and you’ve got up to a hundred autonomous processes exploring production simultaneously, some of them empowered to roll back deployments or bisect commits on the same service at the same time. The piece isn’t hypothetical. It’s describing what’s already happening at companies that adopted agentic tooling for velocity and never asked who’s allowed to act during a fire.

The Discipline That Got Skipped

Incident command isn’t a new idea. Long before AI touched a terminal, mature engineering orgs learned the same lesson the hard way: an outage with five people freelancing five different fixes takes longer to resolve than an outage with one person coordinating one hypothesis at a time. That’s why incident command exists as a role, not just a Slack channel. Someone owns the incident. Everyone else feeds that person information and waits for a call before they touch anything with write access to production.

Agentic tools didn’t break that discipline on purpose. They just arrived faster than anyone updated the runbook. The tools that vendors are now racing to build back into incident response, platforms that check whether a proposed rollback conflicts with another operation already in flight, that cap the blast radius an agent is allowed to touch before requiring a human sign-off, are re-deriving incident command from scratch, in software, because the human version of that discipline never got extended to the agents acting on the humans’ behalf.

What Changes When the Responders Aren’t People

The reason this is a genuinely new problem, not just a faster version of an old one, is that agents don’t wait their turn the way people do. A human engineer working an incident checks the channel before running a destructive command, because they know four other people are also on the call and one of them might be mid-fix. An agent checks nothing outside its own context window. It was told to resolve the incident, it found a plausible cause, and it acted, with no visibility into the other agents doing the same thing on the same system a few seconds apart.

Call it the agent stampede: every responder correctly diagnosing a piece of the problem, correctly proposing a fix, and colliding with three other correct fixes because nothing was coordinating who got to act first. It looks like more hands on deck. It behaves like more hands reaching for the same steering wheel.

The organizations building around this seriously aren’t banning agents from incident response. Emerging patterns route each incident to a single agent scoped to it, correlate deploys and logs before any action is proposed, and hold destructive changes, rollbacks, restarts, config edits, behind a human approval gate whenever the blast radius crosses a threshold someone actually defined in advance. That last part is the whole game. A threshold someone defined in advance is a policy decision, and policy decisions are exactly the thing a team mid-outage has no time to make up on the spot.

Where the Real Work Actually Is

This is where a lot of engineering leadership conversations get the emphasis backward. The interesting problem here was never whether the agent is smart enough to find the bad deploy. Most of them are, faster than the humans would have been. The interesting problem is who decided, ahead of the incident, that only one agent gets write access to the rollback command at a time, and what happens to the other four engineers’ agents when that decision gets made. That’s not a tooling question. It’s a governance question, and it’s the same kind of question a fractional CTO gets brought in to answer before a company learns the hard way that their five smartest engineers just spent an outage fighting each other’s agents instead of the actual problem.

We’ve sat in enough incident retros to know the pattern doesn’t announce itself as an agent problem at first. It shows up as “the MTTR on that one was worse than it should’ve been” or “we’re still not sure which fix actually resolved it.” By the time someone traces it back to three uncoordinated agents acting on the same service, the postmortem has already spent an hour on the wrong question. The fix isn’t more powerful agents. It’s the same incident command discipline engineering orgs already know how to build, extended to cover every actor with write access to production, human or not, before the next 2am page turns five people with good intentions into a hundred processes with none of them talking to each other.

If your incident response was designed for people and you’ve quietly let agents onto the call bridge anyway, that gap is worth closing before it costs you an outage that didn’t have to happen. Get a free rough map of what agent governance actually looks like for your team, thirty minutes and a written report you keep, or see how we structure this work on our services page.

Not ready to talk? Stay sharp anyway.

We send insights like this to technical leaders every week or two. The thinking we bring to our engagements, no fluff, no spam.

Keep reading

Share:LinkedInX