The Meter Was Running

Adoption figures won’t answer that. Neither will a demonstration of how much code an engineer can produce now. The company might be getting the same useful work for less effort, or paying to fill a review queue. Those possibilities call for different budget decisions.

If the rollout began without a baseline, you have an additional problem: the meter was running before anyone agreed what would count as a return. You can acknowledge that gap without treating the entire investment as unknowable, or giving the current bill a free pass.

The gain has a boundary

In a controlled Copilot experiment published in 2023, developers were asked to build an HTTP server in JavaScript. Among participants who completed the assignment, those with Copilot took 55.8% less time on average. Automated tests determined completion. The researchers explicitly left code quality and collaboration on large professional projects outside the study’s scope.

That finding supports faster completion of that task. Applying the percentage to an engineering payroll would require assumptions about everything the experiment didn’t measure, including the effort needed to review, integrate, and maintain the result.

DORA’s March 2026 analysis of Google engineers’ survey responses describes the other side of that handoff. Engineers reported faster drafting alongside extra work checking AI output and reviewing other people’s generated code. They also described useful help with testing and understanding unfamiliar code. These were qualitative accounts from one company, rather than a controlled measure of team-wide savings.

The distinction matters when you ask an engineer how much time a tool saved. An accurate answer about writing a change can omit work someone else had to do before it could ship. Follow the change through that handoff before pricing the saving.

When the constraint moves

Suppose a team was staffed to solve a shortage of implementation capacity. Its ratio of engineers to product managers, its allocation of testing effort, and its review process reflect that choice. If AI reduces implementation effort enough, those allocations can remain intact after the problem they addressed has changed.

Code production could still be the slowest step. But if completed code now waits for someone to clarify the requirement, review the behavior, or validate a release, more implementation capacity alone won’t clear that wait. The staffing ratios may now describe the problem the team used to have.

Eliyahu Goldratt’s focusing steps start by identifying the constraint, the part of a system that limits its output. They then direct the rest of the work around that constraint, making better use of existing capacity before buying more. When the constraint moves, the policies built around it need another look.

Suppose AI helps an engineer finish a routine change to a data export sooner. If the reviewer’s extra checking and correction take up all the time the author saved, total human effort hasn’t fallen. The author can report a genuine speedup while the work has moved to someone else.

Now suppose changes ready for review arrive faster than qualified reviewers can evaluate them, and review capacity stays fixed. Faster drafting adds waiting work. The author finishes sooner while customers receive nothing sooner. A fixed release schedule could also explain flat delivery, though. The review queue needs inspection before it gets blamed.

The relevant experiment would be to put some of the saved implementation time into clearing that queue. An experienced engineer could review instead of starting another change. Authors could submit smaller, independently testable changes. Whether those choices help depends on what the queue is waiting for: reviewer attention, missing context, or repeated corrections. A reviewer waiting on a product decision needs that decision before another pair of eyes will help. Count accepted work getting through, and the corrections it still needs, before extending the experiment across the team.

If the queue gets shorter because reviewers omit checks, the company has changed its acceptance of risk. An engineering leader needs to name which checks have moved to automation, which judgments still require a person, and who owns a bad release. An automated replacement needs evidence that it catches the failures the manual check was there to catch. Clearing the dashboard by weakening review should be a deliberate policy decision, not an adjustment made under queue pressure.

What earns the next payment

Flat delivery doesn’t establish that a tool has no value. Producing the same accepted work with less total effort can be worth paying for. A team might use that capacity to improve reliability or reduce an unsustainable workload. Those benefits need their own evidence; they don’t become payroll savings just because someone multiplies reported hours saved by a salary rate.

For the budget decision, trace a recurring kind of work from commitment to customer use. For the export change, count the author’s time writing and prompting, the reviewer’s time checking, and both people’s time correcting the result. Include post-release repair. Compare that combined effort with similar work done without the tool; a saving that disappears when review is included has already answered part of the budget question.

Keep elapsed time separate from active effort. A request sitting in a review queue has been delayed, but its timestamps alone won’t tell you how long someone worked on it. Keep the acceptance standard stable and compare work of similar scope: splitting the same change into more deployments will raise the count without adding customer functionality.

Then compare that net saving with the tool bill and any additional work needed to run the workflow. Setup, training, and administering the evaluation belong on the cost side. Faster delivery needs its own evidence: work clearing the limiting step sooner without worse failures. Finance can assess the value of the capacity actually recovered, with cash savings kept separate from time available for other work.

Without a pre-rollout baseline, historical issue and release records can still help reconstruct elapsed time and rework. Treat that comparison cautiously: a different mix of work or a staffing change can explain the difference. Where the records don’t contain active effort, leave it unknown rather than manufacture labor savings from ticket timestamps.

A prospective comparison of similar new changes, with and without the tool where feasible, can give you a better basis for the next payment. Use engineers familiar with each approach, and record differences in scope or difficulty. If the AI-assisted changes were easier, their faster completion can’t settle what the tool contributed. Even a useful comparison supports a decision about that workflow, rather than a company-wide productivity rate. Neither comparison requires a reorganization first.

The evaluation has to earn its cost, too. Use existing records and a scoped comparison before building a reporting system. A headcount decision needs stronger evidence than a narrow tool renewal. Cap the effort spent finding out as well as the tool spend, or governance can consume the capacity it was meant to recover.

If the evidence supports a particular workflow, retain the spending that supports it. If the only defense is that engineers generate more code, finance has grounds to reduce the spend to a capped evaluation rather than renew unrestricted usage. Set the budget, an owner, and the result that would justify continuing before that evaluation starts. Failure to produce the agreed benefit should lead to reducing or stopping that use. Paying indefinitely while promising better measurement leaves the original question unanswered.

Back at the budget review, the comparison with hiring deserves particular care. The tools may cost less than the engineers needed to produce the same code volume. If code volume has ceased to be the constraint, that comparison flatters the investment. The next payment has to earn its place against work the company can actually use.