Three PRs That Pass CI and Still Need Judgment

Claude Code can read a codebase, edit files, and write and run tests. That lets an agent produce both a change and its apparent evidence of correctness. If the tests share the implementation’s missing assumption, they can agree with each other while disagreeing with production.

The three changes need different answers. A duplicated dependency might be unnecessary upkeep. A retry policy can amplify load. A cache design depends on what the application promises about stale data. A human could have written any of them; the useful distinction is the risk each change introduces, not who typed it.

What the green checks leave open

For the retry change, follow the call through both layers. Suppose each allows three total attempts, including the initial call, and every attempt fails with an error both layers retry. Each outer attempt invokes the client’s three attempts: 3 × 3 = 9 database calls before the failure reaches the caller. AWS’s engineering account of retries explains this multiplication and how it can prolong an overload.

Approval depends on an explicit total attempt limit, a deadline, and whether the operation is safe to repeat. A timeout can arrive after a write has taken effect, so bounding the retries alone doesn’t prevent duplicate side effects. If the client’s existing policy already meets those requirements, the extra loop can go. Test a persistent failure through the client and wrapper together; substituting a one-shot stub for the retrying client would hide the multiplication you’re trying to catch.

The second HTTP library deserves a different conversation. It may supply a capability the existing client lacks. If it does, the extra dependency has a reason to exist, and its timeout and connection behavior need checking. If the existing client already does the job, reusing it avoids another library to patch and another set of behavior to maintain. Neither conclusion follows from whether the new code compiles.

For the cache change, start with the old value cached in both regions. Write a new value in one region, let the proposed invalidation run, then read through the other. If that region still returns its cached copy, clearing the first region hasn’t made the second read fresh. Starting the test with an empty cache could let it fetch the new value and miss the failure.

Whether the stale read blocks the PR depends on the application’s promise; a stale recommendation and a stale access permission carry different consequences. If bounded staleness is acceptable, approval needs evidence that this read becomes fresh within that bound, including any lag in the data it refreshes from. If the service promises immediate revocation of access, the local-only change is incomplete.

The useful engineering judgment is specific: find the assumption, decide whether it fits the service, and make the relevant behavior observable. A reviewer can ask an agent to do much of that investigation and test writing. Someone still has to check that the chosen test asks the right question.

For an immediate-freshness requirement, run the cross-region test against the local-only implementation before accepting a fix. It should expose the stale read there and accept the repaired behavior. A check that accepts both gives the reviewer no evidence about that failure, however reassuring its name.

The experienced engineers got slower

A reviewer who recognizes every risk above might still be faster writing the change directly.

GitHub’s 2022 experiment randomly split 95 professional developers into groups with and without Copilot, then asked them to write an HTTP server in JavaScript. GitHub reported roughly 55% shorter completion times with Copilot. The task was bounded, and submissions were assessed with a test suite.

METR’s July 2025 study randomly allowed or disallowed AI on 246 tasks completed by 16 experienced open-source developers in repositories they knew well. Allowing AI increased their implementation time by an estimated 19%. They primarily used Cursor with Claude 3.5 and 3.7 Sonnet, rather than Claude Code. The measure was the developers’ own implementation time, including revisions after pull-request review. Even afterward, their estimates put AI’s average effect at a speedup.

Screen recordings of a subset of METR’s tasks showed developers spending a smaller share of time actively coding when AI was allowed. Prompting, waiting on AI, and reviewing its output took up time instead. The study didn’t find a statistically significant difference in time spent revising after PR review, so blaming the slowdown on worse submitted code would go beyond its evidence.

These experiments used different tools, tasks, and acceptance conditions. Comparing their headline percentages cannot isolate the effect of seniority, and METR’s early-2025 result cannot settle what later tools will do. It does directly challenge the idea that putting an expert in the seat is enough to make the productivity gain appear.

Our inference is narrower: expertise gives a reviewer a basis for recognizing missing constraints, while the benefit of delegating a particular task still needs measuring. If prompting, checking, and correcting take more effort than doing and checking the same task directly, delegation loses on time. Verification belongs on both sides of that comparison. A capable engineer can reach that result while producing perfectly acceptable code.

Count the work after generation

The same distinction matters at team level. If changes arrive for review faster than the team can evaluate them, the queue grows. More generation can increase waiting without increasing delivery, even if every proposed change is useful.

DORA’s 2024 report asks respondents about production outcomes, including change lead time, deployment frequency, failures, and rework. Its AI analysis found an association with faster code reviews alongside poorer delivery throughput and stability. The researchers describe larger changes as a possible explanation, not an established cause. Their survey captured a moment in time rather than following an experiment, so it cannot demonstrate that AI created a review bottleneck.

Faster approval can flatter the dashboard too. The useful question is whether a change reaches users with less total effort and an acceptable failure rate. Compare similar changes with the same acceptance requirements, with and without AI. Count the author’s work through revisions, the reviewer’s work, and known production repair linked to the change.

If lower author effort is matched by extra review and repair, the total hasn’t improved. Keep waiting time separate from active work: a change can require less labor and still take longer to reach users. Changing the task mix or lowering the quality bar can make either comparison misleading, so these records need context before they become a claim about AI.

When the delay is repeated explanation of the same service constraints, an internal owner can put those constraints into executable checks and help other engineers learn the system. Building and maintaining those checks counts as work too; their value depends on avoiding enough repeated investigation or repair to justify it. When a task is already quicker to write directly, doing so remains a valid choice. Buying more agent seats only addresses the part of the work the agents can take off the team’s hands.

For the cache PR, ask the agent to exercise a write in one region and a read in the other. Before judging its result, someone on the team needs to decide how stale that second read is allowed to be. The test can expose the behavior. It can’t choose the promise your service makes to its users.