Name the Skill

You have a version of that question, arrived at more quietly. You bought the licenses. Somebody ran the training. A few of your people got noticeably better, a few produce confident nonsense faster than before, and when you ask what separates them, the answer is that it takes skill. Then the answer stops. You can’t hire for a capability nobody will name, you can’t train what nobody can describe, and you can’t tell this week which of your people have it.

While that argument was running, something rarer than another opinion was already sitting in a working paper: a measurement.

The output rose. The finished work did not keep up.

Fiona Chen and James Stratton of Harvard studied AI adoption inside software companies using telemetry from the engineering analytics vendor Jellyfish. Their job market paper, current version August 4, 2026, covers roughly 300 million work events from about 700,000 workers at 718 firms between January 2021 and March 2026, comparing firms by when they adopted.

Coding output rose. At firms adopting AI agents, lines of code went up 30 percent, commits 20 percent, pull requests 23 percent, all statistically significant. Then they looked at finished software, measured as resolved Jira issues and completed epics, and found small positive effects that were not statistically significant. Against that 30 percent gain in individual output, their estimates rule out any increase larger than 12 percent of the baseline.

The step that absorbed it was review. Time from a pull request being submitted to merged rose 49 percent on a baseline of 7.03 days, though that number alone proves little: more submissions arriving at the same reviewers lengthens a queue by arithmetic. The per-submission measures are the interesting ones. Formal requests for changes rose 0.12 on a baseline of 0.13, close to double, and comments per pull request rose 35 percent, while submission size did not significantly change.

That reads like worse output needing more correction, and the same paper undercuts it. Those firms also widened the bench: the share of workers who reviewed at least one pull request rose by 14 percent against itself, from a baseline of 0.29 to roughly 0.33. Our inference from there, not theirs, is that a wider bench is a different bench, and reviewers newer to a codebase have reason to send more back regardless of what the code looks like. The authors test whether the mix of tasks shifted; they do not test whether the mix of reviewers did. So whether the generated work was actually worse is unsettled, and anyone claiming this study proves it is overreading. Either way, the checking step is where the gain stopped.

Other limits: a working paper, not peer reviewed, measuring firm-level adoption rather than who used the tools. The finished-work result is a null and not a zero: what the data rules out is a gain above 12 percent of the baseline, so the 30 percent is excluded and smaller real gains are not. A ceiling is not an estimate, and nothing here tells you where in that range the truth sits. And nobody has excluded the reading that much of the extra output was waste which review correctly discarded, arguing for generating less rather than staffing more checkers. Hold onto that one, because it points at the same first move.

The largest limit is the setting. Software already runs a formal review step, which is what generated work piles up against. Most functions in your business have no such step, and that does not make you the lucky case. It means nothing of yours queues up where anyone can count it: the quote goes to the customer, the memo goes to the board, and the review is whoever read it on the way past. Gated, unchecked output costs you delay. Ungated, it reaches the customer.

So what follows is our inference, not their finding, drawn from firm-level throughput containing no individual people at all. These firms bought generation and got it. What they did not buy is whatever decides whether the output is right, and that is where the gain went. Where that holds, the capability separating your people sits on the checking side of the handoff rather than the asking side. Whether it holds in your shop is a question about your shop, and the last section answers it cheaply.

What the skill is, in words you can test

The answer most people reach for is asking-skill, and it is worth less than it looks. A commenter on that thread posting as extr described the prompt-craft era as already over: the workflow tweaks and “special instruction recipes” he invested in are “all obsolete now,” and what is left is “knowing what to ask for.” One practitioner’s report is not proof, and asking well is a real skill. It is a perishable one, written against this year’s models, so the people who got good at it get to do that again. Knowing the work well enough to judge the answer does not expire on that schedule.

That version is at least testable. The skill is the ability to say, before acting on an answer, what would have to be true for it to be wrong, and then to go find out. One question gets at it in any role, asked about a specific piece of work rather than in the abstract: how do you know?

Listen for whether the answer points at anything outside the model. A quote somebody actually called for. The invoice they opened. The one account where that mapping has always gone wrong. Nobody has run the study that would prove this test sorts people correctly on a bid, a reconciliation, a pricing memo, or a support reply. What we can say is that the shape of a real answer is the same in each, and that finding out costs you one question.

Then ask a second, because a fluent answer is cheap to produce once: not whether they checked, but what the check would have looked like if it had failed. Somebody who opened the invoice can tell you which number would have stopped them. Somebody telling you what you want to hear cannot. It also gives you the hiring test, since a capability with a name is one you can interview for. Hand a candidate a work sample with a plausible error in it, let them use whatever tools they like, and ask both questions. You are not testing whether they catch your planted error. You are testing whether they can tell you what they checked.

The half that does not demo well

Here is our reasoning for why, and you can test it against whatever training you were last offered. Generation travels: one demonstration works for any audience, with a certificate at the end. Checking does not travel. It is specific to your work, your customers, and the three places your data is wrong, so it has to be practiced on live work by somebody who knows your domain well enough to tell when an answer is wrong. That makes it costly to deliver and hard to show in a room, and we would expect the market to supply more of whichever half is easier to sell. Check it against the last proposal you were sent: count how much of the agenda is producing output versus judging it, and whether anyone offered to work on your own files rather than a sample set.

Rich Hickey’s argument, the one that essay was answering, sets the honest boundary here. He holds this is not a tool at all but work handed off, and that outsourcing the reasoning costs you the learning it used to produce. Where nobody present can tell a right answer from a wrong one, that is right, and there is no skill to coach, only a gap. A contract review, a tax position, a structural calculation. Keep that work in human hands until somebody can judge it, because where the only available check is that it read well, speed is not what you bought.

One counterexample deserves weight: maybe review capacity is simply lagging, and the tools themselves close it. The paper tested that. By March 2026 nearly 80 percent of the firms had used AI tools in code review, yet only 23.3 percent of review comments were AI-generated and only 10.8 percent of pull requests received even one. Those firms reached for people instead, as above. That is evidence the quick fix had not arrived by March 2026, not proof it never will.

What this costs you nothing to run

Take the last five pieces of work in your business where AI touched the output. For each, write down who checked it and what they checked it against. Where the answer names a source, a record somebody actually opened, you have found a person worth keeping close to that work. Where the answer is that it looked right, you have found the gap, and you know it now by name and by seat instead of by vague worry. If all five come back with real checks performed by somebody other than you, nothing above describes your business and you can stop reading. If all five were checked by you personally, you have not passed the exercise. You have found its answer.

Because that is the part that stings. The firms in the study met the new review demand by pulling more people into reviewing, which was the response they took rather than the only one on the table: capping output and gating what merges were both available. They had a bench to widen. If your own name is in that column five times out of five, or one person’s name is, you do not, and speeding up generation makes your real constraint worse while more licenses do nothing about it.

Two moves follow, in order. Narrow where the tools are allowed to produce finished work, down to the functions where somebody other than you can check the output, and let the rest stay drafts nobody sends unread. Then take the function with the worst ratio of volume to checkers and build a second checker there before widening anything. Building one is not a course. It is your senior person narrating the check out loud on live work, the second person doing the next one while they watch, and the two of them arguing about a real case often enough that the newer one learns where these answers go wrong. There is no compressed version of that, which is the part a rollout plan has no line item for.

If your team already has the judgment and only needs reps, run that yourself, weekly, on your own work. It beats anything we would sell you and it is most of the value here. Where we earn money is the harder case, where the second checker has to be built while the work keeps shipping and the only person who could teach it is the bottleneck. That is what a coaching engagement is: $400 per named person each month, four-person minimum, your people asking us in their own Slack or Teams channel the day the question comes up, and us doing that narrating until your own people can. Hold it to this piece’s standard. The monthly report comes in your revenue, your margin, and your hours back rather than in our activity, and if your answer to how do you know is still that it felt better, we have failed the test too.

The question on that thread was fair, and the test above is cheaper than the argument it came from. What nobody has produced, ours included, is the verifiable case it actually demanded.