Feeding the Meter

To satisfy his boss, the IT consultant started sending Teams Copilot questions he’d previously put into Google. Some answers helped. Others didn’t. He reported spending three hours trying an AI-suggested approach to a software setup problem before finding the answer through Google in two minutes.

A weekly-use count would record either outcome as compliance. His account can’t tell us the division’s overall productivity, but it exposes what that particular measure leaves out: whether the answer helped him finish the job.

If you grade raw usage as performance, someone can meet your target without improving their work. They’re feeding the meter because feeding the meter is what you asked for. The executive who chose the measure owns that problem.

Required practice can still have a purpose. An employee’s first difficult attempt doesn’t settle a tool’s value, any more than a successful demo does. A team needs access to a tool suited to its work and a chance to learn it. Vandervort’s division was being measured on the integration it could reach.

A usage count can establish that someone tried the tool. Low use gives a manager a reason to ask about access, training or task fit. But a trial must also allow an employee to demonstrate that the tool makes a particular task worse, and stop using it there. Penalize that answer for lowering the usage number and you’ve made compliance safer than judgment.

A gain you can actually examine

A 2025 study by Erik Brynjolfsson, Danielle Li and Lindsey Raymond followed the staggered introduction of an AI assistant in a business-software company’s support operation. The study covered 5,172 agents; the rollout happened primarily in late 2020 and early 2021. During customer chats, the assistant suggested replies and supplied links to relevant internal documentation. Agents could edit or ignore the suggestions.

The researchers estimated a 15% average increase in issues resolved per hour. They compared performance before and after access with agents without access, accounting for differences including experience. Managers chose who received access and when, so this was an analysis of a rollout rather than a randomly assigned trial.

They also examined resolution rates and customer satisfaction, rather than treating a shorter conversation as sufficient proof of a better one. Quality records were incomplete, so the resolutions-per-hour estimate covered a subset of the agents.

The gains were uneven. Less experienced and lower-skilled agents improved both speed and quality. The most experienced and highest-skilled agents saw small gains in speed and small declines in quality. Even within the same job, the value of the same assistance differed.

The same study also found larger gains among agents who followed more suggestions. Following the advice might improve performance, but the authors identify another possibility: agents may follow it because they have more to gain. That relationship gives a manager a reason to investigate use. It doesn’t establish what would happen if the manager imposed a higher usage target.

A novice accepting helpful suggestions and an expert rejecting poor ones could both be making good decisions. A usage leaderboard alone can’t evaluate either decision. The useful question is how assistance changes this person’s work, under these conditions. The study supports a gain in a specific support process; it doesn’t establish a company-wide profit gain.

Follow the work past the prompt

Now consider a process where AI makes drafting faster, but every draft still needs approval from the same person. If that person’s review capacity limits how much work finishes, producing more drafts adds to the queue. Your adoption dashboard can turn green while the revenue tab stays flat, even when the drafting tool works.

In that situation, the process owner needs to inspect the approvals and handoffs. Can routine decisions safely move to someone else? Does the reviewer lack information that should arrive with the draft? Are corrections consuming the time saved in writing? The fix follows from the delay you find. More compulsory prompts don’t answer those questions.

Start with a completed unit of work you can inspect: a support issue resolved, for example. Compare similar work with and without the changed process where practical, and account for differences in difficulty or staffing. Use existing records to track both elapsed time and staff effort through review and rework. Include failed attempts, rather than assembling a gallery of the tool’s best answers.

Keep initial training effort visible but separate from ongoing work. That lets you check whether performance improves with practice without hiding training costs or treating a slow start as a permanent result. An employee’s claim that the tool wasted time needs the same scrutiny as a manager’s claim that it saved time.

Check quality alongside speed. A target for closing tickets can reward premature closure just as a prompt target can reward unnecessary prompts; reopened cases and customer feedback help test what the count claims. Read the work behind the total before putting the new measure into someone’s performance review.

Revenue, margin and hours back still matter. They need a connection to the work. If staff effort falls but nobody uses the freed capacity, you have capacity available, not an automatic reduction in payroll. If revenue rises while prices or demand change, the AI rollout can’t claim the whole increase. And asking people to estimate their hours saved leaves you with an estimate that needs checking, however attractive the total looks.

At your next performance review, what happens to the employee who demonstrates that an AI workflow wasted time, then stops using it?