The Dashboard Says You Did Nothing

The graph is following its rules. The mistake is treating a filtered record as a complete account of someone’s work. That error comes before the familiar argument about whether commit counts measure value: the report can misrepresent the activity itself.

The authors of the 2021 SPACE paper on developer productivity warned about this explicitly. Even counts of pull requests, commits and code reviews are vulnerable to gaps in data and measurement errors. Before debating what a number means, you have to establish what it includes.

How a missing feed becomes a personnel problem

Consider a report that starts each engineer at zero and fills in activity counts from connected repositories. Suppose a repository import fails, and the report leaves those initial zeros in place. A successful import with no events and an unsuccessful import now produce the same result. You can no longer tell which happened by looking at the count.

Now give an AI summarizer only that table and ask it to identify underperformers. It could warn that the evidence is insufficient, or it could produce a confident explanation. What it can’t discover from those counts is that the import failed. The report discarded that information before the prompt was written.

If the explanation reaches a performance review, the engineer has to reconstruct their work to challenge a conclusion that the evidence never supported. The software lost sight of them. They get asked where they went.

Keeping a failed import separate from a successful zero preserves the distinction. It doesn’t make a zero a verdict on effort: a complete commit count can still omit design work or a debugging session that produced no commit. GitHub’s attribution rules add another limit, even when every connection works.

The distinction has to survive beyond the dashboard. If a warning appears beside a chart but disappears from the exported table or the AI-generated summary, the manager reading that summary still lacks it. The report needs to carry what it couldn’t observe wherever it carries a judgment about the person.

A manager can still ask why an agreed change is late or investigate a verified defect. Neither conversation requires a complete record of someone’s day. What needs to stop is the extra inference: that a blank cell proves they weren’t working. Use the work you can inspect, and leave the unobserved interval unresolved.

The AI gain still needs an answer

A CTO owes the business an answer about AI spending. Faster code production is a real possibility. In a 2023 controlled GitHub Copilot experiment, the group with access to the tool completed an assigned JavaScript HTTP-server task 55.8% faster than the control group. That measured gain concerned a prescribed coding task, rather than the full cost of getting a production change safely into use.

In a February 2026 update, METR described an unreliable productivity estimate from its newer experiment. Some developers were declining to participate or withholding tasks because they didn’t want to work without AI. A reduced pay rate could also have affected participation, and concurrent agents complicated measurements of time spent. METR believed developers were likely benefiting more than in its earlier study, but called its data only very weak evidence for the size of that improvement.

The missing work here sits outside the study, rather than outside a repository feed. Even an accurate record of the submitted tasks leaves the speedup on excluded tasks unknown.

If AI gets code ready for review sooner, but the code then spends longer waiting for approval, the coding gain and unchanged delivery time can both be real. Counting more generated code won’t resolve that distinction. Following the work through review and deployment can tell you where the saved time went.

That makes activity data useful for investigating a delay. A growing review queue gives you something specific to examine. It still leaves the business question open: did the change do what the company needed?

For a reliability fix, the evidence might be whether the failure recurs under the conditions that used to trigger it. For a feature, it might be whether customers can complete the task it was meant to support. Choose the result before the work starts, so the team doesn’t have to reverse-engineer a success story from whichever number rose afterward.

Apply the same coverage test to those outcomes. If error reporting stops after a reliability change, the absence of reported failures can’t establish that the fix worked. And even with sound measurement, a before-and-after improvement can reflect changes in scope or staffing as well as AI. Record those changes before attributing the gain.

Put the burden on the report

Once a report names an engineer as a problem, their objection to its data also becomes a defense of their performance. That makes the measurement harder to discuss on its own merits. A CTO can separate those questions before the report enters anyone’s review: establish the report’s limits while nobody’s standing is at stake.

Ask the person who owns it to show which repositories and accounts it covers, the period each source actually supplied, and how it marks incomplete collection. If an import stops partway through, the positive count it already collected is incomplete too. Check the report against underlying work, including a contribution you know should appear. An attribution mismatch like GitHub’s email requirement calls for correcting the record, rather than asking the engineer to generate more visible activity.

Keep the explanation of what the work accomplished close enough to the actual change that someone can inspect it, including the engineer whose work is being discussed. A dashboard can support that conversation without becoming the judge.

In the next demo, ask for a test copy with the repository connection disabled. Compare its report with a successful import that genuinely contains no events, then read the exported table and the performance summary. The failed source should remain visibly unavailable in each. If both runs become the same zero and the same verdict, you’ve found a defect to fix before asking an engineer to explain a blank day.