When AI Passes Your Take-Home

Tristan Hume’s January 21, 2026 account includes a useful complication: he helped the model past a sticking point by telling it what performance was possible. The leading human submission had also used Claude heavily, with steering. Both results involved a person and a tool. The final score left their respective contributions unresolved.

Hume redesigned the assessment around puzzles with an unusual, constrained set of instructions. He reported promising early results, but acknowledged giving up some of the original test’s realism. Making the problem harder for a model had pulled it further from the job.

If the job includes AI, why insist on proving which ideas came from the person? A candidate who uses an assistant to do the work well deserves credit. The lesson we draw from Hume’s account is narrower: the submission alone leaves a hiring question open. What happens when that person has to decide whether a convincing answer is safe to accept?

Give the explanation something to answer to

Asking candidates to explain their work is a reasonable start. Simon Willison describes his own rule for production code: he won’t commit code he couldn’t explain to somebody else. He applies that rule to AI-generated code too. Using the tool and taking responsibility for its output belong together.

An interview needs to get beyond a fluent explanation, though. If the assistant supplied both the implementation and the explanation, hearing the explanation back leaves the same gap. Ask for a prediction that the candidate can check against the running code.

A test that only asserts 3 == 3 passes even if you delete the feature it supposedly protects. That defect is easy to spot. The more interesting question is whether a realistic-looking test also survives broken behavior. Ask the candidate which change to the implementation would make an existing test fail, then make that change and run it.

If the test stays green, the discussion now has something concrete to resolve. Perhaps the candidate misunderstood what the test exercises. Perhaps they’ve found a genuine gap in the suite. Watch whether they trace the mismatch and improve the test, rather than defend their first answer. A corrected prediction gives you more to assess than a rehearsed account of why the original code was good.

Keep AI in the debugging round

You can make that follow-up part of an existing take-home review or pair on a small, sanitized bug from your own codebase. State the tool policy in advance and provide the context needed to investigate. If AI is part of the job, give the candidate a round where they can use it and show how they check its suggestions.

Consider a deliberately simplified reservation exercise. There’s one item left in shared storage. Each request reads the stock count and, if it’s positive, writes back that count minus one and reports success. Pause request A after its read. Let B read the same positive count, then allow both writes. Both requests report success while the stored count ends at zero. A test that checks only for nonnegative stock would bless the oversale.

Give the candidate the requirement: with one item and two valid requests, one reservation succeeds, the other gets a sold-out result, and the stored stock ends at zero. Ask them to reproduce the broken ordering before evaluating a fix. Running the requests one after another won’t exercise that failure.

State up front that requests can arrive through separate server processes sharing that inventory. A lock inside one process only coordinates work in that process. Each server could hold its own lock while both read the last item. A single-process check would make this fix look better than it is.

One valid approach is a conditional change in the shared store: check that stock is positive and reduce it as a single indivisible operation. Once the first reservation consumes the item, the next must check the new count and refuse. The candidate needs to demonstrate that the store actually provides that behavior. Check the success responses as well as the final stock; a final count of zero also occurred in the broken version.

The useful evidence includes the request ordering they construct, the result they expect, and what they do when the result disagrees. The model may help with all of it. Credit a candidate who uses it to build a convincing check, catches an incorrect suggestion through execution, or revises an assumption. Requiring them to reject every bad suggestion on sight would throw away the debugging behavior you’re trying to observe.

The assistant can propose the checks as well as the fix. This round observes someone working with AI; it leaves their unaided ability unresolved. If the role requires diagnosis without an assistant, assess that separately on the same code, with the restriction explained in advance. Keep those observations distinct in the hiring notes.

Keep the assessment bounded and relevant to the role. The reservation exercise tests reasoning about shared state; choose a different failure when the job calls for different work. Supply the intended behavior and accept a trace, a diagram, or a failing test as evidence of understanding. Polished narration shouldn’t be the only way to demonstrate it.

Use consistent follow-up questions and decide what acceptable evidence looks like before meeting candidates. For a junior role, recognizing uncertainty and using help well may be enough. For someone expected to approve other engineers’ work, ask what the successful check establishes and what remains untested.

Hume valued the take-home partly because candidates could understand the system in their own editor, without someone watching or expecting narration. Preserve that advantage: let a candidate prepare a trace or failing test for discussion if live commentary is getting in the way. If the existing take-home review already shows how they investigate and verify, use that evidence rather than add another hurdle.

The proposed round still needs scrutiny. A rehearsed debugging trick can conceal a gap just as a polished submission can. Try the exercise with engineers doing the job and check which assumptions need supplying. Keep prior-work discussions and other role-relevant evidence in the decision; whether this round predicts performance still needs checking against what the people you hire actually do. A missed race condition in an unfamiliar exercise cannot carry the entire verdict.

The person who will review everyone else

The stakes change when the hire will set the standard for other engineers. If that person accepts a convincing implementation and explanation without checking behavior, their approval adds no check on the gap your interview missed. If they use the same standard to interview and promote others, the gap can reproduce through the team. That’s a conditional hiring risk, not a finding about how widespread it has become.

Your load-bearing engineering capacity is the people who can decide whether a change is safe enough to ship, including when the tools and tests give misleading reassurance. Headcount alone doesn’t tell you how much of that capacity you have. For a senior hire, establish which decisions they can own without routing every uncertain result back to the same experienced reviewer.

After the interview, write down one claim about the code that the candidate checked, and how they checked it. If the check changed their answer, record that too. If all your notes concern the finished submission, you still have an assessment to make.