The Paved Demo

Here’s the thing you already half-know but can’t quite act on: you have no way to check. You can’t read the code. You can’t see what the model does when the input is uglier than the one the salesperson just typed. You’re a plumber, a clinic owner, a distributor, a firm with sixteen people and a phone that never stops. You know your business cold. You don’t know whether the thing you’re watching is a genuine capability or a good actor reading from a script the vendor wrote.

So you do the only thing available to you. You grade the demo. And the demo was built to be graded well.

The Demo Always Works

Of course it works. Nobody schedules a demo of the case where their product falls apart. The salesperson picked the input, picked the record, picked the happy path, and rehearsed the drive. You’re not watching the product. You’re watching the one lap the vendor practiced.

And “it uses AI” tells you almost nothing about what’s under it, because the label got cheap. In September 2024 the FTC ran a sweep it called Operation AI Comply, taking action against five companies for deceptive AI claims. One of them, DoNotPay, had marketed an “AI lawyer” that could “generate perfectly valid legal documents” and file lawsuits for you. The FTC found it had never tested that output against the work of an actual lawyer. The same year, the SEC brought its first AI-washing charges, fining two investment advisers a combined $400,000 for overstating AI they didn’t really use. When regulators have to invent a word for lying about AI, the word on the box has stopped being evidence.

That’s the trap you’re in. You’re being asked to buy something whose entire value lives in a place you can’t inspect, on the strength of a performance designed by the party selling it. Grading the demo harder doesn’t help. A rehearsed lap is a rehearsed lap no matter how closely you watch it.

The Route Was Paved

Think about how you’d actually buy a used car if the seller wouldn’t let you open the hood. No engine inspection, no mechanic, no history report. All you get is a test drive, and the seller picks the route. He takes you around a flat, empty parking lot at fifteen miles an hour. Smooth. Quiet. Turns fine. You’d never sign, because you know the route was chosen to hide exactly what a route can hide: the cold start, the highway merge, the hill, the brakes under real weight.

Every AI demo is that drive, on a lot the vendor paved. Call it the paved demo. The salesperson controls the input, the pace, and the scenario, and every one of those choices routes around the places the product is weakest. The clean sample record instead of your real data with the duplicate customers and the fields nobody filled in. The polite request instead of the caller who’s furious and talking over himself. The one workflow that’s finished instead of the six that are half-built. You’re being shown the parking lot and asked to buy the mountain pass.

The mistake isn’t trusting the vendor. The mistake is accepting that evaluation is the job you’re supposed to do here. It isn’t, because you can’t. Your leverage was never inspection. It’s specification. You don’t get to open the hood, but you absolutely get to pick the route.

Bring Your Own Hill

Take the wheel. Before any demo, hand the vendor the messiest real situation your business actually produces and make them run the product on that, live, in front of you. Not their sample. Yours. This is the single move that turns a sales performance back into a test, and it costs you nothing but the nerve to ask.

A few routes worth paving with your own asphalt:

• Your real data, not their sample. Give them a real export, warts included: the duplicate records, the blank fields, the customer whose name is spelled three ways. A tool that only works on clean inputs doesn’t work, because your inputs are never clean.
• The angry edge case. Hand them your worst real interaction. The furious cancellation, the caller who interrupts, the request that requires saying no. Watch whether the product handles it or confidently makes something up. Confident wrong is the failure mode that costs you a customer.
• The moment it should stop. Ask them to show you what happens when the AI doesn’t know. A product with judgment has a “let me get a human” path. A product without one will guess, and it will guess in your name.
• The handoff. Make them show the boring part: how the output lands in the tool your team already uses, who checks it, what breaks when it’s wrong. The demo lives in the product. Your business lives in the seams.

None of that requires you to understand a single line of the model underneath. That’s the point. You don’t have to become technical to stop being sold to. You have to stop grading the lap they practiced and insist on driving a road they didn’t.

The vendors selling honest tools will welcome this. Their product survives contact with your ugliest case, and they’d rather you find that out now than churn in ninety days. The ones selling a paved demo will suddenly need to “set that up for a follow-up,” and the follow-up will keep slipping. That flinch is the most useful thing you’ll learn all quarter. It’s the reading you couldn’t get from the code.

You came into that conference room without a way to check, and that was the real problem, deeper than any one product. It’s also the one we built The Bushido Collective to solve: people who build with these tools every day, sitting on your side of the table, running the ugly cases with you before you sign anything. Our first conversation is free, and if the honest answer is that the tool in front of you is fine, that’s what the report says. Either way, you stop buying the parking lot and start driving the hill.