Why Vibe Coding Needs More Than a Working Demo

A demo in which each customer opens their own invoice cannot distinguish a portal that enforces permissions from one that retrieves whatever invoice number it’s given. Both produce the screen you expected.

That’s a specific way vibe coding can fail: the behavior you watch gives you confidence in a requirement you haven’t tested.

The login form is only the visible part

Authentication checks who you are. Authorization checks what you’re allowed to do. A customer portal needs both: establishing that someone has a valid account still leaves the question of which invoices that account may read.

OWASP’s 2023 guidance on record-level permission failures describes the problem directly. A signed-in user changes the record identifier in a request, and the server returns data they shouldn’t receive. The guidance calls for checking the user’s permission to perform the requested action on the particular record.

The login form is there. The server-side enforcement may be missing. Hiding another customer’s invoice from a menu doesn’t settle this; a person can request it without using that menu.

An AI assistant might generate the permission check correctly. Your chosen platform might already provide it. The useful question is where that rule is enforced and how you know it works. A screenshot cannot answer either.

Why the prompt-and-check loop misses it

Suppose your working loop is to describe a feature, try it with your own account, and ask the AI to fix whatever looks wrong. The missing permission check creates no visible problem on that path. You can improve the layout and add a download button without learning anything about who else can retrieve the invoice.

This follows from the test you’re running. The same blind spot would exist in hand-written code tested the same way. AI makes it possible to produce the feature without understanding its implementation; accepting the demonstration as proof leaves that gap unexamined.

Automating that same demo preserves its blind spot. If the code and its tests both assume that any signed-in customer may read any invoice, a green test run confirms the wrong rule. More passing tests only help if they examine the permission the first demonstration left out.

Simon Willison’s March 2025 definition of vibe coding puts the boundary at building with AI without reviewing the code it writes. He distinguishes that from AI-assisted development in which someone reviews, tests, and understands the result. The distinction leaves plenty of room for AI to write the code, and for a founder to learn how it works.

In a study first published in 2022, researchers compared 47 participants working with or without an OpenAI Codex assistant. The assistant group produced insecure solutions more often on four of the five programming tasks analyzed. It also rated its answers as more secure, on average, than the group without the assistant.

The fifth task had inconclusive results. The participants were mostly university students working on short, security-related exercises with an older model. Those results provide no failure rate for today’s AI-built products. They do show why a builder’s increased confidence deserves a separate check against what the code actually does.

Make the refusal part of the demo

For the portal above, write the rule in ordinary language: a customer can read invoices belonging to their account and cannot read another customer’s invoices. That sentence gives you something more precise to investigate than whether the application seems secure.

Use a test copy with fake data and ordinary customer accounts with separate invoices. An administrator, or someone who legitimately shares access to both sets of invoices, would test a different permission rule.

First prove that each test customer can open their own invoice. Copy an invoice link that works for its owner, sign in as the other customer, and open that same link. Where the portal loads invoices without changing the page address, have its maintainer perform the equivalent test on the underlying data request using the other customer’s login.

OWASP’s testing guide uses this invoice example and recommends testing with multiple users who own different records. An error from requesting a made-up invoice number could mean there’s nothing to retrieve, rather than that a permission check protected anything. The customer who owns the invoice must still be able to read it.

Have whoever maintains the application inspect the server’s response as well as the screen. If the data reaches the browser and the page merely hides it, the information has already crossed the boundary.

If the wrong customer receives the invoice, inspect the lookup on the server. Fetching a record by its invoice number answers which invoice was requested; the server must also check whether the signed-in customer may read it. That decision has to happen before the data leaves the server. Making the number harder to guess doesn’t establish permission to use it.

Your platform may already have the right authorization mechanism. If it does, the repair may be to apply it to a route that missed it, rather than inventing another permission system. Either way, rerun the permitted and forbidden requests after the fix. A patch that refuses both customers has broken the product.

The download and export routes need the same permission rule as the page. Keep the allowed and refused requests as repeatable tests for each route when it changes.

This exercise checks the routes and customer permissions you actually exercised. Other roles or legitimate sharing arrangements need cases of their own. Password recovery and handling failed operations remain separate requirements; a passing invoice test cannot certify the whole application.

For a disposable prototype using fake data, with no access to live systems or paid services, you can keep experimenting within those limits. Before a customer depends on the result, have someone who can inspect and test the relevant behavior take responsibility for it. That can be you as you learn, an engineer already on the team, or a specialist reviewing a particular risk. The work requires that capability, without requiring a new executive role.

If you’re adding the invoice download next, take the other customer’s test account with you.