Back to The Haunt
Building with AI2 min read

The hard part starts after the demo.

A demo runs on the inputs you chose. A product has to work for the inputs you did not expect.

Fast prototyping is useful. You can put an idea in front of someone, discover a misunderstanding, and throw away an approach before investing too much in it. AI can shorten that feedback loop. The mistake is treating the first convincing result as the end of engineering.

Try the uncomfortable inputs

What happens with an empty response, a duplicate request, a large file, an expired session, or a slow connection? What happens when a user can access a record by guessing its identifier? If your product calls a model, what happens when its answer is plausible but wrong?

Turn those questions into a small set of checks. Choose them from the failures that would matter to your users, rather than adding tests only to increase a coverage number.

Make failures observable

When something goes wrong, you need enough information to understand it without exposing private data. Decide what to log, where errors surface, and how to tell whether the system is getting worse.

For an AI feature, a successful HTTP response does not establish that the output was useful. Define what quality means for the task. Keep representative examples, compare changes against them, and review the cases where automation should defer to a person.

A good demo shows the possibility. A good product makes its limits understandable.

Keep the first version small

Pick one workflow and give it clear boundaries. Make the fallback understandable. Set limits on expensive operations. Decide how to undo a bad change. These choices are often more valuable than a second impressive feature.

When you use generated code, ask for an explanation of its assumptions and failure modes, then verify that explanation against the code. Tool output is a starting point for review, not evidence that review has happened.

The next time a prototype feels finished, spend an hour trying to break its central promise. What you learn is the beginning of the product.

That’s it. Come back soon; the ghost keeps writing.Follow by RSS
Read next

You don’t need to learn everything.