Product design
Design for the model being wrong
Every LLM feature fails sometimes. The difference between a product people trust and one they abandon is entirely in what happens next.
5 min read
An LLM feature will produce a confidently wrong answer. Not might — will. The rate can be pushed down but not to zero, and it will not warn you when it happens.
Most teams treat this as an accuracy problem to be solved before launch. It is better understood as a design constraint to be built around, in the same way a payments system is built around the certainty that some transactions fail.
Confidence is the wrong signal
The instinct is to show a confidence score and let the user decide. This mostly does not work.
Token probabilities tell you how expected the text was, not how true it was. A model can be fluent and certain about something it invented. Worse, users read a number next to an answer as a quality guarantee and stop reading critically — so a badly calibrated score is worse than none.
Where confidence is genuinely useful is upstream, in retrieval. If the best-matching document scores poorly, that is a meaningful signal that the system lacks the information to answer, and it is a far better trigger for “I don’t know” than anything derived from the generated text.
Make the failure legible
The most valuable design move is showing where an answer came from.
An answer with its source attached lets a user verify in two seconds. It converts a black box into something checkable, and it changes the user’s relationship with the feature: they stop asking “is this right?” and start asking “is this the right source?” — a question people are actually equipped to answer.
It also changes failure from betrayal to inconvenience. A wrong answer with a visible source is a system that showed its work and got it wrong. A wrong answer with no provenance is a system that lied. Users forgive the first and abandon the second.
Give it a way to decline
A model asked a question it cannot answer will answer anyway. Fixing this requires an explicit path for refusal, and something in the system with the authority to take it — usually a retrieval threshold rather than the model’s own judgement.
“I don’t have information about that” is a good outcome. It is honest, it is fast, and it tells the user to look elsewhere instead of acting on something false. Teams resist building it because it feels like admitting the feature is incomplete. In use it is one of the strongest trust signals a system has, because a system that sometimes declines is one whose answers mean something.
Match the stakes to the autonomy
The right amount of human involvement follows from what a mistake costs.
- Low stakes, reversible — drafting text, suggesting tags, ordering a list. Let it run. The cost of a bad suggestion is a click.
- Moderate stakes — customer-facing responses, data extraction feeding a report. Generate, but have a person approve. Design the approval to be fast, because a review step that takes as long as doing the work manually will be bypassed within a fortnight.
- High stakes, hard to reverse — anything financial, legal, medical, or that sends something irretrievable. The model assists a person who decides. It does not act.
The common failure is treating this as a fixed property of the product rather than a per-action decision. The same application can autonomously reorder a list and require sign-off before emailing a customer.
Fail toward the old way
Model providers have outages, deprecate versions, and rate-limit. If your AI feature going down takes the whole product with it, you built a dependency you cannot survive.
The question to answer at design time is what the system does when the model is unavailable. Usually there is an acceptable degraded mode — search instead of answers, the manual form instead of the extraction, a queue instead of an immediate response. Users tolerate a feature being temporarily unavailable far better than they tolerate the application being broken.
Instrument it, because you will not notice otherwise
Traditional software fails loudly. AI features fail silently and stay up.
Log the inputs, the retrieved context, and the outputs, so a reported problem can actually be investigated rather than guessed at. Track refusal rate and retrieval scores over time — a slow drift usually means your content or your users changed, and it is the earliest signal you will get. Give users a one-click way to flag a bad answer, and read what comes back; it is the cheapest evaluation data you will ever collect, and it comes from exactly the cases you failed to anticipate.
The reframe
Reliability engineering has known for decades that the goal is not preventing all failure, which is impossible, but ensuring failures are contained, visible and recoverable.
AI features are the same discipline with an unfamiliar failure mode. The teams that ship LLM products people keep using are not the ones with the best prompts. They are the ones who assumed the model would be wrong and designed the surrounding system accordingly.