# Avanest — full content > Complete text of Avanest’s published writing on building AI features that hold up in production. Source: https://avanest.com/insights --- # Design for the model being wrong Source: https://avanest.com/insights/designing-for-model-failure Published: 2026-07-30 · Product design An LLM feature will produce a confidently wrong answer. Not might — will. The rate can be pushed down but not to zero, and it will not warn you when it happens. Most teams treat this as an accuracy problem to be solved before launch. It is better understood as a design constraint to be built around, in the same way a payments system is built around the certainty that some transactions fail. ## Confidence is the wrong signal The instinct is to show a confidence score and let the user decide. This mostly does not work. Token probabilities tell you how expected the text was, not how true it was. A model can be fluent and certain about something it invented. Worse, users read a number next to an answer as a quality guarantee and stop reading critically — so a badly calibrated score is worse than none. Where confidence is genuinely useful is upstream, in retrieval. If the best-matching document scores poorly, that is a meaningful signal that the system lacks the information to answer, and it is a far better trigger for "I don't know" than anything derived from the generated text. ## Make the failure legible The most valuable design move is showing where an answer came from. An answer with its source attached lets a user verify in two seconds. It converts a black box into something checkable, and it changes the user's relationship with the feature: they stop asking "is this right?" and start asking "is this the right source?" — a question people are actually equipped to answer. It also changes failure from betrayal to inconvenience. A wrong answer with a visible source is a system that showed its work and got it wrong. A wrong answer with no provenance is a system that lied. Users forgive the first and abandon the second. ## Give it a way to decline A model asked a question it cannot answer will answer anyway. Fixing this requires an explicit path for refusal, and something in the system with the authority to take it — usually a retrieval threshold rather than the model's own judgement. "I don't have information about that" is a good outcome. It is honest, it is fast, and it tells the user to look elsewhere instead of acting on something false. Teams resist building it because it feels like admitting the feature is incomplete. In use it is one of the strongest trust signals a system has, because a system that sometimes declines is one whose answers mean something. ## Match the stakes to the autonomy The right amount of human involvement follows from what a mistake costs. - **Low stakes, reversible** — drafting text, suggesting tags, ordering a list. Let it run. The cost of a bad suggestion is a click. - **Moderate stakes** — customer-facing responses, data extraction feeding a report. Generate, but have a person approve. Design the approval to be fast, because a review step that takes as long as doing the work manually will be bypassed within a fortnight. - **High stakes, hard to reverse** — anything financial, legal, medical, or that sends something irretrievable. The model assists a person who decides. It does not act. The common failure is treating this as a fixed property of the product rather than a per-action decision. The same application can autonomously reorder a list and require sign-off before emailing a customer. ## Fail toward the old way Model providers have outages, deprecate versions, and rate-limit. If your AI feature going down takes the whole product with it, you built a dependency you cannot survive. The question to answer at design time is what the system does when the model is unavailable. Usually there is an acceptable degraded mode — search instead of answers, the manual form instead of the extraction, a queue instead of an immediate response. Users tolerate a feature being temporarily unavailable far better than they tolerate the application being broken. ## Instrument it, because you will not notice otherwise Traditional software fails loudly. AI features fail silently and stay up. Log the inputs, the retrieved context, and the outputs, so a reported problem can actually be investigated rather than guessed at. Track refusal rate and retrieval scores over time — a slow drift usually means your content or your users changed, and it is the earliest signal you will get. Give users a one-click way to flag a bad answer, and read what comes back; it is the cheapest evaluation data you will ever collect, and it comes from exactly the cases you failed to anticipate. ## The reframe Reliability engineering has known for decades that the goal is not preventing all failure, which is impossible, but ensuring failures are contained, visible and recoverable. AI features are the same discipline with an unfamiliar failure mode. The teams that ship LLM products people keep using are not the ones with the best prompts. They are the ones who assumed the model would be wrong and designed the surrounding system accordingly. --- # You cannot ship an AI feature you cannot measure Source: https://avanest.com/insights/evaluating-llm-features Published: 2026-07-30 · Evaluation There is a moment on most LLM projects where someone changes a prompt, the demo looks better, and everyone agrees to ship it. Two weeks later a customer reports that the thing which used to work has stopped working. Nobody can say when it broke, because nobody was measuring. This is the central engineering problem with building on models, and it is not a prompt problem. It is a testing problem wearing an unfamiliar hat. ## Why ordinary tests do not help A conventional test asserts an exact result. Given this input, return exactly this output. That assertion is worthless against a model, which will return something slightly different every time, and where "slightly different" is usually fine. So teams skip the tests, and fall back on looking at the output. Looking works for about three examples. Past that, nobody can hold the comparison in their head, and judgement quietly degrades into "seems alright." The fix is not to abandon testing. It is to change what you assert. You stop asserting exact equality and start asserting acceptable behaviour across a fixed set of realistic cases. ## The evaluation set An evaluation set is a collection of inputs paired with some notion of a correct response. It is a test suite that tolerates variation. Building one is unglamorous and mostly consists of collecting real examples. Twenty to fifty is enough to be useful. A few hundred is enough to be confident. What matters far more than volume is that the examples are real — drawn from actual usage, support tickets, or the documents your users will really send — because the failures you care about live in the messy cases nobody would think to invent. Each case needs a way to decide pass or fail. In rough order of preference: **Deterministic checks.** Did it return valid JSON? Does the extracted date parse? Is the cited document ID one that actually exists? These are cheap, fast and completely reliable. Use them wherever the problem allows, and shape your feature so that more of it is checkable this way. A feature that returns structured data is easier to trust than one that returns prose. **Reference comparison.** For each input, store a known-good output and compare semantically rather than exactly. Useful for extraction, classification and summarisation, where "correct" is reasonably well defined. **Model-graded evaluation.** Use a model to judge whether a response meets stated criteria. This is the weakest option and it is the one everyone reaches for first. It works, but the grader has its own failure modes, and you need to spot-check the grader against human judgement or you have simply moved the unmeasured problem one level up. **Human review.** Slowest, most reliable, and unavoidable for genuinely subjective qualities like tone. Reserve it for a small subset and for periodically auditing whatever automated grading you rely on. ## Write the evaluations first The order matters more than it looks. If you build the feature first, your evaluation set gets written against the behaviour you already have. You will unconsciously choose examples it passes, because those are the ones in front of you. The set will encode your current implementation rather than the requirement, and it will confirm rather than test. Writing the cases first forces the harder conversation, which is what "correct" actually means here. That conversation frequently reveals that the requirement was underspecified, and occasionally that the feature does not need a model at all. It also gives you an honest baseline. Run the evaluation set against the simplest possible implementation before you optimise anything. Sometimes the naive version scores 80% and the question becomes whether the remaining 20% is worth the complexity of chasing it. ## Run it on every change An evaluation set that runs occasionally is a document. One that runs on every change is infrastructure. Wire it into CI alongside the ordinary test suite. Because model calls cost money and take time, most teams run a fast subset on every commit and the full set before release. Record the score for every run so you can see the trend, and fail the build on regression past a threshold you have agreed in advance. The payoff arrives the first time somebody proposes swapping the underlying model. Without evaluations that is a leap of faith, and the honest answer to "is the new one better for us?" is that nobody knows. With them it is an afternoon's work and a number. ## What this buys you Three things, roughly in order of value. You can change things. Prompts, models, retrieval strategy, chunking — all of it becomes adjustable, because you can tell afterwards whether you improved anything. Without evaluations, working code becomes untouchable and the system ossifies. You can answer the client's real question. Not "is the AI good", which nobody can answer, but "it handles these 200 representative cases at this accuracy, and here are the specific ones it gets wrong." That is a conversation about trade-offs rather than vibes. You find out where the model does not belong. A category of input that consistently fails is usually not a prompting problem. It is the system telling you that this part of the job wants a rule, a lookup, or a person. ## The uncomfortable part Building an evaluation set is roughly a fifth of the work on a serious LLM feature, and it produces nothing a client can see in a demo. It is the least demonstrable thing you will build. It is also the difference between a feature that improves over two years and one that nobody dares touch after the engineer who wrote it leaves. Of everything in an AI project, this is the part most worth insisting on and the part most often cut. --- # Most problems that sound like fine-tuning are retrieval problems Source: https://avanest.com/insights/rag-before-fine-tuning Published: 2026-07-30 · Architecture A request that arrives regularly, in various forms: we want to train a model on our company's data so it can answer questions about our business. It is a reasonable thing to want and almost always the wrong way to get it. The distinction that matters is simple, and getting it wrong costs months. ## Behaviour versus knowledge Fine-tuning adjusts a model's weights on examples of desired behaviour. It is how you teach a model to respond in a particular format, adopt a house style, or handle a specialised task the base model handles clumsily. It changes *how* the model responds. Retrieval supplies relevant information at the moment of the request, in the prompt. It changes *what the model knows* when it answers. "Answer questions about our business" is a knowledge problem. Your policies, prices, inventory and documentation are facts. Facts belong in retrieval. ## Why fine-tuning on facts goes wrong **Your facts change; the weights do not.** Fine-tune on this quarter's pricing and the model knows this quarter's pricing until you retrain. Retrieval reads from a source of truth, so when the price changes, the answer changes. **It cannot cite.** A fine-tuned model produces an answer with no provenance. A retrieval system can show which document a claim came from, which is often the difference between a system people trust and one they quietly stop using. In regulated work it is not optional. **It still invents things.** Fine-tuning does not install a fact lookup; it shifts a distribution. A model trained on your documents will produce plausible-sounding statements about your business that are not in any document, and now they carry your house style, so they are harder to spot. **Access control disappears.** Facts baked into weights cannot be filtered per user. With retrieval, permissions are applied at the retrieval step, and a user only ever gets context they were allowed to see. Anyone building an internal tool across departments with different access needs discovers this constraint eventually — better before the build than after. **It is far more expensive to iterate.** Retraining and re-evaluating on every content change is a slow loop. Reindexing a document is a fast one. ## When fine-tuning is right It has real uses, and the pattern is consistent: it is about form, not content. - **Output format**, when prompting alone does not hold a rigid structure reliably enough. - **Tone and voice**, where the requirement is genuinely stylistic and hard to specify in words. - **Narrow, repetitive classification** at volume, where a small fine-tuned model is cheaper and faster than a large general one. - **Domain language**, where a field's vocabulary is unusual enough that the base model misreads it. Notice that none of these are "know our facts." They are all "behave a particular way." That is the line. ## The combination Serious systems often use both, and the division of labour is clear once you see it: retrieval supplies the facts, fine-tuning shapes the delivery. Retrieval first, always — establish that the right information reaches the model at the right moment, then ask whether the response format still needs work. Doing it the other way around means debugging a system where two things could be wrong and no way to isolate which. ## Where retrieval actually gets hard Retrieval being the right answer does not make it an easy one. The failure modes just move somewhere more tractable. **Chunking** decides what a retrievable unit is. Too small and you lose the context that makes a passage meaningful; too large and you dilute the match, burying the relevant sentence among irrelevant ones. Document structure usually beats fixed-size splitting — sections, clauses and headings are boundaries the author already decided for you. **Retrieval quality** is where most systems actually fail, and the symptom is misleading. Pure vector similarity is good at topical closeness and bad at exact identifiers — part numbers, error codes, proper nouns. Combining keyword and vector search covers each other's weaknesses. If your answers are wrong, measure retrieval separately from generation before touching the prompt, because a perfect prompt cannot rescue the wrong document. **Staleness.** A retrieval system is only as current as its index. Reindexing needs to be automatic and monitored, or it becomes the thing that silently stops running in month four. None of these are model problems. They are ordinary information-retrieval and data-pipeline problems, which is good news — they are well understood, debuggable, and fixable by engineers rather than by hoping a training run comes out better. ## The scoping question When someone describes a problem as needing a custom model, one question usually resolves it: *If a new employee had to answer this, would they need training, or would they need to look something up?* If they would look it up, you need retrieval. If they would need to learn a skill or a way of responding, fine-tuning might genuinely help. Most of the time, they would look it up.