You cannot ship an AI feature you cannot measure

There is a moment on most LLM projects where someone changes a prompt, the demo looks better, and everyone agrees to ship it. Two weeks later a customer reports that the thing which used to work has stopped working. Nobody can say when it broke, because nobody was measuring.

This is the central engineering problem with building on models, and it is not a prompt problem. It is a testing problem wearing an unfamiliar hat.

Why ordinary tests do not help

A conventional test asserts an exact result. Given this input, return exactly this output. That assertion is worthless against a model, which will return something slightly different every time, and where “slightly different” is usually fine.

So teams skip the tests, and fall back on looking at the output. Looking works for about three examples. Past that, nobody can hold the comparison in their head, and judgement quietly degrades into “seems alright.”

The fix is not to abandon testing. It is to change what you assert. You stop asserting exact equality and start asserting acceptable behaviour across a fixed set of realistic cases.

The evaluation set

An evaluation set is a collection of inputs paired with some notion of a correct response. It is a test suite that tolerates variation.

Building one is unglamorous and mostly consists of collecting real examples. Twenty to fifty is enough to be useful. A few hundred is enough to be confident. What matters far more than volume is that the examples are real — drawn from actual usage, support tickets, or the documents your users will really send — because the failures you care about live in the messy cases nobody would think to invent.

Each case needs a way to decide pass or fail. In rough order of preference:

Deterministic checks. Did it return valid JSON? Does the extracted date parse? Is the cited document ID one that actually exists? These are cheap, fast and completely reliable. Use them wherever the problem allows, and shape your feature so that more of it is checkable this way. A feature that returns structured data is easier to trust than one that returns prose.

Reference comparison. For each input, store a known-good output and compare semantically rather than exactly. Useful for extraction, classification and summarisation, where “correct” is reasonably well defined.

Model-graded evaluation. Use a model to judge whether a response meets stated criteria. This is the weakest option and it is the one everyone reaches for first. It works, but the grader has its own failure modes, and you need to spot-check the grader against human judgement or you have simply moved the unmeasured problem one level up.

Human review. Slowest, most reliable, and unavoidable for genuinely subjective qualities like tone. Reserve it for a small subset and for periodically auditing whatever automated grading you rely on.

Write the evaluations first

The order matters more than it looks.

If you build the feature first, your evaluation set gets written against the behaviour you already have. You will unconsciously choose examples it passes, because those are the ones in front of you. The set will encode your current implementation rather than the requirement, and it will confirm rather than test.

Writing the cases first forces the harder conversation, which is what “correct” actually means here. That conversation frequently reveals that the requirement was underspecified, and occasionally that the feature does not need a model at all.

It also gives you an honest baseline. Run the evaluation set against the simplest possible implementation before you optimise anything. Sometimes the naive version scores 80% and the question becomes whether the remaining 20% is worth the complexity of chasing it.

Run it on every change

An evaluation set that runs occasionally is a document. One that runs on every change is infrastructure.

Wire it into CI alongside the ordinary test suite. Because model calls cost money and take time, most teams run a fast subset on every commit and the full set before release. Record the score for every run so you can see the trend, and fail the build on regression past a threshold you have agreed in advance.

The payoff arrives the first time somebody proposes swapping the underlying model. Without evaluations that is a leap of faith, and the honest answer to “is the new one better for us?” is that nobody knows. With them it is an afternoon’s work and a number.

What this buys you

Three things, roughly in order of value.

You can change things. Prompts, models, retrieval strategy, chunking — all of it becomes adjustable, because you can tell afterwards whether you improved anything. Without evaluations, working code becomes untouchable and the system ossifies.

You can answer the client’s real question. Not “is the AI good”, which nobody can answer, but “it handles these 200 representative cases at this accuracy, and here are the specific ones it gets wrong.” That is a conversation about trade-offs rather than vibes.

You find out where the model does not belong. A category of input that consistently fails is usually not a prompting problem. It is the system telling you that this part of the job wants a rule, a lookup, or a person.

The uncomfortable part

Building an evaluation set is roughly a fifth of the work on a serious LLM feature, and it produces nothing a client can see in a demo. It is the least demonstrable thing you will build.

It is also the difference between a feature that improves over two years and one that nobody dares touch after the engineer who wrote it leaves. Of everything in an AI project, this is the part most worth insisting on and the part most often cut.