How to know whether your AI system actually works
Evaluation
Almost every AI system looks good in a demo, because the demo runs the examples that were tried while the system was built. The problems live in the tail: the requests nobody thought of, and the documents missing a field.
The question is not whether the system can do the task, but how often it does not, and what happens the next time somebody edits a prompt.
3 questions an evaluation has to answer
- How often is the answer right, measured on something other than gut feeling?
- When it is wrong, is it wrong in a way the user notices, or in a way that looks right?
- Did it get better or worse than the previous version?
The third matters most and is missing most often. Without it every change is a gamble, and teams that cannot answer it stop changing anything.
The loop
What makes the loop worth anything is the dashed line back. Every failure you find in production goes into the dataset. After six months it describes precisely what your system has been bad at.
Start with 30 examples
Evaluation usually does not happen because people believe it needs a thousand labelled examples. 30 chosen because they are hard say more than a thousand random ones.
Assemble them before you build, so they describe the task rather than the solution. Include the boring ones: empty input, wrong language, questions the system is not supposed to answer.
Choose the method by what the failure costs
| Type of task | How to measure | What it takes |
|---|---|---|
| One right answer | Exact match against a known answer | Almost nothing |
| The answer is prose | A model scores it against a reference answer | Reference answers written once |
| The failure is expensive | Sampled human review | A few hours a week from domain experts |
The second row carries two caveats. A judge from the same model family tends to like its own style, and a judge without a reference assesses whether the response looks sensible, not whether it is true. Hence the reference answer.
Set a threshold and let it block
An evaluation that only produces a report is a report. The value arrives when it can say no.
The rule can be as simple as this: if accuracy drops more than 2 percentage points from the previous run, the change does not ship. The margin is deliberate, because a threshold with no slack blocks on noise until somebody turns it off.
Put the The time from sending a request to having the answer. The tail often matters more than the average: p95 is what the slowest five percent wait. in the same rule. An average of 2 seconds can hide every twentieth user waiting fifteen, and it is the twentieth user who calls.
What this makes possible
The point is not a number for the board. It is that you can change things: switch model, rewrite a prompt, cut a step, without wondering whether you just broke something. Teams that can measure dare to optimise. Teams that cannot pay for it every month.