How to — build a small eval set for your own AI feature
You changed a prompt, and now you can't tell whether the feature got better or you just remembered the good answers. An eval set is how you stop guessing: a fixed batch of real cases with a written pass mark, run again every time you touch anything. You can build a usable one in an afternoon.
There is nothing ceremonial about this. A good eval for a product team is 30 to 100 real cases, each with a short statement of what a pass looks like, kept in a file that gets re-run. The public leaderboards don't help here — they measure general capability, not whether your support bot keeps inventing a refund window that doesn't exist. (If the distinction between a product eval and a vendor benchmark is blurry, How to — tell a real benchmark from a marketing one sorts it out.)
1. Harvest failures from real traffic, not from imagination
Open your logs, your support inbox, your own notes, and pull the times the feature actually let someone down. Twenty is enough to start. The discipline is to include two kinds of cases: the ones that break — the wrong format, the invented policy, the answer that ignored the user's language — and the ones that currently work, which exist only so you can catch yourself breaking them later. Team-written examples are fine as a supplement, but they skew easy and offline-written cases mostly test what you already thought of.
2. Write down what a pass looks like before you collect a single score
For each case, write one line the answer can be checked against: "does not state a refund window that isn't in the policy document," "returns valid JSON with the four required fields," "answers in the user's language." Keep them narrow enough to be answered yes or no. This step feels like paperwork and is actually the whole job — a case without a written pass mark is just an anecdote, and a set of anecdotes is what you already have. A binary verdict per case also beats a 1-to-10 score, because nobody on your team will agree on the difference between a 6 and a 7 twice in a row.
3. Split the set in two, and keep the score honest
Divide the cases into a failure half and a don't-break half, and always report both numbers. A single blended score hides the trade you are actually worried about: the fix that cures five bad outputs while quietly degrading fifty good ones. It is the same failure mode that makes retraining risky — the fine-tuning route carries the identical measurement obligation, which is why our guide How to — decide if your model needs fine-tuning puts building the eval before touching the weights. Fix the cases, then re-check the keepers in the same run.
4. Grade by hand first, then add a model judge and calibrate it
Nobody needs an automated judge to score 40 cases — read them yourself, record pass or fail, and note why the failures failed. Do that once and you own something more valuable than a number: a picture of how your feature breaks. Once the set is big enough or the runs frequent enough to be tedious, bring in a stronger model to grade and check it against your own calls on the same cases. That comparison is the point. LLM judges carry known biases — they favor the longer answer, they favor the first option in a comparison, they favor output that looks like their own — and the MT-Bench work that made judges mainstream also showed the mitigation: swap the order of the candidates and see whether the verdict survives the flip.
5. Freeze the set, version it, and re-run on every change
The value comes from repetition. Same cases, same pass marks, same judge, every time you touch the prompt, the retrieval, the model version, or the temperature. When you add cases — and you will, every time a new failure ships — note the date, because a score jump caused by new cases is not an improvement. And when you compare two versions of your feature, compare them on the same cases rather than against last month's memory; the paired comparison is what strips out the noise that comes from easy and hard cases being mixed differently. Anthropic makes the same point in its own guidance on eval statistics: pair the models, or the differences you are reading are partly question-difficulty luck.
Don't do this
Don't let the model grade its own homework and don't build an eval out of nothing but a rubric. Published work accepted at EMNLP 2026 found classifiers could partially predict a judge's verdict from the rubric text alone, without seeing the answer being graded — which means a rubric can carry the score in advance of any evidence. The same study found judges often failed to flip their decision when the answer or the criterion was reversed, and reversed expectations are something you can test in ten minutes. Related trap: treating "the model prefers its own output" as validation, which our own column on the practice — The Take — An agent grading its own homework is an alibi, not proof — calls out directly. And never grade on public benchmarks and call it a product eval.
How you'll know it worked
Four signs. You can run a change and get two numbers back in minutes — failures fixed, keepers held — instead of an argument. A bad answer can be traced to a cause, because you kept the input, the retrieved context, and the output together. Your team stops saying "it feels better" in review threads. And when the number does move, you can say how much of the movement is noise: with a 30-case set, two or three cases flipping is not a trend, and knowing that stops you from shipping on a coin toss.
What's the failure that made you finally build a test set — and did the eval catch it again later? Tell us in the comments.
Sources: Hamel Husain — Your AI product needs evals · Anthropic — A statistical approach to model evaluations · arXiv — Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena · arXiv — Rubric artifacts in LLM-based evaluation (EMNLP 2026) · OpenAI Evals (GitHub)
