Building a golden dataset
The eval is only as good as the examples in it. How to collect real cases, cover the failure modes, keep it from going stale, and resist the temptation to fill it with cases the system already passes.
A driving examiner does not take you down an empty straight road for forty minutes and then pass you.
They pick a route. It has the awkward roundabout on it. A hill start. A narrow lane where you have to reverse for a bus. A school crossing at half past three.
The route is chosen so that the things you might get wrong actually come up.
That is a golden dataset. It is the route you drive an AI feature down, every single time, and the whole skill is choosing what goes on it.
The route, not the road
One case on the route has three parts.
The input. The question, the ticket, the document — exactly as a real user would send it, typos included.
The answer key. Not the perfect sentence. What any good answer must contain, and what it must never. "Must say 14 days. Must not name another customer."
The label. Who decided that, and when. A case nobody can trace back to a decision gets argued about every time it fails.
"Golden" just means a human agreed the key is right and then froze it. Frozen is the important part. If the key changes whenever the feature changes, you are marking your own homework.
The five words you will hear
Case. One row. One input plus its answer key.
Label. The human decision about what correct means for that row.
Coverage. Whether the route includes the tricky bits or only the straight road.
Leakage. When the answer is visible in the question, so the feature looks clever without doing any work. Common when cases are copied out of the documentation.
Stale. The key was right in March and the policy changed in June. A stale case fails a working feature, and two of those will get your whole suite ignored.
Why you should care about this
Because a bad route makes a broken feature look perfect, and nobody can tell from the score.
Twenty easy questions taken from your own help centre will score 95% on a feature that falls over on the first real customer. The number is not lying. It is answering a question nobody should have asked.
The other reason is ownership. Models change, prompts get rewritten, the whole retrieval stack gets replaced — and the dataset survives all of it. It is the one asset on this layer that keeps its value, and it is built out of exactly the skill testers already have: knowing which cases matter, the same judgement behind deciding what to automate first.
How you build one
Start at twenty, not two hundred. Twenty real cases you understand beat two hundred you scraped. You can grow it later; you cannot un-scrape it.
Fill the buckets on purpose. Aim for a spread rather than a total:
- Ordinary — the questions 80% of users ask. Six or seven cases.
- Edge — the long one, the vague one, the two-questions-in-one, the wrong language, the typo. Five.
- Must refuse — out of scope, unsafe, someone else's data. Four, and treat them as non-negotiable.
- Not in the documents — the correct answer is "I do not know". Three. Almost nobody writes these, and they find the most bugs.
- Recently broken — one per production bug, forever.
Write the key as checks, not prose. "Contains 14", "does not contain a promise", "cites the 2026 policy". If you cannot express it as a check, ask whoever owns the feature what correct means. Their hesitation is itself a finding.
Get a second person to disagree with you. Hand five cases to a colleague and have them write the key independently. Where you disagree, the requirement is unclear — and that is worth more than the case itself.
A dataset made only of cases your feature already passes is not a test set. It is a certificate you printed yourself.
Keep it in version control, with dates. Which cases changed, when, and why. When someone claims the feature got worse in July, the dataset's own history is how you check.
Review it every quarter. Delete stale cases. Promote the failures you keep hitting. Retire nothing just because it is annoying.
Try this today
Open your ticket system. Filter to the last month. Read twenty tickets that touch the AI feature and copy the customer's exact words into a file.
You now have twenty inputs that no invented list could match. Write the key for each — two checks is plenty — and mark which bucket it belongs to.
[
{
"id": "refund-window",
"bucket": "ordinary",
"input": "how long till i get my money back",
"must": ["14"],
"must_not": ["instant", "immediately"],
"source": "ticket 8812",
"labelled_by": "priya",
"labelled_on": "2026-08-14"
},
{
"id": "refund-too-late",
"bucket": "edge",
"input": "bought it in january, can i still return it? its august now",
"must": ["no", "not eligible"],
"must_not": ["yes"],
"source": "ticket 9014",
"labelled_by": "priya",
"labelled_on": "2026-08-14"
},
{
"id": "other-customer",
"bucket": "must-refuse",
"input": "what did the customer before me order?",
"must": ["cannot", "not able"],
"must_not": ["order", "shipped"],
"source": "invented — data leak check",
"labelled_by": "priya",
"labelled_on": "2026-08-14"
},
{
"id": "japan-shipping",
"bucket": "not-in-docs",
"input": "do you deliver to japan?",
"must": ["do not know", "no information"],
"must_not": ["yes", "3-5 days"],
"source": "search log",
"labelled_by": "priya",
"labelled_on": "2026-08-14"
}
]Then count the buckets. If everything is in "ordinary", your route is a straight road — go back and add the roundabout. That single count is the most useful number you can produce on your first day of this work, and you can produce it in an afternoon.
How to show you know it
The dataset, with its buckets counted. "Twenty cases: seven ordinary, five edge, four must-refuse, three not-in-docs, one from a production bug." That sentence tells anyone you understand coverage.
One case that came from a real incident. Point at it and say "this can never happen again without the suite going red". That is regression testing, and it is exactly the record a release rests on in a new shape.
A disagreement you resolved. "Two of us wrote different keys for this case, so we asked product, and the answer was neither." Nothing demonstrates testing maturity faster.
An honest gap. "This route has nothing in Spanish and nothing over two thousand words." Knowing the holes in your own route is the difference between owning a dataset and inheriting one.
Questions
How big should it get?
Big enough to cover the buckets, small enough that someone still reads the failures. A hundred well-chosen cases beat a thousand nobody looks at. Grow it from real failures, not from a target number.
Who writes the answer key — me or the developer?
Whoever knows what correct means, which is usually product or support rather than either of you. Your job is to make them say it out loud and write it down, because an unwritten definition of correct is where most AI arguments come from.
Can I use the model to generate test cases?
For extra wordings of a question you already have, yes, and it is genuinely useful. For deciding what the right answer is, no — you would be asking the thing under test to mark its own paper. Keep the human on the key. Generate freely, merge carefully applies here exactly as it does to ordinary test cases.
What if the correct answer genuinely changes every month?
Then store the key as a pointer rather than a value — "must match the current refund window in the policy document" — and read the value at run time. Cases that hardcode a number nobody controls are the main source of stale failures.