Building a small golden dataset from real cases

A golden dataset is a curated set of inputs paired with either an expected output or the criteria for grading one. It is the single most important artifact in evaluation: without it, every prompt or model change is a guess. You run your app against this set repeatedly to measure quality and catch regressions.

Start small and real. A reviewed set of twenty to fifty cases already catches gross regressions, and it beats a thousand synthetic examples that do not match how people actually use the feature. Pull cases from real traffic where possible: typical requests, the edge cases that confused the app, and any input tied to a past incident. Every incident becomes a permanent test so the same failure cannot return unnoticed.

Label each case with what a good answer requires, not just a single blessed string. For open-ended tasks an exact expected output rarely exists, so store grading criteria instead. Version the dataset like code, with an owner and a changelog, because it will grow and its meaning must stay stable.

The reliability angle: this set is your regression suite and your safety net. Grow it toward a few hundred cases that mirror real usage, and keep a smaller fast subset for quick checks. When production surfaces a new failure, add it here before you fix it, so the fix is proven and protected.