Golden datasets: building test sets for marketing agent evaluation
AI-assisted research. This note was prepared with the assistance of AI tools. It may contain errors and is not professional advice. See our Legal Disclosures.
Marketing agents change constantly. Instructions get revised, models get swapped, and reference material gets updated, often in the same month.
Every one of those changes raises the same question: did that make things better or worse? Without a consistent way to check, teams judge from a handful of examples, and those impressions are often wrong.
A golden dataset is the answer. It is a curated set of realistic inputs, each paired with an expected output or a clear description of what a good output looks like. You run the agent against it before and after any change and compare the results case by case.
Why a fixed set beats spot checks
Spot checks feel informative because they use real work. Their weakness is that the examples change every time. If this week's samples are easier than last week's, the agent looks better without having improved.
A fixed set removes that variation. The same inputs go in before and after a change, so any difference in the outputs comes from the change itself. This is the same logic as a controlled experiment, applied to a single agent.
What makes a good golden dataset
Five properties separate a useful set from a pile of examples. Each one is a choice you make while building it.
- Realistic: inputs come from real work, such as actual briefs, real leads, and genuine customer questions, with sensitive details removed.
- Representative: common cases appear in roughly the proportion they occur, so the overall score reflects typical performance.
- Challenging: the set includes hard cases, such as ambiguous briefs, incomplete data, sensitive topics, and inputs that caused problems before.
- Reviewable: it is small enough that a person can read the results, which for most marketing agents means 30 to 100 cases to start.
- Clearly graded: every case has a written definition of what good looks like.
Representative and challenging pull in opposite directions. We handle this by tagging every case as typical or hard and reporting the two scores separately. A change that improves the typical score while lowering the hard score is a different result from one that improves both.
Grading criteria that two people agree on
Grading is the hardest part to get right. Some tasks have an exact answer, such as the correct segment for a lead or the right product for a query. Others need a set of checkable criteria, such as "mentions the integration", "stays under 150 words", or "includes the required disclosure".
Write criteria that a second person would apply the same way. "Sounds on brand" is not checkable. "Ends with exactly one call to action and uses no phrase from the banned list" is. For softer qualities, link the case to your scoring rubric, such as the one in our starter rubric for content agents.
Then test the criteria. Have two people grade the same baseline outputs independently and compare. Where they disagree, the criterion is ambiguous, and it needs rewriting before the dataset is used to make decisions.
Building one in a week
A first version does not need a long project. This sequence fits into a normal working week for one owner and one reviewer.
- Collect: pull 50 recent real inputs for the agent's task, removing personal data.
- Select: choose a mix of typical and hard cases, and add every case that caused a problem before.
- Define expectations: write the expected output or grading criteria for each case, agreed by two people.
- Baseline: run the current agent, grade the outputs, and record the scores per case.
- Store it: keep the dataset with the agent's versioned instructions, so it is easy to rerun.
Storing the set beside the instructions matters more than it looks. When someone asks why a score changed in March, you want the instructions, the dataset version, and the results in one place. Our post on versioning agent instructions describes a structure that holds all three.
A worked example
Consider a hypothetical lead-routing agent with a golden dataset of 60 cases: 45 typical leads and 15 hard ones, including leads with missing company data. At baseline, suppose it routes 42 typical cases and 9 hard cases correctly.
The team then changes the instructions to handle missing data better. On the rerun, suppose the hard score rises to 12 while the typical score falls to 40. The total barely moves, from 51 to 52. A team watching only the total would call this a wash.
Case-by-case comparison tells a different story. The two typical cases that broke both involve a region the new instructions describe ambiguously. That is a specific fix, found in minutes, because the comparison was per case rather than in aggregate.
Using it well
A dataset is a living tool. These practices keep it useful after the first month.
- Compare per case: look at which cases changed, not only at the total score.
- Add real failures: turn every production failure into a new case, so it cannot return unnoticed.
- Refresh quarterly: add cases from current work and retire ones that no longer reflect it.
- Keep a holdout: hold back a set of cases you never use while tuning, so you can tell real improvement from fitting the test.
The holdout guards against a quiet failure. If you adjust instructions until every visible case passes, the agent can become good at the test and no better at the job. Our post on evaluation drift explains how this happens and how to separate a frozen core from a rotating set. Adversarial cases from red-teaming make good additions to the hard tier.
Grading at scale
For large datasets, teams sometimes use a second model to grade outputs against the written criteria. This can work well for clear, checkable criteria. It needs calibration first: grade a sample by hand, compare with the model's grades, and measure how often they agree.
We suggest humans review every case where the model grader is uncertain or disagrees with the previous run. Recheck the calibration whenever the grader model or the criteria change. Golden datasets are the first gate in our evaluation gates diagram.
Open questions
We are now studying how large the hard tier needs to be before its score is stable from run to run. A set with only a few hard cases can swing widely on one result. We expect the answer depends on how varied the agent's inputs are, and we are testing ways to estimate it before a set is built.
How to apply this: build a 30-case golden dataset for your most important agent this month, with at least five cases that caused problems before and grading criteria two people agree on. Run it before every change and compare results case by case.
Related research
Synthetic audiences are a thinking tool, not an evidence tool
Asking AI personas to react to messaging is fast and cheap. It can sharpen early thinking, but it cannot replace evidence from real customers.
Incrementality testing for agent-driven campaigns
The reliable way to know whether an agent changed results is to compare against what would have happened without it. Incrementality testing does that.
Shadow mode: run agents alongside humans before going live
The safest way to learn how an agent will perform on real work is to let it do the work in parallel, without its output reaching anyone. That is shadow mode.
Put this research to work
Marketix turns these patterns into software: agents, automated workflows, and a hub that keeps people in control. Sign up for early access.
