Evaluation drift: why last quarter's tests stop working
AI-assisted research. This note was prepared with the assistance of AI tools. It may contain errors and is not professional advice. See our Legal Disclosures.
Good practice for marketing agents now includes testing them before they ship, with test sets and rubrics kept alongside the prompts. That progress creates a quieter risk. Once an agent passes its tests consistently, the tests stop getting attention.
Evaluations are snapshots of what mattered when they were written. The business changes, products change, audiences change, and the agent's real work changes with them. The tests can keep passing while the agent performs worse on the work that matters now.
We call this evaluation drift. In our research, we treat it as a normal property of any test set that is left alone, in the same way that product data goes stale without upkeep.
How drift happens
Drift has four common sources. Each one widens the gap between what the test set measures and what the agent actually does.
The work changes. New products, new campaigns, and new audiences produce inputs that do not appear in the test set. An agent tested on spring promotions may never have seen a bundle offer with a shipping cutoff.
The standards change. The brand voice evolves, legal requirements shift, or leadership redefines what a qualified lead is. The rubric still encodes the old standard, so a draft can score well against rules the team no longer holds.
The agent is tuned to the test. Repeated adjustments made to pass specific cases can make an agent good at the test and no better, or worse, at the real task. This is the marketing version of teaching to the exam.
The hard cases get stale. Edge cases that were difficult six months ago may now be handled well. Meanwhile, new kinds of difficulty arrive and go untested, so the test set gets easier without anyone noticing.
A worked example
Consider a hypothetical lifecycle email agent with a test set of 200 cases, built in the spring. It scores well each week, and the team stops looking closely at the results.
Over the summer, the company launches a subscription product and tightens its rules on discount language. Neither change appears in the test set. The agent now writes subscription emails it was never tested on, and it sometimes uses discount phrasing the new rules forbid.
The test score stays flat. What moves is the human edit rate on subscription emails, which climbs week by week. Reviewers start leaving the same comment on several drafts. The dashboard says the agent is fine, and the reviewers say it is not.
Neither signal is wrong. They are measuring different work, and only the reviewers are looking at the current one.
Signs of drift
Drift rarely announces itself. These are the signals we suggest watching for, in rough order of how early they appear.
- Diverging measures: test scores stay high while human edit rates or rejection rates rise.
- Uncovered complaints: reviewers mention problems that no test case covers.
- Missing work: new products, campaigns, or audiences have no cases in the test set.
- Stale timestamps: the test set or rubric has not changed in months.
- Too-perfect scores: the agent passes nearly all cases, so the set no longer separates good from great.
The first signal matters most. Our post on measuring human edit rate explains why we treat edit rate as one of the earliest honest measures of agent quality.
Preventing drift
Prevention is mostly routine maintenance. We recommend five practices, run on a fixed schedule rather than when someone remembers.
- Refresh the test set each quarter with cases from recent real work, especially recent failures, and retire cases that no longer reflect current work.
- Keep a holdout set that you never use while adjusting the agent, so you can tell real improvement from tuning to the visible cases.
- Revisit rubric definitions whenever standards change, and recalibrate reviewers on a shared sample afterward.
- Check each quarter whether test scores still predict real measures such as edit rate, rejection rate, and campaign outcomes.
- Sample live production outputs each week and review them against the current rubric.
The holdout set needs care. If it is ever used to tune the agent, it stops being a holdout, and you lose your only clean read on whether changes generalize. We suggest storing it separately, with access limited to the person who runs the quarterly check.
Refreshing also has a cost. Adding new cases resets historical comparisons, because this quarter's score is no longer measured on last quarter's set. One way to handle this is to keep a frozen core of cases for trend lines and a rotating set for current work. Our guide to golden datasets covers how to structure both.
What to measure
The single most useful measure of drift is the gap between test performance and production performance. Track both on the same chart, by agent and by task type, and watch the distance between them.
We recommend reviewing three numbers each quarter: test score on the frozen core, test score on the refreshed set, and the production edit or rejection rate. If the frozen core holds steady while production quality falls, the test set has drifted. If both fall together, the agent itself has changed, often through a model or instruction update.
Coverage is worth tracking too. Map each product line, campaign type, and audience segment to the cases that test it. Gaps in that map are where drift starts.
Scale changes the picture. A program with one agent can refresh its test set by hand in an afternoon. A program with a dozen agents that share facts and rubrics needs an owner for evaluation itself. A single change to the brand voice can make several test sets stale at once.
We suggest logging each standards change with a list of the test sets it touches, so refreshes happen together rather than one agent at a time.
Evaluation is never finished
It is tempting to treat evaluation as a project that ends when the test set is built. In practice, it is an ongoing activity, like maintaining the agent itself. Budget time for it in your operating rhythm, and treat a stale test set as seriously as stale product data.
We are now studying how often a refresh is enough. Quarterly may be too slow for agents whose inputs change weekly, such as paid media or promotional email. We expect the right cadence to track how fast the underlying work changes, and we are testing ways to measure that rate directly.
How to apply this: compare your main agent's test scores with its edit and rejection rates over the last three months. If they have diverged, refresh the test set with recent real work before the busy season, and set aside a holdout you never tune against.
Related research
Year-end measurement plan: proving the value of agentic marketing
In three months, leadership will ask what agents delivered this year. A measurement plan set up now makes that answer credible.
Leading indicators for agent programs: what to watch weekly
Business outcomes take months to show up. A few weekly leading indicators, checked against normal ranges, show much sooner whether an agent program is healthy.
Failure budgets: deciding how much agent error is acceptable
No agent is perfect, and neither is any human process. A failure budget set before launch turns vague worry about agent mistakes into a managed decision.
Put this research to work
Marketix turns these patterns into software: agents, automated workflows, and a hub that keeps people in control. Sign up for early access.
