Incrementality testing for agent-driven campaigns
AI-assisted research. This note was prepared with the assistance of AI tools. It may contain errors and is not professional advice. See our Legal Disclosures.
Agents now run real parts of campaigns: personalizing lifecycle email, following up on leads, and adjusting paid media budgets. When results improve after an agent takes over, it is natural to give the agent credit.
Results change for many reasons, though: seasonality, market conditions, competitor moves, other campaigns, and plain chance. Without a comparison, you cannot tell whether the agent caused the improvement or simply happened to be running when things got better.
Incrementality testing answers the question directly: what difference did the agent make compared with not using it?
Why before-and-after comparisons mislead
The most common evidence for an agent is a before-and-after chart. Conversion was one number last quarter and a higher number this quarter, and the agent launched in between. That chart mixes the agent's effect with everything else that changed over the same period.
Attribution models do not solve this either. They assign credit among touches that happened, but they cannot say what would have happened without the agent. Our post on attribution in an agentic funnel explains why that gap widens when agents touch every step.
The basic design
An incrementality test splits comparable units into two groups. In the treatment group, the agent handles the work. In the control group, the existing process handles it.
You then compare outcomes over the same period. Because both groups face the same season and market, the difference between them is a fair estimate of the agent's effect.
Units can be contacts, accounts, regions, campaigns, or time periods, depending on the workflow. Choose the unit at which the agent's work actually lands. If an agent writes one email per account, randomize accounts, not individual contacts, or colleagues at the same account will see both versions.
Examples by workflow
The design adapts to most agent workflows. These three show how the unit and the metric change with the work.
- Lifecycle email: randomly assign customers to agent-personalized messages or the current template, and compare activation or renewal rates.
- Lead follow-up: randomly assign new leads to agent-assisted follow-up or the current process, and compare meetings booked.
- Paid media: run agent-managed and human-managed budgets in comparable markets, and compare cost per outcome.
Paid media is the hardest of the three, because markets are few and never identical. Geographic tests need matched pairs of markets and longer windows. Our framework for agentic paid media covers where the agent's loops sit, which helps decide what to hold constant.
Getting the design right
Problems in a test are far easier to prevent in the design than to correct in the analysis. Five rules cover the common ones.
- Randomize when you can: random assignment is the best protection against hidden differences between groups.
- Keep groups separate: make sure the control group receives no agent-driven work, because leakage shrinks the measured difference.
- Decide the metric in advance: write down one primary outcome and the comparison before the test starts.
- Size the test first: estimate how many units you need to detect the effect you care about.
- Account for novelty: new messages sometimes win simply because they are new, so run long enough for that to fade.
Writing the plan down before launch is the cheapest safeguard available. Checking many metrics afterward and reporting whichever looks best will produce false conclusions, even when everyone is acting in good faith.
A worked sizing example
Sizing is easy to skip, and it decides whether a test can answer anything. A common rule of thumb for comparing two conversion rates is that each group needs about 16 times p(1 − p), divided by the square of the difference you want to detect. Here p is the baseline rate. That approximation assumes conventional settings: about 80% power and a 5% significance level.
Consider a hypothetical renewal email program with a baseline renewal rate of 5%. The team wants to detect a one-point lift, to 6%. The arithmetic gives 16 × 0.05 × 0.95, divided by 0.0001, or about 7,600 customers per group.
If the program only reaches 3,000 customers a quarter, that test cannot work as designed. The team has three honest options: run it for longer, look for a larger effect, or choose a metric closer to the agent's work. Click-through is one such metric, with renewals kept as a secondary read. Discovering this before launch saves a quarter of inconclusive data.
When full tests are impractical
Not every workflow allows a clean randomized test. These alternatives give weaker evidence, but evidence nonetheless.
- Holdouts: keep a small share of contacts permanently outside agent-driven activity as an ongoing baseline.
- Staggered rollout: introduce the agent in different regions or segments at different times, and compare trends.
- Matched comparison: compare against similar units that did not receive the agent, while stating that the evidence is weaker.
Of these, we usually recommend a standing holdout, because it keeps paying off. Holding back 5% to 10% of eligible contacts gives a running baseline for every agent that touches them. The cost is the forgone benefit to that group, which is worth stating openly when you propose it.
Holdouts need clean data to work. You must be able to show that held-out contacts received no agent-driven touches, which means logging every touch with its source. A shared timeline such as the one in unified event streams makes that check possible.
Reporting results honestly
Report the estimated difference along with its uncertainty, usually as a confidence interval. "Meetings booked were higher in the agent group, but the difference is within the range we would expect by chance" is an honest and useful result.
State what the test did not cover as well. A test on one segment in one season says little about another segment in a different season. Results like these feed directly into year-end reporting, which we cover in the year-end measurement plan.
What we are studying next
Agents change during a test. Instructions get revised and models get updated, so the treatment at week eight may differ from the treatment at week one. We are studying whether to freeze agents during tests or to treat the agent as a moving system and measure the program as a whole.
How to apply this: for your next agent rollout that affects customers, size the test before launch, keep a random control group on the existing process, and write down the primary metric in advance. Report the difference with its uncertainty, not as a single number.
Related research
Golden datasets: building test sets for marketing agent evaluation
A small set of test cases with agreed good answers is the most useful evaluation tool a marketing team can build. It can be built in a week.
Failure budgets: deciding how much agent error is acceptable
No agent is perfect, and neither is any human process. A failure budget set before launch turns vague worry about agent mistakes into a managed decision.
Synthetic audiences are a thinking tool, not an evidence tool
Asking AI personas to react to messaging is fast and cheap. It can sharpen early thinking, but it cannot replace evidence from real customers.
Put this research to work
Marketix turns these patterns into software: agents, automated workflows, and a hub that keeps people in control. Sign up for early access.
