Leading indicators for agent programs: what to watch weekly
AI-assisted research. This note was prepared with the assistance of AI tools. It may contain errors and is not professional advice. See our Legal Disclosures.
Agent programs are now judged on business outcomes: more pipeline, better retention, lower cost per campaign. Those outcomes take weeks or months to measure reliably, and waiting that long to learn something is wrong is too slow.
Leading indicators fill the gap. They are measures that move quickly and, if chosen well, predict whether outcomes will follow. This post sets out the weekly set we recommend for each significant agent, how to read it, and how to check that it predicts anything at all.
Why lagging measures arrive too late
Consider what a lagging measure can and cannot tell you. Pipeline from an agent-assisted nurture program reflects work done weeks earlier, mixed with sales effort, seasonality, and market conditions. By the time it dips, the cause may be a month old.
Agent problems usually start small and specific. A product fact goes stale, an input format changes, or a review queue backs up. Each one shows up first in operational measures, long before it shows up in revenue. The job of a leading indicator is to catch that early signal while it is still cheap to fix.
Quality indicators
Quality indicators tell you whether the agent's work is still good enough to use. We recommend three.
- Rubric score: the mean reviewer score on a random weekly sample of outputs.
- Human edit rate: how much output changes before use, as described in measuring human edit rate.
- Rejection rate and reasons: the share of outputs rejected, grouped by reason.
Rising edit or rejection rates are often the first sign of a problem, such as stale reference data or a changed input format. The reasons matter as much as the rate. A jump in rejections tagged "wrong price" points to a specific fix.
Reliability indicators
Reliability indicators tell you whether the agent is running as designed. They come from run logs rather than reviewers, so they cost little to collect.
- Run success rate: the share of runs that complete without error.
- Escalation rate: how often the agent asks for help, and why.
- Guardrail triggers: how often hard limits block an action.
A sudden change in any of these deserves a look, even if outputs seem fine. A drop in escalations can be as worrying as a rise, because it may mean the agent has stopped recognizing cases it should hand over. Our post on observability for marketing agents covers the traces that make these counts possible.
Efficiency and adoption indicators
Efficiency indicators tell you whether the agent is worth what it costs. Adoption indicators tell you whether people actually want it. We group them because both answer the question of value.
- Cost per task: model and tool spend per completed task, including retries and checks.
- Time to completion: from request to usable output, including time spent waiting for review.
- Review backlog: items waiting for human approval at the end of each week.
- Usage: how often people use the agent when using it is optional.
- Coverage: the share of eligible work that goes through the agent.
Review backlog deserves special attention. When it grows, quality controls weaken and reviewers start approving faster than they can read. See approval queues without bottlenecks for ways to keep it in check. Declining usage is also a strong signal, since it means people have found the agent unhelpful, whatever the other measures say.
Cost per task needs a consistent definition to be useful. Our post on cost per task explains what to include.
Defining normal ranges
An indicator is useful only if you know what normal looks like. We suggest a simple method that needs no special tools. For each indicator, compute the median of the previous four to eight weeks, and flag any week that moves well outside the usual spread.
Consider a hypothetical agent whose weekly edit rate has sat between 18% and 24% for two months. A week at 31% is worth investigating. A week at 25% probably is not, unless it is the third rise in a row. Trends of three or more weeks in one direction deserve a look even inside the normal band.
Watch sample sizes too. If a rubric score comes from five reviewed outputs, one bad draft can move it sharply. Record the sample size beside each weekly figure, and treat movements in small samples with caution.
Making the review useful
A weekly review should take about 15 minutes per agent. We suggest a fixed sequence, so the review stays short and nothing gets skipped.
- Compare each indicator with its normal range from the previous four weeks.
- Note anything outside the range or trending for three weeks.
- Read five recent outputs and two recent rejections.
- Decide on one action, or confirm that no action is needed.
Reading real outputs is not optional. Indicators summarize, and summaries hide things. Five outputs a week keeps the reviewer close to the work the numbers describe.
Keep a short log of these reviews. Over time, it becomes a record of how the agent has changed and why.
Scale changes the shape of the review. With one or two agents, a single owner can run it. With many, we suggest each agent owner runs their own review. One person then scans the flags across all agents for shared causes, such as a reference source that feeds several of them.
That two-level review fits naturally into a team's operating rhythm.
Checking that indicators lead
Leading indicators are useful only if they actually predict outcomes. Every quarter, compare periods of strong indicators with the outcomes that followed a few weeks later. If edit rate rose in a month and conversion held steady, edit rate may still matter for cost but not for results.
Expect some indicators to fail this check. That is useful information, not a setback. Drop the ones that predict nothing, and keep the ones that cost little to collect and warn you early.
Be careful about turning indicators into targets. Once a team is rewarded for a low edit rate, reviewers may edit less rather than the agent improving. Indicators work best as diagnostics, not goals.
What we are studying next
We are studying how long the lag is between quality indicators and business outcomes for different kinds of agents. A lifecycle email agent may show effects within weeks, while an account research agent may take a full sales cycle. Knowing that lag would tell teams which indicators to trust for which decisions.
How to apply this: set up a one-page weekly view for your most important agent with quality, reliability, efficiency, and adoption indicators, each with its normal range. Review it for 15 minutes a week, read five real outputs, and log one decision.
Related research
Failure budgets: deciding how much agent error is acceptable
No agent is perfect, and neither is any human process. A failure budget set before launch turns vague worry about agent mistakes into a managed decision.
Evaluation drift: why last quarter's tests stop working
Test sets and rubrics age. When the business, the audience, or the agent's job changes, evaluations can keep passing while real quality falls.
Incrementality testing for agent-driven campaigns
The reliable way to know whether an agent changed results is to compare against what would have happened without it. Incrementality testing does that.
Put this research to work
Marketix turns these patterns into software: agents, automated workflows, and a hub that keeps people in control. Sign up for early access.
