← Research library

Shadow mode: run agents alongside humans before going live

AI-assisted research. This note was prepared with the assistance of AI tools. It may contain errors and is not professional advice. See our Legal Disclosures.

Marketing teams now test agents before they ship, usually against prepared sets of examples. That testing tells you a lot, but real work brings messy inputs, unusual cases, and timing pressure that test sets miss.

Putting an unproven agent straight into a live workflow carries the opposite risk: its mistakes reach customers, prospects, or sales reps before anyone has seen them. Shadow mode sits between the two.

What shadow mode is

In shadow mode, the agent receives the same real inputs as the people doing the work and produces its own outputs. Those outputs are recorded rather than used. People keep doing the work as usual, and afterward you compare the two.

The agent gets real work without any real consequences. You learn how it handles the inputs it will actually see in production, and nothing it produces leaves the building. We treat shadow mode as one of the gates in our evaluation gates diagram, the step between offline testing and any live role.

How shadow mode works

A shadow run has four steps. Each one is simple, but the second needs the most care.

  1. Duplicate the input: when a real task arrives, such as a new lead to score or a renewal email to draft, a copy goes to the agent.
  2. Let the agent work: the agent uses the same tools and information it would have in production, with every outward action disabled or redirected to a log.
  3. Record both outputs: store the human output and the agent output side by side, with the shared input and a timestamp.
  4. Compare on a schedule: reviewers score a sample of pairs each week and note the differences.

The comparison is the point. A shadow run that records outputs but never reviews them is only a cost.

What shadow mode reveals

Offline tests answer whether an agent can do the task. Shadow mode answers a different question: how it does on the work that actually arrives. We look for four kinds of finding.

  • Coverage: which real cases the agent handles well and which it struggles with.
  • Edge cases: unusual inputs that never appeared in your test set.
  • Disagreement patterns: where the agent consistently differs from people, and who is usually right.
  • Operational issues: latency, cost, and tool failures under real load.

Edge cases found in shadow mode are worth keeping. Each one is a candidate for your test set, which is how a golden dataset stays current after launch.

A worked example

Consider a hypothetical B2B software team that wants an agent to qualify inbound demo requests. Today, two sales development reps read each request and mark it qualified, nurture, or disqualified.

The team runs the agent in shadow mode for three weeks. In this example, about 600 requests arrive in that period. The agent labels every one, and the labels go to a log that no rep sees.

Each week, a reviewer pulls 30 random pairs and every pair where the agent and the rep disagree by more than one category. Suppose the agent matches the reps on most routine requests but marks several students and job seekers as qualified. It also flags a handful of requests from existing customers as new leads, because it cannot see the customer list.

Neither problem would have shown up in the original test set, which contained only clean prospect requests. The first needs a rule in the agent's instructions. The second needs a tool change, giving the agent read access to customer status. Both get fixed before the agent goes live, and both cases join the test set.

Designing a good shadow run

A few design choices decide whether a shadow run produces evidence or noise. We suggest settling them before the run begins.

Run long enough. A single week may miss weekly cycles, month-end spikes, or unusual events. We recommend at least two weeks for recurring workflows, and four weeks when inputs vary by week.

Sample for review. You do not need to review every pair. A random sample plus every case of strong disagreement is usually enough to find the main failure patterns.

Keep the humans independent. If people see the agent's output before doing their own work, their work drifts toward it. The comparison then measures agreement with the agent, not quality.

Block real actions completely. Make sure the agent cannot send, publish, update a record, or notify anyone during the shadow period. Test this before starting by triggering each outward action and confirming it lands in the log.

Failure modes to watch for

Shadow runs have their own ways of going wrong. The most common is treating the human output as ground truth. People make mistakes too, and some disagreements will turn out to favor the agent. Reviewers should judge each pair on its merits rather than scoring the agent on agreement alone.

A second failure is a leak. A shadow agent with write access to a shared CRM field can change real data without anyone noticing. Audit permissions, not just send buttons.

A third is a run that never ends. Shadow mode feels safe, so teams extend it indefinitely. Set an end date and a decision meeting when the run starts.

What to measure

We suggest tracking four numbers through the run, by task type.

  • Acceptance rate: the share of sampled agent outputs a reviewer would have accepted as is or with light edits.
  • Agreement rate: how often the agent and the human reached the same answer.
  • Disagreement verdicts: in disagreements, how often the agent was judged right.
  • Catchability: the share of agent failures a normal review step would have caught.

Acceptance rate is a preview of the human edit rate you will see after launch. Catchability tells you how much review the agent will need.

Deciding what comes next

At the end of a shadow period, you should know how often the agent's output would have been acceptable and which cases it fails on. You should also know whether review would catch those failures. That evidence supports a specific decision: stay in shadow, move to an assist role where a person approves each output, or retire the idea. The autonomy ladder describes what each of those roles involves.

We are now studying how to run shadow mode for tasks without a human counterpart, such as work nobody currently has time to do. Without a paired output, the comparison has to come from reviewers alone. We expect that to need stricter sampling, and we are testing what that looks like.

How to apply this: before giving any agent a live role, run it in shadow mode for at least two weeks on real inputs with every outward action blocked. Review a sample of paired outputs weekly, and add each failure you find to the test set.

Related research

Put this research to work

Marketix turns these patterns into software: agents, automated workflows, and a hub that keeps people in control. Sign up for early access.