From pilot to program: graduating an agent into production
AI-assisted research. This note was prepared with the assistance of AI tools. It may contain errors and is not professional advice. See our Legal Disclosures.
Agentic marketing has moved quickly from demos to pilots, and the harder step now is turning a pilot into a program a team can rely on. That step is where agents tend to stall, or get pushed forward too early.
This post sets out a graduation checklist for making that decision deliberately. It covers what to check, who should decide, how to handle gaps, and what happens after graduation, including the decision to retire a pilot instead.
Two ways pilots go wrong
Some pilots linger in pilot mode indefinitely. A few people use them, nobody fully trusts them, and nobody formally retires them. They consume attention, model spend, and maintenance without producing evidence either way.
Others are pushed into wide use before they are ready, often because a demo impressed someone senior. Both waste effort, and the second can cause real harm. An agent that writes customer emails at scale with untested guardrails turns small errors into large ones.
The common cause is that the graduation decision is never made explicitly. Pilots drift into production or drift into neglect. A checklist turns that drift into a decision, with criteria agreed before anyone is attached to the outcome.
The graduation checklist
The checklist has five areas. Safety and governance items are mandatory, while gaps in the other areas can be accepted with a plan, as described in the next section.
Purpose and ownership
An agent without a clear owner has no one to notice when it degrades. These items establish who is responsible and for what.
- The agent's goal, scope, and boundaries are documented
- A named owner is accountable for its performance
- A backup owner is identified
Evidence of value
Value evidence should come from real work. A strong test score shows that an agent can do the job, and real use shows whether it does the job well under ordinary conditions.
- Time saved, quality improvement, or outcome effect is measured against a baseline
- Evidence comes from real work, such as shadow mode or live assist use, as well as test cases
- Users report that it helps and choose to use it
The third item is easy to dismiss and hard to fake. If practitioners route around the agent when they are busy, that tells you more than a dashboard does.
Quality and evaluation
Quality checks make sure the agent stays good after graduation, not only on the day of the decision.
- A golden dataset exists and is run before each change
- Rubric scores and edit rates are tracked
- A failure budget is defined, and the agent has stayed within it
Safety and governance
These items protect customers and the brand. Any unchecked item here blocks graduation, because these are the gaps that turn a contained mistake into a public one.
- Autonomy level is documented with the evidence that justified it
- Important rules are enforced in code
- Permissions follow least privilege
- A red-team session has been completed and its findings addressed
- Each output can be traced to the run that produced it
Traceability is easy to treat as optional, yet it is the item you need most after an incident. If a customer complains about a message, you should be able to find the run, the instructions version, and the facts the agent used. Without that trail, investigation turns into guesswork.
Operations
Operations items make the agent maintainable by someone other than the person who built it.
- Instructions are versioned with change notes
- Cost per task is tracked and budgeted
- Alerts exist for failures, spikes, and guardrail triggers
- The agent has a place in the team's operating rhythm
- A pause procedure exists and has been tested
A useful test for this area is a handover. If the owner went on leave for two weeks, could the backup owner keep the agent running from the documentation alone? If the honest answer is no, the agent is still a pilot that depends on one person.
Using the checklist
Review the checklist in one meeting with three people: the agent's owner, a reviewer who has read its outputs, and someone responsible for governance. The owner presents evidence for each item. The others decide whether the evidence is enough.
Any unchecked item in safety and governance blocks graduation. Gaps elsewhere can be accepted with a plan and a date to close them. Record each accepted gap with an owner, so it does not quietly become permanent.
Consider a hypothetical lifecycle email agent that has run in shadow mode for eight weeks. Its edit rate has been stable, the email team uses it daily, and it has a golden dataset and a defined failure budget. The review finds two gaps: the backup owner is unnamed, and the pause procedure has never been tested.
The first gap can be closed within a week, so it is accepted with a date. The second is listed under operations, but an untested pause is a safety risk in practice. A careful review holds graduation until a test pause has run.
Graduation is not the end
Production agents still need attention. Models change, data changes, and the business changes. An agent that passed the checklist in spring can drift out of line with it by autumn without any single change looking significant.
Plan a review of each production agent at least quarterly, using the same checklist. Be willing to move an agent back to pilot status if quality or relevance declines. Moving back should be an ordinary decision, recorded like any other, rather than a judgment on the team that built it.
Retiring pilots
The checklist also supports the opposite decision. A pilot that cannot meet the value criteria after a reasonable period should be retired. We suggest setting that period when the pilot starts, for example one quarter, so the retirement decision does not depend on anyone's mood.
Retirement should come with a short write-up: what the pilot tried, what evidence it produced, and why it stopped. That write-up can prevent another team from repeating the same experiment. It also preserves useful parts, such as test cases or instructions, that a later agent can reuse.
What we are studying next is graduation for agents that work together. A single agent can pass the checklist while the system it belongs to has gaps at the handoffs. We do not yet have a checklist for the seams between agents, and we expect it to look different from this one.
How to apply this: run your longest-running pilot through this checklist this month. Either schedule its graduation with a plan to close the gaps, or retire it and write down what you learned.
Related research
2027 planning framework: mapping your marketing org to agents and humans
Next year's plan should say who and what will do the work, not only what marketing will do. Mapping activities to people and agents makes it realistic.
Q4 readiness: stress-test your agents before your busiest quarter
The fourth quarter brings the year's highest volume and little room for error. September is the time to learn how your agents behave under pressure.
Q4 execution checklist: keeping agent programs steady through year-end
Planning and readiness testing are done. This checklist covers the daily, weekly, and as-needed practices that keep agent programs steady through Q4.
Put this research to work
Marketix turns these patterns into software: agents, automated workflows, and a hub that keeps people in control. Sign up for early access.
