Evaluating content agents before they ship: a starter rubric
AI-assisted research. This note was prepared with the assistance of AI tools. It may contain errors and is not professional advice. See our Legal Disclosures.
Content agents are moving from experiments into real production work: product emails, landing page drafts, social posts, and sales enablement. The tools for writing have improved faster than the tools for judging what gets written.
Most teams still evaluate content agents the way they evaluate a new freelancer. They read a few samples and decide whether the work feels right. That works for a first impression, but it breaks down quickly: reviewers disagree, standards shift, and nobody can say whether last month's change helped.
A simple rubric fixes most of this. This post sets out the one we use as a starting point, along with the sampling and calibration method that makes its scores worth trusting.
Why impressions fail
Impressions fail for three reasons, and each one has a measurable cause. The first is reviewer variance. Two capable editors can read the same draft and reach different verdicts, because each weighs tone, accuracy, and structure differently without saying so.
The second is anchoring. A reviewer who has just read a weak draft will rate an average one generously. Scores then depend on reading order as much as on quality.
The third is memory. Without recorded scores, a team cannot compare this month with last month. Every instruction change becomes a matter of opinion, and the loudest opinion tends to win.
The five criteria
Score each criterion on a four-point scale, where 4 is the best. We avoid a middle option so reviewers have to commit to a side. Each criterion asks one question, and the anchors describe the two ends of the scale.
- Accuracy: are all facts, claims, and numbers correct and supported by approved sources? A 4 means everything is verifiable; a 1 means the piece contains a false or unapproved claim.
- Brief fit: does the piece serve the stated objective, audience, and message? A 4 is clearly built for this reader and goal; a 1 is generic or aimed at the wrong reader.
- Voice: does it follow the brand voice specification? A 4 reads like strong human work; a 1 is off brand in ways a customer would notice.
- Clarity: is it easy to read and act on? A 4 has a clear structure and one obvious next step; a 1 is confusing, padded, or missing a call to action.
- Compliance: does it meet legal, disclosure, and policy requirements? A 4 is fully compliant; a 1 needs changes before it can ship.
Write the anchors for 2 and 3 as well, using real examples from your own work. Anchors built from your drafts are far easier to apply than abstract descriptions. The voice criterion depends on having a written voice standard, which we cover in brand voice as a specification.
Treat accuracy and compliance as gates
Not all criteria carry equal weight. A well-written email with a false claim is worse than a plain email that is correct. We treat accuracy and compliance as gates: a score of 1 or 2 on either one fails the piece, whatever the other scores say.
The other three criteria are averaged into a quality score. Keeping gates and quality separate matters for reporting. If you average all five, a strong voice score can hide a compliance failure, and the summary number will look healthier than the work.
We suggest reporting two figures for any batch: the gate pass rate and the mean quality score among pieces that passed. A falling gate pass rate calls for immediate action. A falling quality score calls for a closer look at instructions or reference material.
Calibrate your reviewers
A rubric is only as consistent as the people applying it. Before using it at scale, have two or three reviewers score the same set of pieces independently. We recommend at least 20 pieces for this round, chosen to include both strong and weak drafts.
Then compare the scores criterion by criterion. Three simple measures are enough to start.
- Exact agreement: the share of pieces where reviewers gave the identical score.
- Adjacent agreement: the share where scores differ by no more than one point.
- Gate agreement: the share where reviewers agree on pass or fail for each gate.
Gate agreement matters most, because gates decide what ships. We suggest treating anything below 90% gate agreement as a sign that the gate definitions need rewriting. For teams that want a chance-corrected measure, a weighted kappa statistic suits ordinal scales like this one, and most spreadsheet tools can compute it.
Where scores differ by more than one point, discuss the piece and write down the reason. Those notes become rubric guidance. Repeat calibration whenever reviewers change, and at least once a quarter, because reviewers drift too.
Sample with a plan
Once the rubric is calibrated, you do not need to score everything. You need a sample that represents the work. Draw it at random from each content type, rather than letting reviewers pick the pieces they happen to see.
Consider a hypothetical team whose agent produces 200 pieces a month across email, social, and web copy. Scoring 30 to 40 pieces, stratified across the three types, gives a usable monthly read without consuming a reviewer's week. Every piece that a reviewer flags in normal work should be scored as well, but kept apart from the random sample so it does not skew the averages.
Blind the sample where you can. If reviewers know which drafts came from the agent and which came from people, their expectations shape the scores. Mixing a few human-written pieces into the batch also gives you a baseline on the same scale.
Add one effort measure
Alongside the rubric, capture how long it took to bring each piece to a shippable state. Rubric scores tell you about quality, and edit time tells you about value. A draft that scores well but takes as long to fix as writing from scratch is not saving anyone time.
Edit time also catches a weakness in the rubric itself. If two drafts score the same but one took three times as long to finish, the rubric is missing something the editors care about. Our post on measuring human edit rate explains how to collect this without adding much work.
What good looks like
The rubric supports a decision, so set the decision rule before you collect scores. A content agent is ready for more volume when, over a meaningful sample, three conditions hold.
- Gates hold: it never fails the accuracy or compliance gates.
- Quality matches: its mean quality score is close to your human baseline on the same blind sample.
- Effort falls: edit time is clearly lower than writing from scratch.
Until then, the agent stays at the assist level of the autonomy ladder, with every piece reviewed. When the rules are met, the scored sample becomes the evidence for the next gate in the evaluation gates process.
Where the rubric goes next
A starter rubric is a first version, not a finished instrument. As the agent improves, scores bunch at the top and the scale stops separating good work from great work. At that point, the anchors need raising or the criteria need splitting.
We are now studying whether a second model can apply the gate criteria reliably enough to pre-screen drafts, with humans scoring a sample. The open question is how much agreement between model and human graders is enough to trust that screen on customer-facing work.
How to apply this: calibrate two or three reviewers on 20 shared pieces, then score a random, blind sample of your agent's next month of work with edit time recorded. Expand the agent's role only when the gates hold and quality matches your human baseline.
Related research
A Q1 retrospective template for marketing programs that use AI
A first quarter of AI work produces plenty of activity and little evidence. This 90-minute retrospective turns three months of experiments into decisions.
Measuring human edit rate: the quiet metric for agent quality
How much people change agent output before using it may be the most useful sign that an agent is helping. Here is how to define, collect, and read it.
Attribution in an agentic funnel: credit decisions, not just channels
When agents research, write, target, and follow up, the question shifts from which channel worked to which decisions worked. Attribution has to follow.
Put this research to work
Marketix turns these patterns into software: agents, automated workflows, and a hub that keeps people in control. Sign up for early access.
