← Research library

Measuring human edit rate: the quiet metric for agent quality

AI-assisted research. This note was prepared with the assistance of AI tools. It may contain errors and is not professional advice. See our Legal Disclosures.

Content agents are now part of everyday marketing work, drafting emails, pages, and posts that people finish and ship. Most evaluation still focuses on quality scores and reviewer impressions of the drafts.

Those matter, but they miss what happens between the draft and the final version. If a reviewer rewrites half of every draft, the agent is not saving much time, however good its drafts look at first glance.

We call this measure human edit rate, and we think it deserves a permanent place on every agent dashboard.

What edit rate measures

Edit rate is the proportion of an agent's output that people change before it is used. It captures a judgment reviewers make anyway, as they fix what needs fixing. There are three common ways to measure it, and each tells you something different.

  • Text difference: compare the draft and the final version and calculate how much changed.
  • Edit time: record how long a reviewer spends bringing the draft to a usable state.
  • Edit categories: tag what kind of changes were made, such as factual fixes, tone, structure, or polish.

Text difference is easy to automate. Edit time captures value. Edit categories tell you what to fix. We recommend collecting all three, at different sampling rates.

Defining the text measure precisely

"How much changed" needs a definition before two teams can compare numbers. We suggest a word-level measure: the number of words inserted, deleted, or replaced to turn the draft into the final, divided by the number of words in the draft. Standard diff tools produce these counts.

Word level is a deliberate choice. Character-level measures overstate small fixes, since changing one product name can touch a dozen characters. Sentence-level measures understate them, because one changed word marks the whole sentence as edited.

Two edge cases need rules. If a reviewer discards the draft and writes from scratch, record it as a rejection rather than an edit rate of 100%. Mixing the two hides how often the agent fails outright. If the final is much longer than the draft, cap the rate at 100% and note the added length separately.

Consider a hypothetical 200-word email draft. The reviewer replaces 12 words, deletes eight, and inserts 10 new ones. That is 30 word changes against a 200-word draft, an edit rate of 15%. The same arithmetic applies to every piece, so rates stay comparable across long and short formats.

Why it is so useful

Edit rate has three properties that most quality measures lack. Together, they make it one of the earliest honest signals of agent quality.

It reflects real use. Reviewers do not grade drafts in the abstract. They fix what blocks publication, so edit rate captures their working judgment rather than a score given for the sake of scoring.

It connects to cost. Edit time translates directly into labor. Consider a hypothetical team whose drafts take 10 minutes to finalize, against 40 minutes to write from scratch. That 30-minute difference per piece is the agent's value, before model costs.

It responds to good fixes. When you fix a recurring problem, such as stale product facts or a vague voice specification, edit rate should drop. That makes it a direct check on whether a change helped.

Collecting categories without slowing reviewers

Text difference can run on every piece. Categories need a person, so collect them on a sample. We suggest tagging a random 20% of edit sessions, plus every session with an unusually high edit rate.

Keep the category list short, with five or six options at most. A typical set covers factual fixes, compliance changes, tone or voice, structure, and polish. Ask reviewers to pick the one or two categories that account for most of the edit, not to label every change.

Check that reviewers apply the categories consistently. Once a quarter, have two reviewers tag the same 20 edit sessions and compare. If they disagree often on a category, its definition needs work. Tone and structure are the pair we expect to blur first.

Interpreting the numbers

A falling edit rate is usually good news, but it can fall for the wrong reasons. Watch for three traps.

  • Reviewer fatigue: overloaded reviewers edit less while quality drops, so pair edit rate with periodic blind quality checks.
  • Light edits on heavy problems: one changed sentence can fix a serious factual error, so watch the factual and compliance categories on their own.
  • Reviewer mix: some editors change more than others, so compare rates within the same reviewer before drawing conclusions.

Reviewer fatigue is the most important to catch. A useful check is to score a small sample of shipped pieces with a rubric, such as our starter rubric for content agents. If edit rate falls while rubric scores also fall, reviewers are letting more through.

Edit rate also catches a gap that test sets miss. When test scores stay high while edit rate climbs, the test set has likely drifted from current work, a pattern we describe in evaluation drift.

Setting it up

A basic version takes a few days to set up. The main requirement is keeping both versions of every piece.

  1. Store the agent's draft and the final published version for every piece, linked by a shared identifier.
  2. Calculate the word-level difference automatically.
  3. Tag a random sample of edit sessions with one or two categories.
  4. Review the rate and the category mix weekly with the team that owns the agent.

Edit time is the hardest of the three to collect, because timers in editing tools count idle minutes too. We suggest asking reviewers to log time on a sample of pieces for two weeks each quarter. That gives a working estimate of value without asking anyone to track every minute.

Over a few weeks, a pattern usually appears. Edits tend to concentrate in two or three categories, and fixing those categories is often the fastest path to a better agent. Edit rate then becomes one of the core weekly leading indicators for that agent.

Open questions

Edit rate measures how far the draft was from acceptable, but not how far acceptable was from excellent. A low edit rate on mediocre work is still mediocre work. We are studying how to pair edit rate with outcome measures, so teams can tell when an agent has reached good enough and when it is holding results back.

How to apply this: start saving drafts and finals side by side for every agent-assisted piece, and compute a word-level edit rate. Tag a sample with edit categories, and the categories will show you where your agent needs help.

Related research

Put this research to work

Marketix turns these patterns into software: agents, automated workflows, and a hub that keeps people in control. Sign up for early access.