Evaluation gates
- Stage 1Offline evalsGolden dataset and rubric
- Gate: meets quality bar on test set
- Stage 2Red teamDeliberate failure hunting
- Gate: severe failures fixed
- Stage 3Shadow modeReal inputs, no real actions
- Gate: matches human baseline
- Stage 4Limited rolloutAssist or approve to act
- Gate: within failure budget
- Stage 5ProductionMonitoring and failure budget
- New failures in production become new test cases at stage 1.
Before an agent touches live work, it passes through a series of stages separated by gates. Each gate has explicit criteria, and failures found later feed back into the earliest tests.
What each part does
Offline evals
The agent runs against a golden dataset of realistic and difficult cases, scored with a calibrated rubric.
Red team
A small group deliberately tries to make the agent fabricate, go off brand, violate policy, or misuse tools.
Shadow mode
The agent works on real inputs alongside people, with every external action blocked, and outputs are compared.
Limited rollout
The agent takes a live role at the assist or approve-to-act level, with full review.
Production
The agent runs at its earned autonomy level with weekly indicators and a failure budget.
Gates
Each gate has written criteria: meeting the quality bar, fixing severe failures, matching the human baseline, staying within budget.
Feedback
Every real failure becomes a new test case, so the same problem cannot return unnoticed.
See the architecture in the product
Marketix is built on these diagrams: specialist agents, governed connectors, and a hub that plans, routes, and records every run. Sign up for early access.
