What changed
AWS published the evaluation framework used for an internal multi-agent sales system, spanning golden sets, experiments, CI, deployment gates, canaries, adversarial tests, drift, live monitoring and feedback. AWS says the process caught unsupported financial claims and a mismatched release that cut a pass rate from 78% to 38%.
Why it matters
The case study shows why uptime and latency cannot detect fluent but wrong agent outputs, and why tools, sub-agents and orchestration need separate evaluation contracts. It turns continuous agent evaluation into an inspectable operational sequence rather than a final QA step.
What to do
Start with one agent and define its business metric, failure taxonomy, golden set, promotion threshold and owner. Add a CI regression gate and a small production canary before attempting all nine stages or automated optimization.
Caveat
The results are AWS-reported from an internal system built on AgentCore, CloudWatch and other AWS services; datasets, full evaluator prompts and matched independent reproduction were not published. Transfer to another stack and domain is unproven.