What changed
A new analysis models one person reviewing a small share of many agents' outputs. Shared task difficulty can correlate failures across model families, while five tested open models produced confidence estimates that were nearly constant and operationally weak.
Why it matters
Confidence-based review can systematically miss correlated failures when confidence is poorly calibrated. A diverse model fleet does not guarantee diverse failure patterns.
What to do
Combine random review, risk-tier sampling, behavioral anomaly detection and deterministic checks. Calibrate confidence against real outcomes before using it to allocate human attention.
Caveat
Worse-than-random auditing occurs only beyond the paper's modeled miscalibration threshold. Results are mainly theoretical and simulated, with a small open-model sample and wide uncertainty.