Agent confidence can misdirect audits across large fleets
A new analysis models one person reviewing a small share of many agents' outputs. Shared task difficulty can correlate failures across model families, while five tested open models produced confidence estimates that were nearly constant and operationally weak.
Why it matters: Confidence-based review can systematically miss correlated failures when confidence is poorly calibrated. A diverse model fleet does not guarantee diverse failure patterns.