What changed
A manual audit of 150 public trajectories labelled as failures across five benchmarks classified 10.7% as evaluator false negatives and 4.7% as broken tasks. Among genuine failures, verification, feedback and planning problems outweighed low-level execution errors.
Why it matters
An aggregate success rate can measure grader and task quality alongside model ability. That can send engineering effort toward the wrong component of an agent system.
What to do
Manually review a meaningful sample of passes and failures. Report grader error, broken-task rate, planning failure and execution failure separately from the headline score.
Caveat
The 15.3% applies only to 150 public trajectories already labelled as failures, not to a random sample of all attempts or every computer-use benchmark.