Computer-use benchmarks mis-score a meaningful share of labelled failures
A manual audit of 150 public trajectories labelled as failures across five benchmarks classified 10.7% as evaluator false negatives and 4.7% as broken tasks. Among genuine failures, verification, feedback and planning problems outweighed low-level execution errors.
Why it matters: An aggregate success rate can measure grader and task quality alongside model ability. That can send engineering effort toward the wrong component of an agent system.