A domain-expert audit found 263 defects across SciCode's original 65 problems, with 192 rejecting correct work; corrected evaluation raised twelve models from 45–60% to 84–98% on subproblems and 9–27% to 69–92% on main problems.
Why it matters: A widely cited capability plateau can come from broken specifications, frozen answers, tolerances, and graders rather than stalled model performance.
A manual audit of 150 public trajectories labelled as failures across five benchmarks classified 10.7% as evaluator false negatives and 4.7% as broken tasks. Among genuine failures, verification, feedback and planning problems outweighed low-level execution errors.
Why it matters: An aggregate success rate can measure grader and task quality alongside model ability. That can send engineering effort toward the wrong component of an agent system.