What changed
A domain-expert audit found 263 defects across SciCode's original 65 problems, with 192 rejecting correct work; corrected evaluation raised twelve models from 45–60% to 84–98% on subproblems and 9–27% to 69–92% on main problems.
Why it matters
A widely cited capability plateau can come from broken specifications, frozen answers, tolerances, and graders rather than stalled model performance.
What to do
Audit samples of passes and failures before trusting procurement scores, and include domain experts when tests depend on physics, mathematics, or numerical tolerances.
Caveat
The runnable release excludes one underdetermined task and records 262 corrections across 64 problems; the auditing team still needs independent validation and wider adoption.