SciCode defects hid large gains in scientific coding
A domain-expert audit found 263 defects across SciCode's original 65 problems, with 192 rejecting correct work; corrected evaluation raised twelve models from 45–60% to 84–98% on subproblems and 9–27% to 69–92% on main problems.
Why it matters: A widely cited capability plateau can come from broken specifications, frozen answers, tolerances, and graders rather than stalled model performance.