SWE-bench Pro makes coding-agent evaluation harder
Scale released SWE-bench Pro with 1,865 human-verified tasks across 41 repositories, including 731 public tasks and held-out commercial code. Initial unified-scaffold results placed the best tested agents below 25% Pass@1.
Why it matters: High scores on older coding benchmarks did not transfer to longer, multi-file work on less familiar repositories, exposing a more realistic autonomy ceiling.