ExpertCrowd-SWE tests coding agents on expert-authored repository work
Researchers released ExpertCrowd-SWE, a 100-task coding-agent benchmark across 87 open-source projects and 11 languages. Tasks use expert-authored change requests, sealed reference solutions, partial-credit graders and adversarial control patches; 24 tasks are public and 76 remain held out.
Why it matters: The benchmark targets larger, merge-quality changes rather than short bug fixes and repeats each model five times. Its leading scores around two-thirds show that frontier coding agents still fail substantial portions of realistic unfamiliar-repository work.