CWE-bench v2 opens 100 repository-level vulnerability repair tasks
Collinear AI released CWE-bench v2 with 100 public tasks covering 91 Common Weakness Enumerations, deterministic graders and Docker or Harbor execution. Its provider-reported leaderboard spans 11 models; the leading listed system reached 67.8% pass@1 and 81% pass@4, leaving 19 tasks unsolved across four attempts.
Why it matters: A runnable, repository-level security benchmark gives coding-agent teams a more concrete way to test vulnerability repair than isolated snippets and makes repeated-attempt failure visible.