What changed
PatchBench tests C and C++ vulnerability-patching agents against memorization and superficial crash suppression. Across eleven agents, proof-of-concept-only validation inflated solve rates by 1.83 times on average, while roughly 25% of generated patches substantially resembled historical developer fixes.
Why it matters
A patch can silence one crash without repairing the root cause, so headline agent scores may reward benchmark leakage or narrow test gaming instead of secure remediation.
What to do
Add held-out mutations, fixes outside the crash stack, semantic regression tests and exploit-independent security checks to one patching evaluation before accepting agent-generated remediation.
Caveat
This is a preprint focused on C and C++ benchmarks. The reported averages do not establish the failure rate of every agent, language, repository or live security workflow.