PatchBench finds crash-only tests overstate agent vulnerability repairs
PatchBench tests C and C++ vulnerability-patching agents against memorization and superficial crash suppression. Across eleven agents, proof-of-concept-only validation inflated solve rates by 1.83 times on average, while roughly 25% of generated patches substantially resembled historical developer fixes.
Why it matters: A patch can silence one crash without repairing the root cause, so headline agent scores may reward benchmark leakage or narrow test gaming instead of secure remediation.