What changed
Researchers reran four agent-safety benchmarks on up to 22 models using official implementations. On R-Judge, an always-positive policy scored 0.690 F1, above five models that actually discriminated, while benchmark rankings changed across the same model panel.
Why it matters
An aggregate safety number can reflect label prevalence, metric choice, model capability, or panel selection rather than a stable safety property.
What to do
For every safety claim, name the benchmark, metric, target behavior, model version, and comparison panel, then include trivial baselines and inspect the confusion matrix.
Caveat
This is a preprint auditing four benchmarks. It demonstrates measurement fragility, not that agent safety cannot be measured or compared under well-defined conditions.