What changed
APort Vault reports 225,964 evaluations replaying 4,371 human-authored attacks across 14 models, five policy levels and two tracks. The authors observed 140 unpermitted transfers in model-only runs at policy levels 2–4 and zero of 69,297 evaluations when a deterministic pre-action authorization layer was enforced; they released data and scoring code.
Why it matters
The benchmark separates a model's decision from the system's final authority, supplying unusually direct evidence that payment policy should be enforced at the tool boundary rather than left to prompting.
What to do
Download the frozen dataset and scoring code, reproduce one matched model-policy cell, then apply an independent pre-action check to a synthetic payment workflow. Verify policy parsing, recipient binding, denial evidence and retry behavior.
Caveat
This is a preprint from the system's authors, not peer reviewed. The zero-result spans 790 source sessions with a reported 0.38% per-session upper bound, and several planned multi-turn cells were incomplete.