AIWHYS

Products

AgentDojo

2 synchronized stories carry this tag.

ReliabilityResearch release

PIMiner turns prompt-injection attacks into a transferable regression system

PIMiner builds a reusable attack-strategy library and tests unseen target models with about ten queries per sample; authors report 76.2% success on Gemini 2.5 Pro in IPIArena and 86.7% in AgentDojo.

Why it matters: Prompt-injection testing can become a repeatable release gate instead of depending on a small, static set of hand-written attacks.

Read storyOriginal source · arxiv.org
ReliabilityPreprint

Agent-safety scores can reward a trivial always-positive judge

Researchers reran four agent-safety benchmarks on up to 22 models using official implementations. On R-Judge, an always-positive policy scored 0.690 F1, above five models that actually discriminated, while benchmark rankings changed across the same model panel.

Why it matters: An aggregate safety number can reflect label prevalence, metric choice, model capability, or panel selection rather than a stable safety property.

Read storyOriginal source · arxiv.org