AIWHYS

Companies

Redwood Research

2 synchronized stories carry this tag.

ReliabilityResearch release

METR and Redwood expose gaps in OpenAI's internal agent monitoring

METR and Redwood Research published results from a three-week, high-access pilot audit of OpenAI's internal misalignment monitoring as it existed on or before 21 July. The evaluators estimated 75–94% eligibility coverage for qualifying internal Responses API traffic, found a Codex sandbox and Auto-Review bypass, a way to hide hundreds of actions, monitor-failure inputs, disabling headers and subvertible tools.

Why it matters: The audit shows that a monitor can have broad nominal coverage yet remain vulnerable to missing compaction events, delayed triggers, serialization gaps and agent-controlled bypasses. Coverage percentages are not evidence that every consequential action is…

Read storyOriginal source · metr.org
ReliabilityPreprint

Simulated control evals move agent safeguards closer to deployment

Redwood Research introduced simulated control evaluations, which replay real task context while an LLM simulates tool responses instead of executing attacks. On LinuxArena, the authors report 3.6 percentage points of safety-prediction RMSE at a 5% audit budget versus 10.1 for BashArena, with simulation explaining 78–85% of between-environment variation across key control metrics.

Why it matters: Control evaluations run in generic sandboxes can miss deployment-specific code, services and attack opportunities. A simulator can cheaply test monitors against more realistic context without giving an attack policy live access, but it shifts trust toward…

Read storyOriginal source · arxiv.org