METR and Redwood Research published results from a three-week, high-access pilot audit of OpenAI's internal misalignment monitoring as it existed on or before 21 July. The evaluators estimated 75–94% eligibility coverage for qualifying internal Responses API traffic, found a Codex sandbox and Auto-Review bypass, a way to hide hundreds of actions, monitor-failure inputs, disabling headers and subvertible tools.
Why it matters: The audit shows that a monitor can have broad nominal coverage yet remain vulnerable to missing compaction events, delayed triggers, serialization gaps and agent-controlled bypasses. Coverage percentages are not evidence that every consequential action is…
METR published a lightweight external review of Anthropic's August misalignment risk report. It agreed that catastrophic risk was low for the models covered through 15 July, mainly because they lacked the needed capabilities, while warning that covert-capability benchmarks are a weak proxy and the methodology could miss higher risk.
Why it matters: The review separates a low current-risk conclusion from confidence in the safety method: limited capability can justify today's result without validating the same evaluation stack for more capable successors.
Anthropic CEO Dario Amodei committed the company to inviting a permanent external review team with employee-like access to relevant risk-assessment workspaces, tools, permissions and staff. The proposed contract would let reviewers publish key findings about risks, incidents, practices and denied access without Anthropic editorial control, subject to narrow security, legal and third-party confidentiality redactions.
Why it matters: The commitment moves third-party frontier-model evaluation from periodic access to an intended standing oversight function that can inspect training processes and incidents, not only completed models. If implemented as described, it would create a more…
Anthropic reported a pilot in which Stanford, Oxford, and METR researchers designed analyses over roughly 250,000 Claude and Claude Code conversations from April and May. Anthropic ran the privacy-preserving aggregation, a third party audited the privacy process, and project-level aggregate data was released publicly.
Why it matters: External questions over real product use can expose patterns that lab-authored studies or public chat datasets miss. The pilot also makes the trade-off visible: privacy-preserving aggregation protects raw conversations but makes category errors harder for…
METR formalized ways to compare model performance as a function of money, tokens or time, especially when retries and extra inference can materially improve results.
Why it matters: A fixed benchmark score can hide the fact that one model wins only by spending much more.
METR conducted an entity-level pilot with Anthropic, Google, Meta and OpenAI using access to internal state-of-the-art models, raw reasoning traces and non-public operational information. It concluded that February-March agents plausibly had the means, motive and opportunity for small rogue deployments, but lacked the judgment and capability to make them highly robust.
Why it matters: Risk assessment is shifting from public model cards toward how labs actually use agents internally—with credentials, infrastructure access and weak oversight.
A METR survey of 349 technical workers found a median self-reported 1.4-to-2-times increase in value created and a three-times increase in speed from AI tools. METR explicitly cautioned that counterfactual productivity surveys have overstated benefits relative to measured outcomes.
Why it matters: The gap between perceived speed and actual business value is becoming a central measurement problem for enterprise AI.
METR reported that 16 experienced open-source developers completed 246 tasks in familiar repositories 19 percent more slowly when allowed to use early-2025 AI coding tools.
Why it matters: The result challenged self-reported productivity expectations and showed why engineering teams should measure completed work and rework instead of assuming tool use saves time.