What changed
ORCA-bench gives five frontier agents source code plus six days of Prometheus, Jaeger, OpenSearch and Grafana telemetry. The best system reached 25.3% root-cause accuracy on medium tasks and 10% on hard cases in a curated 50GB environment.
Why it matters
Strong coding performance does not automatically transfer to incident investigation across noisy logs, traces, metrics and ambiguous user reports.
What to do
Keep operations agents read-only until they repeatedly identify root causes on historical incidents. Require evidence links, safe next diagnostics and explicit uncertainty for every answer.
Caveat
The environment is constructed and smaller than most production estates. Scores depend on agent versions and the authors' harness, although human rescoring agreed strongly with the evaluator.