On-call agents reach only 25.3% realistic root-cause accuracy
ORCA-bench gives five frontier agents source code plus six days of Prometheus, Jaeger, OpenSearch and Grafana telemetry. The best system reached 25.3% root-cause accuracy on medium tasks and 10% on hard cases in a curated 50GB environment.
Why it matters: Strong coding performance does not automatically transfer to incident investigation across noisy logs, traces, metrics and ambiguous user reports.