E-Bench tests whether product agents change state reliably
E-Bench introduces 323 synthetic tasks across gaming, music and meeting products. Agents must discover hidden information, compose tool calls and commit state changes, while deterministic database diffs grade outcomes. Across 11 models, three-run reliability stayed below 60%, or below 70% with code execution.
Why it matters: A single successful agent demo can conceal high variance. Repeated trials and exact state transitions are closer to the reliability a real product needs.