XYBench tests whether assistants challenge mistaken premises
Ai2, Carnegie Mellon, NVIDIA and Johns Hopkins released XYBench with 8,115 real-world questions whose requested approach conflicts with the user's underlying goal. Public data, code and a paper evaluate whether assistants identify the misconception and redirect toward a more effective solution.
Why it matters: Assistants often optimize the request they were given even when the request encodes a bad assumption. The benchmark turns that common support failure into a reproducible test for troubleshooting, advice and coding assistants.