What changed
Ai2, Carnegie Mellon, NVIDIA and Johns Hopkins released XYBench with 8,115 real-world questions whose requested approach conflicts with the user's underlying goal. Public data, code and a paper evaluate whether assistants identify the misconception and redirect toward a more effective solution.
Why it matters
Assistants often optimize the request they were given even when the request encodes a bad assumption. The benchmark turns that common support failure into a reproducible test for troubleshooting, advice and coding assistants.
What to do
Run a representative private slice through the published evaluator, audit automated-judge disagreements with human reviewers and add high-consequence examples from your own support domain before using the score for model selection.
Caveat
The reported model results use single-turn prompts and a Claude Sonnet 4.5 judge. Human agreement is lower for pragmatic emphasis than solution presence, and benchmark performance does not establish safety or quality in unseen domains.