Ai2 releases BenchMIRT to audit what benchmark questions measure
Ai2 released BenchMIRT, code, data and a technical report for applying multidimensional item-response theory to model evaluations. Trained on results from 100 open-weight models across 16 benchmarks and more than 34,000 questions, it recovered safety and general reasoning dimensions without being given those labels.
Why it matters: A single benchmark score can hide that questions measure different abilities or even reward opposite behavior. BenchMIRT gives evaluators an inspectable way to find mixed signals, identify informative items and test whether a safety or reasoning claim is…