Google releases a simulator for sizing human-rated AI benchmarks
Google's evaluation study shows that the common one-to-five-raters-per-item design can miss genuine disagreement. It models the trade-off between items and raters under a fixed annotation budget and publishes the Variance Estimation Toolkit for power analysis.
Why it matters: A benchmark can look precise while being irreproducible if its annotation budget is allocated to too many shallowly rated items.