What changed
Google's evaluation study shows that the common one-to-five-raters-per-item design can miss genuine disagreement. It models the trade-off between items and raters under a fixed annotation budget and publishes the Variance Estimation Toolkit for power analysis.
Why it matters
A benchmark can look precise while being irreproducible if its annotation budget is allocated to too many shallowly rated items.
What to do
Run the simulator with your expected effect size and budget before collecting labels, then pre-register the item count, raters per item and disagreement metric.
Caveat
The optimal allocation depends on the rating distribution, task and comparison metric; the public VET repository was archived in April 2026 and is unsupported.