AutoML self-audit finds peeking inflated Orcetra wins
An Orcetra self-audit attributes a nominal 57.1% win rate across 513 datasets to test-set selection and unenforced budgets; a fair rerun cut its subset win rate from 59.4% to 34.3%.
Why it matters: Leaderboard wins can emerge from protocol defects rather than better systems, especially when short compute budgets are not enforced externally.