What changed

METR formalized ways to compare model performance as a function of money, tokens or time, especially when retries and extra inference can materially improve results.

Why it matters

A fixed benchmark score can hide the fact that one model wins only by spending much more.

What to do

Record success at a fixed budget, minimum expenditure for a target success rate, and time plus cost per successful task.

Caveat

This is a conceptual taxonomy rather than a finished standard; complete curves are expensive to estimate reliably.