Matched-compute study reverses apparent gains from dense GRPO rewards
A preregistered preprint compared dense process rewards with ordinary GRPO across 1,668 runs on six models while matching generated-token and GPU-hour budgets. In the tested setup, Monte Carlo continuations consumed 91% of generated tokens and none of four dense variants beat standard GRPO at equal compute.
Why it matters: The result challenges training studies that compare reward methods per update while ignoring the cost of generating the reward. It gives post-training teams a concrete evaluation rule: equalize tokens and time before calling a method better.