Computer-use judges systematically accept unfinished work
OSReward evaluates 27 vision-language judges on 1,019 human-labelled trajectories across web, Windows, Ubuntu and mobile. Two-thirds of observed errors accepted incomplete tasks as successful, while removing action and reasoning text reduced average accuracy by 7.2 points.
Why it matters: An agent's own closing claim can make both evaluation and reinforcement learning reward work that never reached the required final state.