What changed
OSReward evaluates 27 vision-language judges on 1,019 human-labelled trajectories across web, Windows, Ubuntu and mobile. Two-thirds of observed errors accepted incomplete tasks as successful, while removing action and reasoning text reduced average accuracy by 7.2 points.
Why it matters
An agent's own closing claim can make both evaluation and reinforcement learning reward work that never reached the required final state.
What to do
Grade final state independently, treat completion statements as untrusted and report false-success recall separately from aggregate accuracy. Manually audit the hardest accepted runs.
Caveat
The project calls itself work in progress. Its page says code, corpus and model checkpoints are still arriving, so the central findings are not fully reproducible yet.