SWE-Lancer tested coding agents on paid freelance tasks
OpenAI researchers introduced SWE-Lancer, more than 1,400 software-engineering tasks drawn from Upwork contracts worth about $1 million, including implementation work and managerial choices over competing proposals.
Why it matters: The benchmark moved evaluation closer to economically meaningful work and showed that strong models still left most task value unsolved.