OpenAI and Ironclad turn contract workflows into agent evaluations
OpenAI and Ironclad built 11 contracting tasks with 8 to 50 criteria each and hosted practice environments for training and evaluation. OpenAI reports GPT-6 Astra averaging a 55.0% rubric score versus 41.6% for GPT-5.6 Sol, with simulated time per attempt falling from 37.0 to 19.2 minutes.
Why it matters: The work shows a concrete method for evaluating long, business-rule-heavy software tasks instead of relying on generic benchmarks. The absolute score also shows that the tested agent still missed a substantial share of requirements.