Ai2 releases FairStress to test decision robustness under identity cues
Ai2 released FairStress, an open benchmark that expands 300 scenarios across 29 domains into more than 13 million cases and a 50,000-example core set. In author-run tests across 15 models, explicit or implied demographic cues caused larger decision shifts than neutral prefixes, including a reported 6.9- to 9.3-point abandonment gap for initially correct decisions.
Why it matters: The benchmark measures whether a model changes an otherwise matched decision when demographic information is added, exposing instability that aggregate accuracy or one-shot fairness scores can hide.