Google's Gemma-2 9b and OpenAI's GPT-4o achieved near-perfect scores on Anthropic's DiscrimEval bias benchmark, released in December 2023.
Notes on verification
Confirmed by original Stanford paper (arXiv 2502.01926) stating Gemma-2 9b and GPT-4o saturate DiscrimEval/BBQ benchmarks, plus independent secondary coverage (Sri Lanka Guardian) and contemporaneous reporting on DiscrimEval's December 2023 release (TechCrunch, VentureBeat). [tier=gold indep_score=0.913 clusters=4 claim_tier=notable]
Sources
- These new AI benchmarks could help make models less biased (seed:technology_and_ai)
- https://arxiv.org/abs/2502.01926 (corroboration)
- https://slguardian.org/new-ai-benchmarks-aim-to-reduce-bias-and-improve-fairness/ (corroboration)
- https://techcrunch.com/2023/12/07/anthropics-latest-tactic-to-stop-racist-ai-asking-it-really-really-really-really-nicely/ (corroboration)
- https://venturebeat.com/ai/anthropic-leads-charge-against-ai-bias-and-discrimination-with-new-research (corroboration)