Review of 445 LLM benchmarks finds widespread construct-validity weaknesses

A 42-author team with lead authors at the University of Oxford, using 29 expert reviewers, systematically reviewed 445 benchmarks from leading NLP and ML venues and found recurring problems in how phenomena are defined, tasks chosen and scores computed that undermine the claims drawn from them. It sets out eight recommendations for benchmark design and was accepted to the NeurIPS 2025 Datasets and Benchmarks track.