ทันเอไอ

Study Finds Nearly Half of LLM Benchmarks Are "Saturated" and No Longer Measure Anything

New research published at ICML 2026 analyzed 60 LLM benchmarks and found that nearly half have lost the ability to distinguish top-performing models from one another

A research team led by Mubashara Akhtar and Anka Reuel, along with 36 co-researchers from the EvalEval Coalition project, published their work in the Proceedings of the 43rd International Conference on Machine Learning (ICML) 2026, systematically analyzing the problem of benchmark saturation. The team selected 60 LLM benchmarks from technical reports published by major model developers for study.

The team defined saturation as a benchmark "losing the ability to reliably distinguish between top-performing models" — in simple terms, strong models all score near the top, making it impossible to tell which one is better. They developed an uncertainty-aware saturation index based on leaderboard data to measure results, and defined 14 benchmark attributes covering task design, data construction, and evaluation formats, in order to test five hypotheses about what makes benchmarks saturate quickly or slowly.

The analysis found that nearly half of the benchmarks already show signs of saturation, and this rate increases with benchmark age. One notable finding that contradicts common beliefs in the field is that keeping test data private (private test sets) does not help prevent saturation at all, while benchmarks curated by experts resist saturation better than crowdsourced benchmarks (built with contributions from large numbers of people).

State of AI Substack, an independent analysis blog, described the work as a rigorous analytical framework, noting that the age and size of a test set are important predictors of saturation — more so than measures many believed to be safe, such as keeping test data secret or adversarial design. Meanwhile, Irene Solaiman, one of the researchers, summarized on X that as AI models advance, many benchmarks are becoming saturated and can no longer differentiate between models.