ทันเอไอ

Hugging Face study reveals speech AI may not actually be good, just memorizing standard tests

A recent report from Hugging Face points out that ASR models achieve soaring benchmark scores by memorizing test patterns rather than true listening skills.

On August 21, 2026, Hugging Face published an article titled "Measuring benchmark optimization in speech recognition," exposing the issue of benchmark optimization in ASR (Automatic Speech Recognition) systems. While these models currently score exceptionally high on public tests—making them seem on par with humans—they fail to reflect real-world usage.

Following a study and analysis conducted in collaboration with Artificial Analysis on 11 open-source ASR models, researchers discovered an interesting behavior: these models tend to transcribe according to test datasets like VoxPopuli, even when the actual audio in the file does not match the text in that test set.

Additionally, the research found reference errors in the VoxPopuli test set as high as 40%. Shockingly, the highest-scoring models replicated these reference errors from the test set in 18 to 30 percent of the tested cases.

To address this issue, the authors recommend that developers avoid relying solely on independent and identically distributed (IID) test sets. Instead, they should split test data temporally or by speaker to prevent models from being overfitted to ad-hoc evaluation sets.