Private ASR test sets land on the Open ASR Leaderboard
The Open ASR Leaderboard has become a daily habit for speech folks. It's been visited over 710,000 times since its September 2023 launch, and that attention is both humbling and a problem. When a measure becomes a target, it ceases to be a good measure. Goodhart's Law is in full effect, and the team at Hugging Face has been watching models get good at the leaderboard without getting meaningfully better in the wild.
That's why Appen Inc. and DataoceanAI have donated high-quality English ASR datasets and why they're staying private. The post published May 6, 2026 by Eric Bezzam, Steven Zheng, Eustache Le Bihan and Sergio Bruccoleri frames it plainly: keep the data out of training loops, keep test-set contamination down, and use it as a benchmaxxing repellant.
For context on Hugging Face's broader mission, advancing AI through open source and open science informs their approach to benchmark design.
The default Average WER isn't changing. By default the leaderboard still averages only public datasets. Users can toggle Private data on to see the impact, and a new Rank Δ column shows how model ordering shifts relative to the default macroaverage. Including or excluding public datasets also moves the macroaverage, so you can tailor the view to the distributions that matter for your use case.
Why standardization and openness need a private check
The leaderboard was built on two tensions.
Standardization means unifying test sets into a single Hub dataset and running everything through a normalizer based on Whisper that strips punctuation and casing and maps to American spelling. Models have different output conventions, datasets have different conventions, and you can't compare WER fairly until you normalize.
Openness means the UI code and evaluation scripts are open-sourced. That invites community feedback, faster model additions, and better evaluation quality over time.
Together they make the benchmark more meaningful. Together they also make it easier to optimize for the benchmark. Benchmaxxing isn't just a theoretical risk; it's what happens when developers chase leaderboard scores without corresponding gains in real-world robustness. The team notes there's no single catch-all ASR model. Some models win on American English, others on diverse accents or multilingual settings, others on speed or conversational audio. The goal is a more holistic view, not a single number.
What Appen and DataoceanAI contributed
The new private track covers scripted and conversational speech across multiple accents and transcription styles. The providers have been asked not to share these datasets with clients, and even similar-distribution data could give an edge. Using multiple vendors helps balance that out.
Appen datasets
Appen Scripted AU, CA, IN and US are read speech, punctuated and cased, with roughly 1.42h, 1.53h, 1.02h and 1.45h respectively and balanced male/female splits around 49/51.
Appen Conversational IN is Indian English, conversational and spontaneous, 1.37h, 51/49 male/female, punctuated with disfluencies noted.
Appen Conversational US003 is American English, conversational and spontaneous, 1.64h, 49/51, punctuated, cased, disfluencies.
Appen Conversational US004 is American English, conversational and spontaneous, 1.65h, 49/51, punctuated, cased, disfluencies.
DataoceanAI datasets
DataoceanAI Scripted US is American read speech, 2.43h, 54/46 male/female, punctuated, cased for proper nouns, disfluencies.
DataoceanAI Scripted GB is British read speech, 2.43h, 47/53, punctuated, disfluencies.
DataoceanAI Conversational US is American conversational and spontaneous, 8.82h, male/female not available, punctuated, disfluencies.
DataoceanAI Conversational GB is British conversational and spontaneous, 5.96h, male/female not available, punctuated, disfluencies.
| Dataset | Accent | Duration h | Male % / Female % | Style | Transcription |
|---|---|---|---|---|---|
| Appen Scripted AU | Australian | 1.42 | 49 / 51 | Read | Punctuated, cased |
| Appen Scripted CA | Canadian | 1.53 | 52 / 48 | Read | Punctuated, cased |
| Appen Scripted IN | Indian | 1.02 | 49 / 51 | Read | Punctuated, cased |
| Appen Scripted US | American | 1.45 | 49 / 51 | Read | Punctuated, cased |
| Appen Conversational IN | Indian | 1.37 | 51 / 49 | Conversational, spontaneous | Punctuated, disfluencies |
| Appen Conversational US003 | American | 1.64 | 49 / 51 | Conversational, spontaneous | Punctuated, cased, disfluencies |
| Appen Conversational US004 | American | 1.65 | 49 / 51 | Conversational, spontaneous | Punctuated, cased, disfluencies |
| DataoceanAI Scripted US | American | 2.43 | 54 / 46 | Read | Punctuated, cased (proper nouns), disfluencies |
| DataoceanAI Scripted GB | British | 2.43 | 47 / 53 | Read | Punctuated, disfluencies |
| DataoceanAI Conversational US | American | 8.82 | NA | Conversational, spontaneous | Punctuated, disfluencies |
| DataoceanAI Conversational GB | British | 5.96 | NA | Conversational, spontaneous | Punctuated, disfluencies |
The split lets the team surface gaps between controlled, often saturated settings like scripted American speech and messier conditions like conversational non-American accents. No split-level scores are published, deliberately, to avoid developers boosting performance on a single provider or accent.
How the new metrics work
The leaderboard now offers several macroaverage variants.
Average WER computes a macroaverage of data-provider averages, weighted equally. By default private datasets are excluded.
Avg Scripted is a macroaverage of all scripted datasets.
Avg Conversational is a macroaverage of all conversational datasets.
Avg US is a macroaverage of all American-accent datasets.
Avg non-US is a macroaverage of all non-American-accent datasets.
Toggle Private data on and the macroaverage includes the new sets. Rank Δ shows how ordering changes versus the default configuration. You can also include or exclude public datasets, which changes the macroaverage and lets you tailor evaluation to your application.
Getting your model evaluated on private data
The process hasn't changed much. Add your model to the Open ASR Leaderboard via a pull request on GitHub. A model checklist appears. Report results on the public datasets, the team verifies those results, and then computes metrics on the private sets.
While you wait, you can self-report public-set metrics via a YAML file on your model card. That puts your model on an unverified leaderboard shown on the dataset page. The approach is decentralized and documented in the repo.
Could models trained on data from the providers have an advantage? Possibly. The team asked Appen and DataoceanAI not to provide this data to clients, but even similar-distribution data could help. Multiple providers balance that risk, and the team is open to more vendors and eval sets for the Private data tab.
What's next and community pushback
The team wants feedback on how the new track and toggling help users identify models that fit their apps. They're also looking into evaluations that better reflect real-world noisy conditions.
Preparing the private sets took extra care. The team built tooling to identify low signal-to-noise conditions and transcript mismatches, because those factors meaningfully affect WER. More on that is promised in a future post.
Community comments have been pragmatic. One researcher noted the two vendors sell datasets directly to ASR service providers for training and asked for extra prudence mixing private data into evaluations. The authors agree that's why private data isn't in the default average, and they point to the new splits on scripted/conversational and American/non-American accents as a way to get nuance.
Another common request is for more open-source speech datasets. The team says a channel where researchers can recommend open datasets directly would be a huge step forward, and contributions are possible via the GitHub repo's checklist.
Private data doesn't kill openness. It's a pressure valve. It makes the leaderboard harder to game, and it gives users a clearer sense of where models actually break. That's worth the trade.
