Speech recognition for the world’s languages needs benchmarks models cannot easily overfit.

@huggingface now has partnered with @voicearena_ai to add Hindi and Indian English to the Open ASR Leaderboard.

The evaluation design is so very interesting here:

- 4,888 speakers across 4 speaker-disjoint splits
- spontaneous conversation recorded on people's own phones
- 16 metadata columns per clip, held-out private splits
- OIWER against reference lattices where 1 written form is not enough.

A model has fewer shortcuts here: it can’t lean on familiar speakers, clean studio audio, or one rigid transcript. The score has to survive messier speech and genuinely held-out evaluation.

So the evaluation design is doing more work here than data volume.

read their full thread below, which has got so much more interesting details, on the benchmark design.