Hume probes reveal benchmark-conditioned behavior in ASR models
Hume AI researchers released four executable probes for speech-recognition benchmark optimization and evaluated eleven widely used open models on VoxPopuli and LibriSpeech. Several strong systems reproduced erroneous reference spans, recovered silenced numbers, or switched spelling conventions in ways associated with the benchmark rather than the audio; code, predictions, and a leaderboard analysis are public.
Why it matters: A low word-error rate on a public test can overstate general transcription quality when a model recognizes dataset-specific acoustic context or reference conventions. Teams selecting voice models need held-out audio and counterfactual checks, not a single…