bench-1
June 2026
Speech-to-Text on Real Multilingual Conversation
15 models4 languageshuman-verified ground truth
TL;DR
- ElevenLabs leads overall. Strong Spanish and code-switching results do not eliminate language-specific weaknesses.
- Dialectal Arabic is difficult. Even the lowest observed error rate exceeds 50%.
- Failures are not graceful. Loops, invented content, and dropped audio need separate inspection.
01
Results
Error rate of each model against human-verified transcripts. Word error rate throughout, character error rate for Japanese.
| Model | ||||
|---|---|---|---|---|
| ElevenLabsscribe_v2 | 2.9best | 53.6 | 8.1best | 19.9 |
| Deepgramnova-3 | 9.0 | 53.1best | 19.3 | 24.9 |
| Googlegemini-3.5-flash | 8.3 | 54.8 | 35.0 | 22.2 |
| Googlegemini-3.1-pro-preview | 9.7 | 54.9 | 52.6 | 23.4 |
| AssemblyAIspeech_models: universal-3-pro, universal-2 | 11.5 | 79.8 | 53.3 | 16.6best |
| Microsoftmai-transcribe-1.5 | 8.9 | 96.4 | 52.3 | 22.0 |
| Groqwhisper-large-v3 · open weights | 13.8 | 77.3 | 75.7 | 19.7 |
| OpenAIgpt-4o-transcribe-diarize | 15.4 | 83.9 | 44.8 | — |
| OpenAIwhisper-1 | 18.0 | 99.4 | 94.5 | 23.4 |
| OpenAIgpt-4o-transcribe | 15.1 | 91.2 | 91.3 | 38.2 |
| Sarvamsaarika:v2.5 | — | — | 63.6 | — |
| Cartesiaink-whisper | 11.4 | 97.2 | 108.4* | 58.6 |
| Mistralvoxtral-small-24b-2507 · open weights | 10.2 | 130.1* | 182.3* | 53.1 |
| OpenAIgpt-4o-mini-transcribe-2025-12-15 | 12.1 | 384.9* | 63.0 | 23.2 |
| NVIDIAnemotron-3-nano-omni-30b-a3b · open weights | 749.1* | 2291.5* | 678.7* | 1281.5* |
Complete 15-model matrix from the retained public report. Text labels replace provider marks not recovered in this reference. Asterisks retain the source outlier notation.
Language rankings
Captured report values. Lower is better. Bars stop at 100%; outlier values remain printed in full. Missing results are omitted, not treated as zero.
- ElevenLabsscribe_v22.9
- Googlegemini-3.5-flash8.3
- Microsoftmai-transcribe-1.58.9
- Deepgramnova-39.0
- Googlegemini-3.1-pro-preview9.7
- Mistralvoxtral-small-24b-2507 · open weights10.2
- Cartesiaink-whisper11.4
- AssemblyAIspeech_models: universal-3-pro, universal-211.5
- OpenAIgpt-4o-mini-transcribe-2025-12-1512.1
- Groqwhisper-large-v3 · open weights13.8
- OpenAIgpt-4o-transcribe15.1
- OpenAIgpt-4o-transcribe-diarize15.4
- OpenAIwhisper-118.0
- NVIDIAnemotron-3-nano-omni-30b-a3b · open weights749.1*
- Deepgramnova-353.1
- ElevenLabsscribe_v253.6
- Googlegemini-3.5-flash54.8
- Googlegemini-3.1-pro-preview54.9
- Groqwhisper-large-v3 · open weights77.3
- AssemblyAIspeech_models: universal-3-pro, universal-279.8
- OpenAIgpt-4o-transcribe-diarize83.9
- OpenAIgpt-4o-transcribe91.2
- Microsoftmai-transcribe-1.596.4
- Cartesiaink-whisper97.2
- OpenAIwhisper-199.4
- Mistralvoxtral-small-24b-2507 · open weights130.1*
- OpenAIgpt-4o-mini-transcribe-2025-12-15384.9*
- NVIDIAnemotron-3-nano-omni-30b-a3b · open weights2291.5*
- ElevenLabsscribe_v28.1
- Deepgramnova-319.3
- Googlegemini-3.5-flash35.0
- OpenAIgpt-4o-transcribe-diarize44.8
- Microsoftmai-transcribe-1.552.3
- Googlegemini-3.1-pro-preview52.6
- AssemblyAIspeech_models: universal-3-pro, universal-253.3
- OpenAIgpt-4o-mini-transcribe-2025-12-1563.0
- Sarvamsaarika:v2.563.6
- Groqwhisper-large-v3 · open weights75.7
- OpenAIgpt-4o-transcribe91.3
- OpenAIwhisper-194.5
- Cartesiaink-whisper108.4*
- Mistralvoxtral-small-24b-2507 · open weights182.3*
- NVIDIAnemotron-3-nano-omni-30b-a3b · open weights678.7*
- AssemblyAIspeech_models: universal-3-pro, universal-216.6
- Groqwhisper-large-v3 · open weights19.7
- ElevenLabsscribe_v219.9
- Microsoftmai-transcribe-1.522.0
- Googlegemini-3.5-flash22.2
- OpenAIgpt-4o-mini-transcribe-2025-12-1523.2
- Googlegemini-3.1-pro-preview23.4
- OpenAIwhisper-123.4
- Deepgramnova-324.9
- OpenAIgpt-4o-transcribe38.2
- Mistralvoxtral-small-24b-2507 · open weights53.1
- Cartesiaink-whisper58.6
- NVIDIAnemotron-3-nano-omni-30b-a3b · open weights1281.5*
02
The Data
The recordings are natural conversations between native speakers who know each other. Nothing is scripted, read aloud, or staged.
03
Findings
Clean-language scores do not transfer. Code-switching is a separating dimension. Failure is catastrophic, not graceful.