Automatic human utility evaluation of ASR systems: does WER really predict performance?
Abstract
We propose an alternative evaluation metric to Word Error Rate (WER) for the decision audit task of meeting recordings, which exemplifies how to evaluate speech recognition within a legitimate application context. Using machine learning on an initial seed of human-subject experimental data, our alternative metric handily outperforms WER, which correlates very poorly with human subjectsf success in finding decisions given ASR transcripts with a range of WERs.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The WER does not reflect text understanding and only weakly correlates with human judgement of a transcript's quality.
“But the WER has been criticised, as it does not relect text understanding, and only weakly correlates with human judgement of a transcripts' quality [Favre et al. 2013;Mishra et al. 2011;Wang et al. 2003].”
From Measuring the Accuracy of Automatic Speech Recognition Solutions (2023) · cites this paper as Favre et al. 2013
Despite criticism that the WER does not necessarily reflect the usefulness of a text, it still holds potential as a complementary accessibility metric.
“Despite criticism that the WER does not necessarily relect the usefulness of a text [Favre et al. 2013;Mishra et al. 2011;Wang et al. 2003], it still holds potential as a complementary accessibility metric.”
From Measuring the Accuracy of Automatic Speech Recognition Solutions (2023) · cites this paper as Favre et al. 2013
Cites
The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries
Carbonell, J., & Goldstein, J. (2017) · ACM SIGIR Forum · doi:10.1145/3130348.3130369 · Cited by 2,203 · see citing works
Learning Accurate, Compact, and Interpretable Tree Annotation
Petrov, S., Barrett, L., Thibaux, R., & Klein, D. (2006) · Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the ACL - ACL '06 · doi:10.3115/1220175.1220230 · Cited by 793 · see citing works
Re-evaluating the Role of Bleu in Machine Translation Research
Callison-Burch, C., Osborne, M., & Koehn, P. (2006) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 632 · see citing works
The automated acquisition of topic signatures for text summarization
Lin, C.-Y., & Hovy, E. H. (2000) · International Conference on Computational Linguistics · doi:10.3115/990820.990892 · Cited by 471 · see citing works
McCowan, I. A., Carletta, J., Kraaij, W., Ashby, S., Bourban, S., Flynn, M., Guillemot, M., Hain, T., Kadlec, J., Vasilis Karaiskos, B., et al. (2005) · University of Twente Research Information · Cited by 384 · see citing works
Findings of the 2009 workshop on statistical machine translation
Callison-Burch, C., Koehn, P., Monz, C., & Schroeder, J. (2009) · Conference on Machine Translation (WMT) · doi:10.3115/1626431.1626433 · Cited by 347 · see citing works
Making Tree Kernels Practical for Natural Language Learning.
Moschitti, A. (2006) · Cited by 294 · see citing works
From WER and RIL to MER and WIL: improved evaluation measures for connected speech recognition
Morris, A. C., Maier, V., & Green, P. (2004) · Interspeech 2004 · doi:10.21437/interspeech.2004-668 · Cited by 290 · see citing works
The TREC Spoken Document Retrieval Track: A Success Story.
Garofolo, J. S., Auzanne, C. G. P., & Voorhees, E. M. (1999) · Cited by 203 · see citing works
A binary linear programming formulation of the graph edit distance
Justice, D. H., & Hero, A. O. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.152 · Cited by 184 · see citing works
Is word error rate a good indicator for spoken language understanding accuracy
Wang, Y., Acero, A., & Chelba, C. I. (2004) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2003.1318504 · Cited by 149 · see citing works
On the Use of Information Retrieval Measures for Speech Recognition Evaluation
McCowan, I. A., Moore, D. C., Dines, J., Gática-Pérez, D., Flynn, M., Wellner, P. D., & Bourlard, H. A. (2004) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 90 · see citing works
The effect of speech recognition accuracy rates on the usefulness and usability of webcast archives
Munteanu, C., Baecker, R. M., Penn, G., Toms, E. G., & James, D. A. (2006) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1124772.1124848 · Cited by 82 · see citing works
Self-citationExtrinsic summarization evaluation
Murray, G., Kleinbauer, T., Poller, P., Becker, T., Renals, S. J., & Kilgour, J. (2009) · ACM Transactions on Speech and Language Processing · doi:10.1145/1596517.1596518 · Cited by 30 · see citing works
Predicting human perceived accuracy of ASR systems
Mishra, T., Ljolje, A., & Gilbert, M. (2011) · Interspeech 2011 · doi:10.21437/interspeech.2011-364 · Cited by 28 · cited 1× · see citing works
“[16] developed a metric called Human Perceived Accuracy, and showed in a voice mail recognition task that it correlates more highly with human judgments of ASR accuracy than WER does.”
MethodEffects of word error rate in the DARPA communicator data during 2000 and 2001
Sanders, G. A., Le, A. N., & Garofolo, J. S. (2002) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.2002-134 · Cited by 14 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Automatic Speech Recognition Errors Detection and Correction: A Review
Errattahi, R., El Hannani, A., & Ouahmane, H. (2018) · Procedia Computer Science · doi:10.1016/j.procs.2018.03.005 · Cited by 155 · see citing works
Kafle, S., & Huenerfauth, M. (2017) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3132525.3132542 · Cited by 60 · see citing works
Siri, Echo and Performance: You have to Suffer Darling
Aylett, M., Cowan, B. R., & Clark, L. (2019) · CHI Extended Abstracts · doi:10.1145/3290607.3310422 · Cited by 57 · see citing works
Measuring the Accuracy of Automatic Speech Recognition Solutions
Kuhn, K., Kersken, V., Reuter, B., Egger, N., & Zimmermann, G. (2023) · ACM Transactions on Accessible Computing · doi:10.1145/3636513 · Cited by 44 · see citing works
“But the WER has been criticised, as it does not relect text understanding, and only weakly correlates with human judgement of a transcripts’ quality [Favre et al. 2013; Mishra et al. 2011; Wang et al. 2003].”
BackgroundButler, J., Trager, B., & Behm, B. (2019) · Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3308561.3353772 · Cited by 31 · see citing works
Kafle, S., & Huenerfauth, M. (2019) · ACM Transactions on Accessible Computing · doi:10.1145/3325862 · Cited by 29 · see citing works
“A Comparative Analysis of Automatic Speech Recognition Errors in Small Group Classroom Discourse.”
Cao, J., Ganesh, A., Cai, J., Southwell, R., Perkoff, E. M., Regan, M., Kann, K., Martin, J. H., Palmer, M., & D'Mello, S. (2023) · Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization · doi:10.1145/3565472.3595606 · Cited by 25 · see citing works
Hollands, S., Blackburn, D. J., & Christensen, H. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10433 · Cited by 24 · see citing works
Miró, J. D. V., Silvestre-Cerdà, J. A., Civera, J., Turró, C., & Juan, A. (2015) · Speech Communication · doi:10.1016/j.specom.2015.09.006 · Cited by 21 · see citing works
Cave, R. (2024) · Journal of Speech Language and Hearing Research · doi:10.1044/2024_jslhr-24-00097 · Cited by 17 · see citing works
Towards Speech to Speech Machine Translation focusing on Indian Languages
Mujadia, V., & Sharma, D. M. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-demo.19 · Cited by 13 · see citing works
Quantifying the Dialect Gap and its Correlates Across Languages
Kantharuban, A., Vulić, I., & Korhonen, A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.481 · Cited by 14 · see citing works
Enhancing Communication Equity: Evaluation of an Automated Speech Recognition Application in Ghana
Ayoka, G., Barbareschi, G., Cave, R., & Holloway, C. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641903 · Cited by 10 · see citing works
Reitmaier, T., Wallington, E., Klejch, O., Markl, N., Lam-Yee-Mui, L.-M., Pearson, J. S., Jones, M., Bell, P., & Robinson, S. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581385 · Cited by 9 · see citing works
Qualitative Evaluation of ASR Adaptation in a Lecture Context: Application to the PASTEL Corpus
Mdhaffar, S., Estève, Y., Hernández, N., Laurent, A., Dufour, R., & Quiniou, S. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2661 · Cited by 8 · see citing works
H EVAL : A New Hybrid Evaluation Metric for Automatic Speech Recognition Tasks
Sasindran, Z., Yelchuri, H., Prabhakar, T. V., & Rao, S. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389717 · Cited by 9 · see citing works
Real-Time Sentiment Analysis for Polish Dialog Systems Using MT as Pivot
Wołk, K. (2021) · Electronics · doi:10.3390/electronics10151813 · Cited by 7 · see citing works
Accuracy of AI-generated Captions With Collaborative Manual Corrections in Real-Time
Kuhn, K., Kersken, V., & Zimmermann, G. (2023) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3544549.3585724 · Cited by 7 · see citing works
Semantic-WER: A Unified Metric for the Evaluation of ASR Transcript for End UsabilityPreprint
Roy, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.02016 · Cited by 6 · see citing works
Benchmarking ASR Systems Based on Post-Editing Effort and Error Analysis
Papadopoulou, M. M., Zaretskaya, A., TransPerfect, U., & Mitkov, R. (2021) · Translation and Interpreting Technology Online Conference (TRITON) · doi:10.26615/978-954-452-071-7_023 · Cited by 5 · see citing works
Kuhn, K., Kersken, V., & Zimmermann, G. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3719984 · Cited by 6 · see citing works
Giacobello, D., Wung, J., Pichevar, R., & Atkins, J. D. (2014) · International Workshop on Acoustic Signal Enhancement · doi:10.1109/iwaenc.2014.6953998 · Cited by 4 · see citing works
Hughes, S. E., Wu, L.-Y., Ma, L. J., Jain, D., & McKee, M. M. (2025) · JMIR Rehabilitation and Assistive Technologies · doi:10.2196/79073 · Cited by 3 · see citing works
Do smart speaker skills support diverse audiences?
Shafei, H. A., & Tan, C. C. (2022) · Pervasive and Mobile Computing · doi:10.1016/j.pmcj.2022.101716 · Cited by 3 · see citing works
Un mot pour un autre ? Analyse et comparaison de huit plateformes de transcription automatique
Tancoigne, É., Corbellini, J.-P., Deletraz, G., Gayraud, L., Ollinger, S., & Valero, D. (2022) · Bulletin of Sociological Methodology/Bulletin de Méthodologie Sociologique · doi:10.1177/07591063221088322 · Cited by 3 · see citing works
Wung, J. (2015) · SMARTech Repository (Georgia Institute of Technology) · Cited by 3 · see citing works
Corrección de errores del reconocedor de voz de Google usando métricas de distancia fonética
Campos-Sobrino, D., Campos-Soberanis, M., Chin, I. M., & Uc-Cetina, V. (2019) · Research in Computing Science · doi:10.13053/rcs-148-1-6 · Cited by 2 · see citing works
Tündik, M. Á., Tarján, B., & Szaszák, G. (2020) · Computer Speech & Language · doi:10.1016/j.csl.2020.101076 · Cited by 2 · see citing works
Using language models for improving speech recognition for U.S. Open Tennis Championships
Baughman, A. K., Hammer, S., & Provan, D. (2017) · IBM Journal of Research and Development · doi:10.1147/jrd.2017.2712019 · Cited by 2 · see citing works
Bañeras-Roux, T., Wottawa, J., Rouvier, M., Merlin, T., & Dufour, R. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-40498-6_15 · Cited by 1 · see citing works
Diseño de un programa de subtitulación de vídeos pedagógicos
Quintas, L. C. (2019) · doi:10.6035/14110.2019.762203 · Cited by 1 · see citing works
Comment evaluer la qualite des transcriptions automatiques pour la detection d’entites nommees ?
Jannet, M. A. B., Adda‐Decker, M., Galibert, O., Kahn, J., & Rosset, S. (2014)
Different Contributions to Cost-Effective Transcription and Translation of Video Lectures.
Silvestre-Cerdà, J. A. (2016) · doi:10.4995/thesis/10251/62194
TASK- AND DOMAIN-DEPENDENT NATURAL LANGUAGE PROCESSING FOR SUPPORTING SPOKEN DOCUMENT USE
Kazemian, S. (2018) · TSpace (University of Toronto)
Self-citationFixing Errors of the Google Voice Recognizer through Phonetic Distance MetricsPreprint
Campos-Sobrino, D., Campos-Soberanis, M., Chin, I. M., & Uc-Cetina, V. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.09680
Exploring Practical Metrics to Support Automatic Speech Recognition Evaluations
Draffan, E. A., Wald, M., Ding, C., & Li, Y. (2023) · Studies in health technology and informatics · doi:10.3233/shti230636 · Cited by 1 · see citing works
A Paradigm for Interpreting Metrics and Measuring Error Severity in Automatic Speech Recognition
Bañeras-Roux, T., Rouvier, M., Wottawa, J., & Dufour, R. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70566-3_16
Disfluency processing for cascaded speech translation involving English and Indian languages
Mujadia, V., Mishra, P., & Sharma, D. M. (2025) · Language Resources and Evaluation · doi:10.1007/s10579-025-09818-3 · Cited by 1 · see citing works
Hughes, S. E., Wu, L.-Y., Ma, L. J., Jain, D., & McKee, M. M. (2025) · doi:10.2196/preprints.79073
- Authors
- Favre, B., Cheung, K., Kazemian, S., Lee, A., Liu, Y., Munteanu, C., Nenkova, A., Ochei, D., Penn, G., Tratz, S., Voss, C., & Zeller, F.
- Affiliations
- Columbia University (US)
- University of Toronto (CA)
- City University of New York (US)
- The University of Texas at Dallas (US)
- University of Pennsylvania (US)
- DEVCOM Army Research Laboratory (US)
- University College London (GB)
- Venue
- Interspeech 2013
- Publisher
- ISCA
- Published
- 2013-08-25
- Pages
- 3463-3467
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W2404260248
- Cited by
- 44
- References
- 16
- Citations by year
- 2025: 4 · 2024: 4 · 2023: 9 · 2022: 3 · 2021: 4 · 2020: 2 · 2019: 7 · 2018: 2 · 2017: 3 · 2016: 1 · 2015: 3 · 2014: 2
- FWCI
- 1.29
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Interspeech (USB) (publishedVersion)
- HAL (submittedVersion)
- Edinburgh Napier Research Repository (Edinburgh Napier University) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- speech recognition, word error rate, human evaluation, performance prediction, speech recognition evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
