Predicting human perceived accuracy of ASR systems
Abstract
Word error rate (WER), which is the most commonly used method of measuring automatic speech recognition (ASR) ac-curacy, penalizes all types of ASR errors equally. However, hu-mans differentially weigh different types of ASR errors. They judge ASR errors that distort the meaning of the spoken mes-sage more harshly than those that do not. Aiming to align more closely with human perception of ASR accuracy, we developed a new metric HPA (Human Perceived Accuracy) that predicts the subjective perceived accuracy of ASR transcriptions. HPA is computed based on the central idea of differential weighting of different ASR errors. Applied to the particular task of auto-matically recognizing voicemails, we found that the correlation between HPA and the human judgement of ASR accuracy was significantly higher (r-value=0.91) than the correlation between WER and human judgement (r-value=0.65). 1.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The WER does not reflect text understanding and only weakly correlates with human judgement of a transcript's quality.
“But the WER has been criticised, as it does not relect text understanding, and only weakly correlates with human judgement of a transcripts' quality [Favre et al. 2013;Mishra et al. 2011;Wang et al. 2003].”
From Measuring the Accuracy of Automatic Speech Recognition Solutions (2023) · cites this paper as Mishra et al. 2011
Despite criticism that the WER does not necessarily reflect the usefulness of a text, it still holds potential as a complementary accessibility metric.
“Despite criticism that the WER does not necessarily relect the usefulness of a text [Favre et al. 2013;Mishra et al. 2011;Wang et al. 2003], it still holds potential as a complementary accessibility metric.”
From Measuring the Accuracy of Automatic Speech Recognition Solutions (2023) · cites this paper as Mishra et al. 2011
Cites
From WER and RIL to MER and WIL: improved evaluation measures for connected speech recognition
Morris, A. C., Maier, V., & Green, P. (2004) · Interspeech 2004 · doi:10.21437/interspeech.2004-668 · Cited by 290 · see citing works
The TREC Spoken Document Retrieval Track: A Success Story.
Garofolo, J. S., Auzanne, C. G. P., & Voorhees, E. M. (1999) · Cited by 203 · see citing works
Is word error rate a good indicator for spoken language understanding accuracy
Wang, Y., Acero, A., & Chelba, C. I. (2004) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2003.1318504 · Cited by 149 · see citing works
On the Use of Information Retrieval Measures for Speech Recognition Evaluation
McCowan, I. A., Moore, D. C., Dines, J., Gática-Pérez, D., Flynn, M., Wellner, P. D., & Bourlard, H. A. (2004) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 90 · see citing works
Stochastic language models for speech recognition and understanding
Riccardi, G., & Gorin, A. L. (1998) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.1998-502 · Cited by 29 · see citing works
Bacchiani, M., Hirschberg, J., Rosenberg, A. E., Whittaker, S., Hindle, D. M., Isenhour, P., Jones, M., Stark, L., & Zamchick, G. (2001) · International Conference on Human Language Technology Research · doi:10.3115/1072133.1072216 · Cited by 18 · see citing works
Information Retrieval on Noisy Text
Grangier, D., Vinciarelli, A., & Bourlard, H. A. (2003) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 10 · see citing works
Exploring the stability of IDF term weighting
Fu, X., & Chen, M. (2008) · Asia Information Retrieval Symposium · doi:10.5555/1786374.1786377 · Cited by 3 · see citing works
Visual Voice Mail to Text on the iPhone/iPad.
Ljolje, A., Goffin, V., Caseiro, D., Mishra, T., & Gilbert, M. (2011) · Cited by 2 · see citing works
Self-citation
Flagged first, then by citations
1 of the 10 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Exploring the use of speech input by blind people on mobile devices
Azenkot, S., & Lee, N. B. (2013) · Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/2513383.2513440 · Cited by 151 · see citing works
Assessing the accuracy of automatic speech recognition for psychotherapy
Miner, A. S., Haque, A., Fries, J., Fleming, S. L., Wilfley, D. E., Wilson, G. T., Milstein, A. S., Jurafsky, D., Arnow, B. A., Agras, W. S., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0285-8 · Cited by 82 · see citing works
Kafle, S., & Huenerfauth, M. (2017) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3132525.3132542 · Cited by 60 · see citing works
“…that are even closer to our focus on automatic captioning: For instance, some have proposed a metric for evaluating ASR output on a speech transcription task [31]; their metric was based on opinion scores collected from humans who judged the quality of ASR-generated voicemail-to-text transcripts.”
BackgroundMeasuring the Accuracy of Automatic Speech Recognition Solutions
Kuhn, K., Kersken, V., Reuter, B., Egger, N., & Zimmermann, G. (2023) · ACM Transactions on Accessible Computing · doi:10.1145/3636513 · Cited by 44 · see citing works
“But the WER has been criticised, as it does not relect text understanding, and only weakly correlates with human judgement of a transcripts’ quality [Favre et al. 2013; Mishra et al. 2011; Wang et al. 2003].”
BackgroundMethodAutomatic human utility evaluation of ASR systems: does WER really predict performance?
Favre, B., Cheung, K., Kazemian, S., Lee, A., Liu, Y., Munteanu, C., Nenkova, A., Ochei, D., Penn, G., Tratz, S., et al. (2013) · Interspeech 2013 · doi:10.21437/interspeech.2013-610 · Cited by 44 · see citing works
“[16] developed a metric called Human Perceived Accuracy, and showed in a voice mail recognition task that it correlates more highly with human judgments of ASR accuracy than WER does.”
MethodKafle, S., & Huenerfauth, M. (2019) · ACM Transactions on Accessible Computing · doi:10.1145/3325862 · Cited by 29 · see citing works
“This is a concerning limitation, because researchers have previously found that humans perceive different ASR errors as having different degrees of impact on a text; i.e., some errors might distort the meaning of the text more harshly than others [41].”
BackgroundMethodInfluentialCave, R. (2024) · Journal of Speech Language and Hearing Research · doi:10.1044/2024_jslhr-24-00097 · Cited by 17 · see citing works
“ASR model performance is frequently evaluated using word error rate (WER; Mishra et al., 2011).”
Enhancing Communication Equity: Evaluation of an Automated Speech Recognition Application in Ghana
Ayoka, G., Barbareschi, G., Cave, R., & Holloway, C. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641903 · Cited by 10 · see citing works
“WER is the most commonly used metric for evaluating speech recognition performance [55].”
BackgroundChen, S.-H. K., Saeli, C., & Hu, G. (2023) · Assistive Technology · doi:10.1080/10400435.2023.2260860 · Cited by 9 · see citing works
“However, it is important to acknowledge that word-by-word transcription generated from ASR might affect the understandability of the messages due to ASR errors (Kafle & Huenerfauth, 2019; Mishra et al., 2011).”
BackgroundEnd-to-end speech recognition accuracy metric for voice-search tasks
Levit, M., Chang, S., Buntschuh, B., & Kibre, N. (2012) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2012.6289078 · Cited by 9 · see citing works
“In [4, 5] alternative error metrics were proposed that incorporated word salience into the computation.”
BackgroundMethodA Corpus for Modeling Word Importance in Spoken Dialogue Transcripts
Kafle, S., & Huenerfauth, M. (2018) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/28r4h7tnjefk · Cited by 7 · see citing works
Markl, N., & Lai, C. (2021) · Cited by 6 · see citing works
Crandell, H. A., Silcox, J., Ferguson, S. H., Lohani, M., & Payne, B. (2022) · Journal of Speech, Language and Hearing Research · doi:10.1044/2022_jslhr-21-00416 · Cited by 4 · see citing works
“Other research has similarly shown that deletion, insertion, and substitution errors were the most commonly found errors in ASR compared to other error types (Errattahi et al., 2018; McCowan et al., 2004; Mishra et al., 2011; Quint et al., 2008).”
ResultAccuracy of AI-generated Captions With Collaborative Manual Corrections in Real-Time
Kuhn, K., Kersken, V., & Zimmermann, G. (2023) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3544549.3585724 · Cited by 7 · see citing works
“This measure has, however, been challenged as it does not necessarily refect the usefulness of captions [11, 15, 23, 24, 30, 35].”
BackgroundBenchmarking ASR Systems Based on Post-Editing Effort and Error Analysis
Papadopoulou, M. M., Zaretskaya, A., TransPerfect, U., & Mitkov, R. (2021) · Translation and Interpreting Technology Online Conference (TRITON) · doi:10.26615/978-954-452-071-7_023 · Cited by 5 · see citing works
“Furthermore, with the aim to represent human perception of ASR accuracy, HPA (Human Perceived Accuracy) was developed [11].”
Kuhn, K., Kersken, V., & Zimmermann, G. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3719984 · Cited by 6 · see citing works
“Furthermore, commonly used evaluation metrics offer a limited representation of accessibility, leading to a mismatch between measured accuracy and the perceived transcription quality [14, 38, 59].”
Automatic speech recognition: challenges, enhancements, and evaluation metrics
Nagaraja, B. G., & Thimmaraja Yadava, G. (2025) · Multimedia Tools and Applications · doi:10.1007/s11042-025-20998-6 · Cited by 4 · see citing works
“2 Block diagram illustrating the speech enhancement module as a preprocessing stage for the ASR system tences that were correctly recognized by the ASR system [43]. where Correct is the number of correctly recognized words or sentences and Total is the total number of words or sentences.”
Kim, S., Park, Y. S., Ahn, D., Kwak, J. M., & Kim, J. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3641008 · Cited by 3 · see citing works
“However, Mishra et al. [34] found that while WER considers all words as equally important, in practice, users’ perceptions of errors are not the same.”
BackgroundFusion Strategy for Prosodic and Lexical Representations of Word Importance
Kafle, S., Alm, C. O., & Huenerfauth, M. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1898 · Cited by 3 · see citing works
An event-related brain potential study on the impact of speech recognition errors
Sakti, S., Yu, O., Sasakura, T., Neubig, G., Toda, T., & Nakamura, S. (2014) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipa.2014.7041620 · Cited by 2 · see citing works
“A “Human Perceived Accuracy” metric was developed by the direct regression of human subjective ratings (mean opinion score ratings) of recognition performance [4].”
BackgroundTündik, M. Á., Tarján, B., & Szaszák, G. (2020) · Computer Speech & Language · doi:10.1016/j.csl.2020.101076 · Cited by 2 · see citing works
Gordeeva, L., Ershov, V., Gulyaev, O., & Kuralenok, I. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467372 · Cited by 2 · see citing works
“There are two ways implementing such metrics known in the literature: one is based on information-theoretical approach [14], the other is inspired by information-retrieval techniques [6, 12, 13].”
BackgroundKadoma, K., Shrivastava, P., & Naaman, M. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790911 · Cited by 1 · see citing works
“While standard evaluation metrics such as word error rate treat all errors equally, users often weigh them differently (e.g., misrecognition of names or numbers may be more damaging than the omission of minor function words) [42].”
Acoustic Word Disambiguation with Phonogical Features in Danish ASR
Kirkedal, A. S. (2018) · doi:10.18653/v1/w18-5803 · Cited by 1 · see citing works
“The semantic disambiguation at both sentence and lexical level is appealing because ASR errors that disturb the meaning of an utterance are less acceptable for human consumers of ASR output (Mishra et al., 2011).”
BackgroundAutomatic transcription and phonetic labelling of dyslexic children's reading in Bahasa Melayu
Nurhidayat, N., & Him, N. N. N. (2015) · UUM Electronic Theses and Dissertation [eTheses] (Northern University of Malaysia)
TASK- AND DOMAIN-DEPENDENT NATURAL LANGUAGE PROCESSING FOR SUPPORTING SPOKEN DOCUMENT USE
Kazemian, S. (2018) · TSpace (University of Toronto)
Tonkin, E. L., & Tourte, G. J. L. (2025) · Communications in computer and information science · doi:10.1007/978-3-032-09117-8_10
- Authors
- Mishra, T., Ljolje, A., & Gilbert, M.
- ORCID
- Mishra, Taniya
- Venue
- Interspeech 2011
- Publisher
- ISCA
- Published
- 2011-08-27
- Pages
- 1945-1948
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W2403044103
- Cited by
- 28
- References
- 10
- Citations by year
- 2026: 1 · 2025: 3 · 2024: 3 · 2023: 3 · 2022: 1 · 2021: 3 · 2020: 2 · 2019: 2 · 2018: 4 · 2017: 1 · 2015: 1 · 2014: 1 · 2013: 2 · 2012: 1
- FWCI
- 1.00
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Interspeech (USB) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- speech recognition, word error rate
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
