Evaluating the Usability of Automatically Generated Captions for People who are Deaf or Hard of Hearing
Open accessThe DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
The accuracy of Automated Speech Recognition (ASR) technology has improved, but it is still imperfect in many settings. Researchers who evaluate ASR performance often focus on improving the Word Error Rate (WER) metric, but WER has been found to have little correlation with human-subject performance on many applications. We propose a new captioning-focused evaluation metric that better predicts the impact of ASR recognition errors on the usability of automatically generated captions for people who are Deaf or Hard of Hearing (DHH). Through a user study with 30 DHH users, we compared our new metric with the traditional WER metric on a caption usability evaluation task. In a side-by-side comparison of pairs of ASR text output (with identical WER), the texts preferred by our new metric were preferred by DHH participants. Further, our metric had significantly higher correlation with DHH participants' subjective scores on the usability of a caption, as compared to the correlation between WER metric and participant subjective scores. This new metric could be used to select ASR systems for captioning applications, and it may be a better metric for ASR researchers to consider when optimizing ASR systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
SWITCHBOARD: telephone speech corpus for research and development
Godfrey, J. J., Holliman, E. C., & McDaniel, J. (1992) · [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.1992.225858 · Cited by 2,134 · cited 1×
“We trained our n-gram models (n = 1 to 5) on Switchboard [17], English CALLHOME and TED-LIUM [38] corpora, which contain a total of 1.9 million unique words.”
MethodInfluentialThe microsoft 2016 conversational speech recognition system
Xiong, W., Droppo, J., Huang, X., Seide, F., Seltzer, M., Stolcke, A., Yu, D., & Zweig, G. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7953159 · Cited by 326
From WER and RIL to MER and WIL: improved evaluation measures for connected speech recognition
Morris, A. C., Maier, V., & Green, P. (2004) · Interspeech 2004 · doi:10.21437/interspeech.2004-668 · Cited by 290 · cited 4×
“Multiple reference words can be misrecognized as a single word (substitution followed by deletions) and a single reference word can be misrecognized as multiple words (substitution followed by insertions) [22][32].”
BackgroundInfluentialToward Human Parity in Conversational Speech Recognition
Xiong, W., Droppo, J., Huang, X., Seide, F., Seltzer, M. L., Stolcke, A., Yu, D., & Zweig, G. (2017) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2017.2756440 · Cited by 245
Improving Real-Time Captioning Experiences for Deaf and Hard of Hearing Students
Kawas, S., Karalis, G., Wen, T., & Ladner, R. E. (2016) · Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/2982142.2982164 · Cited by 82
Automatic human utility evaluation of ASR systems: does WER really predict performance?
Favre, B., Cheung, K., Kazemian, S., Lee, A., Liu, Y., Munteanu, C., Nenkova, A., Ochei, D., Penn, G., Tratz, S., et al. (2013) · Interspeech 2013 · doi:10.21437/interspeech.2013-610 · Cited by 44
Predicting human perceived accuracy of ASR systems
Mishra, T., Ljolje, A., & Gilbert, M. (2011) · Interspeech 2011 · doi:10.21437/interspeech.2011-364 · Cited by 28 · cited 3×
“…that are even closer to our focus on automatic captioning: For instance, some have proposed a metric for evaluating ASR output on a speech transcription task [31]; their metric was based on opinion scores collected from humans who judged the quality of ASR-generated voicemail-to-text transcripts.”
Background
Cited by
Evaluation of Text Generation: A Survey
Celikyilmaz, A., Clark, E., & Gao, J. (2020) · arXiv · doi:10.48550/arxiv.2006.14799 · Cited by 207
Accessibility in Action: Co-Located Collaboration among Deaf and Hearing Professionals
Wang, E. Q., & Piper, A. M. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274449 · Cited by 64
Working at the Intersection of Race, Disability and Accessibility
Harrington, C. N., Desai, A., Lewis, A., Moharana, S., Ross, A. S., & Mankoff, J. (2023) · The 25th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3597638.3608389 · Cited by 53
Seita, M., Lee, S., Andrew, S., Shinohara, K., & Huenerfauth, M. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501843 · Cited by 52
“…work has investigated how to evaluate ASR output: Kafe et al. tested a novel method for evaluating caption quality that outperformed the standard word error rate metric [21], and Berke et al. [8] compared the efcacy of various probes for measuring DHH participants’ perception of caption quality.”
MethodSelf-citationBerke, L., Albusays, K., Seita, M., & Huenerfauth, M. (2019) · Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290607.3312921 · Cited by 48
Self-citation
Metadata
- Authors
- Kafle, S., & Huenerfauth, M.
- Venue
- Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility
- Publisher
- ACM
- Published
- 2017-10-19
- Pages
- 165-174
- OpenAlex
- W2765816858
- Cited by
- 59
- References
- 7
- FWCI
- 74.87
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Evaluating the Usability of Automatically Generated Captions for People who are Deaf or Hard of Hearing. https://app.cochat.ai/papers/10.1145/3132525.3132542
- The paper
- Kafle, S., & Huenerfauth, M. (2017). Evaluating the Usability of Automatically Generated Captions for People who are Deaf or Hard of Hearing. Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility. https://doi.org/10.1145/3132525.3132542
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.