Exploration of Automatic Speech Recognition for Deaf and Hard of Hearing Students in Higher Education Classes
Abstract
Automatic speech recognition (ASR) programs that generate real-time speech-to-text captions can be provided as supplemental access technologies for deaf and hard of hearing (DHH) students in higher education classes. As part of a pilot program, we implemented ASR as a supplemental access service in biology, statistics, and other courses at our university. To identify the benefits and limitations of ASR as an access technology, we surveyed 26 DHH students and interviewed 8 of these students about their experiences with ASR in their mainstream classes. Participants believed that ASR was beneficial despite the errors that ASR continued to generate; however, the accuracy and readability of ASR need to improve so that students can better access spoken information through ASR. This paper reviews points for researchers to consider when designing and providing ASR as a supplemental access service in educational settings.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Proceedings of Interspeech 2013
Lu, L., Ghoshal, A. K., & Renals, S. J. (2013) · Proceedings - International Symposium on Computer Architecture/Proceedings · Cited by 1,003 · see citing works
Automatic speech recognition and speech variability: A review
Benzeghiba, M., De Mori, R., Deroo, O., Dupont, S., Erbes, T., Jouvet, D., Fissore, L., Laface, P., Mertins, A., Ris, C., et al. (2007) · Speech Communication · doi:10.1016/j.specom.2007.02.006 · Cited by 529 · see citing works
Active learning: theory and applications to automatic speech recognition
Riccardi, G., & Hakkani-Tür, D. Z. (2005) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/tsa.2005.848882 · Cited by 167 · see citing works
Speech recognition in university classrooms
Bain, K., Basson, S. H., & Wald, M. (2002) · doi:10.1145/638249.638284 · Cited by 87 · see citing works
Accessibility Evaluation of Classroom Captions
Kushalnagar, R., Lasecki, W. S., & Bigham, J. P. (2014) · ACM Transactions on Accessible Computing · doi:10.1145/2543578 · Cited by 84 · see citing works
Improving Real-Time Captioning Experiences for Deaf and Hard of Hearing Students
Kawas, S., Karalis, G., Wen, T., & Ladner, R. E. (2016) · Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/2982142.2982164 · Cited by 82 · see citing works
Universal access to communication and learning: the role of automatic speech recognition
Wald, M., & Bain, K. (2007) · Universal Access in the Information Society · doi:10.1007/s10209-007-0093-9 · Cited by 64 · see citing works
Berke, L., Caulfield, C., & Huenerfauth, M. (2017) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3132525.3132541 · Cited by 62 · see citing works
Kafle, S., & Huenerfauth, M. (2017) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3132525.3132542 · Cited by 60 · see citing works
Inclusion of deaf students in computer science classes using real-time speech transcription
Kheir, R., & Way, T. (2007) · Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/1268784.1268860 · Cited by 49 · see citing works
The effects of automatic speech recognition quality on human transcription latency
Gaur, Y., Lasecki, W. S., Metze, F., & Bigham, J. P. (2016) · International Web for All Conference (W4A) · doi:10.1145/2899475.2899478 · Cited by 48 · see citing works
Enhancing learning accessibility through fully automatic captioning
Federico, M., & Furini, M. (2012) · International Cross-Disciplinary Conference on Web Accessibility (W4A) · doi:10.1145/2207016.2207053 · Cited by 47 · see citing works
College Students' Perceptions of the C-Print Speech-to-Text Transcription System
Elliot, L. B. (2001) · The Journal of Deaf Studies and Deaf Education · doi:10.1093/deafed/6.4.285 · Cited by 46 · see citing works
Automatic human utility evaluation of ASR systems: does WER really predict performance?
Favre, B., Cheung, K., Kazemian, S., Lee, A., Liu, Y., Munteanu, C., Nenkova, A., Ochei, D., Penn, G., Tratz, S., et al. (2013) · Interspeech 2013 · doi:10.21437/interspeech.2013-610 · Cited by 44 · see citing works
Captioning for Deaf and Hard of Hearing People by Editing Automatic Speech Recognition in Real Time
Wald, M. (2006) · Lecture notes in computer science · doi:10.1007/11788713_100 · Cited by 42 · see citing works
Tracked Speech-To-Text Display
Kushalnagar, R., Behm, G. W., Kelstone, A. W., & Ali, S. (2015) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/2700648.2809843 · Cited by 34 · see citing works
Towards More Robust Speech Interactions for Deaf and Hard of Hearing Users
Fok, R., Kaur, H., Palani, S., Mott, M. E., & Lasecki, W. S. (2018) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3234695.3236343 · Cited by 33 · see citing works
A readability evaluation of real-time crowd captions in the classroom
Kushalnagar, R., Lasecki, W. S., & Bigham, J. P. (2012) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/2384916.2384930 · Cited by 32 · see citing works
Červa, P., Silovský, J., Žďánský, J., et al. (2013) · Speech Communication · doi:10.1016/j.specom.2013.06.017 · Cited by 22 · see citing works
Piquard-Kipffer, A., Mella, O., Miranda, J., Jouvet, D., & Orosanu, L. (2015) · Workshop on Speech and Language Processing for Assistive Technologies · doi:10.18653/v1/w15-5107 · Cited by 14 · see citing works
Predicting error rates for unknown data in automatic speech recognition
Meyer, B. T., Mallidi, S. H., Kayser, H., et al. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7953174 · Cited by 10 · see citing works
Fujii, Y., Yamamoto, K., & Nakagawa, S. (2012) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.e95.d.1101 · Cited by 3 · see citing works
Buzo, A., Cucu, H., Burileanu, C., et al. (2011) · doi:10.1109/sped.2011.5940731 · Cited by 2 · see citing works
Post-error Correction in Automatic Speech Recognition Using Discourse Information
Kang, S., Kim, J. H., & Seo, J. (2014) · Advances in Electrical and Computer Engineering · doi:10.4316/aece.2014.02009 · Cited by 2 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
McDonnell, E. J., Liu, P., Goodman, S. M., Kushalnagar, R., Froehlich, J. E., & Findlater, L. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479578 · Cited by 54 · see citing works
McDonnell, E. J., Eagle, T., Sinlapanuntakul, P., Moon, S. H., Ringland, K. E., Froehlich, J. E., & Findlater, L. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642177 · Cited by 34 · see citing works
“Traditional captioning tool design either assumes captions will be generated by professional CART captioners (e.g., [46, 53, 59]), or by automatic speech recognition-based tools (e.g., [23, 70]), and therefore does not consider the needs of non-expert captioners.”
Pataca, C. D. L., Hassan, S., Tinker, N., Peiris, R. L., & Huenerfauth, M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642258 · Cited by 31 · see citing works
“Captions are a widely adopted strategy to make speech accessible [12, 46, 48, 49].”
BackgroundMcDonnell, E. J., Moon, S. H., Jiang, L., Goodman, S. M., Kushalnagar, R., Froehlich, J. E., & Findlater, L. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580809 · Cited by 27 · see citing works
“Strategies to minimize visual dispersion include collating distributed information to one interface [11], projecting captions to follow the speaker [33], and displaying captions underneath lecture slides [10].”
BackgroundWeb accessibility automatic evaluation tools: to what extent can they be automated?
Doush, I. A., Sultan, K., Al‐Betar, M. A., AlMeraj, Z., Alyasseri, Z. A. A., & Awadallah, M. A. (2023) · CCF Transactions on Pervasive Computing and Interaction · doi:10.1007/s42486-023-00127-8 · Cited by 25 · see citing works
A scoping review on U.S. undergraduate students with disabilities in STEM courses and STEM majors
Borrego, M. J., Chasen, A., Tripp, H. C., Landgren, E., & Koolman, E. (2025) · International Journal of STEM Education · doi:10.1186/s40594-024-00522-2 · Cited by 23 · see citing works
“Many technology projects focused on developing and translating scientific terms and mathematical equations into captions (Butler et al., 2019), speech (Isaacson & Michaels, 2015), American Sign Language (Andrei et al., 2013; Kumawat, 2011) and Braille for mathematics (McDermott-Wells, 2015;…”
Artificial intelligence for students in postsecondary education
Martiniello, N., Asuncion, J. V., Fichten, C. S., Jorgensen, M., Havel, A., Harvison, M., Legault, A., Lussier, A., & Vo, C. (2020) · AI Matters · doi:10.1145/3446243.3446250 · Cited by 15 · see citing works
“In spite of increased use of automatic speech recognition, Deaf and hard of hearing students often do not have adequate access to qualified real-time captioning (Butler, Trager, & Behm, 2019).”
BackgroundInclusive Deaf Education Enabled by Artificial Intelligence: The Path to a Solution
Coy, A., Mohammed, P. S., & Skerrit, P. (2024) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-024-00419-9 · Cited by 22 · see citing works
What Factors Motivate Culturally Deaf People to Want Assistive Technologies?
Vogel, A., & Korte, J. L. (2024) · Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613905.3650871 · Cited by 15 · see citing works
“These concerns act as barriers to adoption, although Butler et al.[6] found that DHH students were more accepting of ASR in conjunction with other assistive technologies Historically, the development of ATs has focused on functionality alone [26].”
McDonnell, E. J., & Findlater, L. (2024) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3663548.3675649 · Cited by 13 · see citing works
“Particularly when researching captions in the classroom, lecturers and university access service providers are often implicitly situated as stakeholders and users of captioning technologies, though their perspectives are rarely represented in findings (e.g., [28, 77, 80].”
BackgroundModeling and Improving Text Stability in Live Captions
Liu, X., Zhang, J., Ferrer, L., et al. (2023) · Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3544549.3585609 · Cited by 11 · see citing works
“[4, 24] found that while d/Deaf or Hard of Hearing (DHH) students found the benefts of ASR, they identify accuracy and readability of ASR as major pain points, and struggle with low-quality captions [3, 11].”
BackgroundMcDonnell, E. J. (2022) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3517428.3550417 · Cited by 7 · see citing works
“Many captioning tools are designed for use during classroom lectures [2, 11, 12, 14, 27– 30, 60], which Most related to my work is an ongoing efort by Seita et al. [53–56], who have been exploring how hearing people’s behavior changes when using automatic captioning, behavioral impact on DHH…”
BackgroundConducting HCI Research with the Deaf Community in American Sign Language: Practices and Experiences
Mahajan, S., Boll, R., Alkhudaidi, K., Reis, J., & Solovey, E. (2025) · CHI Extended Abstracts · doi:10.1145/3706599.3706691 · Cited by 5 · see citing works
“The research is spread across domains, including education [8], news [2], meetings [27] and sound detection applications [6].”
Samaradivakara, Y., Ushan, T., Pathirage, A., Sasikumar, P., Karunanayaka, K. T., Keppitiyagama, C., & Nanayakkara, S. C. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3650974 · Cited by 4 · see citing works
“With the recent advancements in speech recognition, caption interfaces have been used as a promising solution to provide real-time access to spoken words for DHH individuals [9, 29, 36].”
MethodReal-Time Speech-to-Text Holographic Communication for the Deaf Children and Elderly
Charan, S. S., Srihitha, V., Palaniswamy, S., et al. (2024) · doi:10.1109/mpcit62449.2024.10892690 · Cited by 3 · see citing works
“Butler [13] investigated the application and efficacy of ASR as a supplemental access technology for DHH students in university courses, identifying benefits and areas for improvement to enhance its effectiveness as an educational tool.”
Dementyev, A., Kanevsky, D., Yang, S., et al. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713631 · Cited by 3 · see citing works
(2022) · The WAC Clearinghouse; University Press of Colorado eBooks · doi:10.37514/pra-b.2022.1541 · Cited by 3 · see citing works
Pucci, M. (2023) · Studies in health technology and informatics · doi:10.3233/shti230394 · Cited by 6 · see citing works
“For example, the simultaneous presentation of auditory input and written transcription help diverse students recover missing or incomplete information [9].”
BackgroundInfluentialSalimi, N., Gere, B., Mora, E., & Chen, R. K. (2024) · Advances in educational marketing, administration, and leadership book series · doi:10.4018/979-8-3693-3342-6.ch006 · Cited by 2 · see citing works
Chapter 5. Digital Story-Mapping
nar, E. �. I., & Beltran, D. Q. (2022) · The WAC Clearinghouse; University Press of Colorado eBooks · doi:10.37514/pra-b.2022.1541.2.05 · Cited by 1 · see citing works
Kang, J., Layton, E., Martin, D., & Starner, T. E. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3650976 · Cited by 1 · see citing works
“Recently, we have seen ASR integrated into HWDs [3, 12, 13, 17, 19, 22], the classroom [4, 23] and video-conferencing [2, 16].”
BackgroundMcDonnell, E. J. (2023) · ACM SIGACCESS Accessibility and Computing · doi:10.1145/3584732.3584735 · Cited by 1 · see citing works
Redesigning Educational Videos for Deaf and Hard-of-Hearing Learners
Chen, S., Cheng, H., Su, S., Ming, L., Masud, S., Wang, Q., & Huang, Y. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791470
“Attention for DHH Learners Efforts to make video learning content accessible to DHH learners often focus on closed captions and descriptive transcripts, including the use of AI technology to support automatic speech recognition [17, 64].”
Code of Ethics for Automatic Speech Recognition
Cecilio, J. E. (2022) · Proceedings of the Wellington Faculty of Engineering Ethics and Sustainability Symposium · doi:10.26686/wfeess.vi.7646
“Another application for ASR is live speech captioning as an aide for individuals that are deaf or hard-of-hearing [5].”
BackgroundMahnke, S., & Wilson, J. B. (2022) · The WAC Clearinghouse; University Press of Colorado eBooks · doi:10.37514/pra-b.2022.1541.2.08
Butler, J. (2022) · The WAC Clearinghouse; University Press of Colorado eBooks · doi:10.37514/pra-b.2022.1541.2.03
Self-citationShivers-McNair, A. (2022) · The WAC Clearinghouse; University Press of Colorado eBooks · doi:10.37514/pra-b.2022.1541.2.01
Poudyal, B. (2022) · The WAC Clearinghouse; University Press of Colorado eBooks · doi:10.37514/pra-b.2022.1541.2.09
Craig, T. (2022) · The WAC Clearinghouse; University Press of Colorado eBooks · doi:10.37514/pra-b.2022.1541.2.04
VanKooten, C., & Hierro, V. D. (2022) · The WAC Clearinghouse; University Press of Colorado eBooks · doi:10.37514/pra-b.2022.1541.1.3
Shi, X., & Wang, X. (2023) · International Conference on Advanced Algorithms and Signal Image Processing (AASIP) · doi:10.1117/12.3006706
McDonnell, E. J., Jones, T. E., & Pratt, W. (2026) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3797867.3829045
“HCI Accessibility research has long-focused on providing critical support for disabled people to access work (e.g., [30, 100]), public life (e.g., [40, 56]), online discourse (e.g.,[7, 39]), and education (e.g., [20, 61]), but relatively little work supports domestic access (e.g., [23, 25]).”
- Authors
- Butler, J., Trager, B., & Behm, B.
- Affiliations
- Rochester Institute of Technology (US)
- ORCID
- Butler, Janine
- Venue
- Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility
- Publisher
- ACM
- Published
- 2019-10-24
- Pages
- 32-42
- OpenAlex
- W2981873888
- Semantic Scholar
- 26912d031fb3af47a4e12441281aa428e5111ef5
- Cited by
- 31
- References
- 24
- Citations by year
- 2026: 1 · 2025: 3 · 2024: 9 · 2023: 6 · 2022: 10 · 2021: 1 · 2020: 1
- FWCI
- 30.52
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) (publishedVersion)
- Topics
- Social Sciences › Arts and Humanities › Language and Linguistics › Subtitles and Audiovisual Media
- Social Sciences › Psychology › Developmental and Educational Psychology › Hearing Impairment and Communication
- Social Sciences › Safety Research › Disability Education and Employment
- Keywords
- speech recognition, deaf and hard of hearing students, higher education accessibility, assistive technology, student experience, mainstream classrooms
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
