Creating accessible educational multimedia through editing automatic speech recognition captioning in real time
Abstract
Lectures can be digitally recorded and replayed to provide multimedia revision material for students who attended the class and a substitute learning experience for students unable to attend. Deaf and hard of hearing people can find it difficult to follow speech through hearing alone or to take notes while they are lip‐reading or watching a sign‐language interpreter. Notetakers can only summarise what is being said while qualified sign language interpreters with a good understanding of the relevant higher education subject content are in very scarce supply. Synchronising the speech with text captions can ensure deaf students are not disadvantaged and assist all learners to search for relevant specific parts of the multimedia recording by means of the synchronised text. Real time stenography transcription is not normally available in UK higher education because of the shortage of stenographers wishing to work in universities. Captions are time consuming and expensive to create by hand and while Automatic Speech Recognition can be used to provide real time captioning directly from lecturers’ speech in classrooms it has proved difficult to obtain accuracy comparable to stenography. This paper describes the development of a system that enables editors to correct errors in the captions as they are created by Automatic Speech Recognition.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Speech can be up to ten times faster than typing, making manual transcription laborious.
“Manually creating accurate transcripts for spoken language is a laborious task, as speech is up to ten times faster than typing [Wald 2006].”
From Measuring the Accuracy of Automatic Speech Recognition Solutions (2023) · cites this paper as Wald 2006
Cites
Learning Styles and Pedagogy in post-16 learning: a systematic and critical review
Coffield, F., Moseley, D., Hall, E., & Ecclestone, K. (2004) · Newcastle University · doi:10.25405/ncl.34085991 · Cited by 1,810 · see citing works
Journal of Computing Sciences in Colleges
(2019) · Journal of computing sciences in colleges · doi:10.36614/jcsc · Cited by 1,035 · see citing works
Gathercole, S. E., Pickering, S. J., Knight, C. J., & Stegmann, Z. (2003) · Applied Cognitive Psychology · doi:10.1002/acp.934 · Cited by 955 · see citing works
Cognitive effort during note taking
Piolat, A., Olive, T., & Kellogg, R. T. (2005) · Applied Cognitive Psychology · doi:10.1002/acp.1086 · Cited by 517 · see citing works
The limits of speech recognition
Shneiderman, B. (2000) · Communications of the ACM · doi:10.1145/348941.348990 · Cited by 279 · see citing works
Patterns of entry and correction in large vocabulary continuous speech recognition systems
Karat, C.-M., Halverson, C. A., Horn, D. B., & Karat, J. (1999) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/302979.303160 · Cited by 276 · see citing works
Brotherton, J. A., & Abowd, G. D. (2004) · ACM Transactions on Computer-Human Interaction · doi:10.1145/1005361.1005362 · Cited by 267 · see citing works
Reading text from computer screens
Mills, C. B., & Weldon, L. J. (1987) · ACM Computing Surveys · doi:10.1145/45075.46162 · Cited by 253 · see citing works
Multimodal error correction for speech user interfaces
Suhm, B., Myers, B., & Waibel, A. (2001) · ACM Transactions on Computer-Human Interaction · doi:10.1145/371127.371166 · Cited by 198 · cited 1× · see citing works
“However Suhm & Myers (2001) found that highlighting likely errors based on these confidence scores didn’t help speed up correction as some correct words were also highlighted as well.”
BackgroundProceedings of Graphics Interface 2005
Inkpen, K. M., & van de Panne, M. (2005) · Cited by 144 · see citing works
Speech recognition in university classrooms
Bain, K., Basson, S. H., & Wald, M. (2002) · doi:10.1145/638249.638284 · Cited by 87 · see citing works
Self-citationMeasuring the readability of automatic speech-to-text transcripts
Jones, D. A., Wolf, F., Gibson, E. A. F., Williams, E., Fedorenko, E. G., Reynolds, D. A., & Zissman, M. A. (2003) · European Conference on Speech Communication and Technology (Eurospeech) · doi:10.21437/eurospeech.2003-463 · Cited by 71 · see citing works
Model-based and empirical evaluation of multimodal interactive error correction
Suhm, B., Myers, B. A., & Waibel, A. (1999) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/302979.303165 · Cited by 67 · see citing works
Accessibility, transcription, and access everywhere
Bain, K., Basson, S. H., Faisman, A., & Kanevsky, D. (2005) · IBM Systems Journal · doi:10.1147/sj.443.0589 · Cited by 56 · see citing works
Speech-Based Real-Time Subtitling Services
Lambourne, A., Hewitt, J. A., Lyon, C., & Warren, S. (2004) · International Journal of Speech Technology · doi:10.1023/b:ijst.0000037071.39044.cc · Cited by 40 · see citing works
Whittaker, S., & Amento, B. (2004) · ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) · doi:10.1145/985692.985759 · Cited by 37 · see citing works
Karat, J., Horn, D. B., Halverson, C. A., & Karat, C.-M. (2000) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/633292.633372 · Cited by 30 · see citing works
Effect of Error Correction Strategy on Speech Dictation Throughput
Lewis, J. R. (1999) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193129904300514 · Cited by 26 · see citing works
Baecker, R., Wolf, P., & Rankin, K. (2004) · E-Learn: World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education · Cited by 16 · see citing works
Hearing disability and technology
Wald, M. (2002) · ePrints Soton (University of Southampton) · Cited by 15 · see citing works
Self-citationPortability issues for speech recognition technologies
Lamel, L. F., Lefèvre, F., Gauvain, J., & Adda, G. (2001) · International Conference on Human Language Technology Research · doi:10.3115/1072133.1072211 · Cited by 14 · see citing works
Phonetic searching applied to on-line distance learning modules
Clements, M. A., Robertson, S., & Miller, M. S. (2005) · IEEE Digital Signal Processing Workshop and IEEE Signal Processing Education Workshop (DSP/SPE) · doi:10.1109/dspws.2002.1231100 · Cited by 13 · see citing works
Closed captioning in distance education
Carroll, J., & McLaughlin, K. (2005) · Journal of computing sciences in colleges · doi:10.5555/1047846.1047875 · Cited by 4 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Real-time captioning by groups of non-experts
Lasecki, W. S., Miller, C. D., Sadilek, A., Abumoussa, A. L., Borrello, D., Kushalnagar, R., & Bigham, J. P. (2012) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2380116.2380122 · Cited by 199 · see citing works
Beyond lecture capture: Creating digital video content for online learning – a case study
Dinmore, S. P. (2019) · Journal of University Teaching and Learning Practice · doi:10.53761/1.16.1.7 · Cited by 52 · see citing works
Measuring the Accuracy of Automatic Speech Recognition Solutions
Kuhn, K., Kersken, V., Reuter, B., Egger, N., & Zimmermann, G. (2023) · ACM Transactions on Accessible Computing · doi:10.1145/3636513 · Cited by 44 · see citing works
“Manually creating accurate transcripts for spoken language is a laborious task, as speech is up to ten times faster than typing [Wald 2006].”
The Case for Captioned Lectures in Australian Higher Education
Kent, M., Ellis, K., Latter, N., & Peaty, G. (2018) · TechTrends · doi:10.1007/s11528-017-0225-x · Cited by 35 · see citing works
A readability evaluation of real-time crowd captions in the classroom
Kushalnagar, R., Lasecki, W. S., & Bigham, J. P. (2012) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/2384916.2384930 · Cited by 32 · see citing works
Online quality control for real-time crowd captioning
Lasecki, W. S., & Bigham, J. P. (2012) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/2384916.2384942 · Cited by 28 · see citing works
Lee, P.-J., Liu, Y., & Tseng, W. (2019) · Language Teaching Research · doi:10.1177/1362168819856451 · Cited by 26 · see citing works
Text Alignment for Real-Time Crowd Captioning
Naim, I., Gildea, D., Lasecki, W. S., & Bigham, J. P. (2013) · Cited by 25 · see citing works
Lasecki, W. S., Miller, C. D., Naim, I., Kushalnagar, R., Sadilek, A., Gildea, D., & Bigham, J. P. (2017) · Communications of the ACM · doi:10.1145/3068663 · Cited by 23 · see citing works
Wald, M., & Bain, K. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-73283-9_50 · Cited by 22 · see citing works
Self-citationMiró, J. D. V., Silvestre-Cerdà, J. A., Civera, J., Turró, C., & Juan, A. (2015) · Speech Communication · doi:10.1016/j.specom.2015.09.006 · Cited by 21 · see citing works
Evaluating intelligent interfaces for post-editing automatic transcriptions of online video lectures
Miró, J. D. V., Spencer, R. N., Pérez-González-de-Martos, A., Díaz-Munío, G. V. G., Turró, C., Civera, J., & Juan, A. (2014) · Open Learning The Journal of Open Distance and e-Learning · doi:10.1080/02680513.2014.909722 · Cited by 20 · see citing works
Synote: Accessible and Assistive Technology Enhancing Learning for All Students
Wald, M. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-14100-3_27 · Cited by 12 · see citing works
Self-citationEfficient Generation of High-Quality Multilingual Subtitles for Video Lecture Repositories
Miró, J. D. V., Silvestre-Cerdà, J. A., Civera, J., Turró, C., & Juan, A. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24258-3_44 · Cited by 12 · see citing works
E-Inclusion for People with Disabilities in E-Government Services through Accessible Multimedia
García-Crespo, Á., Martín, F. P., Colomo‐Palacios, R., & Gómez-Berbís, J. M. (2012) · International Journal of Information Systems and Social Change · doi:10.4018/jissc.2012070103 · Cited by 11 · see citing works
Synchronised Annotation of Multimedia
Wald, M., Wills, G. B., Millard, D., Gilbert, L. H., Khoja, S. A., Kajaba, J., & Li, Y. (2009) · IEEE International Conference on Advanced Learning Technologies (ICALT) · doi:10.1109/icalt.2009.16 · Cited by 10 · see citing works
Self-citationCaptioning Multiple Speakers Using Speech Recognition to Assist Disabled People
Wald, M. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-70540-6_88 · Cited by 10 · see citing works
Self-citationLearning Through Multimedia: Speech Recognition Enhancing Accessibility and Interaction
Wald, M. (2008) · Journal of educational multimedia and hypermedia · Cited by 10 · see citing works
Self-citationSilvestre-Cerdà, J. A., Pérez-González-de-Martos, A., Jimenez, M., Turró, C., Juan, A., & Civera, J. (2013) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc.2013.682 · Cited by 9 · see citing works
Accuracy of AI-generated Captions With Collaborative Manual Corrections in Real-Time
Kuhn, K., Kersken, V., & Zimmermann, G. (2023) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3544549.3585724 · Cited by 7 · see citing works
Kuhn, K., Kersken, V., & Zimmermann, G. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3719984 · Cited by 6 · see citing works
Pucci, M. (2023) · Studies in health technology and informatics · doi:10.3233/shti230394 · Cited by 6 · see citing works
Li, Y., Wald, M., Khoja, S. A., Wills, G. B., Millard, D. E., Kajaba, J., Singh, P., & Gilbert, L. H. (2009) · doi:10.1145/1631111.1631114 · Cited by 6 · see citing works
Self-citationReal-time correction of closed-captions
Cardinal, P., Boulianne, G., Comeau, M., & Boisvert, M. (2007) · doi:10.3115/1557769.1557803 · Cited by 6 · see citing works
22. Transcribe Your Class: Using Speech Recognition to Improve Access for At-Risk Students
Bain, K., Lund‐Lucas, E., & Stevens, J. (2012) · Collected Essays on Learning and Teaching · doi:10.22329/celt.v5i0.3419 · Cited by 5 · see citing works
Net4voice: new technologies for voice-converting in barrier-free learning environments
Luppi, E., Primiani, R., Raffaelli, C., Tibaldi, D., Traina, I., & Violi, A. (2009) · Archivio istituzionale della ricerca (Alma Mater Studiorum Università di Bologna) · Cited by 3 · see citing works
Kustritz, M. R., Rupprecht, R., & Zhitnitskiy, P. E. (2023) · Clinical Theriogenology · doi:10.58292/ct.v15.9596 · Cited by 2 · see citing works
Using automatic speech transcriptions in lecture recommendation systems
Pérez-González-de-Martos, A., Silvestre-Cerdà, J. A., Rihtar, M., Císcar, A. J., & Saiz, J. C. (2014) · RiuNet (Universitat Politècnica de València) · Cited by 2 · see citing works
Voice-to-Text Transcription of Lecture Recordings
Dinmore, S. P., & Gao, J. P. (2016) · ASCILITE Publications · doi:10.14742/apubs.2016.864 · Cited by 2 · see citing works
Online Sequence Alignment for Real-Time Audio Transcription by Non-Experts
Lasecki, W. S., Miller, C., Borrello, D., & Bigham, J. P. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v26i1.8420 · Cited by 2 · see citing works
Exploring Practical Metrics to Support Automatic Speech Recognition Evaluations
Draffan, E. A., Wald, M., Ding, C., & Li, Y. (2023) · Studies in health technology and informatics · doi:10.3233/shti230636 · Cited by 1 · see citing works
Self-citationDiseño de un programa de subtitulación de vídeos pedagógicos
Quintas, L. C. (2019) · doi:10.6035/14110.2019.762203 · Cited by 1 · see citing works
Enhancing learning using synchronised multimedia annotation
Wald, M., Wills, G. B., Millard, D. E., Gilbert, L. H., Khoja, S. A., Kajaba, J., Li, Y., & Singh, P. (2009) · ePrints Soton (University of Southampton)
Self-citationSliding Alignment Windows for Real-Time Crowd Captioning
Kazemi, M., Lavaee, R., Naim, I., & Gildea, D. (2014) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/v1/p14-2039
Different Contributions to Cost-Effective Transcription and Translation of Video Lectures.
Silvestre-Cerdà, J. A. (2016) · doi:10.4995/thesis/10251/62194
Faculty Development through Collaboratively Coordinated Teaching
Mingus, T. T. Y., & Koelling, M. E. (2024) · Emerald Group Publishing Limited eBooks · doi:10.1108/979-8-88730-446-520251019
Accuracy and Efficiency of Articicial Intelligence Tools for Transcription in Qualitative Research
Hennink, M. M., Kazzi, B., Yoon, E., Alami, N. H., & Farhad, G. (2026) · American Journal of Qualitative Research · doi:10.66815/ajqr/19595
- Authors
- Wald, M.
- Affiliations
- University of Southampton (GB)
- Corresponding author
- Wald, Mike
- ORCID
- Wald, Mike
- Venue
- Interactive Technology and Smart Education
- Publisher
- Emerald
- Published
- 2006-05-01
- Volume
- 3
- Issue
- 2
- Pages
- 131-141
- ISSN
- 1741-5659 (ISSN-L)
- OpenAlex
- W1983009749
- Cited by
- 38
- References
- 23
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 1 · 2023: 5 · 2021: 1 · 2019: 4 · 2017: 2 · 2016: 2 · 2015: 2 · 2014: 3 · 2013: 2 · 2012: 5
- FWCI
- 22.55
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Interactive Technology and Smart Education (publishedVersion)
- ePrints Soton (University of Southampton) (submittedVersion)
- Topics
- Social Sciences › Arts and Humanities › Language and Linguistics › Subtitles and Audiovisual Media
- Social Sciences › Education › Innovations in Educational Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- speech recognition, real-time captioning, deaf and hard of hearing students, accessible education, lecture recordings, stenography, higher education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
