Toward Automatic Audio Description Generation for Accessible Videos
The DOI matches its Crossref record; 53 of the 54 references it cites were found in a registry or an index.
Abstract
Video accessibility is essential for people with visual impairments. Audio descriptions describe what is happening on-screen, e.g., physical actions, facial expressions, and scene changes. Generating high-quality audio descriptions requires a lot of manual description generation [50]. To address this accessibility obstacle, we built a system that analyzes the audiovisual contents of a video and generates the audio descriptions. The system consisted of three modules: AD insertion time prediction, AD generation, and AD optimization. We evaluated the quality of our system on five types of videos by conducting qualitative studies with 20 sighted users and 12 users who were blind or visually impaired. Our findings revealed how audio description preferences varied with user types and video types. Based on our study’s analysis, we provided recommendations for the development of future audio description generation technologies.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“Starting from word embeddings [7, 32, 59], recent popular neural network methods evolved to use sentence-level embedding for distance metrics [5, 30, 36].”
MethodLearning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593 · cited 1×
“Research on dense video captioning, such as C3D [53] and I3D [11], showed that multiple features can improve video captioning models [37, 38, 54].”
BackgroundQuo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
A Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,479
HuggingFace's Transformers: State-of-the-art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.03771 · Cited by 3,347
A neural probabilistic language model
BengioYoshua, DucharmeRéjean, VincentPascal, & JanvinChristian (2003) · Journal of Machine Learning Research · doi:10.5555/944919.944966 · Cited by 2,644
Sequence to Sequence -- Video to Text
Venugopalan, S., Rohrbach, M., Donahue, J., Mooney, R., Darrell, T., & Saenko, K. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.515 · Cited by 1,352 · cited 1×
“Research on dense video captioning, such as C3D [53] and I3D [11], showed that multiple features can improve video captioning models [37, 38, 54].”
BackgroundDense-Captioning Events in Videos
Krishna, R., Hata, K., Ren, F., Fei-Fei, L., & Niebles, J. C. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.83 · Cited by 1,251
Attention on Attention for Image Captioning
Huang, L., Wang, W., Chen, J., & Wei, X.-Y. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00473 · Cited by 1,092 · cited 1×
“Similar to image captioning, it requires not only recognizing salient objects, but also understanding actions and interactions [26].”
Background(2017) · International Conference on Learning Representations · Cited by 1,075
What in the World Do We Hear?: An Ecological Approach to Auditory Event Perception
Gaver, W. W. (1993) · Ecological Psychology · doi:10.1207/s15326969eco0501_1 · Cited by 903 · cited 3×
“As demonstrated in [18], humans are able to perceive events and environments based on sounds.”
BackgroundInfluentialArandjelovic, R., & Zisserman, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.73 · Cited by 782 · cited 4×
“Researchers investigate different techniques along this direction, such as detecting audio-visual events [4], improving human speech consistency [34, 57], and generating scene-aware sound [25, 58].”
BackgroundMethodInfluentialUnsupervised Learning of Sentence Embeddings Using Compositional n-Gram Features
Pagliardini, M., Gupta, P., & Jaggi, M. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1049 · Cited by 678 · cited 1×
“Starting from word embeddings [7, 32, 59], recent popular neural network methods evolved to use sentence-level embedding for distance metrics [5, 30, 36].”
MethodEnd-to-End Dense Video Captioning with Masked Transformer
Zhou, L., Zhou, Y., Corso, J. J., Socher, R., & Xiong, C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00911 · Cited by 605
Jointly Modeling Embedding and Translation to Bridge Video and Language
Pan, Y., Mei, T., Yao, T., Li, H., & Rui, Y. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.497 · Cited by 563 · cited 1×
“Research on dense video captioning, such as C3D [53] and I3D [11], showed that multiple features can improve video captioning models [37, 38, 54].”
BackgroundAn efficient framework for learning sentence representations
Logeswaran, L., & Lee, H. (2018) · arXiv · doi:10.48550/arxiv.1803.02893 · Cited by 427 · cited 1×
“Starting from word embeddings [7, 32, 59], recent popular neural network methods evolved to use sentence-level embedding for distance metrics [5, 30, 36].”
MethodVideo Captioning with Transferred Semantic Attributes
Pan, Y., Yao, T., Li, H., & Mei, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.111 · Cited by 364 · cited 1×
“Research on dense video captioning, such as C3D [53] and I3D [11], showed that multiple features can improve video captioning models [37, 38, 54].”
BackgroundRohrbach, A., Torabi, A., Rohrbach, M., Tandon, N., Pal, C., Larochelle, H., Courville, A., & Schiele, B. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0987-1 · Cited by 300
Blind Navigation and the Role of Technology
Giudice, N. A., & Legge, G. E. (2008) · The Engineering Handbook of Smart Technology for Aging, Disability, and Independence · doi:10.1002/9780470379424.ch25 · Cited by 232 · cited 1×
“For visually-impaired people, it is important to train their listening skills to enhance vision-to-sound mapping early in their life, since sound is a major source of information [8, 19].”
BackgroundBidirectional Attentive Fusion with Context Gating for Dense Video Captioning
Wang, J., Jiang, W., Ma, L., Liu, W., & Xu, Y. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00751 · Cited by 223
Video Description: A Survey of Methods, Datasets, and Evaluation Metrics
Aafaq, N., Mian, A., Liu, W., Gilani, S. Z., & Shah, M. (2020) · ACM Computing Surveys · doi:10.1145/3355390 · Cited by 211
Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning
Subramanian, S., Trischler, A., Bengio, Y., & Pal, C. J. (2018) · arXiv · doi:10.48550/arxiv.1804.00079 · Cited by 207 · cited 1×
“, syntax, length, and word order, proved to improve the ability to measure semantic textual similarity [52].”
BackgroundDeep Multimodal Clustering for Unsupervised Audiovisual Learning
Hu, D., Nie, F., & Li, X. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00947 · Cited by 194 · cited 1×
“In computer vision, audio-visual related research explores algorithms for achieving audiovisual understanding [4, 6, 24] akin to training visually-impaired people to understand their environments from audio.”
BackgroundJointly Localizing and Describing Events for Dense Video Captioning
Li, Y., Yao, T., Pan, Y., Chao, H., & Mei, T. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00782 · Cited by 191
Weakly Supervised Dense Event Captioning in VideosPreprint
Duan, X., Huang, W., Gan, C., Wang, J., Zhu, W., & Huang, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.03849 · Cited by 171 · cited 1×
“To generate high-quality descriptions for all events of interest of a video, the fine-grained video clips were fed into an attention-based model [12], which consists of a sentence localizer and a caption generator.”
MethodSpeech2Face: Learning the Face Behind a Voice
Oh, T.-H., Dekel, T., Kim, C., Mosseri, I., Freeman, W. T., Rubinstein, M., & Matusik, W. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00772 · Cited by 168 · cited 1×
“Researchers investigate different techniques along this direction, such as detecting audio-visual events [4], improving human speech consistency [34, 57], and generating scene-aware sound [25, 58].”
BackgroundStangl, A., Morris, M. R., & Gurari, D. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376404 · Cited by 139
Walczak, A., & Fryer, L. (2017) · British Journal of Visual Impairment · doi:10.1177/0264619616661603 · Cited by 135 · cited 1×
“, cinematic language style [17], standard and creative style [55]).”
BackgroundTwitter A11y: A Browser Extension to Make Twitter Images Accessible
Gleason, C., Pavel, A., McCamey, E., Low, C., Carrington, P., Kitani, K. M., & Bigham, J. P. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376728 · Cited by 121 · cited 1×
“developed various tools to help generate alt texts of images [22], memes [21], and GIFs [20].”
MethodCaption Crawler: Enabling Reusable Alternative Text Descriptions using Reverse Image Search
Guinness, D., Cutrell, E., & Morris, M. R. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174092 · Cited by 108 · cited 1×
“Guinness et al. leveraged reverse image search to collect similar images that have been annotated [23].”
BackgroundCinematic language and the description of film: keeping AD users in the frame
Fryer, L., & Freeman, J. (2013) · Perspectives · doi:10.1080/0907676x.2012.693108 · Cited by 98 · cited 1×
“, cinematic language style [17], standard and creative style [55]).”
BackgroundSalisbury, E., Kamar, E., & Morris, M. (2017) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v5i1.13301 · Cited by 83 · cited 1×
“proposed a real-time crowd-sourcing experience which brought human in the loop to compensate for the shortcomings of existing AI algorithms [46].”
BackgroundConsciousness Inside and Out: Phenomenology, Neuroscience, and the Nature of Experience
Brown, R. (2013) · Springer eBooks · doi:10.1007/978-94-007-6001-1 · Cited by 79
An Efficient Framework for Sentence Similarity Modeling
Quan, Z., Wang, Z.-J., Le, Y., Yao, B., Li, K., & Yin, J. P. (2019) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2019.2899494 · Cited by 73 · cited 1×
“Sentence similarity and perplexity modeling lie at the core of many natural language processing applications [41].”
BackgroundGleason, C., Pavel, A., Liu, X., Carrington, P., Chilton, L. B., & Bigham, J. P. (2019) · Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3308561.3353792 · Cited by 69 · cited 1×
“developed various tools to help generate alt texts of images [22], memes [21], and GIFs [20].”
MethodUsability of a Multimodal Video Game to Improve Navigation Skills for Blind Children
Sánchez, J., Sáenz, M., & Garrido, J. M. (2010) · ACM Transactions on Accessible Computing · doi:10.1145/1857920.1857924 · Cited by 69 · cited 1×
“Moreover, we hope that our work will not only contribute to the design of audio description generation for social and video streaming platforms but also improve navigation skill training for blind children via multi-modal video games [47].”
BackgroundCreating Coherence in Audio Description
Braun, S. (2012) · Meta Journal des traducteurs · doi:10.7202/1008338ar · Cited by 53 · cited 2×
“The effects carried by the edits need to be illustrated by the audio descriptions to support the BVI users in the creation of a coherent understanding of the video content [9].”
BackgroundA Comparison of Alternative Narrative Approaches to Video Description for Animated Comedy
Fels, D. I., Udo, J. P., Diamond, J. E., & Diamond, J. I. (2006) · Journal of Visual Impairment & Blindness · doi:10.1177/0145482x0610000507 · Cited by 52 · cited 1×
“Blind users find it valuable to watch videos, television, or movies independently, which, without an automatic AD generator, is hard to achieve [14].”
BackgroundCould Audio-Described Films Benefit from Audio Introductions? An Audience Response Study
Romero-Fresco, P., & Fryer, L. (2013) · Journal of Visual Impairment & Blindness · doi:10.1177/0145482x1310700405 · Cited by 50 · cited 1×
“, science programs [48], film [45], and live theater [31]), different audiences (i.”
BackgroundAudible Panorama: Automatic Spatial Audio Generation for Panorama Imagery
Huang, H., Solah, M., Li, D., & Yu, L.-F. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300851 · Cited by 40 · cited 1×
“Researchers investigate different techniques along this direction, such as detecting audio-visual events [4], improving human speech consistency [34, 57], and generating scene-aware sound [25, 58].”
BackgroundSelf-citationGleason, C., Pavel, A., Gururaj, H., Kitani, K., & Bigham, J. (2020) · Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3373625.3417027 · Cited by 40 · cited 4×
“Recently, Gleason et al. developed various tools to help generate alt texts of images [22], memes [21], and GIFs [20].”
MethodInfluentialAn introduction to audio description: a practical guide
Nikolić, K. (2017) · Perspectives · doi:10.1080/0907676x.2017.1355345 · Cited by 36
Spatial Mapping of Familiar Locations in Blind Children
Bigelow, A. (1991) · Journal of Visual Impairment & Blindness · doi:10.1177/0145482x9108500307 · Cited by 34 · cited 1×
“For visually-impaired people, it is important to train their listening skills to enhance vision-to-sound mapping early in their life, since sound is a major source of information [8, 19].”
BackgroundAn Overview of Video Description: History, Benefits, and Guidelines
Packer, J., Vizenor, K., & Miele, J. A. (2015) · Journal of Visual Impairment & Blindness · doi:10.1177/0145482x1510900204 · Cited by 34 · cited 1×
“sighted users [40] and visually impaired users [35]), and different description styles (e.”
BackgroundAn Introduction to Audio Description: A practical guide
Fryer, L. (2016) · UCL Discovery (University College London) · Cited by 34
Video Captions for Online Courses: Do YouTube’s Auto-generated Captions Meet Deaf Students’ Needs?
Parton, B. S. (2016) · Journal of Open Flexible and Distance Learning · doi:10.61468/jofdl.v20i1.255 · Cited by 30 · cited 1×
“Thanks to deep learning and automatic speech recognition, platforms like YouTube automatically generate closed captions for all videos in spoken English [39].”
BackgroundGains and losses of watching audio described films for sighted viewers
Perego, E. (2016) · Target International Journal of Translation Studies · doi:10.1075/target.28.3.04per · Cited by 27 · cited 2×
“sighted users [40] and visually impaired users [35]), and different description styles (e.”
BackgroundWang, Y., Wang, W., Liang, W. G., & Yu, L.-F. (2019) · ACM Transactions on Graphics · doi:10.1145/3355089.3356487 · Cited by 25
Self-citationScene-Aware Background Music Synthesis
Wang, Y., Liang, W. G., Li, W., Li, D., & Yu, L.-F. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413894 · Cited by 19 · cited 1×
“Researchers investigate different techniques along this direction, such as detecting audio-visual events [4], improving human speech consistency [34, 57], and generating sceneaware sound [25, 58].”
BackgroundSelf-citationEllis, K. M. (2015) · Murdoch University Research Portal · Cited by 9
Audio Description-Seeing Theater with Your Ears
Miers, J. (1995) · Information technology and disabilities · Cited by 2
Lipsynching : popular song recordings and the disembodied voice
Snell, M. (2015) · Cited by 1
Research works printed but not linked: 7
- Yusuf Aytar, Carl Vondrick, and Antonio Torralba. 2016. Soundnet: Learning sound representations from unlabeled video. In Advances in neural information processing systems. 892–900.
- Pat Fletcher. 2002. Seeing with sound: A journey into sight. Retrieved September 21(2002), 2015.
- 10.4324/9781315707228 · doi:10.4324/9781315707228
- R Kingett. 2014. The Accessible Netflix Project Advocates Taking Steps to Ensure Netflix Accessibility for Everyone. The Accessible Netflix Project 26 (2014).
- Aline Remael and Gert Vercauteren. 2011. Basisprincipes voor audiobeschrijving voor televisie en film [Basics of audio description for television and film]. Antwerp: Departement Vertalers and Tolken, Artesis Hogeschool (2011).
- Emilie Schmeidler and Corinne Kirchner. 2001. Adding audio description: Does it make a difference?Journal of Visual Impairment & Blindness 95, 4 (2001), 197–212.
- Joel Snyder. 2014. The visual made verbal: A comprehensive training manual and guide to the history and applications of audio description. American Council of the Blind, Incorporated.
Cited by
Mukhiddinov, M., Abdusalomov, A. B., & Cho, J. (2022) · Sensors · doi:10.3390/s22093307 · Cited by 102
“Assistive systems aid BVI people with daily activities such as distinguishing banknotes [2], crossing a road [3,4], video media accessibility [5,6], image sonification for navigation [7], recognizing people [8], recognizing private visual information [9], selecting clothing [10], and navigating both outdoors and indoo…”
BackgroundA Literature Review of Video-Sharing Platform Research in HCI
Bartolome, A., & Niu, S. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581107 · Cited by 82
“video navigation and recommendation [14, 23, 24, 84, 187, 191, 207], live streaming support [38, 55, 107, 108, 127, 152, 179, 211], closed captions [69], user protection techniques [74, 118, 129] accessibility features [125, 193]”
BackgroundInfluentialAVscript: Accessible Video Editing with Audio-Visual Scripts
Huh, M., Yang, S., Peng, Y.-H., Chen, X. '., Kim, Y.-H., & Pavel, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581494 · Cited by 45
“Beyond helping creators author accessible videos, prior work makes inaccessible videos accessible on demand by generating automatic [83] or interactive [29, 59] visual descriptions.”
AutoAD: Movie Description in Context
Han, T., Bain, M., Nagrani, A., Varol, G., Xie, W., & Zisserman, A. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01815 · Cited by 43
“This is a relatively unexplored task in the vision community with previous work targeting ActivityNet videos [91], a very different domain to long-term feature films with storylines, and the LSMDC challenge [71], where the descriptions and character names are treated separately.”
Making Short-Form Videos Accessible with Hierarchical Video Summaries
Van Daele, T., Iyer, A., Zhang, Y., Derry, J. C., Huh, M., & Pavel, A. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642839 · Cited by 38
Self-citation
Metadata
- Authors
- Wang, Y., Liang, W., Huang, H., Zhang, Y., Li, D., & Yu, L.-F.
- Affiliations
- Beijing Institute of Technology (CN)
- George Mason University (US)
- University of Massachusetts Boston (US)
- Adobe Systems (United States) (US)
- Venue
- Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2021-05-06
- Pages
- 1-12
- OpenAlex
- W3160237369
- Cited by
- 87
- References
- 54
- Citations by year
- 2026: 7 · 2025: 26 · 2024: 26 · 2023: 17 · 2022: 6 · 2021: 5
- FWCI
- 121.88
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Funders
- NSF CAREER Award (1942531)
- Topics
- Social Sciences › Arts and Humanities › Language and Linguistics › Subtitles and Audiovisual Media
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Analysis and Summarization
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Keywords
- video accessibility, visual impairment, audiovisual content analysis, blind and visually impaired users
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.