Toward Automatic Audio Description Generation for Accessible Videos
The DOI matches its Crossref record; 53 of the 54 references it cites were found in a registry or an index.
Abstract
Video accessibility is essential for people with visual impairments. Audio descriptions describe what is happening on-screen, e.g., physical actions, facial expressions, and scene changes. Generating high-quality audio descriptions requires a lot of manual description generation [50]. To address this accessibility obstacle, we built a system that analyzes the audiovisual contents of a video and generates the audio descriptions. The system consisted of three modules: AD insertion time prediction, AD generation, and AD optimization. We evaluated the quality of our system on five types of videos by conducting qualitative studies with 20 sighted users and 12 users who were blind or visually impaired. Our findings revealed how audio description preferences varied with user types and video types. Based on our study’s analysis, we provided recommendations for the development of future audio description generation technologies.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“Starting from word embeddings [7, 32, 59], recent popular neural network methods evolved to use sentence-level embedding for distance metrics [5, 30, 36].”
MethodLearning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593 · cited 1×
“Research on dense video captioning, such as C3D [53] and I3D [11], showed that multiple features can improve video captioning models [37, 38, 54].”
BackgroundQuo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
A Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,479
HuggingFace's Transformers: State-of-the-art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.03771 · Cited by 3,347
A neural probabilistic language model
BengioYoshua, DucharmeRéjean, VincentPascal, & JanvinChristian (2003) · Journal of Machine Learning Research · doi:10.5555/944919.944966 · Cited by 2,644
Sequence to Sequence -- Video to Text
Venugopalan, S., Rohrbach, M., Donahue, J., Mooney, R., Darrell, T., & Saenko, K. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.515 · Cited by 1,352 · cited 1×
“Research on dense video captioning, such as C3D [53] and I3D [11], showed that multiple features can improve video captioning models [37, 38, 54].”
BackgroundDense-Captioning Events in Videos
Krishna, R., Hata, K., Ren, F., Fei-Fei, L., & Niebles, J. C. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.83 · Cited by 1,251
Attention on Attention for Image Captioning
Huang, L., Wang, W., Chen, J., & Wei, X.-Y. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00473 · Cited by 1,092 · cited 1×
“Similar to image captioning, it requires not only recognizing salient objects, but also understanding actions and interactions [26].”
Background
Research works printed but not linked: 7
- Yusuf Aytar, Carl Vondrick, and Antonio Torralba. 2016. Soundnet: Learning sound representations from unlabeled video. In Advances in neural information processing systems. 892–900.
- Pat Fletcher. 2002. Seeing with sound: A journey into sight. Retrieved September 21(2002), 2015.
- 10.4324/9781315707228 · doi:10.4324/9781315707228
- R Kingett. 2014. The Accessible Netflix Project Advocates Taking Steps to Ensure Netflix Accessibility for Everyone. The Accessible Netflix Project 26 (2014).
- Aline Remael and Gert Vercauteren. 2011. Basisprincipes voor audiobeschrijving voor televisie en film [Basics of audio description for television and film]. Antwerp: Departement Vertalers and Tolken, Artesis Hogeschool (2011).
- Emilie Schmeidler and Corinne Kirchner. 2001. Adding audio description: Does it make a difference?Journal of Visual Impairment & Blindness 95, 4 (2001), 197–212.
- Joel Snyder. 2014. The visual made verbal: A comprehensive training manual and guide to the history and applications of audio description. American Council of the Blind, Incorporated.
Cited by
Mukhiddinov, M., Abdusalomov, A. B., & Cho, J. (2022) · Sensors · doi:10.3390/s22093307 · Cited by 102
“Assistive systems aid BVI people with daily activities such as distinguishing banknotes [2], crossing a road [3,4], video media accessibility [5,6], image sonification for navigation [7], recognizing people [8], recognizing private visual information [9], selecting clothing [10], and navigating both outdoors and indoo…”
BackgroundA Literature Review of Video-Sharing Platform Research in HCI
Bartolome, A., & Niu, S. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581107 · Cited by 82
“video navigation and recommendation [14, 23, 24, 84, 187, 191, 207], live streaming support [38, 55, 107, 108, 127, 152, 179, 211], closed captions [69], user protection techniques [74, 118, 129] accessibility features [125, 193]”
BackgroundInfluentialAVscript: Accessible Video Editing with Audio-Visual Scripts
Huh, M., Yang, S., Peng, Y.-H., Chen, X. '., Kim, Y.-H., & Pavel, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581494 · Cited by 45
“Beyond helping creators author accessible videos, prior work makes inaccessible videos accessible on demand by generating automatic [83] or interactive [29, 59] visual descriptions.”
AutoAD: Movie Description in Context
Han, T., Bain, M., Nagrani, A., Varol, G., Xie, W., & Zisserman, A. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01815 · Cited by 43
“This is a relatively unexplored task in the vision community with previous work targeting ActivityNet videos [91], a very different domain to long-term feature films with storylines, and the LSMDC challenge [71], where the descriptions and character names are treated separately.”
Making Short-Form Videos Accessible with Hierarchical Video Summaries
Daele, T. V., Iyer, A., Zhang, Y., Derry, J. C., Huh, M., & Pavel, A. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642839 · Cited by 74
Self-citation
Metadata
- Authors
- Wang, Y., Liang, W., Huang, H., Zhang, Y., Li, D., & Yu, L.-F.
- Affiliations
- Beijing Institute of Technology (CN)
- George Mason University (US)
- University of Massachusetts Boston (US)
- Adobe Systems (United States) (US)
- Venue
- Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2021-05-06
- Pages
- 1-12
- OpenAlex
- W3160237369
- Cited by
- 87
- References
- 54
- Citations by year
- 2026: 7 · 2025: 26 · 2024: 26 · 2023: 17 · 2022: 6 · 2021: 5
- FWCI
- 121.88
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Funders
- NSF CAREER Award (1942531)
- Topics
- Social Sciences › Arts and Humanities › Language and Linguistics › Subtitles and Audiovisual Media
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Analysis and Summarization
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Keywords
- video accessibility, visual impairment, audiovisual content analysis, blind and visually impaired users
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Toward Automatic Audio Description Generation for Accessible Videos. https://app.cochat.ai/papers/10.1145/3411764.3445347
- The paper
- Wang, Y., Liang, W., Huang, H., Zhang, Y., Li, D., & Yu, L.-F. (2021). Toward Automatic Audio Description Generation for Accessible Videos. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3411764.3445347
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.