Human-in-the-Loop Machine Learning to Increase Video Accessibility for Visually Impaired and Blind Users
The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
Abstract
Video accessibility is crucial for blind and visually impaired individuals for education, employment, and entertainment purposes. However, professional video descriptions are costly and time-consuming. Volunteer-created video descriptions could be a promising alternative, however, they can vary in quality and can be intimidating for novice describers. We developed a Human-in-the-Loop Machine Learning (HILML) approach to video description by automating video text generation and scene segmentation and allowing humans to edit the output. The HILML approach facilitates human-machine collaboration to produce high quality video descriptions while keeping a low barrier to entry for volunteer describers. Our HILML system was significantly faster and easier to use for first-time video describers compared to a human-only control condition with no machine learning assistance. The quality of the video descriptions and understanding of the topic created by the HILML system compared to the human-only condition were rated as being significantly higher by blind and visually impaired users.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“In particular, Convolutional Neural Networks (CNNs) [6] are the state of the art for modeling visual data recognition [6, 65, 67] and Long Short-Term Memory (LSTMs) [34] are now dominating the area of sequence modeling such as NLP [16, 31, 66].”
BackgroundGoing deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893 · cited 1×
“…modeling visual data recognition [6, 65, 67] and Long Short-Term Memory (LSTMs) [34] are now dominating the area of sequence modeling such as NLP [16, 31, 66] Our approach assists sighted users to create video descriptions by combining automation using deep learning with human input and editing.”
MethodOn the Properties of Neural Machine Translation: Encoder–Decoder Approaches
Cho, K., van Merrienboer, B., Bahdanau, D., & Bengio, Y. (2014) · Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation · doi:10.3115/v1/w14-4012 · Cited by 7,099 · cited 1×
“…modeling visual data recognition [6, 65, 67] and Long Short-Term Memory (LSTMs) [34] are now dominating the area of sequence modeling such as NLP [16, 31, 66] Our approach assists sighted users to create video descriptions by combining automation using deep learning with human input and editing.”
MethodVQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
Bourne, R. R. A., Flaxman, S. R., Braithwaite, T., Cicinelli, M. V., Das, A., Jonas, J. B., Keeffe, J., Kempen, J. H., Leasher, J., Limburg, H., et al. (2017) · The Lancet Global Health · doi:10.1016/s2214-109x(17)30293-0 · Cited by 2,213 · cited 2×
“Request permissions from Permissions@acm.org. million of them are blind [11].”
BackgroundInfluentialHierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1082 · Cited by 1,266
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · arXiv · doi:10.48550/arxiv.1805.04833 · Cited by 1,242
Expertise Reversal Effect and Its Implications for Learner-Tailored Instruction
Kalyuga, S. (2007) · Educational Psychology Review · doi:10.1007/s10648-007-9054-3 · Cited by 1,056 · cited 1×
“There is a phenomenon in learning called the expertise reversal effect whereby instructional techniques that are benefcial to beginners can have the reverse effects on more experienced learners [41, 42].”
Background
Research works printed but not linked: 13
- Nayyer Aafaq , Syed Zulqarnain Gilani , Wei Liu, Ajmal Mian, and Mubarak Shah. 2018 . Video Description : A Survey of Methods, Datasets and Evaluation Metrics. CoRR abs/1806.00186 (2018). Nayyer Aafaq, Syed Zulqarnain Gilani, Wei Liu, Ajmal Mian, and Mubarak Shah. 2018. Video Description: A Survey of Methods, Datasets and Evaluation Metrics. CoRR abs/1806.00186 (2018). · arXiv
- Andrei Barbu , Alexander Bridge , Zachary Burchill , Dan Coroian , Sven Dickinson , Sanja Fidler , Aaron Michaux , Sam Mussman , Siddharth Narayanaswamy , Dhaval Salvi , Lara Schmidt , Jiangnan Shangguan , Jeffrey Mark Siskind , Jarrell Waggoner, Song Wang, Jinlian Wei, Yifan Yin, and Zhiqi Zhang. 2012 . Video In Sentences Out . (2012). Andrei Barbu, Alexander Bridge, Zachary Burchill, Dan Coroian, Sven Dickinson, Sanja Fidler, Aaron Michaux, Sam Mussman, Siddharth Narayanaswamy, Dhaval Salvi, Lara Schmidt, Jiangnan Shangguan, Jeffrey Mark Siskind, Jarrell Waggoner, Song Wang, Jinlian Wei, Yifan Yin, and Zhiqi Zhang. 2012. Video In Sentences Out. (2012).
- Matthew Brand. 1997. The" Inverse hollywood problem": from video to scripts and storyboards via causal analysis. In AAAI/IAAI. Citeseer 132--137. Matthew Brand. 1997. The" Inverse hollywood problem": from video to scripts and storyboards via causal analysis. In AAAI/IAAI. Citeseer 132--137.
- Carmen Branje , Susan Marshall , Ashley Tyndall , and Deborah Fels . 2006 . LiveDescribe. AMCIS 2006 Proceedings (2006), 368. Carmen Branje, Susan Marshall, Ashley Tyndall, and Deborah Fels. 2006. LiveDescribe. AMCIS 2006 Proceedings (2006), 368.
- Abhishek Das , Satwik Kottur , Khushi Gupta , Avi Singh , Deshraj Yadav , Jose M. F. Moura , Devi Parikh , and Dhruv Batra . 2017 . Visual Dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR. Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, Jose M. F. Moura, Devi Parikh, and Dhruv Batra. 2017. Visual Dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR.
- Pradipto Das , Chenliang Xu , Richard F. Doell , and Jason J. Corso . 2013. A Thousand Frames in Just a Few Words: Lingual Description of Videos through Latent Topics and Sparse Object Stitching . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR. Pradipto Das, Chenliang Xu, Richard F. Doell, and Jason J. Corso. 2013. A Thousand Frames in Just a Few Words: Lingual Description of Videos through Latent Topics and Sparse Object Stitching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR.
- Patrick Hanckmann , Klamer Schutte , and Gertjan J . Burghouts . 2012 . Automated Textual Descriptions for a Wide Range of Video Events with 48 Human Actions. Lecture Notes in Computer Science ( 2012), 372--380. Patrick Hanckmann, Klamer Schutte, and Gertjan J. Burghouts. 2012. Automated Textual Descriptions for a Wide Range of Video Events with 48 Human Actions. Lecture Notes in Computer Science (2012), 372--380.
- SG Hart and others. 1988. Development of NASA-TLX: Results of empirical and theoretical research." inP . A. Hancock and N. Meshkati (eds.), Human Mental Workload . ( 1988 ). SG Hart and others. 1988. Development of NASA-TLX: Results of empirical and theoretical research." inP. A. Hancock and N. Meshkati (eds.), Human Mental Workload. (1988).
- 10.18653/v1 · doi:10.18653/v1
- Slava Kalyuga . 2003. The expertise reversal effect. Educational psychologist 38, 1 ( 2003 ), 23--31. Slava Kalyuga. 2003. The expertise reversal effect. Educational psychologist 38, 1 (2003), 23--31.
- Kishore Papineni , Salim Roukos , Todd Ward , and Wei-Jing Zhu . 2002 . BLEU: a method for automatic evaluation of machine translation . In Proceedings of the 40th annual meeting on association for computational linguistics. Association for Computational Linguistics, 311--318 . Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics. Association for Computational Linguistics, 311--318.
- Karen Simonyan and Andrew Zisserman . 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. CoRR abs/1409.1556 ( 2014 ). Karen Simonyan and Andrew Zisserman. 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. CoRR abs/1409.1556 (2014). · arXiv
- Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Lukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 5998--6008. Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Lukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 5998--6008.
Cited by
Large-scale Text-to-Image Generation Models for Visual Artists’ Creative Works
Ko, H.-K., Park, G., Jeon, H., Jo, J., Kim, J., & Seo, J. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584078 · Cited by 189
“%) such as podcast listening [44], video captioning [107], and conversation [35] which can be encountered in our everyday life.”
BackgroundSmart Glass System Using Deep Learning for the Blind and Visually Impaired
Mukhiddinov, M., & Cho, J. (2021) · Electronics · doi:10.3390/electronics10222756 · Cited by 161
“Assistive systems exist to aid BVI people with navigation and daily activities, such as distinguishing banknotes [2,3], crossing a road [4–6], video media accessibility [7,8], image sonification for navigation [9,10], recognizing people [11–13], recognizing private visual information [14], selecting clothing [15], and…”
BackgroundMukhiddinov, M., & Kim, S.-Y. (2021) · Processes · doi:10.3390/pr9101726 · Cited by 110
“Researchers have also developed a touchscreen tablet with audio feedback [37], an accessibility system for web documents [38], a tactile graphics finder [35], video description [39,40], bas-reliefs [41], and touchscreen interactions for environmental perception [42] using AI technologies to make visual information eas…”
MethodSelf-citationWhat Makes Videos Accessible to Blind and Visually Impaired People?
Liu, X., Carrington, P., Chen, X. '., & Pavel, A. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445233 · Cited by 87
“Prioritizing Content for Description: Our work helping people fnd videos that already have high-quality and built-in descriptions during video search can advance existing work on helping authors add high-quality audio descriptions to individual inaccessible videos [18, 19, 29, 40, 42, 66].”
BackgroundMethodMaking Short-Form Videos Accessible with Hierarchical Video Summaries
Van Daele, T., Iyer, A., Zhang, Y., Derry, J. C., Huh, M., & Pavel, A. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642839 · Cited by 38
“Prior work has explored making videos accessible with manual [36], semi-automated [15, 46, 60, 76] and automated [17, 73] audio descriptions — or, narrations of visual content in the video.”
Metadata
- Authors
- Yuksel, B. F., Fazli, P., Mathur, U., Bisht, V., Kim, S. J., Lee, J. J., Jin, S. J., Siu, Y.-T., Miele, J. A., & Yoon, I.
- Affiliations
- University of San Francisco (US)
- San Francisco State University (US)
- University of San Francisco, San Francisco, CA, USA
- University of Washington (US)
- Smith-Kettlewell Eye Research Institute (US)
- Venue
- Proceedings of the 2020 ACM Designing Interactive Systems Conference
- Publisher
- ACM
- Published
- 2020-07-03
- Pages
- 47-60
- OpenAlex
- W3038896047
- Cited by
- 60
- References
- 45
- Citations by year
- 2026: 3 · 2025: 17 · 2024: 12 · 2023: 10 · 2022: 8 · 2021: 10
- FWCI
- 49.82
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Designing Interactive Systems Conference (publishedVersion)
- Funders
- Craig Newmark Philanthropies
- Center for Computing for Life Sciences, San Francisco State University
- Topics
- Social Sciences › Arts and Humanities › Language and Linguistics › Subtitles and Audiovisual Media
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- video accessibility, human-in-the-loop learning, audio description, blind and visually impaired users, text generation, human-machine collaboration, accessibility for people with disabilities
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Human-in-the-Loop Machine Learning to Increase Video Accessibility for Visually Impaired and Blind Users. https://app.cochat.ai/papers/10.1145/3357236.3395433
- The paper
- Yuksel, B. F., Fazli, P., Mathur, U., Bisht, V., Kim, S. J., Lee, J. J., Jin, S. J., Siu, Y.-T., Miele, J. A., & Yoon, I. (2020). Human-in-the-Loop Machine Learning to Increase Video Accessibility for Visually Impaired and Blind Users. Proceedings of the 2020 ACM Designing Interactive Systems Conference. https://doi.org/10.1145/3357236.3395433
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.