CoChat

Human-in-the-Loop Machine Learning to Increase Video Accessibility for Visually Impaired and Blind Users

Yuksel, Beste F., Fazli, Pooyan, Mathur, Umang, Bisht, Vaishali, Kim, Soo Jung, Lee, Joshua Junhee, Jin, Seung Jung, Siu, Yue-Ting, Miele, Joshua A., Yoon, Ilmi · 2020 · Proceedings of the 2020 ACM Designing Interactive Systems Conference

The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.

Abstract

Video accessibility is crucial for blind and visually impaired individuals for education, employment, and entertainment purposes. However, professional video descriptions are costly and time-consuming. Volunteer-created video descriptions could be a promising alternative, however, they can vary in quality and can be intimidating for novice describers. We developed a Human-in-the-Loop Machine Learning (HILML) approach to video description by automating video text generation and scene segmentation and allowing humans to edit the output. The HILML approach facilitates human-machine collaboration to produce high quality video descriptions while keeping a low barrier to entry for volunteer describers. Our HILML system was significantly faster and easier to use for first-time video describers compared to a human-only control condition with no machine learning assistance. The quality of the video descriptions and understanding of the topic created by the HILML system compared to the human-only condition were rated as being significantly higher by blind and visually impaired users.

Overview

Integrity signals

Registry

Crossref · 10.1145/3357236.3395433

Cites

45 of 45 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Citations

Cites

Show all 45 references

Research works printed but not linked: 13

  1. Nayyer Aafaq , Syed Zulqarnain Gilani , Wei Liu, Ajmal Mian, and Mubarak Shah. 2018 . Video Description : A Survey of Methods, Datasets and Evaluation Metrics. CoRR abs/1806.00186 (2018). Nayyer Aafaq, Syed Zulqarnain Gilani, Wei Liu, Ajmal Mian, and Mubarak Shah. 2018. Video Description: A Survey of Methods, Datasets and Evaluation Metrics. CoRR abs/1806.00186 (2018). · arXiv
  2. Andrei Barbu , Alexander Bridge , Zachary Burchill , Dan Coroian , Sven Dickinson , Sanja Fidler , Aaron Michaux , Sam Mussman , Siddharth Narayanaswamy , Dhaval Salvi , Lara Schmidt , Jiangnan Shangguan , Jeffrey Mark Siskind , Jarrell Waggoner, Song Wang, Jinlian Wei, Yifan Yin, and Zhiqi Zhang. 2012 . Video In Sentences Out . (2012). Andrei Barbu, Alexander Bridge, Zachary Burchill, Dan Coroian, Sven Dickinson, Sanja Fidler, Aaron Michaux, Sam Mussman, Siddharth Narayanaswamy, Dhaval Salvi, Lara Schmidt, Jiangnan Shangguan, Jeffrey Mark Siskind, Jarrell Waggoner, Song Wang, Jinlian Wei, Yifan Yin, and Zhiqi Zhang. 2012. Video In Sentences Out. (2012).
  3. Matthew Brand. 1997. The" Inverse hollywood problem": from video to scripts and storyboards via causal analysis. In AAAI/IAAI. Citeseer 132--137. Matthew Brand. 1997. The" Inverse hollywood problem": from video to scripts and storyboards via causal analysis. In AAAI/IAAI. Citeseer 132--137.
  4. Carmen Branje , Susan Marshall , Ashley Tyndall , and Deborah Fels . 2006 . LiveDescribe. AMCIS 2006 Proceedings (2006), 368. Carmen Branje, Susan Marshall, Ashley Tyndall, and Deborah Fels. 2006. LiveDescribe. AMCIS 2006 Proceedings (2006), 368.
  5. Abhishek Das , Satwik Kottur , Khushi Gupta , Avi Singh , Deshraj Yadav , Jose M. F. Moura , Devi Parikh , and Dhruv Batra . 2017 . Visual Dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR. Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, Jose M. F. Moura, Devi Parikh, and Dhruv Batra. 2017. Visual Dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR.
  6. Pradipto Das , Chenliang Xu , Richard F. Doell , and Jason J. Corso . 2013. A Thousand Frames in Just a Few Words: Lingual Description of Videos through Latent Topics and Sparse Object Stitching . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR. Pradipto Das, Chenliang Xu, Richard F. Doell, and Jason J. Corso. 2013. A Thousand Frames in Just a Few Words: Lingual Description of Videos through Latent Topics and Sparse Object Stitching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR.
  7. Patrick Hanckmann , Klamer Schutte , and Gertjan J . Burghouts . 2012 . Automated Textual Descriptions for a Wide Range of Video Events with 48 Human Actions. Lecture Notes in Computer Science ( 2012), 372--380. Patrick Hanckmann, Klamer Schutte, and Gertjan J. Burghouts. 2012. Automated Textual Descriptions for a Wide Range of Video Events with 48 Human Actions. Lecture Notes in Computer Science (2012), 372--380.
  8. SG Hart and others. 1988. Development of NASA-TLX: Results of empirical and theoretical research." inP . A. Hancock and N. Meshkati (eds.), Human Mental Workload . ( 1988 ). SG Hart and others. 1988. Development of NASA-TLX: Results of empirical and theoretical research." inP. A. Hancock and N. Meshkati (eds.), Human Mental Workload. (1988).
  9. 10.18653/v1 · doi:10.18653/v1
  10. Slava Kalyuga . 2003. The expertise reversal effect. Educational psychologist 38, 1 ( 2003 ), 23--31. Slava Kalyuga. 2003. The expertise reversal effect. Educational psychologist 38, 1 (2003), 23--31.
  11. Kishore Papineni , Salim Roukos , Todd Ward , and Wei-Jing Zhu . 2002 . BLEU: a method for automatic evaluation of machine translation . In Proceedings of the 40th annual meeting on association for computational linguistics. Association for Computational Linguistics, 311--318 . Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics. Association for Computational Linguistics, 311--318.
  12. Karen Simonyan and Andrew Zisserman . 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. CoRR abs/1409.1556 ( 2014 ). Karen Simonyan and Andrew Zisserman. 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. CoRR abs/1409.1556 (2014). · arXiv
  13. Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Lukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 5998--6008. Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Lukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 5998--6008.

Cited by

See all 70

Metadata

Authors
Yuksel, B. F., Fazli, P., Mathur, U., Bisht, V., Kim, S. J., Lee, J. J., Jin, S. J., Siu, Y.-T., Miele, J. A., & Yoon, I.
Affiliations
  • University of San Francisco (US)
  • San Francisco State University (US)
  • University of San Francisco, San Francisco, CA, USA
  • University of Washington (US)
  • Smith-Kettlewell Eye Research Institute (US)
Venue
Proceedings of the 2020 ACM Designing Interactive Systems Conference
Publisher
ACM
Published
2020-07-03
Pages
47-60
OpenAlex
W3038896047
Cited by
60
References
45
Citations by year
2026: 3 · 2025: 17 · 2024: 12 · 2023: 10 · 2022: 8 · 2021: 10
FWCI
49.82
Type
conference paper
Language
en
Open access
closed
Locations
Designing Interactive Systems Conference (publishedVersion)
Funders
  • Craig Newmark Philanthropies
  • Center for Computing for Life Sciences, San Francisco State University
Topics
  • Social Sciences › Arts and Humanities › Language and Linguistics › Subtitles and Audiovisual Media
  • Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
video accessibility, human-in-the-loop learning, audio description, blind and visually impaired users, text generation, human-machine collaboration, accessibility for people with disabilities

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Human-in-the-Loop Machine Learning to Increase Video Accessibility for Visually Impaired and Blind Users. https://app.cochat.ai/papers/10.1145/3357236.3395433
The paper
Yuksel, B. F., Fazli, P., Mathur, U., Bisht, V., Kim, S. J., Lee, J. J., Jin, S. J., Siu, Y.-T., Miele, J. A., & Yoon, I. (2020). Human-in-the-Loop Machine Learning to Increase Video Accessibility for Visually Impaired and Blind Users. Proceedings of the 2020 ACM Designing Interactive Systems Conference. https://doi.org/10.1145/3357236.3395433

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.