Human-in-the-Loop Machine Learning to Increase Video Accessibility for Visually Impaired and Blind Users
The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
Abstract
Video accessibility is crucial for blind and visually impaired individuals for education, employment, and entertainment purposes. However, professional video descriptions are costly and time-consuming. Volunteer-created video descriptions could be a promising alternative, however, they can vary in quality and can be intimidating for novice describers. We developed a Human-in-the-Loop Machine Learning (HILML) approach to video description by automating video text generation and scene segmentation and allowing humans to edit the output. The HILML approach facilitates human-machine collaboration to produce high quality video descriptions while keeping a low barrier to entry for volunteer describers. Our HILML system was significantly faster and easier to use for first-time video describers compared to a human-only control condition with no machine learning assistance. The quality of the video descriptions and understanding of the topic created by the HILML system compared to the human-only condition were rated as being significantly higher by blind and visually impaired users.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“In particular, Convolutional Neural Networks (CNNs) [6] are the state of the art for modeling visual data recognition [6, 65, 67] and Long Short-Term Memory (LSTMs) [34] are now dominating the area of sequence modeling such as NLP [16, 31, 66].”
BackgroundGoing deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893 · cited 1×
“…modeling visual data recognition [6, 65, 67] and Long Short-Term Memory (LSTMs) [34] are now dominating the area of sequence modeling such as NLP [16, 31, 66] Our approach assists sighted users to create video descriptions by combining automation using deep learning with human input and editing.”
MethodOn the Properties of Neural Machine Translation: Encoder–Decoder Approaches
Cho, K., van Merrienboer, B., Bahdanau, D., & Bengio, Y. (2014) · Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation · doi:10.3115/v1/w14-4012 · Cited by 7,099 · cited 1×
“…modeling visual data recognition [6, 65, 67] and Long Short-Term Memory (LSTMs) [34] are now dominating the area of sequence modeling such as NLP [16, 31, 66] Our approach assists sighted users to create video descriptions by combining automation using deep learning with human input and editing.”
MethodVQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
Bourne, R. R. A., Flaxman, S. R., Braithwaite, T., Cicinelli, M. V., Das, A., Jonas, J. B., Keeffe, J., Kempen, J. H., Leasher, J., Limburg, H., et al. (2017) · The Lancet Global Health · doi:10.1016/s2214-109x(17)30293-0 · Cited by 2,213 · cited 2×
“Request permissions from Permissions@acm.org. million of them are blind [11].”
BackgroundInfluentialHierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1082 · Cited by 1,266
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · arXiv · doi:10.48550/arxiv.1805.04833 · Cited by 1,242
Expertise Reversal Effect and Its Implications for Learner-Tailored Instruction
Kalyuga, S. (2007) · Educational Psychology Review · doi:10.1007/s10648-007-9054-3 · Cited by 1,056 · cited 1×
“There is a phenomenon in learning called the expertise reversal effect whereby instructional techniques that are benefcial to beginners can have the reverse effects on more experienced learners [41, 42].”
Background
Research works printed but not linked: 13
- Nayyer Aafaq , Syed Zulqarnain Gilani , Wei Liu, Ajmal Mian, and Mubarak Shah. 2018 . Video Description : A Survey of Methods, Datasets and Evaluation Metrics. CoRR abs/1806.00186 (2018). Nayyer Aafaq, Syed Zulqarnain Gilani, Wei Liu, Ajmal Mian, and Mubarak Shah. 2018. Video Description: A Survey of Methods, Datasets and Evaluation Metrics. CoRR abs/1806.00186 (2018). · arXiv
- Andrei Barbu , Alexander Bridge , Zachary Burchill , Dan Coroian , Sven Dickinson , Sanja Fidler , Aaron Michaux , Sam Mussman , Siddharth Narayanaswamy , Dhaval Salvi , Lara Schmidt , Jiangnan Shangguan , Jeffrey Mark Siskind , Jarrell Waggoner, Song Wang, Jinlian Wei, Yifan Yin, and Zhiqi Zhang. 2012 . Video In Sentences Out . (2012). Andrei Barbu, Alexander Bridge, Zachary Burchill, Dan Coroian, Sven Dickinson, Sanja Fidler, Aaron Michaux, Sam Mussman, Siddharth Narayanaswamy, Dhaval Salvi, Lara Schmidt, Jiangnan Shangguan, Jeffrey Mark Siskind, Jarrell Waggoner, Song Wang, Jinlian Wei, Yifan Yin, and Zhiqi Zhang. 2012. Video In Sentences Out. (2012).
- Matthew Brand. 1997. The" Inverse hollywood problem": from video to scripts and storyboards via causal analysis. In AAAI/IAAI. Citeseer 132--137. Matthew Brand. 1997. The" Inverse hollywood problem": from video to scripts and storyboards via causal analysis. In AAAI/IAAI. Citeseer 132--137.
- Carmen Branje , Susan Marshall , Ashley Tyndall , and Deborah Fels . 2006 . LiveDescribe. AMCIS 2006 Proceedings (2006), 368. Carmen Branje, Susan Marshall, Ashley Tyndall, and Deborah Fels. 2006. LiveDescribe. AMCIS 2006 Proceedings (2006), 368.
- Abhishek Das , Satwik Kottur , Khushi Gupta , Avi Singh , Deshraj Yadav , Jose M. F. Moura , Devi Parikh , and Dhruv Batra . 2017 . Visual Dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR. Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, Jose M. F. Moura, Devi Parikh, and Dhruv Batra. 2017. Visual Dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR.
- Pradipto Das , Chenliang Xu , Richard F. Doell , and Jason J. Corso . 2013. A Thousand Frames in Just a Few Words: Lingual Description of Videos through Latent Topics and Sparse Object Stitching . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR. Pradipto Das, Chenliang Xu, Richard F. Doell, and Jason J. Corso. 2013. A Thousand Frames in Just a Few Words: Lingual Description of Videos through Latent Topics and Sparse Object Stitching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR.
- Patrick Hanckmann , Klamer Schutte , and Gertjan J . Burghouts . 2012 . Automated Textual Descriptions for a Wide Range of Video Events with 48 Human Actions. Lecture Notes in Computer Science ( 2012), 372--380. Patrick Hanckmann, Klamer Schutte, and Gertjan J. Burghouts. 2012. Automated Textual Descriptions for a Wide Range of Video Events with 48 Human Actions. Lecture Notes in Computer Science (2012), 372--380.
- SG Hart and others. 1988. Development of NASA-TLX: Results of empirical and theoretical research." inP . A. Hancock and N. Meshkati (eds.), Human Mental Workload . ( 1988 ). SG Hart and others. 1988. Development of NASA-TLX: Results of empirical and theoretical research." inP. A. Hancock and N. Meshkati (eds.), Human Mental Workload. (1988).
- 10.18653/v1 · DOI
- Slava Kalyuga . 2003. The expertise reversal effect. Educational psychologist 38, 1 ( 2003 ), 23--31. Slava Kalyuga. 2003. The expertise reversal effect. Educational psychologist 38, 1 (2003), 23--31.
- Kishore Papineni , Salim Roukos , Todd Ward , and Wei-Jing Zhu . 2002 . BLEU: a method for automatic evaluation of machine translation . In Proceedings of the 40th annual meeting on association for computational linguistics. Association for Computational Linguistics, 311--318 . Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics. Association for Computational Linguistics, 311--318.
- Karen Simonyan and Andrew Zisserman . 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. CoRR abs/1409.1556 ( 2014 ). Karen Simonyan and Andrew Zisserman. 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. CoRR abs/1409.1556 (2014). · arXiv
- Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Lukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 5998--6008. Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Lukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 5998--6008.
Cited by
Large-scale Text-to-Image Generation Models for Visual Artists’ Creative Works
Ko, H.-K., Park, G., Jeon, H., Jo, J., Kim, J., & Seo, J. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584078 · Cited by 189
“%) such as podcast listening [44], video captioning [107], and conversation [35] which can be encountered in our everyday life.”
BackgroundSmart Glass System Using Deep Learning for the Blind and Visually Impaired
Mukhiddinov, M., & Cho, J. (2021) · Electronics · doi:10.3390/electronics10222756 · Cited by 163
“Assistive systems exist to aid BVI people with navigation and daily activities, such as distinguishing banknotes [2,3], crossing a road [4–6], video media accessibility [7,8], image sonification for navigation [9,10], recognizing people [11–13], recognizing private visual information [14], selecting clothing [15], and…”
BackgroundMukhiddinov, M., & Kim, S.-Y. (2021) · Processes · doi:10.3390/pr9101726 · Cited by 110
“Researchers have also developed a touchscreen tablet with audio feedback [37], an accessibility system for web documents [38], a tactile graphics finder [35], video description [39,40], bas-reliefs [41], and touchscreen interactions for environmental perception [42] using AI technologies to make visual information eas…”
MethodSelf-citationWhat Makes Videos Accessible to Blind and Visually Impaired People?
Liu, X., Carrington, P., Chen, X. '., & Pavel, A. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445233 · Cited by 87
“Prioritizing Content for Description: Our work helping people fnd videos that already have high-quality and built-in descriptions during video search can advance existing work on helping authors add high-quality audio descriptions to individual inaccessible videos [18, 19, 29, 40, 42, 66].”
BackgroundMethodMaking Short-Form Videos Accessible with Hierarchical Video Summaries
Van Daele, T., Iyer, A., Zhang, Y., Derry, J. C., Huh, M., & Pavel, A. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642839 · Cited by 38
“Prior work has explored making videos accessible with manual [36], semi-automated [15, 46, 60, 76] and automated [17, 73] audio descriptions — or, narrations of visual content in the video.”
Li, F. M., Spektor, F., Xia, M., Huh, M., Cederberg, P., Gong, Y., Shinohara, K., & Carrington, P. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517490 · Cited by 56
“” Therefore, we suggest future research on designing accessible makeup videos by creating a database [100] to store the detailed information and leverage human-in-the-loop approaches [98] to obtain specific answers to questions from visually impaired individuals as an add-on tool to the video stream synchronously.”
BackgroundJiang, L., Phutane, M., & Azenkot, S. (2023) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3597638.3608381 · Cited by 46
“Increase AD Quality and Quantity Researchers have studied how AI can make human processes for creating AD more efcient by automating parts of the workfow [12, 16, 17, 32, 49, 66, 86, 87].”
BackgroundMethodNing, Z., Wimer, B. L., Jiang, K., Chen, K., Ban, J., Tian, Y., Zhao, Y., & Li, T. J.-J. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642632 · Cited by 49
“Previous studies have employed AI to expedite AD creation, albeit with varying degrees of success [8, 16, 17, 66, 72].”
BackgroundJiang, L., Jung, C., Phutane, M., Stangl, A. J., & Azenkot, S. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642238 · Cited by 59
“…organizations serving BLV people [3, 26–28, 117 ], and industry practitioners [ 34, 73, Additional insights about which details, and levels of detail, that BLV users want in AD have emerged through the development of tools to streamline the AD creation process [11, 61, 66, 93, 126, 128, 134].”
Automated Video Description for Blind and Low Vision Users
Bodi, A., Fazli, P., Ihorn, S. M., Siu, Y.-T., Scott, A. T., Narins, L. D., Kant, Y., Das, A., & Yoon, I. (2021) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3411763.3451810 · Cited by 40
“In a recent work, we developed a human-inthe-loop machine learning (HILML) approach to facilitate novice describers’ ease of authoring video descriptions [36, 37].”
MethodSelf-citationImage Accessibility for Screen Reader Users: A Systematic Review and a Road Map
Oh, U., Joh, H., & Lee, Y. (2021) · Electronics · doi:10.3390/electronics10080953 · Cited by 37
Audio Description Customization
Natalie, R., Chang, R.-C., Sheshadri, S., Guo, A., & Hara, K. (2024) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3663548.3675617 · Cited by 38
“Yuksel et al. showed that BLV individuals seek precise direction and measurements, and emphasizing such information in cooking videos to be beneficial [107].”
CrossA11y: Identifying Video Accessibility Issues via Cross-modal Grounding
Liu, X., Wang, R., Li, D., Chen, X., & Pavel, A. (2022) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3526113.3545703 · Cited by 33
The Efficacy of Collaborative Authoring of Video Scene Descriptions
Natalie, R., Loh, J., Tan, H. S., Tseng, J., Chan, I. L. Y.-R., Jarjue, E., Kacorri, H., & Hara, K. (2021) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3441852.3471201 · Cited by 36
“We are excited to see a few efforts this past year in this direction ( e.g., [49, 51, 52]), though there is much to be done for achieving high-quality audio descriptions.”
BackgroundDescribe Now: User-Driven Audio Description for Blind and Low Vision Individuals
Cheema, M. S., Seifi, H., & Fazli, P. (2025) · Designing Interactive Systems Conference · doi:10.1145/3715336.3735685 · Cited by 34
“To assist sighted users in creating descriptions, several tools have been built to streamline different parts of the process [7, 17, 25, 28, 32, 37, 38, 46, 59].”
InfluentialSelf-citationSlidecho: Flexible Non-Visual Exploration of Presentation Videos
Peng, Y.-H., Bigham, J. P., & Pavel, A. (2021) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3441852.3471234 · Cited by 40
“tecting key visual content [21, 22], generating descriptions of the visual content [22, 56, 70, 73], and editing descriptions to ft within”
BackgroundSupporting Novices Author Audio Descriptions via Automatic Feedback
Natalie, R., Tseng, J., Kacorri, H., & Hara, K. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581023 · Cited by 26
“Prior work has explored ways to minimize the cost of generating AD and turn-around time by (semi-)automating the AD authoring process [31, 54, 57, 58].”
BackgroundDang, K., Korreshi, H., Iqbal, Y., & Lee, S. (2023) · ACM Symposium on Spatial User Interaction (SUI) · doi:10.1145/3607822.3614540 · Cited by 25
“The strategies to enhance visual content accessibility extend further into video accessibility, focusing on methods like automatic AD generation systems [99, 148], web-based authoring tools exemplified by ViScene [100], and collaborative video description techniques using human-in-the-loop machine learning [158].”
BackgroundJung, J.-Y.-O., Steinberger, T., Kim, J., & Ackerman, M. (2022) · Conference on Designing Interactive Systems · doi:10.1145/3532106.3533522 · Cited by 23
Mahmoud, A. Y., Neagu, D. C., Scrimieri, D., & Abdullatif, A. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107295 · Cited by 26
Nevsky, A., Neate, T., Simperl, E. P. B., & Vatavu, R.-D. (2023) · ACM International Conference on Interactive Media Experiences · doi:10.1145/3573381.3596159 · Cited by 24
Co-Designing Systems to Support Blind and Low Vision Audio Description Writers
Jiang, L., & Ladner, R. E. (2022) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3517428.3550394 · Cited by 21
“Yuksel et al. reported that a human-in-the-loop machine learning approach reduced barriers for creating AD [14].”
BackgroundHow Older Adults Use Online Videos for Learning
Kim, S., Shin, D.-H., Kim, J., Kwon, S., & Kim, J. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580671 · Cited by 19
“Previous work also highlighted the importance of customizing the design of video platforms for underrepresented user groups, such as visually or hearing impaired users [42, 57, 90], to improve the accessibility of video.”
BackgroundSelf-citationA Survey of Accessible Explainable Artificial Intelligence Research
Nwokoye, C. H., Peixoto, M. J. P., Pandey, A., Pardy, L., Sukhai, M. A., & Lewis, P. R. (2025) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2025.3636017 · Cited by 20
“The findings are as follows: [33], [35], [36], [37], [38].”
Exploring Community-Driven Descriptions for Making Livestreams Accessible
Killough, D., & Pavel, A. (2023) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3597638.3608425 · Cited by 17
“…for recorded videos such as films [3, 36, 46], user-generated videos [19, 28, 29, 37, 50], slide presentations [38, 39] and GIFs [11] by providing computational description support [29, 37, 38, 50, 54] and proposing what to describe for specific video types ( e.g. , GIFs [11], films [46]).”
BackgroundMethodDang, K., Burke, G., Korreshi, H., & Lee, S. (2024) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3663548.3675618 · Cited by 16
“Contrary to conventional studies that often depict BLV individuals as passive consumers [12, 14, 15, 21, 34–36, 42, 73, 74, 120], our work highlights their active contributions, building on the foundation laid by blind visionaries in the inception of AD [88].”
VideoA11y: Method and Dataset for Accessible Video Description
Li, C., Padmanabhuni, S., Cheema, M. S., Seifi, H., & Fazli, P. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714096 · Cited by 23
Self-citationMIMOSA: Human-AI Co-Creation of Computational Spatial Audio Effects on Videos
Ning, Z., Zhang, Z., Ban, J., Jiang, K., Gan, R., Tian, Y., & Li, T. J.-J. (2024) · Creativity and Cognition · doi:10.1145/3635636.3656189 · Cited by 18
“This will also allow us to assess the real-world effectiveness of the system for content creators and verify its ecological validity.”
BackgroundDang, K., & Lee, S. (2024) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3663548.3688492 · Cited by 12
“AI-driven improvements in AD generation [29, 44] and web-based authoring tools such as ViScene [30] and collaborative description techniques [45] have enhanced content accessibility.”
DescribePro: Collaborative Audio Description with Human-AI Interaction
Cheema, M. S., Elahimanesh, S., Martin, S., Fazli, P., & Seifi, H. (2025) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3663547.3746320 · Cited by 13
“The Forking feature received the highest usefulness ratings, and participants valued the flexibility offered by variations and the potential of tags.”
Self-citationGenerative AI and Accessibility Workshop: Surfacing Opportunities and Risks
Glazko, K. S., Huh, M., Johnson, J., Pavel, A., & Mankoff, J. C. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3706734 · Cited by 9
“Some of the issues surfaced in these works include receiving inaccurate information from outputs [7, 12, 17], over-reliance and dependence [8, 22], opting for lower-quality results [31, 32, 37], harmful advice [30], or encountering bias and stereotypes [12].”
Interactive wearable digital devices for blind and partially sighted people
Xue, W., Ren, Y., Tang, Y. W., Gong, Z., Zhang, T., Chen, Z., Dong, X., Ma, X., Wang, Z., Xu, H., et al. (2025) · Nature Reviews Electrical Engineering · doi:10.1038/s44287-025-00170-w · Cited by 10
“Moreover, the balance between supplying information of complementary types and sensory occupancy should be 56,58,68,96,171,185–187, 188, 189 . b , Comparison of US National In the context of assistive device design, the delivery of information should be appropriately balanced between primary and…”
Sechayk, Y., Shamir, A., Pavel, A., & Igarashi, T. (2025) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3663547.3746372 · Cited by 12
“Research in AD focused on improving the creation process [30, 57], automating the creation altogether [10, 27, 35, 39, 80, 84], and supporting AD consumption through spatial audio [22] or user exploration [53, 68].”
Improving the Accessibility of Screen-Shared Presentations by Enabling Concurrent Exploration
Fan, D., Junuzovic, S., Tang, J. C., & Jaeger, T. F. (2023) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3597638.3608411 · Cited by 9
“Prior work has looked into ways to automatically generate alt-text for images that lack them [6, 19, 23] and audio descriptions for videos [15, 75, 77].”
MethodLLM-Driven Optimization of HTML Structure to Support Screen Reader NavigationPreprint
Yu, Y., Ryskeldiev, B., Tsutsui, A., Gillingham, M., & Wang, Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.18701 · Cited by 9
“Recent work has explored AI-driven methods to improve web accessibility, including restructuring data tables [60], converting PDFs to HTML [56, 57], and generating image and audio descriptions [23, 31, 59, 68, 69].”
Yu, Y., Ryskeldiev, B., Tsutsui, A., Gillingham, M., & Wang, Y. (2025) · Proceedings of the 27th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3663547.3746353 · Cited by 8
“Recent work has explored AI-driven methods to improve web accessibility, including restructuring data tables [58], converting PDFs to HTML [54, 55], and generating image and audio descriptions [23, 31, 57, 65, 66].”
Khan, A., Choubineh, A., Shaaban, M. A., Akkasi, A., & Komeili, M. (2025) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc58881.2025.11342771 · Cited by 11
“Meanwhile, video description systems [17] improve narrative accessibility yet lack tactile translation, leaving a critical gap in multimodal understanding.”
Uncovering Patterns in Reviewers’ Feedback to Scene Description Authors
Natalie, R., Loh, J., Tan, H. S., Tseng, J., Kacorri, H., & Hara, K. (2021) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3441852.3476550 · Cited by 8
“[20, 23, 24].”
Blind Assistance System using Tensor Flow
Jakka, S. C., Sai, Y. V., Jesudoss, A. G., & Mary.A, V. A. (2022) · International Conference on Electronics and Sustainable Communication Systems (ICESC) · doi:10.1109/icesc54411.2022.9885356 · Cited by 8
“Here they used the machine learning concept.[4]”
SmartLearn: Visual-Temporal Accessibility for Slide-based e-learning Videos
Sechayk, Y., Shamir, A., & Igarashi, T. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3650883 · Cited by 8
“Prior work explored systems and methods to improve AD content creation [25, 45, 48], static visual filters [38] and gaze-estimated video magnification [3].”
MethodNarrationBot and InfoBot: A Hybrid System for Automated Video DescriptionPreprint
Ihorn, S. M., Siu, Y.-T., Bodi, A., Narins, L. D., Castanon, J. M., Kant, Y., Das, A., Yoon, I., & Fazli, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.03994 · Cited by 8
“Previous works show that the time, training, and confidence needed to create high-quality descriptions might deter potential describers from completing more descriptions [52, 79].”
BackgroundSelf-citationUnderstanding the Video Content Creation Journey of Creators with Sensory Impairment in Kenya
Xiao, L., Bandukda, M., Li, F. M., Colley, M., & Holloway, C. (2025) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3663547.3746356 · Cited by 7
“Accessibility in digital content has long been a research focus, particularly concerning consumption by individuals with sensory impairments [23, 32, 34–36, 38, 61].”
WatchWithMe: LLM-Based Interactive Guided Watching of Review Videos
Chi, P., Hu, S.-P., Shi, L., Kraljic, T., Secor, J., Dong, T., Essa, I. A., & Cleron, M. (2025) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3719160.3736624 · Cited by 7
“Researchers have enable in-video activities for on-line communities that provide structured support, including exercises [51], audience feedback [58], visual descriptions [77, 112], and Q&A [108, 116].”
Tang, X., & Piper, A. M. (2026) · Designing Interactive Systems Conference · doi:10.1145/3800645.3813075 · Cited by 5
“…of access has led to extensive research and efforts on developing assistive technologies or accessible artifacts for PWDs [24, 28, 41, 64, 77, 78], accessibility guidelines [68, 79], understanding disabled people’s access needs [30, 37, 63, 66, 70], and identifying accessibility issues of…”
Cost-effective and Collaborative Methods to Author Video’s Scene Description for Blind People.
Natalie, R. (2022) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3491101.3503814 · Cited by 6
“In these studies, the authors let novice describers edit the AD, and it was evident, as evaluated by blind raters, that novices can improve the AD quality [35].”
BackgroundMethodInfluentialBennett, C. L., Pavel, A., Bryant, A., Eberly, L. J. M., Shelby, R. M., & Kane, S. K. (2025) · ACM Transactions on Accessible Computing · doi:10.1145/3777475 · Cited by 5
“Some generate scripts that are usable as-is, or as a basis for human correction or elaboration [50, 73, 116].”
InfluentialCoSight: Exploring Viewer Contributions to Online Video Accessibility Through Descriptive Commenting
Wang, R., Liu, X., Wang, B., Zhang, W., Liao, Z., Li, Z., Pavel, A., & Chen, X. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747747 · Cited by 4
“Hiring professional audio describers to provide high-quality audio is costly and not readily-available [78], as estimated to cost $12 to $75 per video minute and the turnaround time from days to weeks [71].”
Understanding Online Discussion Experiences of Blind Screen Reader Users
Ferdous, M. J., Nayak, A. K., Prakash, Y., Venkatraman, N., Jayarathna, S., Lee, H.-N., & Ashok, V. G. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2589865 · Cited by 4
“…have enabled such usability-enhancing tools that restructure web content for efficient screen reader navigation [149, 177, 194], describe visual media for screen reader users [78, 118, 217, 218, 228], and convert inaccessible formats into screen-reader-compatible narration [96, 126, 140, 169, 201].”
Li, F. M., Liu, M. X., Bennett, C. L., & Kane, S. K. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791158 · Cited by 8
“This has led to a class of semi-automated, Human-in-the-Loop (HITL) systems where AI performs initial analysis and a human provides refinement and oversight [20, 85, 87].”
Automatic Video-to-Audiotactile Conversion of Golf Broadcasting on A Refreshable Pin Array
Kim, H., Chung, M., Kim, E., & Yoo, Y. (2024) · ACM Symposium on Spatial User Interaction (SUI) · doi:10.1145/3677386.3682092 · Cited by 4
“Yuksel et al.[33] attempted to create descriptions and scene segmentation from online videos that were half automatic and half manual.”
YouDescribe: Bridging AI Efficiency and Human Insight for Scalable Audio Description
Do, L., Mirani, S., Pitcher-Cooper, C., Nguyen, X. D. A., Bairaboina, A., & Yoon, I. (2025) · Annual Symposium on Human-Computer Interaction for Work (CHIWORK) · doi:10.1145/3707640.3731916 · Cited by 3
“[22].”
Self-citationNon-Visual Accessibility Assessment of Videos
Aydin, A. S., Ko, Y.-J., Uckun, U., Ramakrishnan, I. V., & Ashok, V. G. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482457 · Cited by 5
“[41] present an approach based on creating video descriptions for improving the video accessibility for both people who are blind and those with low-vision.”
MethodA Taxonomy of Methods, Tools, and Approaches for Enabling Collaborative Annotation
de Souza, J. G. R., Ferreira, J. J., & Segura, V. (2023) · Brazilian Symposium on Human Factors in Computing Systems (IHC) · doi:10.1145/3638067.3638074 · Cited by 3
“Most articles do not focus on a specific domain, but in the annotation process [69, 102, 103] or in general context as for people annotate videos to promote accessibility[100].”
BackgroundMethodInfluentialDang, K., Burke, G., Korreshi, H., & Lee, S. (2026) · ACM Transactions on Accessible Computing · doi:10.1145/3813802 · Cited by 2
MTeacher: A Gamified and Physiological-Based Web Application Designed for Machine Learning Education
Hernández-Cuevas, B. Y., & Crawford, C. S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-05028-2_29 · Cited by 2
Towards An Accurate and Effective Printed Document Reader for Visually Impaired People
Viet, A. P., Duy, D. L., Thị, V. A. T., Duy, H. P., Van, T. V., & Thu, L. B. (2022) · International Conference on Knowledge and Systems Engineering/Proceedings · doi:10.1109/kse56063.2022.9953768 · Cited by 2
“For visual content understanding, Yuksel et al. developed an approach to generate the descriptions for videos [6].”
BackgroundNetizen A11y: Engaging Internet Users in Making Visual Media Accessible
Wang, R. (2024) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3640544.3645247 · Cited by 2
“Another line of research has been focusing on bridging such a gap by developing human-authoring tools to support novice describers such as volunteers and video content creators with writing high-quality descriptions efficiently [1, 10, 12, 22].”
BackgroundHobson, D. G., & Komeili, M. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.07191 · Cited by 5
Design considerations for photosensitivity warnings in visual media
South, L., Yildirim, C., Pavel, A., & Borkin, M. A. (2024) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3663548.3675643 · Cited by 1
“Many videos remain inaccessible to blind and low vision viewers such that prior work has explored crowd-sourced [25, 49], collaborative [32], semi-automated [6, 31, 35, 48], and fully automated [9, 45, 49] approaches to add audio descriptions that narrate the visual content in videos.”
Hera, C. J., Arcana, I. N., Kusumaningrum, B., & Sulaimon, J. T. (2025) · UNION Jurnal Ilmiah Pendidikan Matematika · doi:10.30738/union.v13i1.17079 · Cited by 1
South, L. (2024) · Digital Repository Service (Northeastern University) · doi:10.17760/d20653004
Assessing AI’s Role in Digital Accessibility
Makati, T. (2025) · Springer eBooks · doi:10.1007/978-3-031-93022-5_7
Smart Surveillance for Blinds: A YOLOv5-Based Assistive Vision System
Awais, M., Anees, M., Mehmood, Y., & Shahzad, A. (2025) · International Conference on Emerging Technologies · doi:10.1109/icet66147.2025.11321421
“These systems may assist visually impaired individuals to perform daily activities and navigation (e.g., recognizing banknotes [2, 3], crossing the street[4, 5, 6], watching video media [7, 8]and image rectification for free guidance and people recognition[9,10,11]).”
Toward Scalable Audio Description Quality Control: A Workflow for Evaluating Human and VLM Raters
Do, L., Jung, G., Scott, A. T., Barajas, J. F., Blum, A. M., Ihorn, S. M., Athitsos, V., & Yoon, I. (2026) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3797867.3829038
“Hybrid human-AI systems that combine automated generation with human revision and on-demand question answering show promise, with BLV users reporting preferences for these approaches over AI-only alternatives [43, 52].”
Self-citationADx3: A Collaborative Workflow for High-Quality Accessible Audio DescriptionPreprint
Do, L., Ihorn, S. M., Pitcher-Cooper, C., Barajas, J. F., Jung, G., Nguyen, X. D. A., Mirani, S., & Yoon, I. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.02684
“Prior work established the value of HITL edits and user on-demand queries through extensive studies with BLV participants Yuksel et al. (2020); Stangl et al. (2023).”
InfluentialSelf-citationViDscribe: Multimodal AI for Customizing Audio Description and Question Answering in Online Videos
Cheema, M. S., Elahimanesh, S., Fazli, P., & Seifi, H. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3798744 · Cited by 2
“While human-in-the loop tools capture rich emotional and contextual nuance [38, 39], they cannot scale to the increasing volume of video content [23].”
Self-citationHow Do Novice Describers Edit AI Drafts? The Role of Draft Quality in Audio Description AuthoringPreprint
Do, L., Ihorn, S. M., Pitcher-Cooper, C., Mirani, S., Jung, G., Shim, H., Qin, Z., Nguyen, K. T., Athitsos, V., & Yoon, I. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.05348
“First, prior work established that novice describers require significant time to complete these videos from scratch, making higher-difficulty content impractical within a single session [80].”
InfluentialSelf-citationBirčanin, F., Nevsky, A., Cruice, M. N., & Neate, T. (2026) · Designing Interactive Systems Conference · doi:10.1145/3800645.3812914
“To further support scale without sacrificing quality, semi-automated systems can suggest timings or text that accelerate work under human review for accuracy [110].”
Making Videos Accessible for Blind and Low Vision Users Using a Multimodal Agent Video Player
Sinha, A. K., Olmos, A., Santos, R. D., Rodriguez, R. R., Landay, J. A., Sepah, S., Nelson, P., & Kane, S. K. (2026) · Companion Publication of the International Conference on Multimodal Interaction (ICMI Companion) · doi:10.1145/3776591.3833853
“Our work shows that a modern MLLM can now enable a more fluid interactive dialogue and novel experience across an entire video.”
Proxy Interfaces for Enhancing Non-Visual Web Usability
Ferdous, M. J. (2026) · ODU Digital Commons (Old Dominion University) · doi:10.25777/mk1x-r688
Metadata
- Authors
- Yuksel, B. F., Fazli, P., Mathur, U., Bisht, V., Kim, S. J., Lee, J. J., Jin, S. J., Siu, Y.-T., Miele, J. A., & Yoon, I.
- Affiliations
- University of San Francisco (US)
- San Francisco State University (US)
- University of San Francisco, San Francisco, CA, USA
- University of Washington (US)
- Smith-Kettlewell Eye Research Institute (US)
- Venue
- Proceedings of the 2020 ACM Designing Interactive Systems Conference
- Publisher
- ACM
- Published
- 2020-07-03
- Pages
- 47-60
- OpenAlex
- W3038896047
- Cited by
- 60
- References
- 45
- Citations by year
- 2026: 3 · 2025: 17 · 2024: 12 · 2023: 10 · 2022: 8 · 2021: 10
- FWCI
- 49.82
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Designing Interactive Systems Conference (publishedVersion)
- Funders
- Craig Newmark Philanthropies
- Center for Computing for Life Sciences, San Francisco State University
- Topics
- Social Sciences › Arts and Humanities › Language and Linguistics › Subtitles and Audiovisual Media
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- video accessibility, human-in-the-loop learning, audio description, blind and visually impaired users, text generation, human-machine collaboration, accessibility for people with disabilities
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.