Human-in-the-Loop Machine Learning to Increase Video Accessibility for Visually Impaired and Blind Users
The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
Abstract
Video accessibility is crucial for blind and visually impaired individuals for education, employment, and entertainment purposes. However, professional video descriptions are costly and time-consuming. Volunteer-created video descriptions could be a promising alternative, however, they can vary in quality and can be intimidating for novice describers. We developed a Human-in-the-Loop Machine Learning (HILML) approach to video description by automating video text generation and scene segmentation and allowing humans to edit the output. The HILML approach facilitates human-machine collaboration to produce high quality video descriptions while keeping a low barrier to entry for volunteer describers. Our HILML system was significantly faster and easier to use for first-time video describers compared to a human-only control condition with no machine learning assistance. The quality of the video descriptions and understanding of the topic created by the HILML system compared to the human-only condition were rated as being significantly higher by blind and visually impaired users.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“In particular, Convolutional Neural Networks (CNNs) [6] are the state of the art for modeling visual data recognition [6, 65, 67] and Long Short-Term Memory (LSTMs) [34] are now dominating the area of sequence modeling such as NLP [16, 31, 66].”
BackgroundGoing deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893 · cited 1×
“…modeling visual data recognition [6, 65, 67] and Long Short-Term Memory (LSTMs) [34] are now dominating the area of sequence modeling such as NLP [16, 31, 66] Our approach assists sighted users to create video descriptions by combining automation using deep learning with human input and editing.”
MethodOn the Properties of Neural Machine Translation: Encoder–Decoder Approaches
Cho, K., van Merrienboer, B., Bahdanau, D., & Bengio, Y. (2014) · Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation · doi:10.3115/v1/w14-4012 · Cited by 7,099 · cited 1×
“…modeling visual data recognition [6, 65, 67] and Long Short-Term Memory (LSTMs) [34] are now dominating the area of sequence modeling such as NLP [16, 31, 66] Our approach assists sighted users to create video descriptions by combining automation using deep learning with human input and editing.”
MethodVQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
Bourne, R. R. A., Flaxman, S. R., Braithwaite, T., Cicinelli, M. V., Das, A., Jonas, J. B., Keeffe, J., Kempen, J. H., Leasher, J., Limburg, H., et al. (2017) · The Lancet Global Health · doi:10.1016/s2214-109x(17)30293-0 · Cited by 2,213 · cited 2×
“Request permissions from Permissions@acm.org. million of them are blind [11].”
BackgroundInfluentialHierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1082 · Cited by 1,266
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · arXiv · doi:10.48550/arxiv.1805.04833 · Cited by 1,242
Expertise Reversal Effect and Its Implications for Learner-Tailored Instruction
Kalyuga, S. (2007) · Educational Psychology Review · doi:10.1007/s10648-007-9054-3 · Cited by 1,056 · cited 1×
“There is a phenomenon in learning called the expertise reversal effect whereby instructional techniques that are benefcial to beginners can have the reverse effects on more experienced learners [41, 42].”
BackgroundConvention on the Rights of Persons with Disabilities and Optional Protocol
States, M. (2012) · Cited by 623
Interactive deep learning method for segmenting moving objects
Wang, Y., Luo, Z., & Jodoin, P.-M. (2017) · Pattern Recognition Letters · doi:10.1016/j.patrec.2016.09.014 · Cited by 334 · cited 2×
“[71] used HILML to create a semi-automatic method to segment foreground moving objects in surveillance videos.”
BackgroundMethodHuman-in-the-Loop Optimization of Shared Autonomy in Assistive Robotics
Gopinath, D., Jain, S., & Argall, B. D. (2017) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2016.2593928 · Cited by 168 · cited 1×
“HILML has been used in felds such as assistive robotics [30], data analytics [64], systems [35, 72], and audio labeling [43].”
MethodEvent Representations for Automated Story Generation with Deep Neural Nets
Martin, L., Ammanabrolu, P., Wang, X., Hancock, W., Singh, S., Harrison, B., & Riedl, M. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11430 · Cited by 154 · cited 1×
“al proposed an event representation for neural-network-based story generation [51].”
BackgroundPythia v0.1: the Winning Entry to the VQA Challenge 2018
Jiang, Y. L., Natarajan, V., Chen, X., Rohrbach, M., Batra, D., & Parikh, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.09956 · Cited by 150 · cited 2×
“…several state-of-the-art image captioning deep learning technologies: Microsoft AI Video Insight, Pythia, the winning entry from Facebook AI Research (FAIR)’s A-STAR team to the VQA Challenge 2018 [40] and GLACNet, the winning champion from Visual Story Telling challenge for text generation [44].”
BackgroundMethodShake Table Test of Large-Scale Bridge Columns Supported on Rocking Shallow Foundations
Antonellis, G., Gavras, A. G., Panagiotou, M., Kutter, B. L., Guerrini, G., Sander, A. C., & Fox, P. J. (2015) · Journal of Geotechnical and Geoenvironmental Engineering · doi:10.1061/(asce)gt.1943-5606.0001284 · Cited by 143 · cited 1×
“In particular, Convolutional Neural Networks (CNNs) [6] are the state of the art for modeling visual data recognition [6, 65, 67] and Long Short-Term Memory (LSTMs) [34] are now dominating the area of sequence modeling such as NLP [16, 31, 66].”
BackgroundAutomating tactile graphics translation
Ladner, R. E., Ivory, M. Y., Rao, R., Burgstahler, S., Comden, D., Hahn, S., Renzelmann, M., Krisnandi, S., Ramasamy, M., Slabosky, B., et al. (2005) · Proceedings of the 7th international ACM SIGACCESS conference on Computers and accessibility · doi:10.1145/1090785.1090814 · Cited by 124
Accelerating Human-in-the-loop Machine Learning: Challenges and Opportunities
Xin, D., Ma, L., Liu, J., Macke, S., Song, S., & Parameswaran, A. (2018) · Proceedings of the Second Workshop on Data Management for End-To-End Machine Learning · doi:10.1145/3209889.3209897 · Cited by 120
Talking TMAP: Automated generation of audio-tactile maps using Smith-Kettlewell's TMAP software
Miele, J. A., Landau, S., & Gilden, D. (2006) · British Journal of Visual Impairment · doi:10.1177/0264619606064436 · Cited by 118 · cited 1×
“There have been a wide range of innovative works in the feld of HCI and accessibility for blind and visually impaired users including visual graphics [48], data visualizations [70], maps [53, 56], programming [59], board games [18], video games [29, 62, 8], and appliance displays [26].”
BackgroundSelf-citationFeeling what you hear: tactile feedback for navigation of audio graphs
Wall, S., & Brewster, S. (2006) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1124772.1124941 · Cited by 107
Holzinger, A., Plass, M., Holzinger, K., Crişan, G. C., Pintea, C.-M., & Palade, V. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-45507-5_6 · Cited by 97
Algorithmic characterization of vehicle trajectories from image sequences by motion verbs
Koller, D., Heinze, N., & Nagel, H.-H. (1991) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.1991.139667 · Cited by 96 · cited 1×
“Classical video description approaches combined subject, object, and verb (SVO) detection from visual entities with template based language models to generate sentences [47].”
BackgroundCodeTalk: Improving Programming Environment Accessibility for Visually Impaired Developers
Potluri, V., Vaithilingam, P., Iyengar, S., Vidya, Y., Swaminathan, M., & Srinivasa, G. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174192 · Cited by 91
GLAC Net: GLocal Attention Cascading Networks for Multi-image Cued Story GenerationPreprint
Kim, T., Heo, M.-O., Son, S., Park, K.-W., & Zhang, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.10973 · Cited by 78 · cited 2×
“When picking the machine learning algorithms described above, we tested several state-of-the-art image captioning deep learning technologies: Microsoft AI Video Insight, Pythia, the winning entry from Facebook AI Research (FAIR)’s A-STAR team to the VQA Challenge 2018 [40] and GLACNet, the winning champion from Visual…”
BackgroundMethodSpeech technologies for blind and low vision persons
Freitas, D., & Kouroupetroglou, G. (2008) · Technology and Disability · doi:10.3233/tad-2008-20208 · Cited by 66 · cited 1×
“Video description is particularly important as the speech communication channel is considered to be one of the most important modalities for blind and visually impaired users [25].”
BackgroundWriting Stories with Help from Recurrent Neural Networks
Roemmele, M. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.9810 · Cited by 59 · cited 1×
“al used a recurrent-neural-network architecture to generate stories in a sequence-to-sequence manner [61].”
MethodLet's Play Together: Adaptation Guidelines of Board Games for Players with Visual Impairment
da Rocha Tomé Filho, F., Mirza-Babaei, P., Kapralos, B., & Moreira Mendonça Junior, G. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300861 · Cited by 41
Gagnon, L., Foucher, S., Héritier, M., Lalonde, M., Byrns, D., Chapdelaine, C., Turner, J. M., Mathieu, S., Laurendeau, D., Nguyen, N. T., et al. (2009) · Universal Access in the Information Society · doi:10.1007/s10209-008-0141-0 · Cited by 38
Are synthesized video descriptions acceptable?
Kobayashi, M., O’Connell, T., Gould, B., Takagi, H., & Asakawa, C. (2010) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/1878803.1878833 · Cited by 35 · cited 6×
“[46] tested their describer script editor, which has a visual interface to edit the video description sentences and to specify where on the video timeline to read each sentence aloud, on one novice and one professional describer.”
BackgroundMethodInfluentialBridging the gap between user intention and model parameters for human-in-the-loop data analytics
Self, J. Z., Vinayagam, R. K., Fry, J. T., & North, C. (2016) · Proceedings of the Workshop on Human-In-the-Loop Data Analytics · doi:10.1145/2939502.2939505 · Cited by 35 · cited 2×
“[64] that created a breakthrough with human post-editing for language translation.”
BackgroundMethodProviding synthesized audio description for online videos
Kobayashi, M., Fukuda, K., Takagi, H., & Asakawa, C. (2009) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/1639642.1639699 · Cited by 31 · cited 1×
“created audio description synthesis by using external metadata of videos on an online platform [45, 46].”
MethodEncelle, B., Ollagnier-Beldame, M., Pouchot, S., & Prié, Y. (2011) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/2049536.2049560 · Cited by 31
On presenting audio-tactile maps to visually impaired users for getting directions
Paladugu, D. A., Wang, Z., & Li, B. (2010) · CHI '10 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/1753846.1754085 · Cited by 31 · cited 1×
“There have been a wide range of innovative works in the feld of HCI and accessibility for blind and visually impaired users including visual graphics [48], data visualizations [70], maps [53, 56], programming [59], board games [18], video games [29, 62, 8], and appliance displays [26].”
BackgroundAudiOdyssey: an accessible video game for both sighted and non-sighted gamers
Glinert, E., & Wyse, L. L. (2007) · Conference on Future Play · doi:10.1145/1328202.1328255 · Cited by 29
A computer-vision-assisted system for Videodescription scripting
Gagnon, L., Chapdelaine, C., Byrns, D., Foucher, S., Héritier, M., & Gupta, V. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops · doi:10.1109/cvprw.2010.5543575 · Cited by 28 · cited 3×
“[28, 27] displayed detected features on an interactive timeline which allowed professional describers to have an overview of the frequency and duration of features in the video, as well as the silent areas.”
BackgroundMethodInfluentialAudio haptic videogaming for navigation skills in learners who are blind
Sánchez, J., & Espinoza, M. (2011) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/2049536.2049580 · Cited by 24 · cited 1×
“There have been a wide range of innovative works in the feld of HCI and accessibility for blind and visually impaired users including visual graphics [48], data visualizations [70], maps [53, 56], programming [59], board games [18], video games [29, 62, 8], and appliance displays [26].”
BackgroundUsing computer vision to access appliance displays
Fusco, G., Tekin, E., Ladner, R. E., & Coughlan, J. M. (2014) · Proceedings of the 16th international ACM SIGACCESS conference on Computers & accessibility - ASSETS '14 · doi:10.1145/2661334.2661404 · Cited by 23 · cited 1×
“There have been a wide range of innovative works in the feld of HCI and accessibility for blind and visually impaired users including visual graphics [48], data visualizations [70], maps [53, 56], programming [59], board games [18], video games [29, 62, 8], and appliance displays [26].”
BackgroundI-SED: An Interactive Sound Event Detector
Kim, B., & Pardo, B. (2017) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3025171.3025231 · Cited by 21
Hsu, T.-Y., Huang, C.-Y., Hsu, Y.-C., & Huang, T.-H. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1658 · Cited by 19 · cited 1×
“Such sentences are currently not at the level needed to replace the ‘human’ of the Human-in-Loop [37], but would improve the speed and accuracy substantially more.”
BackgroundAccessible videodescription On-Demand
Chapdelaine, C., & Gagnon, L. (2009) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/1639642.1639685 · Cited by 17 · cited 5×
“in the literature, synthesized video descriptions have been shown to be acceptable to blind and visually impaired users [15, 46].”
BackgroundMethodInfluentialAn adaptive videos enrichment system based on decision trees for people with sensory disabilities
Villamizar, J. F. S., Encelle, B., Prié, Y., & Champin, P.-A. (2011) · International Cross-Disciplinary Conference on Web Accessibility (W4A) · doi:10.1145/1969289.1969299 · Cited by 13
Towards the usage of pauses in audio-described videos
Encelle, B., Ollagnier-Beldame, M., & Prié, Y. (2013) · International Cross-Disciplinary Conference on Web Accessibility (W4A) · doi:10.1145/2461121.2461130 · Cited by 12 · cited 1×
“[21] investigated the use of artifcial pauses in audio described videos in order to transmit more information in the video descriptions and suggest the placing of artifcial pauses right after the soundtrack’s natural gaps [21].”
BackgroundSHARKZOR: Human in the Loop ML for User-Defined Image Classification
Pirrung, M., Hilliard, N., O'Brien, N., Yankov, A., Corley, C. D., & Hodas, N. O. (2018) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3180308.3180337 · Cited by 9
Tangicraft: A Multimodal Interface for Minecraft
Bar-El, D., Large, T., Davison, L., & Worsley, M. B. (2018) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3234695.3241031 · Cited by 8
Specialized DVD player to render audio description and its usability performance
Chapdelaine, C. (2012) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/2384916.2384954 · Cited by 2 · cited 1×
“Chapdelaine [14] demonstrated a DVD player with video description with three extra specialized functions: contextual functions, differing quantities of audio description, and recall assistance.”
Background
Research works printed but not linked: 13
- Nayyer Aafaq , Syed Zulqarnain Gilani , Wei Liu, Ajmal Mian, and Mubarak Shah. 2018 . Video Description : A Survey of Methods, Datasets and Evaluation Metrics. CoRR abs/1806.00186 (2018). Nayyer Aafaq, Syed Zulqarnain Gilani, Wei Liu, Ajmal Mian, and Mubarak Shah. 2018. Video Description: A Survey of Methods, Datasets and Evaluation Metrics. CoRR abs/1806.00186 (2018). · arXiv
- Andrei Barbu , Alexander Bridge , Zachary Burchill , Dan Coroian , Sven Dickinson , Sanja Fidler , Aaron Michaux , Sam Mussman , Siddharth Narayanaswamy , Dhaval Salvi , Lara Schmidt , Jiangnan Shangguan , Jeffrey Mark Siskind , Jarrell Waggoner, Song Wang, Jinlian Wei, Yifan Yin, and Zhiqi Zhang. 2012 . Video In Sentences Out . (2012). Andrei Barbu, Alexander Bridge, Zachary Burchill, Dan Coroian, Sven Dickinson, Sanja Fidler, Aaron Michaux, Sam Mussman, Siddharth Narayanaswamy, Dhaval Salvi, Lara Schmidt, Jiangnan Shangguan, Jeffrey Mark Siskind, Jarrell Waggoner, Song Wang, Jinlian Wei, Yifan Yin, and Zhiqi Zhang. 2012. Video In Sentences Out. (2012).
- Matthew Brand. 1997. The" Inverse hollywood problem": from video to scripts and storyboards via causal analysis. In AAAI/IAAI. Citeseer 132--137. Matthew Brand. 1997. The" Inverse hollywood problem": from video to scripts and storyboards via causal analysis. In AAAI/IAAI. Citeseer 132--137.
- Carmen Branje , Susan Marshall , Ashley Tyndall , and Deborah Fels . 2006 . LiveDescribe. AMCIS 2006 Proceedings (2006), 368. Carmen Branje, Susan Marshall, Ashley Tyndall, and Deborah Fels. 2006. LiveDescribe. AMCIS 2006 Proceedings (2006), 368.
- Abhishek Das , Satwik Kottur , Khushi Gupta , Avi Singh , Deshraj Yadav , Jose M. F. Moura , Devi Parikh , and Dhruv Batra . 2017 . Visual Dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR. Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, Jose M. F. Moura, Devi Parikh, and Dhruv Batra. 2017. Visual Dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR.
- Pradipto Das , Chenliang Xu , Richard F. Doell , and Jason J. Corso . 2013. A Thousand Frames in Just a Few Words: Lingual Description of Videos through Latent Topics and Sparse Object Stitching . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR. Pradipto Das, Chenliang Xu, Richard F. Doell, and Jason J. Corso. 2013. A Thousand Frames in Just a Few Words: Lingual Description of Videos through Latent Topics and Sparse Object Stitching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR.
- Patrick Hanckmann , Klamer Schutte , and Gertjan J . Burghouts . 2012 . Automated Textual Descriptions for a Wide Range of Video Events with 48 Human Actions. Lecture Notes in Computer Science ( 2012), 372--380. Patrick Hanckmann, Klamer Schutte, and Gertjan J. Burghouts. 2012. Automated Textual Descriptions for a Wide Range of Video Events with 48 Human Actions. Lecture Notes in Computer Science (2012), 372--380.
- SG Hart and others. 1988. Development of NASA-TLX: Results of empirical and theoretical research." inP . A. Hancock and N. Meshkati (eds.), Human Mental Workload . ( 1988 ). SG Hart and others. 1988. Development of NASA-TLX: Results of empirical and theoretical research." inP. A. Hancock and N. Meshkati (eds.), Human Mental Workload. (1988).
- 10.18653/v1 · doi:10.18653/v1
- Slava Kalyuga . 2003. The expertise reversal effect. Educational psychologist 38, 1 ( 2003 ), 23--31. Slava Kalyuga. 2003. The expertise reversal effect. Educational psychologist 38, 1 (2003), 23--31.
- Kishore Papineni , Salim Roukos , Todd Ward , and Wei-Jing Zhu . 2002 . BLEU: a method for automatic evaluation of machine translation . In Proceedings of the 40th annual meeting on association for computational linguistics. Association for Computational Linguistics, 311--318 . Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics. Association for Computational Linguistics, 311--318.
- Karen Simonyan and Andrew Zisserman . 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. CoRR abs/1409.1556 ( 2014 ). Karen Simonyan and Andrew Zisserman. 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. CoRR abs/1409.1556 (2014). · arXiv
- Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Lukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 5998--6008. Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Lukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 5998--6008.
Cited by
Large-scale Text-to-Image Generation Models for Visual Artists’ Creative Works
Ko, H.-K., Park, G., Jeon, H., Jo, J., Kim, J., & Seo, J. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584078 · Cited by 189
“%) such as podcast listening [44], video captioning [107], and conversation [35] which can be encountered in our everyday life.”
BackgroundSmart Glass System Using Deep Learning for the Blind and Visually Impaired
Mukhiddinov, M., & Cho, J. (2021) · Electronics · doi:10.3390/electronics10222756 · Cited by 161
“Assistive systems exist to aid BVI people with navigation and daily activities, such as distinguishing banknotes [2,3], crossing a road [4–6], video media accessibility [7,8], image sonification for navigation [9,10], recognizing people [11–13], recognizing private visual information [14], selecting clothing [15], and…”
BackgroundMukhiddinov, M., & Kim, S.-Y. (2021) · Processes · doi:10.3390/pr9101726 · Cited by 110
“Researchers have also developed a touchscreen tablet with audio feedback [37], an accessibility system for web documents [38], a tactile graphics finder [35], video description [39,40], bas-reliefs [41], and touchscreen interactions for environmental perception [42] using AI technologies to make visual information eas…”
MethodSelf-citationWhat Makes Videos Accessible to Blind and Visually Impaired People?
Liu, X., Carrington, P., Chen, X. '., & Pavel, A. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445233 · Cited by 87
“Prioritizing Content for Description: Our work helping people fnd videos that already have high-quality and built-in descriptions during video search can advance existing work on helping authors add high-quality audio descriptions to individual inaccessible videos [18, 19, 29, 40, 42, 66].”
BackgroundMethodMaking Short-Form Videos Accessible with Hierarchical Video Summaries
Van Daele, T., Iyer, A., Zhang, Y., Derry, J. C., Huh, M., & Pavel, A. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642839 · Cited by 38
“Prior work has explored making videos accessible with manual [36], semi-automated [15, 46, 60, 76] and automated [17, 73] audio descriptions — or, narrations of visual content in the video.”
Metadata
- Authors
- Yuksel, B. F., Fazli, P., Mathur, U., Bisht, V., Kim, S. J., Lee, J. J., Jin, S. J., Siu, Y.-T., Miele, J. A., & Yoon, I.
- Affiliations
- University of San Francisco (US)
- San Francisco State University (US)
- University of San Francisco, San Francisco, CA, USA
- University of Washington (US)
- Smith-Kettlewell Eye Research Institute (US)
- Venue
- Proceedings of the 2020 ACM Designing Interactive Systems Conference
- Publisher
- ACM
- Published
- 2020-07-03
- Pages
- 47-60
- OpenAlex
- W3038896047
- Cited by
- 60
- References
- 45
- Citations by year
- 2026: 3 · 2025: 17 · 2024: 12 · 2023: 10 · 2022: 8 · 2021: 10
- FWCI
- 49.82
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Designing Interactive Systems Conference (publishedVersion)
- Funders
- Craig Newmark Philanthropies
- Center for Computing for Life Sciences, San Francisco State University
- Topics
- Social Sciences › Arts and Humanities › Language and Linguistics › Subtitles and Audiovisual Media
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- video accessibility, human-in-the-loop learning, audio description, blind and visually impaired users, text generation, human-machine collaboration, accessibility for people with disabilities
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.