Understanding Blind People's Experiences with Computer-Generated Captions of Social Media Images
Abstract
Research advancements allow computational systems to automatically caption social media images. Often, these captions are evaluated with sighted humans using the image as a reference. Here, we explore how blind and visually impaired people experience these captions in two studies about social media images. Using a contextual inquiry approach (n=6 blind/visually impaired), we found that blind people place a lot of trust in automatically generated captions, filling in details to resolve differences between an image's context and an incongruent caption. We built on this in-person study with a second, larger online experiment (n=100 blind/visually impaired) to investigate the role of phrasing in encouraging trust or skepticism in captions. We found that captions emphasizing the probability of error, rather than correctness, encouraged people to attribute incongruence to an incorrect caption, rather than missing details. Where existing research has focused on encouraging trust in intelligent systems, we conclude by challenging this assumption and consider the benefits of encouraging appropriate skepticism.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing work on evaluating figure captions shows that people over-trust AI-generated information while AI can produce inaccurate responses.
“Existing work on evaluating figure captions shows that people over-trust AI-generated information [49] while at the same time, AI can produce inaccurate responses [46, 70].”
From How University Disability Services Professionals Write Image Descriptions for HCI Figures Using Generative AI (2026) · cites this paper as [49]
Inaccurate outputs risk misleading readers and undermining trust in accessibility resources.
“Inaccurate outputs risk misleading readers and undermining trust in accessibility resources [49].”
From How University Disability Services Professionals Write Image Descriptions for HCI Figures Using Generative AI (2026) · cites this paper as [49]
Cites
The Framing of Decisions and the Psychology of Choice
Tversky, A., & Kahneman, D. (1981) · Science · doi:10.1126/science.7455683 · Cited by 17,604 · cited 1× · see citing works
“We turn to research from psychology on framing effects [28] to explore different ways of wording captions to give people a more accurate sense of how reliable they are.”
BackgroundShow, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · see citing works
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902 · see citing works
VizWiz: nearly real-time answers to visual questions
Bigham, J. P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R. C., Miller, R., Tatarowicz, A., White, B., White, S., et al. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866080 · Cited by 601 · see citing works
Helldin, T., Falkman, G., Riveiro, M., & Davidsson, S. (2013) · Proceedings of the 5th International Conference on Automotive User Interfaces and Interactive Vehicular Applications · doi:10.1145/2516540.2516554 · Cited by 286 · cited 1× · see citing works
“In fact, there is even research on how to encourage people to have more trust in these intelligent systems (e.g., [1,2,15,30]).”
BackgroundWebInSight: making web images accessible
Bigham, J. P., Kaminsky, R. S., Ladner, R. E., Danielsson, O. M., & Hempton, G. L. (2006) · Proceedings of the 8th international ACM SIGACCESS conference on Computers and accessibility · doi:10.1145/1168987.1169018 · Cited by 161 · see citing works
Guiding Novice Web Workers in Making Image Descriptions Using Templates
Morash, V. S., Siu, Y.-T., Miele, J. A., Hasty, L., & Landau, S. (2015) · ACM Transactions on Accessible Computing · doi:10.1145/2764916 · Cited by 69 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
VizWiz Grand Challenge: Answering Visual Questions from Blind People
Gurari, D., Li, Q., Stangl, A. J., Guo, A., Lin, C., Grauman, K., Luo, J., & Bigham, J. P. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00380 · Cited by 621 · see citing works
“In addition, social media platforms, such as Facebook and Twitter, provide people a way to maintain connections with friends by enabling them to identify and tag friends in posted images as well as respond to images automatically described to them [26, 41].”
BackgroundObject Hallucination in Image Captioning
Rohrbach, A., Hendricks, L. A., Burns, K., Darrell, T., & Saenko, K. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1437 · Cited by 420 · see citing works
Accessible Visualization via Natural Language Descriptions: A Four-Level Model of Semantic Content
Lundgard, A., & Satyanarayan, A. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114770 · Cited by 155 · see citing works
Rich Representations of Visual Content for Screen Reader Users
Morris, M. R., Johnson, J., Bennett, C. L., & Cutrell, E. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173633 · Cited by 159 · see citing works
Self-citationStangl, A., Morris, M. R., & Gurari, D. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376404 · Cited by 139 · see citing works
Self-citationToward fairness in AI for people with disabilities SBG@a research roadmap
Guo, A., Kamar, E., Vaughan, J. W., Wallach, H., & Morris, M. R. (2020) · ACM SIGACCESS Accessibility and Computing · doi:10.1145/3386296.3386298 · Cited by 128 · see citing works
Self-citationCaption Crawler: Enabling Reusable Alternative Text Descriptions using Reverse Image Search
Guinness, D., Cutrell, E., & Morris, M. R. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174092 · Cited by 108 · see citing works
Self-citationAn Empirical Study of Older Adult’s Voice Assistant Use for Health Information Seeking
Brewer, R., Pierce, C., Upadhyay, P., & Park, L. (2022) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3484507 · Cited by 89 · see citing works
A Face Recognition Application for People with Visual Impairments: Understanding Use Beyond the Lab
Zhao, Y., Wu, S., Reynolds, L., & Azenkot, S. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173789 · Cited by 84 · see citing works
Stangl, A., Verma, N., Fleischmann, K. R., Morris, M. R., & Gurari, D. (2021) · Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3441852.3471233 · Cited by 81 · see citing works
Self-citationZhao, Y., Wu, S., Reynolds, L., & Azenkot, S. (2017) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3134756 · Cited by 68 · see citing works
BrowseWithMe: An Online Clothes Shopping Assistant for People with Visual Impairments
Stangl, A. J., Kothari, E., Jain, S. D., Yeh, T., Grauman, K., & Gurari, D. (2018) · Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3234695.3236337 · Cited by 66 · see citing works
"I look at it as the king of knowledge": How Blind People Use and Understand Generative AI Tools
Adnin, R., & Das, M. (2024) · The 26th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3663548.3675631 · Cited by 73 · see citing works
Designing Tools for High-Quality Alt Text Authoring
Mack, K., Cutrell, E., Lee, B., & Morris, M. R. (2021) · Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3441852.3471207 · Cited by 58 · see citing works
Self-citationMisfitting With AI: How Blind People Verify and Contest AI Errors
Alharbi, R., Lor, P., Herskovitz, J., Schoenebeck, S., & Brewer, R. N. (2024) · The 26th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3663548.3675659 · Cited by 40 · see citing works
Heuer, H., & Breiter, A. (2018) · Proceedings of the 10th Nordic Conference on Human-Computer Interaction · doi:10.1145/3240167.3240172 · Cited by 27 · see citing works
Blind Users Accessing Their Training Images in Teachable Object Recognizers
Hong, J., Gandhi, J., Mensah, E. E., Zeraati, F. Z., Jarjue, E., Lee, K., & Kacorri, H. (2022) · Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3517428.3544824 · Cited by 22 · see citing works
Tiwary, T., & Mahapatra, R. P. (2022) · 2022 3rd International Conference on Issues and Challenges in Intelligent Computing Techniques (ICICT) · doi:10.1109/icict55121.2022.10064545 · Cited by 12 · see citing works
Strategies for Auditory Display of Social Media
Winters, R. M., Joshi, N., Cutrell, E., & Morris, M. R. (2019) · Ergonomics in Design: The Quarterly of Human Factors Applications · doi:10.1177/1064804618788098 · Cited by 11 · see citing works
Self-citationPerera, M., Ananthanarayan, S., Goncu, C., & Marriott, K. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713634 · Cited by 14 · see citing works
Alluqmani, A., Harvey, M. A., & Zhang, Z. (2023) · Proceedings of the 2023 ACM Designing Interactive Systems Conference · doi:10.1145/3563657.3595978 · Cited by 14 · see citing works
Crowdsourcing Thumbnail Captions: Data Collection and Validation
Aguirre, C., Cao, S., Mahmood, A., & Huang, C.-M. (2023) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3589346 · Cited by 6 · see citing works
Garg, K., Tang, X., Heo, J., Morgan, D. R., Gergle, D., Sudderth, E. B., & Piper, A. M. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791309 · Cited by 3 · see citing works
“Image captioning is a well-studied task in computer vision that aims to generate descriptive text for images and has led to extensive work within accessible computing [56, 73, 90, 96, 116].”
Reference-Based Metrics Are Biased Against Blind and Low-Vision Users’ Image Description Preferences
Kapur, R., & Kreiss, E. (2024) · Proceedings of the Third Workshop on NLP for Positive Impact · doi:10.18653/v1/2024.nlp4pi-1.26 · Cited by 2 · see citing works
“However, these systems often do not directly address the needs of BLV users (MacLeod et al., 2017), which recent work has started to partially attribute to the evaluation meth-ods used (Kreiss et al., 2022a).”
BackgroundRaees, M., Iwamoto, Y., Papangelis, K., Heard, J., & Tigwell, G. W. (2026) · arXiv · doi:10.48550/arxiv.2602.08937 · Cited by 1 · see citing works
“For instance, R4 noted,”
- Authors
- MacLeod, H., Bennett, C. L., Morris, M. R., & Cutrell, E.
- Affiliations
- Indiana University Bloomington (US)
- University of Washington (US)
- Microsoft (United States) (US)
- Venue
- Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2017-05-02
- Pages
- 5988-5999
- OpenAlex
- W2578190051
- Cited by
- 200
- References
- 7
- Citations by year
- 2026: 16 · 2025: 20 · 2024: 27 · 2023: 21 · 2022: 33 · 2021: 31 · 2020: 21 · 2019: 11 · 2018: 16 · 2017: 4
- FWCI
- 27.63
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
- Social Sciences › Arts and Humanities › Language and Linguistics › Subtitles and Audiovisual Media
- Keywords
- blind people, visually impaired people, image captioning, social media images, trust in AI, skepticism, error probability, accessibility, contextual inquiry
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
