CoChat
Sign Up
Cited by 6Digital Accessibility for DisabilitiesAssistive Technology in Communication and MobilityTactile and Sensory Interactions

Surfacing Variations to Calibrate Perceived Reliability of MLLM-generated Image Descriptions

Open access
Chen, Meng, Iyer, Akhil, Pavel, Amy·2025·Proceedings of the 27th International ACM SIGACCESS Conference on Computers and Accessibility
DOI 10.1145/3663547.3746393
Open this paper in CoChat — free accountOpen PDF

Abstract

Multimodal large language models (MLLMs) provide new opportunities for blind and low vision (BLV) people to access visual information in their daily lives.However, these models often produce errors that are difficult to detect without sight, posing safety and social risks in scenarios from medication identification to outfit selection.While BLV MLLM users use creative workarounds such as crosschecking between tools and consulting sighted individuals, these approaches are often time-consuming and impractical.We explore how systematically surfacing variations across multiple MLLM responses can support BLV users to detect unreliable information without visually inspecting the image.We contribute a design space for eliciting and presenting variations in MLLM descriptions, a prototype system implementing three variation presentation styles, and findings from a user study with 15 BLV participants.Our results demonstrate that presenting variations significantly increases users' ability to identify unreliable claims (by 4.9x using our approach compared to single descriptions) and significantly decreases perceived reliability of MLLM responses.14 of 15 participants preferred seeing variations of MLLM responses over a single description, and all expressed interest in using our system for tasks from understanding a tornado's path to posting an image on social media.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3663547.3746393

Cites

15 of 15 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Numerous studies have examined how VLMs can support BLV people's visual access needs, but they largely sidestep image quality issues by asking for better photos or leaving users to triangulate facts across multiple models.

    “While numerous studies have examined how VLMs can support BLV people's visual access needs, they largely sidestep image quality issues by asking for better photos (e.g., Seeing AI, Be My AI, [92]) or leaving users to triangulate facts across multiple models [27].”

    From "It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Models (2026) · cites this paper as [27]

    The Introduction and §2.2 report that BLV users cross-check image descriptions across tools and models. But §3.2.1 describes a system that aggregates descriptions and highlights model disagreements to make that comparison easier, rather than simply leaving users to triangulate facts unaided. The paper also includes image quality as a source of ambiguity in §4.2; it does not support the suggestion that its own approach sidesteps image quality issues. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 52

  1. Ali Abdolrahmani, William Easley, Michele Williams, Stacy Branham, and Amy Hurst. 2017. Embracing Errors: Examining How Context of Use Impacts Blind Individuals' Acceptance of Navigation Aid Errors. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI '17). Association for Computing Machinery, New York, NY, USA, 4158-4169. https://doi.org/10.1145/3025453.3025528 · DOI
  2. Rudaiba Adnin and Maitraye Das. 2024. "I look at it as the king of knowledge": How Blind People Use and Understand Generative AI Tools. In Proceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility (St.
  3. John's, NL, Canada) (ASSETS '24). Association for Computing Machinery, New York, NY, USA, Article 64, 14 pages. https://doi.org/10.1145/3663548.3675631 · DOI · Paper page
  4. Rudaiba Adnin and Maitraye Das. 2024. "I look at it as the king of knowledge": How Blind People Use and Understand Generative AI Tools. ACM ASSETS 16, 54 (2024), 92.
  5. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. 2016. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565 (2016). · arXiv
  6. Akari Asai and Hannaneh Hajishirzi. 2020. Logic-Guided Data Augmentation and Regularization for Consistent Question Answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 5642-5650. https://doi.org/10.18653/v1/2020. acl-main.499 · DOI
  7. Mauro Avila, Katrin Wolf, Anke Brock, and Niels Henze. 2016. Remote assistance for blind users in daily life: A survey about be my eyes. In Proceedings of the 9th ACM International Conference on PErvasive Technologies Related to Assistive Environments. 1-2.
  8. Cynthia L Bennett, Jane E, Martez E Mott, Edward Cutrell, and Meredith Ringel Morris. 2018. How teens with visual impairments take, edit, and share photos on social media. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1-12.
  9. Cynthia L. Bennett, Cole Gleason, Morgan Klaus Scheuerman, Jeffrey P. Bigham, Anhong Guo, and Alexandra To. 2021. "It's Complicated": Negotiating Accessibil- ity and (Mis)Representation in Image Descriptions of Race, Gender, and Disability. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, Yokohama Japan, 1-19. https://doi.org/10.1145/3411764.3445498 · DOI · Paper page
  10. Michele A. Burton, Erin Brady, Robin Brewer, Callie Neylan, Jeffrey P. Bigham, and Amy Hurst. 2012. Crowdsourcing subjective fashion advice using VizWiz: challenges and opportunities. In Proceedings of the 14th International ACM SIGAC- CESS Conference on Computers and Accessibility (Boulder, Colorado, USA) (AS- SETS '12). Association for Computing Machinery, New York, NY, USA, 135-142. https://doi.org/10.1145/2384916.2384941 · DOI
  11. Ruei-Che Chang, Yuxuan Liu, and Anhong Guo. 2024. WorldScribe: Towards Context-Aware Live Visual Descriptions. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (Pittsburgh, PA, USA) (UIST '24). Association for Computing Machinery, New York, NY, USA, Article 140, 18 pages. https://doi.org/10.1145/3654777.3676375 · DOI
  12. Ruei-Che Chang, Yuxuan Liu, Lotus Zhang, and Anhong Guo. 2024. EditScribe: Non-Visual Image Editing with Natural Language Verification Loops. https: //doi.org/10.1145/3663548.3675599 arXiv:2408.06632 [cs]. · DOI
  13. Ruijia Chen, Junru Jiang, Pragati Maheshwary, Brianna R. Cochran, and Yuhang Zhao. 2025. VisiMark: Characterizing and Augmenting Landmarks for People with Low Vision in Augmented Reality to Support Indoor Navigation. https: //doi.org/10.1145/3706598.3713847 arXiv:2502.10561 [cs]. · DOI
  14. Furui Cheng, Vilém Zouhar, Simran Arora, Mrinmaya Sachan, Hendrik Strobelt, and Mennatallah El-Assady. 2024. Relic: Investigating large language model responses using self-consistency. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1-18.
  15. Yoonseo Choi, Eun Jeong Kang, Seulgi Choi, Min Kyung Lee, and Juho Kim. 2025. Proxona: Supporting Creators' Sensemaking and Ideation with LLM-Powered Audience Personas. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI '25). Association for Computing Machinery, New York, NY, USA, Article 149, 32 pages. https://doi.org/10.1145/3706598.3714034 · DOI
  16. Roi Cohen, May Hamri, Mor Geva, and Amir Globerson. 2023. LM vs LM: Detect- ing Factual Errors via Cross Examination. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 12621-12640. https://doi.org/10.18653/v1/2023.emnlp-main.778 · DOI · Paper page
  17. Rachel Gardner, Maya Varma, Clare Zhu, and Ranjay Krishna. 2020. Determining question-answer plausibility in crowdsourced datasets using multi-task learning. arXiv preprint arXiv:2011.04883 (2020). · arXiv
  18. Katy Ilonka Gero, Chelse Swoopes, Ziwei Gu, Jonathan K. Kummerfeld, and Elena L. Glassman. 2024. Supporting Sensemaking of Large Language Model Outputs at Scale. http://arxiv.org/abs/2401.13726 arXiv:2401.13726 [cs]. · arXiv
  19. Ricardo E Gonzalez Penuela, Jazmin Collins, Cynthia Bennett, and Shiri Azenkot. 2024. Investigating Use Cases of AI-Powered Scene Description Applications for Blind and Low Vision People. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1-21.
  20. Ziwei Gu, Ian Arawjo, Kenneth Li, Jonathan K. Kummerfeld, and Elena L. Glassman. 2024. An AI-Resilient Text Rendering Technique for Reading and Skimming Documents. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '24). Association for Computing Machinery, New York, NY, USA, Article 898, 22 pages. https: //doi.org/10.1145/3613904.3642699 · DOI
  21. Ananya Gubbi Mohanbabu and Amy Pavel. 2024. Context-Aware Image Descrip- tions for Web Accessibility. In The 26th International ACM SIGACCESS Conference on Computers and Accessibility. 1-17.
  22. Jaylin Herskovitz, Andi Xu, Rahaf Alharbi, and Anhong Guo. 2023. Hacking, switching, combining: understanding and supporting DIY assistive technology design by blind people. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1-17.
  23. Jonggi Hong and Hernisa Kacorri. 2024. Understanding How Blind Users Handle Object Recognition Errors: Strategies and Challenges. arXiv preprint arXiv:2408.03303 (2024). · arXiv
  24. Mina Huh, Fangyuan Xu, Yi-Hao Peng, Chongyan Chen, Hansika Murugu, Danna Gurari, Eunsol Choi, and Amy Pavel. 2024. Long-Form Answers to Visual Ques- tions from Blind and Low Vision People. arXiv preprint arXiv:2408.06303 (2024). · arXiv
  25. Mina Huh, Saelyne Yang, Yi-Hao Peng, Xiang'Anthony' Chen, Young-Ho Kim, and Amy Pavel. 2023. AVscript: Accessible Video Editing with Audio-Visual Scripts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1-17.
  26. Peiling Jiang, Jude Rayan, Steven P. Dow, and Haijun Xia. 2023. Graphologue: Ex- ploring Large Language Model Responses with Interactive Diagrams. In Proceed- ings of the 36th Annual ACM Symposium on User Interface Software and Technology (San Francisco, CA, USA) (UIST '23). Association for Computing Machinery, New York, NY, USA, Article 3, 20 pages. https://doi.org/10.1145/3586183.3606737 · DOI
  27. Patrick Gage Kelley, Joanna Bresee, Lorrie Faith Cranor, and Robert W. Reeder. 2009. A "nutrition label" for privacy. In Proceedings of the 5th Symposium on Usable Privacy and Security (SOUPS '09). Association for Computing Machinery, New York, NY, USA, 1-12. https://doi.org/10.1145/1572532.1572538 · DOI
  28. Matthias Kellner and Michele Ceriotti. 2024. Uncertainty quantification by direct propagation of shallow ensembles. https://doi.org/10.48550/arXiv.2402.16621 arXiv:2402.16621 [physics] version: 1. · DOI
  29. Masaki Kuribayashi, Kohei Uehara, Allan Wang, Shigeo Morishima, and Chieko Asakawa. 2025. WanderGuide: Indoor Map-less Robotic Guide for Exploration by Blind People. https://doi.org/10.1145/3706598.3713788 arXiv:2502.08906 [cs]. · DOI
  30. Jaewook Lee, Jaylin Herskovitz, Yi-Hao Peng, and Anhong Guo. 2022. Image- Explorer: Multi-Layered Touch Exploration to Encourage Skepticism Towards Imperfect AI-Generated Image Captions. In Proceedings of the 2022 CHI Confer- ence on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI '22). Association for Computing Machinery, New York, NY, USA, Article 462, 15 pages. https://doi.org/10.1145/3491102.3501966 · DOI · Paper page
  31. Tomer Levinboim, Ashish V Thapliyal, Piyush Sharma, and Radu Soricut. 2019. Quality estimation for image captions based on large-scale human evaluations. arXiv preprint arXiv:1909.03396 (2019). · arXiv
  32. Xinyu Li, Ruiyang Zhou, Zachary C. Lipton, and Liu Leqi. 2024. Personalized Language Modeling from Personalized Human Feedback. https://doi.org/10. 48550/arXiv.2402.05133 arXiv:2402.05133 [cs]. · DOI
  33. Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. 2023. Evaluating Object Hallucination in Large Vision-Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 292-305. https://doi.org/10.18653/v1/2023.emnlp-main.20 · DOI
  34. Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2024. Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models. http://arxiv. org/abs/2305.19187 arXiv:2305.19187 [cs, stat]. · arXiv
  35. Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. 2024. A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253 (2024). · arXiv
  36. Haley MacLeod, Cynthia L Bennett, Meredith Ringel Morris, and Edward Cutrell. 2017. Understanding blind people's experiences with computer-generated cap- tions of social media images. In proceedings of the 2017 CHI conference on human factors in computing systems. 5988-5999.
  37. Jennifer Mankoff, Janice Light, James Coughlan, Christian Vogler, Abraham Glasser, Gregg Vanderheiden, and Laura Rice. 2025. Accessibility Considerations in the Development of an AI Action Plan. https://doi.org/10.48550/arXiv.2503. 14522 arXiv:2503.14522 [cs]. · DOI
  38. Meredith Ringel Morris. 2020. AI and Accessibility: A Discussion of Ethical Considerations. Commun. ACM 63, 6 (May 2020), 35-37. https://doi.org/10.1145/ 3356727 arXiv:1908.08939 [cs]. · DOI · Paper page
  39. Vishnu Nair, Hanxiu 'Hazel' Zhu, and Brian A. Smith. 2023. ImageAssist: Tools for Enhancing Touchscreen-Based Image Exploration Systems for Blind and Low Vision Users. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI '23). Association for Computing Machinery, New York, NY, USA, Article 76, 17 pages. https://doi.org/10.1145/ 3544548.3581302 · DOI · Paper page
  40. Zheng Ning, Brianna L Wimer, Kaiwen Jiang, Keyi Chen, Jerrick Ban, Yapeng Tian, Yuhang Zhao, and Toby Jia-Jun Li. 2024. SPICA: Interactive Video Content Exploration through Augmented Audio Descriptions for Blind or Low-Vision Viewers. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1-18.
  41. Amy Pavel, Gabriel Reyes, and Jeffrey P. Bigham. 2020. Rescribe: Authoring and Automatically Editing Audio Descriptions. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST '20). Association for Computing Machinery, New York, NY, USA, 747-759. https://doi.org/10.1145/3379337.3415864 · DOI
  42. Yi-Hao Peng, Ding Li, P Jeffrey Bigham, and Amy Pavel. 2025. Morae: Proactively Pausing UI Agents for User Choices. In Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology.
  43. Marco Tulio Ribeiro, Carlos Guestrin, and Sameer Singh. 2019. Are red roses red? evaluating consistency of question-answering models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 6174-6184.
  44. Sangho Suh, Bryan Min, Srishti Palani, and Haijun Xia. 2023. Sensecape: Enabling Multilevel Exploration and Sensemaking with Large Language Models. In Proceed- ings of the 36th Annual ACM Symposium on User Interface Software and Technology (San Francisco, CA, USA) (UIST '23). Association for Computing Machinery, New York, NY, USA, Article 1, 18 pages. https://doi.org/10.1145/3586183.3606756 · DOI · Paper page
  45. Sree Harsha Tanneru, Chirag Agarwal, and Himabindu Lakkaraju. 2024. Quanti- fying uncertainty in natural language explanations of large language models. In International Conference on Artificial Intelligence and Statistics. PMLR, 1072-1080.
  46. Tess Van Daele, Akhil Iyer, Yuning Zhang, Jalyn C Derry, Mina Huh, and Amy Pavel. 2024. Making Short-Form Videos Accessible with Hierarchical Video Summaries. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1-17.
  47. Helena Vasconcelos, Matthew Jörke, Madeleine Grunde-McLaughlin, Tobias Gerstenberg, Michael Bernstein, and Ranjay Krishna. 2023. Explanations Can Reduce Overreliance on AI Systems During Decision-Making. https://doi.org/ 10.48550/arXiv.2212.06823 arXiv:2212.06823 [cs]. · DOI · Paper page
  48. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 (2022). · arXiv
  49. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. https://doi.org/10.48550/arXiv.2201.11903 arXiv:2201.11903 [cs]. · DOI · Paper page
  50. Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2024. A survey on multimodal large language models. National Science Review 11, 12 (Nov. 2024). https://doi.org/10.1093/nsr/nwae403 · DOI · Paper page
  51. Yunfeng Zhang, Q. Vera Liao, and Rachel K. E. Bellamy. 2020. Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (FAT* '20). Association for Computing Machinery, New York, NY, USA, 295-305. https://doi.org/10.1145/3351095.3372852 · DOI · Paper page
  52. Yichun Zhao, Miguel A Nacenta, Mahadeo A. Sukhai, and Sowmya Somanath. 2024. TADA: Making Node-link Diagrams Accessible to Blind and Low-Vision People. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI '24). Association for Computing Machinery, New York, NY, USA, 1-20. https://doi.org/10.1145/3613904.3642222 · DOI

Cited by

Sorted by citations

Authors
Chen, M., Iyer, A., & Pavel, A.
Affiliations
  • The University of Texas at Austin (US)
  • University of California, Berkeley (US)
ORCID
Venue
Proceedings of the 27th International ACM SIGACCESS Conference on Computers and Accessibility
Publisher
ACM
Published
2025-10-22
Pages
1-17
DOI
10.1145/3663547.3746393
OpenAlex
W4415445023
Cited by
6
References
15
Citations by year
2026: 5 · 2025: 1
FWCI
7.35
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
Notre Dame-IBM Technology Ethics Lab
Topics
  • Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
  • Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
Keywords
multimodal large language models, blind and visually impaired users, assistive technology design, user study

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review