CoChat
Sign Up
Cited by 24Assistive Technology in Communication and MobilityDigital Accessibility for DisabilitiesTactile and Sensory Interactions

Beyond Visual Perception: Insights from Smartphone Interaction of Visually Impaired Users with Large Multimodal Models

Open access
Xie, Jingyi, Yu, Rui, Zhang, He, Billah, Syed Masum, Lee, Sooyeon, Carroll, John M.·2025·Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems
DOI 10.1145/3706598.3714210PMC PMC12338113
Open this paper in CoChat — free accountOpen PDF

Abstract

Large multimodal models (LMMs) have enabled new AI-powered applications that help people with visual impairments (PVI) receive natural language descriptions of their surroundings through audible text. We investigated how this emerging paradigm of visual assistance transforms how PVI perform and manage their daily tasks. Moving beyond basic usability assessments, we examined both the capabilities and limitations of LMM-based tools in personal and social contexts, while exploring design implications for their future development. Through interviews with 14 visually impaired users and analysis of image descriptions from both participants and social media using Be My AI (an LMM-based application), we identified two key limitations. First, these systems' context awareness suffers from hallucinations and misinterpretations of social contexts, styles, and human identities. Second, their intent-oriented capabilities often fail to grasp and act on users' intentions. Based on these findings, we propose design strategies for improving both human-AI and AI-AI interactions, contributing to the development of more effective, interactive, and personalized assistive technologies.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3706598.3714210

Cites

53 of 53 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • VLMs are increasingly used by blind and low-vision (BLV) people to identify and understand products in their everyday lives.

    “AI tools and their underlying VLMs are becoming more integral to how BLV people perform a variety of everyday tasks, including grocery shopping, cooking, cleaning, and personal care [9, 120, 132].”

    From "It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Models (2026) · cites this paper as [132]

    The Findings (Intent-Oriented Capabilities and Context Awareness; page unavailable) document visually impaired participants using Be My AI for cooking, household appliances, clothing, and makeup. The paper therefore supports use in several everyday tasks, but the passages do not establish increasing use over time or specifically document grocery shopping and product identification. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 12

  1. [4]Achiam Josh, Adler Steven, Agarwal Sandhini, Ahmad Lama, Akkaya Ilge, Aleman Florencia Leoni, Almeida Diogo, Altenschmidt Janko, Altman Sam, Anadkat Shyamal, 2023. GPT-4 Technical Report. arXiv:2303.08774 (2023). · arXiv
  2. [5]Adnin Rudaiba and Das Maitraye. 2024. “I look at it as the king of knowledge”: How Blind People Use and Understand Generative AI Tools. people 16, 54 (2024), 92.
  3. [7]Bai Yuntao, Jones Andy, Ndousse Kamal, Askell Amanda, Chen Anna, DasSarma Nova, Drain Dawn, Fort Stanislav, Ganguli Deep, Henighan Tom, 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 (2022). · arXiv
  4. [16]Caldwell Ben, Cooper Michael, Reid Loretta Guarino, Vanderheiden Gregg, Chisholm Wendy, Slatin John, and White Jason. 2008. Web content accessibility guidelines (WCAG) 2.0. WWW Consortium (W3C) 290, 1–34 (2008), 5– 12.
  5. [32]Kupferstein Elizabeth, Zhao Yuhang, Azenkot Shiri, and Rojnirun Hathaitorn. 2020. Understanding the use of artificial intelligence based visual aids for people with visual impairments. Investigative Ophthalmology & Visual Science 61, 7 (2020), 932– 932.
  6. [35]Lee Sooyeon, Reddie Madison, Gurdasani Krish, Wang Xiying, Beck Jordan, Rosson Mary Beth, and Carroll John M.. 2018. Conversations for Vision: Remote Sighted Assistants Helping People with Visual Impairments. arXiv:1812.00148 [cs.HC] · arXiv
  7. [42]Mozannar Hussein and Sontag David. 2020. Consistent estimators for learning to defer to an expert. In International Conference on Machine Learning. PMLR, 7076– 7087.
  8. [48]Raghu Maithra, Blumer Katy, Corrado Greg, Kleinberg Jon, Obermeyer Ziad, and Mullainathan Sendhil. 2019. The algorithmic automation problem: Prediction, triage, and human effort. arXiv preprint arXiv:1903.12220 (2019). · arXiv
  9. [56]Wang Xuezhi, Wei Jason, Schuurmans Dale, Le Quoc, Chi Ed, Narang Sharan, Chowdhery Aakanksha, and Zhou Denny. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 (2022). · arXiv
  10. [58]Wei Jason, Wang Xuezhi, Schuurmans Dale, Bosma Maarten, Xia Fei, Chi Ed, Le Quoc V, Zhou Denny, 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824– 24837.
  11. [64]Yu Weihao, Yang Zhengyuan, Li Linjie, Wang Jianfeng, Lin Kevin, Liu Zicheng, Wang Xinchao, and Wang Lijuan. 2023. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490 (2023). · arXiv
  12. [66]Zhao Yi, Zhang Yilin, Xiang Rong, Li Jing, and Li Hillming. 2024. VIALM: A Survey and Benchmark of Visually Impaired Assistance with Large Models. arXiv:2402.01735 (2024). · arXiv

Cited by

Sorted by citations

Related

Authors
Xie, J., Yu, R., Zhang, H., Billah, S. M., Lee, S., & Carroll, J. M.
Affiliations
  • Pennsylvania State University (US)
  • University of Louisville (US)
  • New Jersey Institute of Technology (US)
ORCID
Venue
Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2025-04-25
Volume
25
Pages
1-17
DOI
10.1145/3706598.3714210
PMC
PMC12338113
OpenAlex
W4409736063
Cited by
23
References
53
Citations by year
2026: 18 · 2025: 5
FWCI
65.54
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • CHI Conference on Human Factors in Computing Systems (publishedVersion)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Institutes of Health (R01 LM013330)
Topics
  • Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
  • Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
Keywords
visual impairment, large multimodal models, assistive technology, smartphone interaction, image description generation, hallucinations, context awareness, user intent understanding, human-AI interaction, social media

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review