"It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Models

May combine two claims.

Numerous studies have examined how VLMs can support BLV people's visual access needs, but they largely sidestep image quality issues by asking for better photos or leaving users to triangulate facts across multiple models.

“While numerous studies have examined how VLMs can support BLV people's visual access needs, they largely sidestep image quality issues by asking for better photos (e.g., Seeing AI, Be My AI, [92]) or leaving users to triangulate facts across multiple models [27].”

• cites [92] [92] Maniratnam Mandal, Deepti Ghadiyaram, Danna Gurari, and Alan C. Bovik. 2023. Helping Visually Impaired People Take Better Quality Pictures. 32 (2023), 3873-3884. doi:10.1109/TIP.2023.3282067 - sourceDisputed

  The Abstract and Section B, “Feedback to Assist Visually Impaired Users” (pages unavailable), say this paper directly addresses image-quality issues: it predicts distortions and offers specific guidance on blur, lighting, and camera movement. It is therefore contrary to presenting this paper as an example of work that sidesteps image quality by merely asking users for better photos. (LLM, conf=0.88)

  Checked by a reasoning model — may be wrong.

  Experimental — accuracy not yet measured
• cites [27] [27] Meng Chen, Akhil Iyer, and Amy Pavel. 2025. Surfacing Variations to Calibrate Perceived Reliability of MLLM-generated Image Descriptions. In Proceedings of the 27th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '25) . Association for Computing Machinery, New York, NY, USA, 1-17. doi:10.1145/3663547.3746393 - sourcePartly supported

  The Introduction and §2.2 report that BLV users cross-check image descriptions across tools and models. But §3.2.1 describes a system that aggregates descriptions and highlights model disagreements to make that comparison easier, rather than simply leaving users to triangulate facts unaided. The paper also includes image quality as a source of ambiguity in §4.2; it does not support the suggestion that its own approach sidesteps image quality issues. (LLM, conf=0.83)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Prior work indicates BLV people expect error-prone output from AI tools.

“Although prior work indicates BLV people expect error-prone output from AI tools [3, 9, 120]”

• cites [3] [3] Rudaiba Adnin and Maitraye Das. 2024. 'I Look at It as the King of Knowledge': How Blind People Use and Understand Generative AI Tools. In The 26th International ACM SIGACCESS Conference on Computers and Accessibility (St. John's NL Canada, 2024-10-27). ACM, 1-14. doi:10.1145/3663548.3675631 - sourceNot checked — no full text of the source
• cites [9] [9] Rahaf Alharbi, Pa Lor, Jaylin Herskovitz, Sarita Schoenebeck, and Robin N. Brewer. 2024. Misfitting With AI: How Blind People Verify and Contest AI Errors. In Proceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility (New York, NY, USA, 2024-10-27) (ASSETS '24) . Association for Computing Machinery, 1-17. doi:10.1145/3663548.3675659 - sourceSupported
• cites [120] [120] Xinru Tang, Ali Abdolrahmani, Darren Gergle, and Anne Marie Piper. 2025. Everyday Uncertainty: How Blind People Use GenAI Tools for Information Access. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems . 1-17. - sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

This finding aligns with Hong and Kacorri's findings on overall verification frequency.

“This aligns with Hong and Kacorri's fi ndings on the overall verification frequency [61].”

• cites [61] [61] Jonggi Hong and Hernisa Kacorri. 2024. Understanding How Blind Users Handle Object Recognition Errors: Strategies and Challenges. In Proceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility (New York, NY, USA, 2024-10-27) (ASSETS '24) . Association for Computing Machinery, 1-15. doi:10.1145/3663548.3675635 - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

The prevalence of low-quality images has made image quality assessment a stand-alone task in developing image captioning tools for BLV individuals.

“The prevalence of low-quality images has made image quality assessment a stand-alone task in developing image captioning tools for BLV individuals [30].”

• cites [30] [30] Tai-Yin Chiu, Yinan Zhao, and Danna Gurari. 2020. Assessing Image Quality Issues for Real-World Problems. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . 3643-3653. doi:10.1109/CVPR42600.2020.00370 - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

When VLMs fail, failures are not due to inference-time or training-time issues but rather to limited data frequency for the objects the model is trying to identify.

“When VLMs fail, recent work suggests that failures are not due to inference-time (e.g., prompts; decoding strategies) or training-time issues (e.g., learning objective) but rather to limited data frequency for the objects the model is trying to identify [148].”

• cites [148] [148] Yuhui Zhang, Alyssa Unell, Xiaohan Wang, Dhruba Ghosh, Yuchang Su, Ludwig Schmidt, and Serena Yeung-Levy. 2024. Why Are Visually-Grounded Language Models Bad at Image Classification? Advances in Neural Information Processing Systems 37 (December 2024), 51727-51753. - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Mann-Whitney U Test was used for inferential statistics because the data being compared are ordinal (Likert scale).

“We use a Mann-Whitney U Test for inferential statistics because the data being compared are ordinal (Likert scale) [93].”

• cites [93] [93] H. B. Mann and D. R. Whitney. 1947. On a Test of Whether One of Two Random Variables Is Stochastically Larger than the Other. 18, 1 (1947), 50-60. jstor:2236101 https://www.jstor.org/stable/2236101 - sourcePartly supported

  On p. 1, Mann and Whitney define U using the relative ranks of two samples, supporting the citation’s identification of a rank-based test. The paper does not discuss ordinal or Likert-scale data as a reason to use it; its stated derivation assumes continuous distributions and no ties. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

May combine two claims.

Initial studies have examined the negative impact of visual variations and the effect of synthetic image degradation on captioning output.

“Initial studies have examined the negative impact of visual variations [43] and the effect of synthetic image degradation [57, 107] on captioning output, but the literature on systematically understanding the impact of real-world image distortions on captioning accuracy is limited.”

• cites [43] [43] Zhiyuan Fan, Yumeng Wang, Sandeep Polisetty, and Yi R. Fung. 2025. Unveiling the Lack of LVLM Robustness to Fundamental Visual Variations: Why and Path Forward. In Findings of the Association for Computational Linguistics: ACL 2025 , Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, Vienna, Austria, 20222-20242. doi:10.18653/v1/2025.findings-acl.1037 - sourceSupported
• cites [57] [57] Dan Hendrycks and Thomas Dietterich. 2019. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. In International Conference on Learning Representations . -Not checked yet

  Not checked in the free run.

• cites [107] [107] Jielin Qiu, Yi Zhu, Xingjian Shi, Florian Wenzel, Zhiqiang Tang, Ding Zhao, Bo Li, and Mu Li. 2024. Benchmarking Robustness of Multimodal Image-Text Models under Distribution Shift. Journal of Data-centric Machine Learning Research} (Jan. 2024). doi:10.48550/arXiv.2212.08044 arXiv:2212.08044 [cs] - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Modern VLMs are highly performant on zero-shot image identification benchmarks, such as ImageNet and MS COCO, which cover a broad range of objects.

“Modern VLMs are highly performant on zero-shot image identification benchmarks, such as ImageNet [37] and MS COCO [28, 82], which cover a broad range of objects [85].”

• cites [85] [85] Huan Liu, Lingyu Xiao, Jiangjiang Liu, Xiaofan Li, Ze Feng, Sen Yang, and Jingdong Wang. 2024. Revisiting MLLMs: An In-Depth Analysis of Image Classification Abilities. CoRR abs/2412.16418 (December 2024). - sourcePartly supported

  The Introduction reports strong image-classification performance by recent multimodal models, and the CLIP-style section lists ImageNet among datasets used for general classification. The paper does not evaluate MS COCO as an image-classification benchmark, so it does not support the claim about model performance on that dataset. Page numbers are unavailable in our copy. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

May combine two claims.

Blind users often struggle to take high-quality photographs, and many visual questions go unanswered because images fail to capture the relevant objects.

“From analyzing VizWiz images, Gurari et al. [55] found that blind users often struggle to take high-quality photographs, and many visual questions go unanswered because images fail to capture the relevant objects [55].”

• cites [55] [55] Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. 2018. VizWiz Grand Challenge: Answering Visual Questions from Blind People. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . 3608-3617. doi:10.1109/CVPR.2018.00380 - sourceSupported

May combine two claims.

Adding images to caption metrics or using reference-free measures can help, but degraded images can be a confound in their scores.

“Adding images to such measures (e.g., Vilbertscore [72], TIGEr [65], SCAN [75]) or using reference-free measures (e.g., CLIPScore [59]) can help, but degraded images can be a confound in their scores.”

• cites [72] [72] Hwanhee Lee, Seunghyun Yoon, Franck Dernoncourt, Doo Soon Kim, Trung Bui, and Kyomin Jung. 2020. Vilbertscore: Evaluating image caption using vision-and-language bert. In Proceedings of the fi rst workshop on evaluation and comparison of NLP systems . 34-39. - sourcePartly supported

  The Introduction and §3 identify ViLBERTScore as a caption metric that incorporates the image, and the Evaluation Methods report improved correlation with human judgments. The paper does not establish that degraded images confound its scores; the reference-free example is attributed to a different citation. (LLM, conf=0.83)

  Experimental — accuracy not yet measured
• cites [65] [65] Ming Jiang, Qiuyuan Huang, Lei Zhang, Xin Wang, Pengchuan Zhang, Zhe Gan, Jana Diesner, and Jianfeng Gao. 2019. TIGEr: Text-to-Image Grounding for Image Caption Evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (Eds.). Association for Computational Linguistics, Hong Kong, China, 2141-2152. doi:10.18653/v1/D19-1220 - sourceSupported
• cites [75] [75] Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. 2018. Stacked cross attention for image-text matching. In Proceedings of the European conference on computer vision (ECCV) . 201-216. - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Llama and Molmo both exhibit comparable performance to closed-source industry models on benchmarks.

“Llama from Meta [53] 4 and Molmo from the Allen Institute for AI [35], which both exhibit comparable performance to closed-source industry models on benchmarks.”

• cites [53] [53] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nik sourceSupported
• cites [35] [35] Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, Andrew Head, Rose Hendrix, Favyen Bastani, Eli VanderBilt, Nathan Lambert, Yvonne Chou, Arnavi Chheda, Jenna Sparks, Sam Skjonsberg, Michael Schmitz, Aaron Sarnat, Byron Bischoff, Pete Walsh, Chris Newell, Piper Wolters, Tanmay Gupta, Kuo-Hao Zeng, Jon Borchardt, Dirk Groeneveld, Crystal Nam, Sophie Lebrecht, Caitli sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Recent work on LLM abstention explores techniques based on self-consistency, in which the model evaluates its own outputs and level of uncertainty before returning a response, demonstrating good performance in question-answering settings.

“In contrast, recent work on LLM abstention explores techniques based on self-consistency, in which the model evaluates its own outputs and level of uncertainty before returning a response, demonstrating good performance in question-answering settings [32, 70, 91, 133].”

• cites [32] [32] Jeremy Cole, Michael Zhang, Daniel Gillick, Julian Eisenschlos, Bhuwan Dhingra, and Jacob Eisenstein. 2023. Selectively Answering Ambiguous Questions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 530-543. doi:10.18653/v1/2023.emnlpmain.35 - sourceSupported
• cites [70] [70] Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation. In The Eleventh International Conference on Learning Representations . - sourcePartly supported

  Section 6.1 reports that semantic entropy uses multiple generated answers to estimate uncertainty and predicts answer correctness on CoQA and TriviaQA better than the tested baselines. Section 6 frames this as predicting whether an answer should be trusted, but the paper does not demonstrate a system that abstains or evaluates uncertainty before returning a response. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites [91] [91] Potsawee Manakul, Adian Liusie, and Mark Gales. 2023. SelfCheckGPT: ZeroResource Black-Box Hallucination Detection for Generative Large Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 9004-9017. doi:10.18653/v1/ 2023.emnlp-main.557 - sourcePartly supported

  The Introduction (p. 2) describes SelfCheckGPT as comparing sampled responses to detect hallucinations, and Section 5.2 (p. 4) describes a question-answering method for measuring that consistency. But the paper evaluates hallucination detection on generated Wikipedia-style passages (Section 6, p. 5); it does not describe a model abstaining before returning a response or demonstrate performance in a question-answering setting. Question answering is used as a consistency-checking method, not the evaluated task. (LLM, conf=0.87)

  Experimental — accuracy not yet measured
• cites [133] [133] Yasin Abbasi Yadkori, Ilja Kuzborskij, David Stutz, András György, Adam Fisch, Arnaud Doucet, Iuliya Beloshapka, Wei-Hung Weng, Yao-Yuan Yang, Csaba Szepesvári, et al. 2024. Mitigating LLM Hallucinations via Conformal Abstention. arXiv preprint arXiv:2405.01563 (2024). - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Academic scholars and industry corporations have emphasized the pressing need for more disability-centered datasets.

“Although academic scholars and industry corporations have emphasized the pressing need for more disability-centered datasets [20, 38, 55, 79, 97, 111, 124]”

• cites [20] [20] Danielle Bragg, Naomi Caselli, Julie A Hochgesang, Matt Huenerfauth, Leah Katz-Hernandez, Oscar Koller, Raja Kushalnagar, Christian Vogler, and Richard E Ladner. 2021. The fate landscape of sign language ai datasets: An interdisciplinary perspective. ACM Transactions on Accessible Computing (TACCESS) 14, 2 (2021), 1-45. - sourcePartly supported

  Section 1.1 says sign-language AI lacks adequate data, and Section 7.1 describes researchers’ need for large datasets and companies’ interest in them. The paper supports a need for more sign-language datasets, but it does not establish the broader claim about disability-centered datasets generally or say that industry corporations have publicly emphasized that need. Page numbers are unavailable in our copy. (LLM, conf=0.85)

  Experimental — accuracy not yet measured
• cites [38] [38] Aashaka Desai, Lauren Berger, Fyodor Minakov, Nessa Milano, Chinmay Singh, Kriston Pumphrey, Richard Ladner, Hal Daumé III, Alex X Lu, Naomi Caselli, et al. 2023. ASL citizen: a community-sourced dataset for advancing isolated sign language recognition. Advances in Neural Information Processing Systems 36 (2023), 76893-76907. - sourcePartly supported

  The Introduction says existing sign-language datasets lack scale and diversity and describes calls by disability scholars for collaboration with Deaf communities and attention to real needs. This supports a need for better Deaf-community-centered data, but the paper does not report a call from industry corporations or establish the broader claim about disability-centered datasets generally. The cited passages have section labels but no page numbers. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites [55] [55] Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. 2018. VizWiz Grand Challenge: Answering Visual Questions from Blind People. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . 3608-3617. doi:10.1109/CVPR.2018.00380 - sourcePartly supported

  The Introduction identifies a gap in existing VQA datasets: none addressed images and questions from blind people. It introduces VizWiz to address that gap. This supports the academic call for disability-centered datasets, but the paper does not establish that industry corporations also emphasized the need. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites [79] [79] Qisheng Li and Shaomei Wu. 2024. 'I Want to Publicize My Stutter': Communityled Collection and Curation of Chinese Stuttered Speech Data. Proceedings of the ACM on Human-Computer Interaction 8, CSCW2 (2024), 1-27. -Not checked yet

  Not checked in the free run.

• cites [97] [97] Cecily Morrison, Martin Grayson, Rita Faia Marques, Daniela Massiceti, Camilla Longden, Linda Wen, and Edward Cutrell. 2023. Understanding Personalized Accessibility through Teachable AI: Designing and Evaluating Find My Things for People Who Are Blind or Low Vision. In Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility (New York, NY, USA, 2023-10-22) (ASSETS '23) . Association for Computing Machinery, 1-12. [doi:10.1145/3597638.3608395](https://doi.org/10.1145/3597638.3608395) - sourcePartly supported

  Section 2.3 (Few-Shot Learning; page unavailable) says research is shifting toward real-world data through the collection of new datasets and identifies ORBIT, recorded by people who are blind or low vision, as an example. This supports interest in disability-relevant datasets, but the paper does not state a pressing need for more disability-centered datasets or attribute that call to both academic scholars and industry corporations. (LLM, conf=0.81)

  Experimental — accuracy not yet measured
• cites [111] [111] Tanusree Sharma, Abigale Stangl, Lotus Zhang, Yu-Yun Tseng, Inan Xu, Leah Findlater, Danna Gurari, and Yang Wang. 2023. Disability-First Design and Creation of A Dataset Showing Private Visual Information Collected With People Who Are Blind. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23) . Association for Computing Machinery, New York, NY, USA, 1-15. doi:10.1145/3544548.3580922 - sourceNot checked — no full text of the source
• cites [124] [124] Lida Theodorou, Daniela Massiceti, Luisa Zintgraf, Simone Stumpf, Cecily Morrison, Edward Cutrell, Matthew Tobias Harris, and Katja Hofmann. 2021. Disability-first dataset creation: Lessons from constructing a dataset for teachable object recognition with blind and low vision data collectors. In Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility . 1-12. - sourcePartly supported

  Section 2.1 (page unavailable) says it is critical to create large disability-first datasets, and the Discussion (page unavailable) says AI innovation requires datasets reflecting people with disabilities. These passages support the need for such datasets, but they do not establish that industry corporations have emphasized it. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim. May combine two claims.

Kapur and Kreiss [67] demonstrates bias in reference-based metrics against BLV people, calling for evaluation methods based on user groups' specific needs.

“For example, Kapur and Kreiss [67] demonstrates bias in referencebased metrics against BLV people, calling for evaluation methods based on user groups' specific needs.”

• cites [67] [67] Rhea Kapur and Elisa Kreiss. 2024. Reference-Based Metrics Are Biased Against Blind and Low-Vision Users' Image Description Preferences. In Proceedings of the Third Workshop on NLP for Positive Impact . 308-314. - sourceSupported

May combine two claims.

Using other VLMs to synthetically generate annotations is a popular approach, but it is likely to perpetuate inaccuracies or biases that the model already has, rather than capturing important nuances.

“Using other VLMs to synthetically generate annotations is a popular approach [84, 119], but it is likely to perpetuate inaccuracies or biases that the model already has (see distribution shift [110]), rather than capturing important nuances.”

• cites [84] [84] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual Instruction Tuning. Advances in Neural Information Processing Systems 36 (December 2023), 34892-34916. - sourceNot checked — no full text of the source
• cites [119] [119] Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. 2024. Large Language Models for Data Annotation and Synthesis: A Survey. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguistics, Miami, Florida, USA, 930-957. doi:10.18653/v1/2024.emnlp-main.54 - sourcePartly supported

  Section 3 describes LLM-generated annotation as an area of significant interest, supporting the approach cited in the first clause. But the Introduction says the survey focuses on pure language models and does not extensively cover multimodal models, so it does not establish that using VLMs specifically is popular. The later claim about perpetuating inaccuracies or biases has a separate citation in the citing sentence. (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites [110] [110] Hope Schroeder, Deb Roy, and Jad Kabbara. 2025. Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective Tasks. In Findings of the Association for Computational Linguistics: ACL 2025 , Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, Vienna, Austria, 25771-25795. - sourcePartly supported

  The Related Work section says LLMs are used for annotation tasks and widely used in computational social science, supporting the claim that model-generated annotation is an established approach. The paper studies text-focused LLM annotation, however, not the use or popularity of VLMs specifically. Its Discussion also warns that LLM-assisted labels can homogenize judgments, but the citation in the sentence is attached to the approach’s popularity, not that later claim. (LLM, conf=0.83)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Expressions of uncertainty can meaningfully influence users' reliance on model outputs.

“Existing work shows that expressions of uncertainty can meaningfully influence users' reliance on model outputs [140].”

• cites [140] [140] Gal Yona, Roee Aharoni, and Mor Geva. 2024. Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguistics, Miami, Florida, USA, 7752-7764. doi:10.18653/v1/2024.emnlpmain.443 - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Recent research suggests that VLMs require fine-tuning for good performance in recognizing products and their attributes.

“For recognizing products and their attributes, recent research suggests that VLMs require fi ne-tuning for good performance [104, 125].”

• cites [104] [104] Vishnu Prabhakaran, Purav Aggarwal, Vishruit Kulshreshtha, Arunita Das, Sahini Venkata Sitaram Sruti, and Anoop Saladi. 2025. VIT-Pro: Visual Instruction Tuning for Product Images. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: Industry Track) , Weizhu Chen, Yi Yang, Mohammad Kachuee, and Xue-Yong Fu (Eds.). Association for Computational Linguistics, Albuquerque, New Mexico, 695-707. - sourceSupported
• cites [125] [125] Ameni Trabelsi, Maria Zontak, Yiming Qian, Brian Jackson, Suleiman Khan, and Umit Batur. 2025. What Matters When Building Vision Language Models for Product Image Analysis? (2025). - sourceNot checked — no full text of the source

May combine two claims.

Molmo already applies an overlapping cropping procedure in its training, which we would expect to make it more resistant to misframed images, but our findings demonstrate that further development is needed to address its sensitivity to image framing.

“For instance, Molmo already applies an overlapping cropping procedure in its training [35], which we would expect to make it more resistant to misframed images, but our fi ndings demonstrate that further development is needed to address its sensitivity to image framing.”

• cites [35] [35] Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, Andrew Head, Rose Hendrix, Favyen Bastani, Eli VanderBilt, Nathan Lambert, Yvonne Chou, Arnavi Chheda, Jenna Sparks, Sam Skjonsberg, Michael Schmitz, Aaron Sarnat, Byron Bischoff, Pete Walsh, Chris Newell, Piper Wolters, Tanmay Gupta, Kuo-Hao Zeng, Jon Borchardt, Dirk Groeneveld, Crystal Nam, Sophie Lebrecht, Caitli sourceSupported

May combine two claims.

Recent work for VLMs has explored techniques to repair captioning errors prior to returning them during the generation process or post-hoc verification, but these techniques can induce further errors during correction, rather than providing a higher-precision output that only includes details that are likely correct.

“While recent work for VLMs has explored techniques to repair captioning errors prior to returning them during the generation process (e.g., controlling what objects are mentioned [144]; strategically adjusting model weights [78, 108, 136, 141] or fi ne-tuning [22, 146]; sampling multiple patches [29]; guided decoding [150]; backtracking when uncertain [40, 131]) or post-hoc verification [139, 155]”

• cites [144] [144] Bohan Zhai, Shijia Yang, Chenfeng Xu, Sheng Shen, Kurt Keutzer, Chunyuan Li, and Manling Li. 2024. HallE-Control: Controlling Object Hallucination in Large Multimodal Models. doi:10.48550/arXiv.2310.01779 arXiv:2310.01779 [cs] - sourceSupported
• cites [78] [78] Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2024. Mitigating Object Hallucinations in Large VisionLanguage Models through Visual Contrastive Decoding. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2024), 1387213882. doi:10.1109/CVPR52733.2024.01316 - sourceSupported
• cites [108] [108] Sreetama Sarkar, Yue Che, Alex Gavin, Peter Anthony Beerel, and Souvik Kundu. 2025. Mitigating Hallucinations in Vision-Language Models through ImageGuided Head Suppression. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (Eds.). Association for Computational Linguistics, Suzhou, China, 12492-12511. doi:10.18653/v1/2025.emnlpmain.631 - sourcePartly supported

  Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured
• cites [136] [136] Le Yang, Ziwei Zheng, Boxu Chen, Zhengyu Zhao, Chenhao Lin, and Chao Shen. 2025. Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projection. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . 14635-14645. doi:10.1109/CVPR52734.2025.01364 - sourceSupported
• cites [141] [141] Dokyoon Yoon, Youngsook Song, and Woomyong Park. 2025. Stop Learning It All to Mitigate Visual Hallucination, Focus on the Hallucination Target. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2025), 4200-4208. doi:10.1109/CVPR52734.2025.00397 - sourceSupported
• cites [22] [22] Peter Carragher, Nikitha Rao, Abhinand Jha, R. Raghav, and Kathleen M. Carley. 2025. SegSub: Evaluating Robustness to Knowledge Conflicts and Hallucinations in Vision-Language Models. Workshop Proceedings of the 19th International AAAI Conference on Web and Social Media 2025 (June 2025), 27. doi:10.36190/2025.27 - sourcePartly supported

  Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured
• cites [146] [146] Jinrui Zhang, Teng Wang, Haigang Zhang, Ping Lu, and Feng Zheng. 2024. Reflective Instruction Tuning: Mitigating Hallucinations in Large Vision-Language Models. In Computer Vision -ECCV 2024: 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part LXVIII . Springer-Verlag, Berlin, Heidelberg, 196-213. doi:10.1007/978-3-031-73113-6\_12 - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

VLM performance is heavily shaped by post-training activities, including fine-tuning on specific tasks and diverse datasets, or training to provide answers in specific formats.

“VLM performance is heavily shaped by post-training activities, including fine-tuning on specific tasks (e.g., PixMoCap for captioning [35]) and diverse datasets [80], or training to provide answers in specific formats (e.g., instruction tuning [84]).”

• cites [35] [35] Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, Andrew Head, Rose Hendrix, Favyen Bastani, Eli VanderBilt, Nathan Lambert, Yvonne Chou, Arnavi Chheda, Jenna Sparks, Sam Skjonsberg, Michael Schmitz, Aaron Sarnat, Byron Bischoff, Pete Walsh, Chris Newell, Piper Wolters, Tanmay Gupta, Kuo-Hao Zeng, Jon Borchardt, Dirk Groeneveld, Crystal Nam, Sophie Lebrecht, Caitli sourceSupported
• cites [80] [80] Zhiqi Li, Guo Chen, Shilong Liu, Shihao Wang, Vibashan VS, Yishen Ji, Shiyi Lan, Hao Zhang, Yilin Zhao, Subhashree Radhakrishnan, Nadine Chang, Karan Sapra, Amala Sanjay Deshmukh, Tuomas Rintamaki, Matthieu Le, Ilia Karmanov, Lukas Voegtle, Philipp Fischer, De-An Huang, Timo Roman, Tong Lu, Jose M. Alvarez, Bryan Catanzaro, Jan Kautz, Andrew Tao, Guilin Liu, and Zhiding Yu. 2025. Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Models. doi:10.48550/arXiv.2501.14818 arXiv:2501.14818 [cs] - sourceSupported
• cites [84] [84] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual Instruction Tuning. Advances in Neural Information Processing Systems 36 (December 2023), 34892-34916. - sourceNot checked — no full text of the source

May combine two claims.

BLV people want key product information and visual details of the product and nearby objects in image captions.

“which BLV people want in captions (shown by our survey and prior work [67, 97])”

• cites [67] [67] Rhea Kapur and Elisa Kreiss. 2024. Reference-Based Metrics Are Biased Against Blind and Low-Vision Users' Image Description Preferences. In Proceedings of the Third Workshop on NLP for Positive Impact . 308-314. - sourceCould not verify

  Section 2.2 reports that BLV participants preferred longer descriptions and were especially sensitive to image context; Section 4.1 reiterates the length preference. Neither section addresses whether they want key product information or visual details of the product and nearby objects. Searches for product information and nearby objects found no supporting passage in our copy. (LLM, conf=0.72)

  Experimental — accuracy not yet measured
• cites [97] [97] Cecily Morrison, Martin Grayson, Rita Faia Marques, Daniela Massiceti, Camilla Longden, Linda Wen, and Edward Cutrell. 2023. Understanding Personalized Accessibility through Teachable AI: Designing and Evaluating Find My Things for People Who Are Blind or Low Vision. In Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility (New York, NY, USA, 2023-10-22) (ASSETS '23) . Association for Computing Machinery, 1-12. [doi:10.1145/3597638.3608395](https://doi.org/10.1145/3597638.3608395) - sourceCould not verify

  The Conclusion mentions personalized audio descriptions and captions only as a possible future application. The paper’s Dynamic Support and Findings sections concern teaching an app to recognize and locate personal objects; they do not report that BLV people want product information, visual details, or nearby objects in image captions. Searched for captions, product information, and nearby-object descriptions. (LLM, conf=0.78)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Cross-cultural bias is a significant limitation of VLMs perceived by BLV users.

“Previous research has identified cross-cultural bias as a significant limitation of VLMs perceived by BLV users [9].”

• cites [9] [9] Rahaf Alharbi, Pa Lor, Jaylin Herskovitz, Sarita Schoenebeck, and Robin N. Brewer. 2024. Misfitting With AI: How Blind People Verify and Contest AI Errors. In Proceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility (New York, NY, USA, 2024-10-27) (ASSETS '24) . Association for Computing Machinery, 1-17. doi:10.1145/3663548.3675659 - sourcePartly supported

  Section 4.1 identifies cross-cultural bias as an error reported by blind participants using AI visual-assistance technologies, including failures with non-English text and cultural artifacts. But the paper does not establish this as a limitation of VLMs specifically; its examples concern a broader range of technologies, including OCR and scene-description tools. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

May combine two claims.

Lighting (m=2.35), distance (m=2.22), and hand position (m=2.12) followed in how well tools helped assess quality issues.

“followed by lighting ( 𝑚 = 2 . 35, 𝑠 = 0 . 96), distance ( 𝑚 = 2 . 22, 𝑠 = 0 . 82), and hand position ( 𝑚 = 2 . 12, 𝑠 = 0 . 80).”

Probably not a substantial claim: may not pass S3 (load-bearing).

19.8% (n=17) said it took 5-9 minutes.

“or 5-9 minutes (19.8%, 𝑛 = 17).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Among general-purpose AI tools, participants reported using ChatGPT (38.4%, n=33), Ray-Ban Meta Glasses (29.1%, n=25), and other tools.

“Among general-purpose AI tools, participants reported using ChatGPT (38.4%, 𝑛 = 33), Ray-Ban Meta Glasses (29.1%, 𝑛 = 25), and other tools.”

May combine two claims.

The top accessibility-focused tools used by respondents to identify and understand products included Be My AI (76.7%, n=66), Microsoft Seeing AI (69.8%, n=60), and AI captioning built into screen readers (51.2%, n=44).

“The top accessibility-focused tools used by our respondents to identify and understand products included Be My AI (76.7%, 𝑛 = 66), Microsoft Seeing AI (69.8%, 𝑛 = 60), and AI captioning built into screen readers (51.2%, 𝑛 = 44); see Table 1.”

Probably not a substantial claim: may not pass S3 (load-bearing).

27.9% (n=24) said it took 0-1 minutes.

“followed by 0-1 minutes (27.9%, 𝑛 = 24)”

May combine two claims.

Hand placement and position (m=3.35), lighting (m=3.15), and rotation (m=3.13) followed in perceived impact on caption quality.

“followed by hand placement and position ( 𝑚 = 3 . 35, 𝑠 = 0 . 74), lighting ( 𝑚 = 3 . 15, 𝑠 = 0 . 71), and rotation ( 𝑚 = 3 . 13, 𝑠 = 0 . 74).”

The quote only partly states this claim.

44.2% (n=38) leaned towards using human-sighted assistance when accuracy mattered most.

“and accuracy (44.2%, 𝑛 = 38) mattered most”

Probably not a substantial claim: may not pass S3 (load-bearing).

Be My AI offered more detailed feedback on photos.

“While Be My AI offered more detailed feedback on photos, it also received mixed responses to its suggestions”

May combine two claims.

Survey respondents provided insight into how product design and packaging affect caption accuracy, confirming prior work regarding medical product packaging.

“Survey respondents also provided insight into how product design and packaging affect caption accuracy, confirming prior work regarding medical product packaging [33].”

• cites [33] [33] Nathan Davis, Bo Xie, and Danna Gurari. 2020. Quality of Images Showing Medication Packaging from Individuals with Vision Impairments: Implications for the Design of Visual Question Answering Applications. 57, 1 (2020), e251. doi:10.1002/pra2.251 - sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

Multiple participants suggested that the tool provide more detailed, real-time feedback on framing, lighting, and orientation.

“with multiple participants suggesting that the tool provide more detailed, real-time feedback on framing, lighting, and orientation.”

May combine two claims.

The final dataset contains 1,859 images annotated with product details, comprising 729 high-quality and 1,130 low-quality images.

“This process yielded a fi nal dataset of 1,859 images annotated with product details, comprising 729 high-quality and 1,130”

Probably not a substantial claim: may not pass S3 (load-bearing).

Feedback from Be My AI was only given after the photo was taken.

“feedback was only given after the photo was taken”

May combine two claims.

Participants described captions that were not accurate, such as recognizing a pregnancy test as a pen, boots as food, protein bars as stuffing mix, and green beans as spark plugs.

“participants described captions that were not accurate (e.g., recognizing a pregnancy test as a pen, a pair of boots as food item, protein bars as stuffing mix, green beans as spark plugs)”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Respondents emphasized frustration and that there is more work to be done.

“Others emphasized frustration (e.g., 'It's really frustrating that I have to go through so many hoops just to be able to fi nd out what's in a box or can') and that there is more work to be done”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

People became cautious about using AI in life-threatening situations and turned to people or to more tested technologies.

“Either way, people became cautious about using AI in life-threatening situations and turned to people or to more tested technologies”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

We identified 622 (33.5%) products with rounded labels, 126 (6.8%) with large text panels, and 49 (2.6%) with both characteristics.

“We identified 622 (33.5% of our dataset) products with rounded labels, 126 (6.8%) with large text panels, and 49 (2.6%) with both characteristics”

Probably not a substantial claim: may not pass S3 (load-bearing).

Respondents across the board emphasized accuracy and precision.

“Respondents across the board emphasized accuracy and precision”

May combine two claims.

Participants with mixed or negative comments often noted that the feedback was limited, lacked clear guidance on resolving issues, and still required trial and error.

“often noting that the feedback was limited, lacked clear guidance on resolving issues, and still required trial and error.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.

Respondents identified these errors based on their life experience or by asking sighted people.

“Respondents identified these errors based on their life experience (e.g., 'it seemed like it was taking me way too long to fi nish the prescriptions and then I called the pharmacy to verify' and 'It told me a name of medicine I knew I never”

May combine two claims.

Captions from crowdworkers in the VizWiz Image Captioning dataset varied in whether they correctly identified products and the level of detail provided, making it difficult to assess whether a VLM was performing poorly or if accurate product identification data was lacking for benchmarking.

“First, captions from crowdworkers varied in whether they correctly identified products and the level of detail provided, making it difficult to assess whether a VLM was performing poorly or if we lacked accurate product identification data to benchmark the model against.”

May combine two claims.

14 people shared positive comments about Seeing AI's beeps for image quality; 27 shared negative or mixed experiences; 21 had not used this feature.

“When asked about this feature, 14 people shared positive comments (e.g., 'Does a great job of letting me know when the object is in full view' and 'This really helps me when adjusting the angle of the camera and increases my confidence').”

• Also stated in: § 3.3 Results · p. 6 · plain

  “However, 27 people shared negative or mixed experiences with this feature, mentioning that it is not always accurate regarding alignment, it is hard to rotate objects to fi nd the barcode, making slight adjustments and holding one's hand steady is problematic, and the feedback can be misleading (e.g., forcing the camera to put a whole object in view when only a small portion is of interest).”

• Also stated in: § 3.3 Results · p. 6 · plain

  “The remainder stated that they had not used this feature ( 𝑛 = 21) or did not answer the question.”

May combine two claims.

All models showed a significant interaction between blur and framing, with positive coefficients (all p < 0.001).

“All models showed a significant interaction between blur and framing, with positive coefficients (all 𝑝 &lt; 0 . 001).”

May combine two claims.

Blurred images were the most likely to be incorrect, reducing the odds of correct product identification by 88.3%.

“Blurred images were the most likely to be incorrect, reducing the odds of correct product identification by 88.3%.”

May combine two claims.

Molmo had significant, positive interaction effects for blur by rotation (p < 0.05) and framing by rotation (p < 0.001).

“while Molmo had significant, positive interaction effects for blur by rotation ( 𝑝 &lt; 0 . 05) and framing by rotation ( 𝑝 &lt; 0 . 001).”

May combine two claims.

All image quality variables (blur, framing, and rotation) were statistically significant and negative, indicating that their presence increases the likelihood that the studied VLM would incorrectly identify a product.

“All image quality variables (blur, framing, and rotation) were statistically significant and negative, indicating that their presence increases the likelihood that the studied VLM would incorrectly identify a product.”

May combine two claims.

The interaction effect for rotation by rounded labels was significant (p < 0.01), with the drop in performance being less steep than when only one variable is true.

“the interaction effect for rotation by rounded labels was significant ( 𝑝 &lt; 0 . 01), with the drop in performance being less steep than when only one variable is true (similar to image quality interactions).”

May combine two claims.

The rounded Manwich sloppy joe can partially shows the 'M' from the logo and an image of prepared sloppy joe, but all models focused on the more visible tomatoes instead, inferring it was just tomato sauce.

“The rounded Manwich sloppy joe can partially shows the 'M' from the logo and an image of prepared sloppy joe, but all models focused on the more visible tomatoes instead, inferring it was just tomato sauce.”

May combine two claims.

In the Select Choice Chewy granola bars example, all models identified the product (granola bars) and variety (chewy, with chocolate chips), but only GPT correctly recognized the brand.

“In the second example of Select Choice Chewy granola bars, all models identified the product (granola bars) and variety (chewy, with chocolate chips), but only GPT correctly recognizes the brand.”

May combine two claims.

Framing was the second-most problematic image-quality issue across models, reducing the odds of correct product identification by 84.5%.

“Framing was the second-most problematic image-quality issue across models, reducing the odds of correct product identification by 84.5%.”

May combine two claims.

GPT and Gemini could fill in 'Tid' and 'DS' for Tide Pods, while Llama could fill in 'Tide' and Molmo filled in neither (despite recognizing it was laundry detergent).

“For example, GPT and Gemini could fi ll in 'Tid' and 'DS' for Tide Pods, while Llama could fi ll in 'Tide' and Molmo fi lled in neither (despite recognizing it was laundry detergent).”

May combine two claims.

GPT and Gemini correctly identify a box of Chewy Lemonhead & Friends candy, while Llama only identifies 'Lemon Head' and that it is candy, and Molmo misses sub-brand and variety details.

“GPT and Gemini correctly identify a box of Chewy Lemonhead &amp; Friends candy, while Llama only identifies 'Lemon Head' (missing '&amp; Friends' sub-brand) and that it is candy (missing 'chewy' variety). Molmo similarly misses sub-brand and variety details.”

May combine two claims.

Only GPT can correctly identify a can of Great Value light red Kidney Beans; Llama can identify the brand but not 'kidney beans'; Gemini and Molmo identify nothing correctly.

“only GPT can correctly identify a can of Great Value light red Kidney Beans; Llama can identify the brand, but not 'kidney beans'; Gemini and Molmo identify nothing correctly.”

May combine two claims.

For high-quality images, GPT recognized 98.5% of products and Gemini 95.7%.

“For high-quality images, GPT and Gemini performed well, recognizing 98.5% and 95.7% of products, respectively.”

May combine two claims.

When images have multiple distortions, GPT accuracy drops to 69.4%, Gemini to 66.9%, Llama to 36.1%, and Molmo to 27.8%.

“accuracy is even worse when images have multiple distortions, with GPT dropping to 69.4%, Gemini to 66.9%, Llama to 36.1%, and Molmo to 27.8%.”

May combine two claims.

No model correctly identified the ground beef as 90% lean, 10% fat, despite it being clearly visible in the upper left, and only Llama noticed the text.

“For instance, no model correctly identified the ground beef as 90% lean, 10% fat, despite it being clearly visible in the upper left, and only Llama noticed the text.”

May combine two claims.

Images with multiple degradations are common in our dataset, comprising nearly 60% of all low-quality images and 36% of the entire dataset.

“images with multiple degradations are common in our dataset, comprising nearly 60% of all low-quality images and 36% of the entire dataset.”

May combine two claims.

Rotation was the least problematic image quality issue, reducing the odds of correct product identification by 79.5%.

“Finally, rotation was the least problematic image quality issue, reducing the odds of correct product identification by 79.5%.”

May combine two claims.

Regression results reveal significant two-way interaction effects between blur and framing (p < 0.001), blur and rotation (p < 0.05), and framing and rotation (p < 0.001).

“The regression results reveal significant two-way interaction effects between blur and framing ( 𝑝 &lt; 0 . 001), blur and rotation ( 𝑝 &lt; 0 . 05), and framing and rotation ( 𝑝 &lt; 0 . 001).”

May combine two claims.

For product and variety details, GPT, Llama, and Molmo failed to identify the product (herb tea) and brand (Bigelow) of a K-Cup pod, whereas Gemini was correct.

“For product and variety details, we observed that GPT, Llama, and Molmo failed to identify the product (herb tea) and brand (Bigelow) of a K-Cup pod, whereas Gemini was correct.”

May combine two claims.

Additional image quality issues reduce performance, but not to the same extent as a single issue.

“inspection of this interaction plot reveals a similar pattern to the two-way interactions, where additional image quality issues reduce performance, but not to the same extent as a single issue.”

May combine two claims.

For high-quality images, Llama correctly recognized 86.1% and Molmo 86.8% of products.

“Accuracy for open-source models was slightly less, with Llama correctly recognizing 86.1% and Molmo 86.8%.”

May combine two claims.

Product images with both rounded labels and text panels had a greater impact on performance; GPT remained unaffected, while Gemini, Llama, and Molmo all dropped to 77.8% for high-quality images and similarly for low-quality images (67.5%, 42.5%, and 30.0%, respectively).

“product images with both rounded labels and text panels had a greater impact on performance. While GPT remained unaffected, Gemini, Llama, and Molmo all dropped to 77.8% for high-quality images and similarly for low-quality images (67.5%, 42.5%, and 30.0%, respectively).”

The quote only partly states this claim.

The studied closed-source models (GPT-4.1, Gemini) perform better on low-quality images than open-source models (Llama, Molmo).

“While the studied closed-source models (i.e., GPT-4.1, Gemini) perform better on low-quality images”

Probably not a substantial claim: may not pass S3 (load-bearing).

Participants mentioned their dexterity makes it difficult to orient the camera in particular ways.

“Others mentioned their dexterity makes it difficult to orient the camera in particular ways.”

May combine two claims.

All models had substantially lower performance on low-quality images, suggesting they could not find enough distinguishing characteristics in those images to support successful identification (as humans could).

“While performing well on high-quality images, all models had substantially lower performance on low-quality images, suggesting they could not fi nd enough distinguishing characteristics in those images to support successful identification (as humans could).”

May combine two claims.

Our study revealed that VLMs frequently produce believable product descriptions that are subtly incorrect, affecting their meaning (e.g., 'Coke Zero' versus 'Diet Coke').

“Our study revealed that VLMs frequently produce believable product descriptions that are subtly incorrect, affecting their meaning (e.g., 'Coke Zero' versus 'Diet Coke').”

May combine two claims.

Off-the-shelf VLMs perform well for U.S.-based products when product images are high-quality, suggesting that the issue is not due to the model's knowledge gaps.

“However, our analysis shows that off-theshelf VLMs perform well for U.S.-based products when product images are high-quality, suggesting that the issue is not due to the model's knowledge gaps.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Participants mentioned difficulty holding the camera steady enough and carefully controlling their breathing to prevent blur.

“For example, participants mentioned difficulty holding the camera steady enough and carefully controlling their breathing to prevent blur.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Survey participants raised concerns that people with multiple disabilities may find interventions to improve photo-taking more difficult.

“However, survey participants also raised concerns that people with multiple disabilities may fi nd such interventions more difficult.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Those 17 images were moved into the high-quality subset because they had no other issues.

“since they had no other issues, we moved these into the high-quality subset.”

May combine two claims.

The 17 high-quality images with rotation ≥ 4 were images that only had a rotation issue but were actually not rotated, as checked by two researchers.

“The 17 high-quality images with rotation ≥ 4 were images that only had a rotation issue (noted by the crowdworkers) but were actually not rotated, as checked by two researchers (see Section 4.2.2)”

Probably not a substantial claim: may not pass S3 (load-bearing).

The Honey Nut Cheerios example provides two alternate framings with varying amounts of the product text visible.

“The Honey Nut Cheerios example provides two alternate framings, with varying amounts of the text visible.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

For images with rotation issues, different vision-language models may only provide high-level information or incorrectly infer what the product is.

“Examples of images with rotation issues where different VLMs may only provide high-level information or incorrectly infer what the product is.”

May combine two claims.

Vision-language models often incorrectly read text panels or miss key information.

“VLMs can read text panels, but often incorrectly or miss key information.”

May combine two claims.

Rounded objects often obscure the label, requiring more inference about the product.

“Rounded objects often obscure the label, requiring more inference about the product, which humans do well, but VLMs still struggle with.”

May combine two claims.

For blurred product images, VLMs may only provide high-level information or incorrectly infer what the product is.

“Examples of blurred product images where VLMs may only provide high-level information or incorrectly infer what the product is.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Of the 1,859 images in the final dataset, 729 were high-quality images.

“with 729 images being high-quality images”

Probably not a substantial claim: may not pass S3 (load-bearing).

Images with multiple issues (blur, framing, and rotation) produce variations in product captions.

“Examples of images with multiple issues (blur, framing, and rotation) with variations in product captions.”

Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing), S4 (based). The quote only partly states this claim.

The final dataset for Study 2 included 1,859 images taken by BLV people.

“for our subset of 1.859 images”

Classified as probably not a substantial claim. May combine two claims.

The average scores for assessing each quality issue range from 'Very Little' and 'Somewhat', suggesting that both tools could do more to make quality issues apparent to BLV people.

“Notably, the average scores for assessing each quality issue range from 'Very Little' and 'Somewhat', suggesting that both tools could do more to make quality issues apparent to BLV people.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The survey study was approved by our university IRB.

“The survey study was approved by our university IRB.”

May combine two claims.

Existing metrics for measuring caption quality (BLEU, METEOR, ROUGE, CIDEr, SPICE, BERTScore) primarily measure text alignment and are unreliable for evaluating correctness of product information in captions.

“existing metrics for measuring caption quality-like BLEU [102], METEOR [12], ROUGE [81], CIDEr [128], SPICE [11], and BERTScore [147]-primarily measure text alignment and are unreliable for evaluating correctness of product information in captions.”

• cites [102] [102] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2001. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 (Philadelphia, Pennsylvania, 2001). Association for Computational Linguistics, 311. doi:10.3115/1073083.1073135 - sourceNot checked — no full text of the source
• cites [12] [12] Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization (Ann Arbor, Michigan, 2005-06), Jade Goldstein, Alon Lavie, Chin-Yew Lin, and Clare Voss (Eds.). Association for Computational Linguistics, 65-72. https://aclanthology.org/W05-0909/ -Not checked yet

  Not checked in the free run.

• cites [81] [81] Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out (Barcelona, Spain, 2004-07). Association for Computational Linguistics, 74-81. https://aclanthology.org/W04-1013/ -Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

Image quality issues affect whether VLMs can accurately identify products with the level of detail BLV people need for safe and effective use.

“Their fi nal reflections underscore the importance of evaluating how the image quality issues BLV people contend with daily-which are often set aside in research-affect whether VLMs can accurately identify products with the level of detail that BLV people need for safe and effective use.”

• AssumptionRests on an assumption: The image quality issues BLV people contend with daily are often set aside in research.

Probably not a substantial claim: may not pass S3 (load-bearing).

When users encounter delays or fail to receive meaningful responses, they may view the system as inaccurate or untrustworthy, even if the underlying model is functioning properly.

“When users encounter delays or fail to receive meaningful responses, they may view the system as inaccurate or untrustworthy, even if the underlying model is functioning properly [152].”

• cites [152] [152] Yuhang Zhao, Shaomei Wu, Lindsay Reynolds, and Shiri Azenkot. 2018. A Face Recognition Application for People with Visual Impairments: Understanding Use Beyond the Lab. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC Canada, 2018-04-21). ACM, 1-14. doi:10. 1145/3173574.3173789 - sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The survey was hosted via Google Forms, which is accessible to screen reader users.

“The survey was hosted via Google Forms, which is accessible to screen reader users”

Classified as probably not a substantial claim.

To ensure the validity of product annotations, a second researcher reviewed and confirmed agreement with each image and annotation.

“To ensure the validity of product annotations, a second researcher then reviewed and confirmed agreement with each image and annotation.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The survey instrument was revised over three iterations, including two researchers taking the survey multiple times and pilot testing with 20 participants.

“We revised the survey over three iterations. First, two researchers took the entire survey multiple times to check for language and length.”

• Also stated in: § 3.1 Method · p. 4 · plain

  “Then, we deployed the survey to 10 participants, including an open-ended question at the end that allowed participants to share any confusion or suggestions for improving the survey.”

• Also stated in: § 3.1 Method · p. 4 · plain

  “Following these corrections, we distributed the survey to another 10 participants. No major issues were noted at this stage, and we proceeded with the fi nal deployment.”

Probably not a substantial claim: may not pass S3 (load-bearing).

External factors, such as low-quality images or unreliable internet connectivity, often exacerbate perceived inaccuracies in image captioning.

“In addition, external factors, such as low-quality images or unreliable internet connectivity, often exacerbate perceived inaccuracies in image captioning [152].”

• cites [152] [152] Yuhang Zhao, Shaomei Wu, Lindsay Reynolds, and Shiri Azenkot. 2018. A Face Recognition Application for People with Visual Impairments: Understanding Use Beyond the Lab. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC Canada, 2018-04-21). ACM, 1-14. doi:10. 1145/3173574.3173789 - sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

After removing 11 responses for duplicates and quality issues, the final sample was 86 BLV participants.

“In total, eleven responses were removed, resulting in a fi nal sample of 86.”

May combine two claims.

Partially correct captions can be more difficult for BLV people to assess and cause potentially life-threatening issues.

“Partially correct captions can be more difficult for BLV people to assess and cause potentially life-threatening issues.”

Classified as probably not a substantial claim. May combine two claims.

Any discrepancies in product annotations were flagged for discussion, and if no agreement was reached, the image was removed from the dataset.

“Any discrepancies were fl agged for discussion, and if no agreement was reached, the image was removed from the dataset.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

All VLM captions were anonymized to minimize potential bias during coding, and image metadata was concealed except for product annotations.

“All VLM captions were anonymized to minimize potential bias during coding (i.e., Models A, B, C, D), and any image metadata (e.g., what quality issues were present) was concealed, except for the product annotations.”

May combine two claims.

We excluded variables for obstruction, too dark, and too bright from the analysis due to extreme class imbalance and insufficient true cases, which violate assumptions required for reliable model estimation.

“We excluded the variables for obstruction, too dark, and too bright from the analysis due to extreme class imbalance and an insufficient number of true cases, which violate the assumptions required for reliable model estimation.”

May combine two claims.

Davis et al. showed how medical packaging presents a challenging task for VLMs and is a case where knowing the correct medicine and dosage is critical.

“Davis et al. showed how medical packaging presents a challenging task for VLMs and is a case where knowing the correct medicine and dosage is critical [33].”

• cites [33] [33] Nathan Davis, Bo Xie, and Danna Gurari. 2020. Quality of Images Showing Medication Packaging from Individuals with Vision Impairments: Implications for the Design of Visual Question Answering Applications. 57, 1 (2020), e251. doi:10.1002/pra2.251 - sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S2 (checkable).

The stakes for misidentifying products are higher for BLV people, especially for products with health or safety-related issues.

“the stakes for misidentifying products are higher for BLV people, especially for products with health or safety-related issues.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The order of images was randomized to reduce ordering effects.

“The order of images was also randomized to reduce any ordering effects.”

Classified as probably not a substantial claim.

Inter-rater reliability for coding VLM captions was computed using Krippendorff's alpha, with an agreement of 0.859.

“IRR was computed using Krippendorff's alpha, with an agreement of 0.859.”

Classified as probably not a substantial claim. May combine two claims.

The researchers recoded rotation as an orientation beyond 45 degrees from the product's natural axis, marking the image as rotated if both agreed.

“Therefore, the researchers recoded rotation as an orientation beyond 45 degrees from the product's natural axis-depending on the product's top and bottom (such as a can) and text orientation-marking the image as rotated if both agreed.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

We observed no outliers in the dataset, nor evidence of multicollinearity in the final models (all VIF scores were less than five).

“We observed no outliers in the dataset, nor evidence of multicollinearity in the fi nal models (all variance inflation factor (VIF) scores were less than fi ve).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Recognizing products with rounded labels is generally challenging, as is identifying products from a large panel of text.

“Recognizing products with rounded labels is generally challenging [33], as is identifying products from a large panel of text.”

• cites [33] [33] Nathan Davis, Bo Xie, and Danna Gurari. 2020. Quality of Images Showing Medication Packaging from Individuals with Vision Impairments: Implications for the Design of Visual Question Answering Applications. 57, 1 (2020), e251. doi:10.1002/pra2.251 - sourceNot checked — no full text of the source

May combine two claims.

Seeing AI, Be My AI, and OpenAI's ChatGPT all use a GPT-4 class model from OpenAI.

“We include GPT-4.1 since the three most commonly used AI tools in our survey-Seeing AI [16], Be My AI [13], and OpenAI's ChatGPT [100]-all use a GPT-4 class model from OpenAI.”

• cites [16] [16] Andy Beatman and Ailsa Leen. 2024. 6 Ways Generative AI Helps Improve Accessibility for All with Azure. -Not checked yet

  Not checked in the free run.

• cites [13] [13] Be My Eyes. 2023. Introducing: Be My AI. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [100] [100] OpenAI. 2025. ChatGPT. https://chatgpt.com/. -Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Open-source models provide access to model architecture, training regime, and in some cases training data, affording greater flexibility for improving performance than closed-source models.

“open-source models provide access to the model architecture, training regime, and, in some cases, training data (e.g., Molmo), affording greater fl exibility for improving performance than closed-source models.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Two researchers observed that agreement of two crowdworkers on the presence of blur or framing issues effectively captured issue quality, but it did not for rotation.

“two researchers observed that while the agreement of two crowdworkers on the presence of blur or framing issues effectively captured issue quality, it did not for rotation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Four researchers spent more than three months reviewing, discussing, validating, and annotating low-quality images.

“In our case, four researchers spent more than three months reviewing, discussing, validating, and annotating low-quality images.”

Classified as probably not a substantial claim.

BLV people may still need to re-take photos even when VLM performance on low-quality images is improved.

“While we emphasize multiple ways to improve VLM performance on low-quality images, BLV people may still need to re-take photos, which participants in our study wanted better guidance on.”

Classified as probably not a substantial claim.

The annotation structure provides a pathway for annotating products.

“Our annotation structure provides a pathway for annotating products”

May combine two claims.

Open-source models make the model's architecture and training details available to researchers, while preserving privacy when run locally.

“In contrast, open-source models make the model's architecture and training details available to researchers 9 , while preserving privacy when run locally.”

May combine two claims.

The study was limited by the information available in images and could not reliably code expiration dates or product ingredient lists.

“we were still limited by the information available in images, and could not reliably code expiration dates or product ingredient lists”

Classified as probably not a substantial claim.

The research team manually reviewed and coded 7,436 model captions for accuracy and completeness, rather than relying on metrics that assess similarity and could lead to false positives.

“the research team manually reviewed and coded 7,436 model captions for accuracy and completeness, rather than relying on metrics that assess similarity and could lead to false positives (see Section 4.1).”

Probably not a substantial claim: may not pass S2 (checkable).

Given the difficulty BLV people reported in assessing errors, let alone the risk of mis-identification, more nuanced and consistent frameworks for data annotation and error analysis are crucial for reliable VLMs, especially for high-stakes uses.

“Given the difficulty BLV people reported in assessing errors, let alone the risk of mis-identification, more nuanced and consistent frameworks for data annotation and error analysis are crucial for reliable VLMs, especially for high-stakes uses, such as identifying food products, medications, and household cleaners.”

Classified as probably not a substantial claim. May combine two claims.

Closed-source models are limited to prompt engineering-which is insufficient for handling distorted images-and fine-tuning to improve performance.

“Closed-source models are limited to prompt engineering-which is insufficient for handling distorted images-and fi ne-tuning to improve performance.”

Probably not a substantial claim: may not pass S2 (checkable).

Open-source models (Llama, Molmo) are likely more fruitful for developing reliable VLMs that meet BLV people's needs.

“open-source models (i.e., Llama, Molmo) are likely more fruitful for developing reliable VLMs that meet BLV people's needs.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Open-source models enable the understanding of training procedures, which can aid in interpreting evaluation results.

“Opensource models also enable the understanding of training procedures, which can aid in interpreting evaluation results.”

Probably not a substantial claim: may not pass S2 (checkable).

Including representative quality variations that reflect the real-world conditions under which BLV people capture images can help us develop VLMs that are more resistant to such variations from the start.

“Including representative quality variations that reflect the real-world conditions under which BLV people capture images can help us develop VLMs that are more resistant to such variations from the start”

Classified as probably not a substantial claim. May combine two claims.

The study focused on evaluating product identification accuracy rather than the caption quality of VLMs generally.

“First, we focus on evaluating product identification accuracy rather than the caption quality of VLMs generally.”

May combine two claims.

Closed-source models may leak private data, compromising data privacy that survey respondents strongly desired.

“Moreover, closed-source models may leak private data [41, 86], compromising data privacy that our survey respondents strongly desired.”

• cites [41] [41] Emily Forlini. 2025. After Backlash, ChatGPT Removes Option to Have Private Chats Indexed by Google. PCMag (August 2025). -Not checked yet

  Not checked in the free run.

• cites [86] [86] Connie Loizos. 08-28-25. Anthropic Users Face a New Choice -Opt out or Share Your Chats for AI Training. TechCrunch (08-28-25). -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Simple techniques involving prompt engineering to abstain are of limited efficacy, with no guarantees that the instruction to abstain will be followed (e.g., best abstention prompting yields only 0.78 accuracy on question-answer tasks with similarly low-quality images).

“Simple techniques involving prompt engineering to abstain are of limited efficacy, with no guarantees that the instruction to abstain will be followed (e.g., best abstention prompting yields only 0.78 accuracy on question-answer tasks with similarly low-quality images [63]).”

• cites [63] [63] Mina Huh, Fangyuan Xu, Yi-Hao Peng, Chongyan Chen, Hansika Murugu, Danna Gurari, Eunsol Choi, and Amy Pavel. 2024. Long-Form Answers to Visual Questions from Blind and Low Vision People. In First Conference on Language Modeling . -Not checked yet

  Not checked in the free run.

May combine two claims.

Relying on crowdworkers is a common approach to annotation, but they may lack insight into disabled people's information needs and may apply varying standards of visual interpretation in BLV-focused datasets.

“Relying on crowdworkers is a common approach to annotation, but they may lack insight into disabled people's information needs and may apply varying standards of visual interpretation in BLV-focused datasets [113].”

• cites [113] [113] Rachel N. Simons, Danna Gurari, and Kenneth R. Fleischmann. 2020. 'I Hope This Is Helpful': Understanding Crowdworkers' Challenges and Motivations for an Image Description Task. 4 (2020), 1-26. Issue CSCW2. doi:10.1145/3415176 - sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

The VLMs studied would likely perform worse on product photos from a non-English-speaking country.

“These VLMs would likely perform worse on product photos from a non-English-speaking country.”

May combine two claims.

Framing affects product identification and resulting captions.

“Examples of images illustrating how framing affects product identification and resulting captions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The study reduced image quality issues to a binary variable.

“A second limitation is reducing image quality issues to a binary variable.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The experiment focused on VLMs and data with a U.S. and English-speaking bias.

“Finally, our experiment focused on VLMs and data with a U.S. and English-speaking bias.”