CoChat
Sign Up
Cited by 24Multimodal Machine Learning ApplicationsDigital Accessibility for DisabilitiesImage Retrieval and Classification Techniques

Context Matters for Image Descriptions for Accessibility: Challenges for Referenceless Evaluation Metrics

Open access
Kreiss, Elisa, Bennett, Cynthia, Hooshmand, Shayan, Zelikman, Eric, Ringel Morris, Meredith, Potts, Christopher·2022·Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
DOI 10.18653/v1/2022.emnlp-main.309
Open this paper in CoChat — free accountOpen PDF

Abstract

Few images on the Web receive alt-text descriptions that would make them accessible to blind and low vision (BLV) users.Imagebased NLG systems have progressed to the point where they can begin to address this persistent societal problem, but these systems will not be fully successful unless we evaluate them on metrics that guide their development correctly.Here, we argue against current referenceless metrics -those that don't rely on human-generated ground-truth descriptions -on the grounds that they do not align with the needs of BLV users.The fundamental shortcoming of these metrics is that they do not take context into account, whereas contextual information is highly valued by BLV users.To substantiate these claims, we present a study with BLV participants who rated descriptions along a variety of dimensions.An in-depth analysis reveals that the lack of context-awareness makes current referenceless metrics inadequate for advancing image accessibility.As a proof-of-concept, we provide a contextual version of the referenceless metric CLIPScore which begins to address the disconnect to the BLV data.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2022.emnlp-main.309

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Web-scraped data for AI model pre-training is skewed towards an underrepresentation of disability data.

    “Web-scraped data for AI model pre-training is skewed towards an underrepresentation of disability data [69, 79].”

    From Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation (2026) · cites this paper as [69]

    Section 2.1, “Image Accessibility” (page not provided), reports that image descriptions are rare on some websites, but does not discuss the representation of disability data in AI pre-training datasets. Searches for web-scraped pre-training data, disability underrepresentation, and training-data bias found no passage addressing the claim. (LLM, conf=0.75)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 36

  1. Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. SPICE: Semantic proposi- tional image caption evaluation. In European Confer- ence on Computer Vision, pages 382-398. Springer. ZSCC: 0000514.
  2. Jeffrey P. Bigham, Ryan S. Kaminsky, Richard E. Lad- ner, Oscar M. Danielsson, and Gordon L. Hempton. 2006. WebInSight:: making web images accessible. In Proceedings of the 8th international ACM SIGAC- CESS conference on Computers and accessibility - Assets '06, page 181, Portland, Oregon, USA. ACM Press. · DOI · Paper page
  3. Maria Claudia Buzzi, Marina Buzzi, and Barbara Lep- orini. 2011. Web 2.0: Twitter and the blind. In Proceedings of the 9th ACM SIGCHI Italian Chapter International Conference on Computer-Human Inter- action: Facing Complexity, CHItaly, pages 151-156, New York, NY, USA. Association for Computing Machinery. · DOI
  4. James Carifio and Rocco Perla. 2008. Resolving the 50- year debate around using and misusing Likert scales. Medical education, 42(12):1150-1152.
  5. Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Uniter: Universal image-text representation learning. In European conference on computer vision, pages 104-120. Springer.
  6. Daniel Deutsch, Tania Bedrax-Weiss, and Dan Roth. 2021. Towards question-answering as an automatic metric for evaluating the content quality of a sum- mary. Transactions of the Association for Computa- tional Linguistics, 9:774-789.
  7. Kawin Ethayarajh and Dan Jurafsky. 2022. How hu- man is human evaluation? Improving the gold stan- dard for NLG with utility theory. arXiv preprint arXiv:2205.11930. · arXiv
  8. Joshua Feinglass and Yezhou Yang. 2021. SMURF: SeMantic and linguistic UndeRstanding Fusion for caption evaluation via typicality analysis. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2250-2260.
  9. Yang Gao, Wei Zhao, and Steffen Eger. 2020. SUPERT: Towards new frontiers in unsupervised evaluation metrics for multi-document summarization. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 1347- 1354, Online. Association for Computational Linguis- tics. · DOI
  10. Cole Gleason, Patrick Carrington, Cameron Cassidy, Meredith Ringel Morris, Kris M. Kitani, and Jeffrey P. Bigham. 2019. "It's almost like they're trying to hide it": How user-provided image descriptions have failed to make Twitter accessible. In The World Wide Web Conference on -WWW '19, pages 549-559, San Francisco, CA, USA. ACM Press. · DOI
  11. Cole Gleason, Amy Pavel, Emma McCamey, Christina Low, Patrick Carrington, Kris M. Kitani, and Jef- frey P. Bigham. 2020. Twitter A11y: A browser extension to make Twitter images accessible. In Pro- ceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1-12.
  12. Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. CLIPScore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 7514-7528.
  13. Susan Jamieson. 2004. Likert scales: How to (ab) use them? Medical education, 38(12):1217-1218.
  14. Jungo Kasai, Keisuke Sakaguchi, Lavinia Dunagan, Jacob Morrison, Ronan Le Bras, Yejin Choi, and Noah A. Smith. 2022a. Transparent human evalua- tion for image captioning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3464-3478, Seattle, United States. Association for Computational Lin- guistics. · DOI
  15. Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Lavinia Dunagan, Jacob Morrison, Alexander Fabbri, Yejin Choi, and Noah A. Smith. 2022b. Bidimen- sional leaderboards: Generate and evaluate language hand in hand. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 3540-3557, Seattle, United States. Association for Computational Linguistics. · DOI
  16. Elisa Kreiss, Fei Fang, Noah D. Goodman, and Christo- pher Potts. 2022. Concadia: Towards image-based text generation with a purpose. In Proceedings of the 2022 Conference on Empirical Methods in Natu- ral Language Processing (EMNLP). Association for Computational Linguistics.
  17. Hwanhee Lee, Thomas Scialom, Seunghyun Yoon, Franck Dernoncourt, and Kyomin Jung. 2021a. QACE: Asking questions to evaluate an image cap- tion. In Findings of the Association for Computa- tional Linguistics: EMNLP 2021, pages 4631-4638.
  18. Hwanhee Lee, Seunghyun Yoon, Franck Dernoncourt, Trung Bui, and Kyomin Jung. 2021b. UMIC: An unreferenced metric for image captioning via con- trastive learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 220-226.
  19. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
  20. Chi-kiu Lo. 2019. YiSi -A unified semantic MT quality evaluation and estimation metric for languages with different levels of available resources. In Proceed- ings of the Fourth Conference on Machine Transla- tion (Volume 2: Shared Task Papers, Day 1), pages 507-513, Florence, Italy. Association for Computa- tional Linguistics. · DOI
  21. Annie Louis and Ani Nenkova. 2013. Automatically assessing machine summary content without a gold standard. Computational Linguistics, 39(2):267-300. Place: Cambridge, MA Publisher: MIT Press. · DOI
  22. Haley MacLeod, Cynthia L. Bennett, Meredith Ringel Morris, and Edward Cutrell. 2017. Understanding blind people's experiences with computer-generated captions of social media images. In Proceedings of the 2017 CHI Conference on Human Factors in Com- puting Systems, pages 5988-5999, Denver Colorado USA. ACM. · DOI · Paper page
  23. Meredith Ringel Morris, Annuska Zolyomi, Catherine Yao, Sina Bahram, Jeffrey P. Bigham, and Shaun K. Kane. 2016. "With most of it being pictures now, I rarely use it": Understanding Twitter's evolving ac- cessibility to blind users. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pages 5506-5516.
  24. Annika Muehlbradt and Shaun K. Kane. 2022. What's in an ALT tag? Exploring caption content priorities through collaborative captioning. ACM Transactions on Accessible Computing, 15(1):6:1-6:32. · DOI · Paper page
  25. Geoff Norman. 2010. Likert scales, levels of measure- ment and the "laws" of statistics. Advances in health sciences education, 15(5):625-632.
  26. Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318.
  27. Maxime Peyrard and Iryna Gurevych. 2018. Objective function learning to match human judgements for optimization-based summarization. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Pa- pers), pages 654-660, New Orleans, Louisiana. As- sociation for Computational Linguistics. · DOI
  28. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763.
  29. PMLR.
  30. Abigale Stangl, Nitin Verma, Kenneth R Fleischmann, Meredith Ringel Morris, and Danna Gurari. 2021. Going beyond one-size-fits-all image descriptions to satisfy the information wants of people who are blind or have low vision. In The 23rd International ACM SIGACCESS Conference on Computers and Accessibility, pages 1-15. · DOI · Paper page
  31. Aditya Vashistha, Pooja Sethi, and Richard Anderson. 2018. BSpeak: An accessible voice-based crowd- sourcing marketplace for low-income blind people. In Proceedings of the 2018 CHI Conference on Hu- man Factors in Computing Systems, CHI '18, page 1-13, New York, NY, USA. Association for Comput- ing Machinery. · DOI
  32. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett, editors, Advances in Neural Information Pro- cessing Systems 30, pages 5998-6008.
  33. Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. CIDEr: Consensus-based image de- scription evaluation. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 4566-4575. ZSCC: 0001489.
  34. Violeta Voykinska, Shiri Azenkot, Shaomei Wu, and Gilly Leshed. 2016. How blind people interact with visual content on social networking services. In Pro- ceedings of the 19th ACM conference on computer- supported cooperative work & social computing, pages 1584-1595.
  35. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Wein- berger, and Yoav Artzi. 2019. BERTScore: Evalu- ating text generation with BERT. In International Conference on Learning Representations.
  36. Wei Zhao, Goran Glavaš, Maxime Peyrard, Yang Gao, Robert West, and Steffen Eger. 2020. On the lim- itations of cross-lingual encoders as exposed by reference-free machine translation evaluation. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 1656- 1671, Online. Association for Computational Linguis- tics. · DOI

Cited by

Sorted by citations

Authors
Kreiss, E., Bennett, C., Hooshmand, S., Zelikman, E., Ringel Morris, M., & Potts, C.
Affiliations
Google (United States) (US)
Corresponding author
Kreiss, Elisa
ORCID
Venue
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
Publisher
Association for Computational Linguistics
Published
2022
Pages
4685-4697
DOI
10.18653/v1/2022.emnlp-main.309
OpenAlex
W4385572843
Cited by
24
References
4
Citations by year
2026: 9 · 2025: 4 · 2024: 6 · 2023: 5
FWCI
1.36
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
National Science Foundation (1950223)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
Keywords
image accessibility, alt text, blind and visually impaired users, context awareness, CLIP score, web accessibility

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review