CoChat
Sign Up
Cited by 22Digital Accessibility for DisabilitiesSubtitles and Audiovisual MediaAssistive Technology in Communication and Mobility

What's in an ALT Tag? Exploring Caption Content Priorities through Collaborative Captioning

Open access
Muehlbradt, Annika, Kane, Shaun K.·2022·ACM Transactions on Accessible Computing
DOI 10.1145/3507659
Open this paper in CoChat — free accountOpen PDF

Abstract

Evaluating the quality of accessible image captions with human raters is difficult, as it may be difficult for a visually impaired user to know how comprehensive a caption is, whereas a sighted assistant may not know what information a user will need from a caption. To explore how image captioners and caption consumers assess caption content, we conducted a series of collaborative captioning sessions in which six pairs, consisting of a blind person and their sighted partner, worked together to discuss, create, and evaluate image captions. By making captioning a collaborative task, we were able to observe captioning strategies, to elicit questions and answers about image captions, and to explore blind users’ caption preferences. Our findings provide insight about the process of creating good captions and serve as a case study for cross-ability collaboration between blind and sighted people.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3507659

Cites

13 of 13 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • In many tasks, disabled people might have limited access to sensory information that is important to the task, or their desired information might come into tension with other values such as privacy or making racial inference from photos.

    “In many tasks, disabled people might have limited access to sensory information that is important to the task [33, 75], or their desired information might come into tension with other values such as privacy or making racial inference from photos [10, 37, 94].”

    From Disability-First AI Dataset Annotation: Co-designing Stuttered Speech Annotation Guidelines with People Who Stutter (2026) · cites this paper as [75]

    Section 2.3 (Crowdsourced Captioning; page unavailable) says captioning services can violate users’ privacy by exposing sensitive information to strangers. The example captions in Table 3 include racial descriptions of people in photos, but the paper does not identify racial inference as a value in tension with users’ desired information. The privacy concern supports part of the clause cited to this paper; the claimed tension around racial inference is not established. (LLM, conf=0.77)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 46

  1. Luis Von Ahn and Laura Dabbish. 2004. Labeling images with a computer game. In Proceedings of the SIGCHI Confer- ence on Human Factors in Computing Systems. 319-326.
  2. Luis Von Ahn, Shiry Ginosar, Mihir Kedia, Ruoran Liu, and Manuel Blum. 2006. Improving accessibility of the web with a computer game. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 79-82.
  3. Luis Von Ahn, Mihir Kedia, and Manuel Blum. 2006. Verbosity: A game for collecting common-sense facts. In Pro- ceedings of the SIGCHI Conference on Human Factors in Computing Systems. 75-78.
  4. Chieko Asakawa. 2005. What's the web like if you can't see it? In Proceedings of the 2005 International Cross- Disciplinary Workshop on Web Accessibility (W4A'05). 1-8. https://doi.org/10.1145/1061811.1061813 · DOI
  5. Mauro Avila, Katrin Wolf, Anke Brock, and Niels Henze. 2016. Remote assistance for blind users in daily life: A survey about be my eyes. In Proceedings of the 9th ACM International Conference on Pervasive Technologies Related to Assistive Environments (PETRA'16). Article 85. https://doi.org/10.1145/2910674.2935839 · DOI · Paper page
  6. Jeffrey P. Bigham, Richard E. Ladner, and Yevgen Borodin. 2011. The design of human-powered access technology. In Proceedings of the 13th International ACM SIGACCESS Conference on Computers and Accessibility. 3-10.
  7. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit. O'Reilly Media.
  8. Erin Brady, Meredith Ringel Morris, and Jeffrey P. Bigham. 2014. Friendsourcing for the greater good: Perceptions of social microvolunteering. In Proceedings of the 2nd AAAI Conference on Human Computation and Crowdsourcing.
  9. Erin Buehler, Stacy Branham, Abdullah Ali, Jeremy J. Chang, Megan Kelly Hofmann, Amy Hurst, and Shaun K. Kane. 2015. Sharing is caring: Assistive technology designs on Thingiverse. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 525-534.
  10. Michele A. Burton, Erin Brady, Robin Brewer, Callie Neylan, Jeffrey P. Bigham, and Amy Hurst. 2012. Crowdsourcing subjective fashion advice using VizWiz: Challenges and opportunities. In Proceedings of the 14th International ACM SIGACCESS Conference on Computers and Accessibility. 135-142.
  11. Wendy Chisholm, Gregg Vanderheiden, and Ian Jacobs. 2001. Web content accessibility guidelines 1.0. Interactions 8, 4 (2001), 35-54.
  12. Juliet Corbin and Anselm Strauss. 2014. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. Sage Publications.
  13. Timothy C. Craven. 2006. Some features of "alt" texts associated with images in web pages. Information Research: An International Electronic Journal 11, 2 (2006), 2.
  14. Maitraye Das, Darren Gergle, and Anne Marie Piper. 2019. "It doesn't win you friends": Understanding accessibility in collaborative writing for people with vision impairments. Proceedings of the ACM on Human-Computer Interactio. 3, CSCW (2019), Article 191, 26 pages.
  15. Ali Farhadi, Mohsen Hejrati, Mohammad Amin Sadeghi, Peter Young, Cyrus Rashtchian, Julia Hockenmaier, and David Forsyth. 2010. Every picture tells a story: Generating sentences from images. In Proceedings of the European Conference on Computer Vision. 15-29.
  16. Cole Gleason, Patrick Carrington, Cameron Cassidy, Meredith Ringel Morris, Kris M. Kitani, and Jeffrey P. Bigham. 2019. "It's almost like they're trying to hide it": How user-provided image descriptions have failed to make Twitter accessible. In Proceedings of the World Wide Web Conference. 549-559.
  17. Morten Goodwin, Deniz Susar, Annika Nietzio, Mikael Snaprud, and Christian S. Jensen. 2011. Global web accessi- bility analysis of national government portals and ministry web sites. Journal of Information Technology & Politics 8, 1 (2011), 41-67.
  18. Gabriella M. Johnson and Shaun K. Kane. 2020. Game changer: Accessible audio and tactile guidance for board and card games. In Proceedings of the 17th International Web for All Conference. 1-12.
  19. Daniel Keysers, Marius Renn, and Thomas M. Breuel. 2007. Improving accessibility of HTML documents by gener- ating image-tags in a proxy. In Proceedings of the 9th International ACM SIGACCESS Conference on Computers and Accessibility. 249-250.
  20. Girish Kulkarni, Visruth Premraj, Vicente Ordonez, Sagnik Dhar, Siming Li, Yejin Choi, Alexander C. Berg, and Tamara L. Berg. 2013. Baby Talk: Understanding and generating simple image descriptions. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 12 (2013), 2891-2903.
  21. Polina Kuznetsova, Vicente Ordonez, Alexander Berg, Tamara Berg, and Yejin Choi. 2012. Collective generation of natural image descriptions. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 359-368.
  22. Scott C. LaBarre. 2007. ABA resolution and report on website accessibility. Mental and Physical Disability Law Reporter 31, 4 (2007), 504-507.
  23. Walter S. Lasecki, Phyo Thiha, Yu Zhong, Erin Brady, and Jeffrey P. Bigham. 2013. Answering visual questions with conversational crowd assistants. In Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility. 1-8.
  24. Eleanor T. Loiacono, Nicholas C. Romano Jr., and Scott McCoy. 2009. The state of corporate website accessibility. Communications of the ACM 52, 9 (2009), 128-132.
  25. Haley MacLeod, Cynthia L. Bennett, Meredith Ringel Morris, and Edward Cutrell. 2017. Understanding blind people's experiences with computer-generated captions of social media images. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 5988-5999.
  26. Alexander Patrick Mathews. 2015. Captioning images using different styles. In Proceedings of the 23rd ACM Interna- tional Conference on Multimedia. 665-668.
  27. Meredith Ringel Morris, Jaime Teevan, and Katrina Panovich. 2010. A comparison of information seeking using search engines and social networks. In Proceedings of the 4th International AAAI Conference on Weblogs and Social Media.
  28. Meredith Ringel Morris, Annuska Zolyomi, Catherine Yao, Sina Bahram, Jeffrey P. Bigham, and Shaun K. Kane. 2016. "With most of it being pictures now, I rarely use it": Understanding Twitter's evolving accessibility to blind users. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 5506-5516.
  29. Julius T. Nganji, Mike Brayshaw, and Brian Tompsett. 2013. Describing and assessing image descriptions for visually impaired web users with IDAT. In Proceedings of the 3rd International Conference on Intelligent Human Computer Interaction (IHCI'11). 27-37.
  30. Abiodun Olalere and Jonathan Lazar. 2011. Accessibility of US federal government home pages: Section 508 compli- ance and site accessibility statements. Government Information Quarterly 28, 3 (2011), 303-309.
  31. Vicente Ordonez, Girish Kulkarni, and Tamara Berg. 2011. Im2Text: Describing images using 1 million captioned photographs. Advances in Neural Information Processing Systems 24 (2011), 1143-1151.
  32. Helen Petrie, Chandra Harrison, and Sundeep Dev. 2005. Describing images on the web: A survey of current practice and prospects for the future. In Proceedings of the 3rd International Conference on Universal Access in Human Computer Interaction (HCII'05).
  33. Elliot Salisbury, Ece Kamar, and Meredith Morris. 2017. Toward scalable social alt text: Conversational crowdsourcing as a tool for refining vision-to-language technology for the blind. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing.
  34. John M. Slatin. 2001. The art of ALT: Toward a more accessible web. Computers and Composition 18, 1 (2001), 73-81.
  35. Lisa Tang and Jim A. Carter. 2011. Communicating image content. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting. 495-499.
  36. Anja Thieme, Cecily Morrison, Nicolas Villar, Martin Grayson, and Siân Lindley. 2017. Enabling collaboration in learning computer programming inclusive of children with vision impairments. In Proceedings of the 2017 Conference on Designing Interactive Systems. 739-752.
  37. Kenneth Tran, Xiaodong He, Lei Zhang, Jian Sun, Cornelia Carapcea, Chris Thrasher, Chris Buehler, and Chris Sienkiewicz. 2016. Rich image captioning in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 49-56.
  38. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3156-3164.
  39. Timo Volkmer, John R. Smith, and Apostol Natsev. 2005. A web-based system for collaborative annotation of large image and video collections: An evaluation and user study. In Proceedings of the 13th Annual ACM International Conference on Multimedia. 892-901.
  40. Violeta Voykinska, Shiri Azenkot, Shaomei Wu, and Gilly Leshed. 2016. How blind people interact with visual content on social networking services. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work and Social Computing. 1584-1595.
  41. Michele A. Williams, Caroline Galbraith, Shaun K. Kane, and Amy Hurst. 2014. "Just let the cane hit it": How the blind and sighted see navigation differently. In Proceedings of the 16th International ACM SIGACCESS Conference on Computers and Accessibility. 217-224.
  42. Fredrik Winberg and John Bowers. 2004. Assembling the senses: Towards the design of cooperative interfaces for visually impaired users. In Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work. 332- 341.
  43. Shaomei Wu, Jeffrey Wieland, Omid Farivar, and Julie Schiller. 2017. Automatic alt-text: Computer-generated image descriptions for blind users on a social network service. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. 1180-1192.
  44. Chien Wen Yuan, Benjamin V. Hanrahan, Sooyeon Lee, Mary Beth Rosson, and John M. Carroll. 2017. I didn't know that you knew I knew: Collaborative shopping practices between people with visual impairment and people with vision. Proceedings of the ACM on Human-Computer Interaction 1, CSCW (2017), Article 118, 18 pages.
  45. Yu Zhong, Walter S. Lasecki, Erin Brady, and Jeffrey P. Bigham. 2015. RegionSpeak: Quick comprehensive spatial descriptions of complex images for blind users. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 2353-2362.
  46. Kathryn Zyskowski, Meredith Ringel Morris, Jeffrey P. Bigham, Mary L. Gray, and Shaun K. Kane. 2015. Accessible crowdwork? Understanding the value in and challenge of microtask employment for people with disabilities. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work and Social Computing. 1682-1693.

Cited by

Sorted by citations

Authors
Muehlbradt, A., & Kane, S. K.
Affiliations
University of Colorado Boulder (US)
ORCID
Venue
ACM Transactions on Accessible Computing
Publisher
Association for Computing Machinery (ACM)
Published
2022-03-04
Volume
15
Issue
1
Pages
1-32
ISSN
  • 1936-7228 (ISSN-L)
  • 1936-7236
DOI
10.1145/3507659
OpenAlex
W4214860063
Cited by
22
References
13
Citations by year
2026: 5 · 2025: 6 · 2024: 7 · 2023: 2 · 2022: 2
FWCI
10.82
Type
journal article
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
ACM Transactions on Accessible Computing (publishedVersion)
Funders
National Science Foundation (IIS-1619384 and IIS-1652907)
Topics
  • Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
  • Social Sciences › Arts and Humanities › Language and Linguistics › Subtitles and Audiovisual Media
  • Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
Keywords
image captioning, accessibility, visually impaired users, user preferences

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review