CoChat
Sign Up
Cited by 38Ethics and Social Impacts of AIDigital Accessibility for DisabilitiesAssistive Technology in Communication and Mobility

Computer Vision and Conflicting Values: Describing People with Automated Alt Text

Open access
Hanley, Margot, Barocas, Solon, Levy, Karen, Azenkot, Shiri, Nissenbaum, Helen·2021·Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society
DOI 10.1145/3461702.3462620
Open this paper in CoChat — free accountOpen PDF

Abstract

Scholars have recently drawn attention to a range of controversial issues posed by the use of computer vision for automatically generating descriptions of people in images. Despite these concerns, automated image description has become an important tool to ensure equitable access to information for blind and low vision people. In this paper, we investigate the ethical dilemmas faced by companies that have adopted the use of computer vision for producing alt text: textual descriptions of images for blind and low vision people. We use Facebook's automatic alt text tool as our primary case study. First, we analyze the policies that Facebook has adopted with respect to identity categories, such as race, gender, age, etc., and the company's decisions about whether to present these terms in alt text. We then describe an alternative---and manual---approach practiced in the museum community, focusing on how museums determine what to include in alt text descriptions of cultural artifacts. We compare these policies, using notable points of contrast to develop an analytic framework that characterizes the particular apprehensions behind these policy choices. We conclude by considering two strategies that seem to sidestep some of these concerns, finding that there are no easy ways to avoid the normative dilemmas posed by the use of computer vision to automate alt text.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3461702.3462620

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Concerns exist regarding the reliability and appropriateness of AI-generated alt text in academic contexts.

    “However, concerns exist regarding the reliability and appropriateness of AI-generated alt text in academic contexts [27, 48, 66].”

    From How University Disability Services Professionals Write Image Descriptions for HCI Figures Using Generative AI (2026) · cites this paper as [27]

    Section 4 (p. 4) discusses accuracy and consistency limitations in automated alt text, while Section 8 (p. 12) discusses whether descriptions of people are appropriate even when accurate. The paper examines Facebook and museum practices, not academic contexts, so it does not establish the claim's setting. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • In many tasks, disabled people might have limited access to sensory information that is important to the task, or their desired information might come into tension with other values such as privacy or making racial inference from photos.

    “In many tasks, disabled people might have limited access to sensory information that is important to the task [33, 75], or their desired information might come into tension with other values such as privacy or making racial inference from photos [10, 37, 94].”

    From Disability-First AI Dataset Annotation: Co-designing Stuttered Speech Annotation Guidelines with People Who Stutter (2026) · cites this paper as [37]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 51 · Cited by 38

Research works printed but not linked: 33

  1. [2][2] Pınar Barlas, Kyriakos Kyriakou, Styliani Kleanthous, and Jahna Otterbacher. Social b(eye) as: Human and machine descriptions of people images. In Proceedings of the International AAAI Conference on Web and Social Media , volume 13, pages 583-591, 2019. -
  2. [3][3] Solon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach. The Problem With Bias: Allocative Versus Representational Harms in Machine Learning. In Proceedings of SIGCIS , 2017. -
  3. [6][6] Cynthia L Bennett and Os Keyes. What is the point of fairness? disability, ai and the complexity of justice. ACM SIGACCESS Accessibility and Computing , (125), 2020. -
  4. [9][9] Abeba Birhane and Vinay Uday Prabhu. Large image datasets: A pyrrhic win for computer vision? In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 15371547, 2021. -
  5. [10][10] Geoffrey C Bowker and Susan Leigh Star. Sorting things out: Classification and its consequences . MIT press, 2000. -
  6. [11][11] Erin Brady, Meredith Ringel Morris, and Jeffrey P Bigham. Gauging receptiveness to social microvolunteering. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems , pages 1055-1064, 2015. -
  7. [12][12] Stacy M Branham, Ali Abdolrahmani, William Easley, Morgan Scheuerman, Erick Ronquillo, and Amy Hurst. 'is someone there? do they have a gun' how visual information about others can improve personal safety management for blind individuals. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility , pages 260-269, 2017. -
  8. [13][13] Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency , pages 77-91. PMLR, 2018. -
  9. [17][17] N Ann Davis. Invisible disability. Ethics , 116(1):153-213, 2005. -
  10. [20][20] Hao Fang, Saurabh Gupta, Forrest Iandola, Rupesh K Srivastava, Li Deng, Piotr Doll´ ar, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C Platt, et al. From captions to visual concepts and back. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1473-1482, 2015. -
  11. [24][24] Cole Gleason, Patrick Carrington, Cameron Cassidy, Meredith Ringel Morris, Kris M Kitani, and Jeffrey P Bigham. 'it's almost like they're trying to hide it': How user-provided image descriptions have failed to make twitter accessible. In The World Wide Web Conference , pages 549-559, 2019. -
  12. [27][27] Foad Hamidi, Morgan Klaus Scheuerman, and Stacy M Branham. Gender recognition or gender reductionism? the social implications of embedded gender recognition systems. In Proceedings of the 2018 chi conference on human factors in computing systems , pages 1-13, 2018. -
  13. [28][28] Margot Hanley, Apoorv Khandelwal, Hadar Averbuch-Elor, Noah Snavely, and Helen Nissenbaum. An ethical highlighter for people-centric dataset creation. arXiv preprint arXiv:2011.13583 , 2020. - · arXiv
  14. [29][29] Mahdi Hashemi and Margeret Hall. Criminal tendency detection from facial images and the gender bias effect. Journal of Big Data , 7(1):1-16, 2020. -
  15. [31][31] Anna Lauren Hoffmann. Terms of inclusion: Data, discourse, violence. New Media & Society , pages 1-18, 2020. -
  16. [32][32] Ting-Hao Huang, Walter Lasecki, Amos Azaria, and Jeffrey Bigham. 'is there anything else i can help you with?' challenges in deploying an on-demand crowd-powered conversational agent. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing , volume 4, 2016. -
  17. [34][34] Eun Seo Jo and Timnit Gebru. Lessons from archives: Strategies for collecting sociocultural data in machine learning. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency , pages 306-316, 2020. -
  18. [35][35] Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3128-3137, 2015. -
  19. [36][36] Gunay Kazimzade and Milagros Miceli. Biased priorities, biased outcomes: Three recommendations for ethics-oriented data annotation practices. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society , pages 71-71, 2020. -
  20. [38][38] Kyriakos Kyriakou, Pınar Barlas, Styliani Kleanthous, and Jahna Otterbacher. Fairness in proprietary image tagging algorithms: A cross-platform audit on people images. In Proceedings of the International AAAI Conference on Web and Social Media , volume 13, pages 313-322, 2019. -
  21. [41][41] Haley MacLeod, Cynthia L Bennett, Meredith Ringel Morris, and Edward Cutrell. Understanding blind people's experiences with computer-generated captions of social media images. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems , pages 5988-5999, 2017. -
  22. [43][43] Milagros Miceli, Martin Schuessler, and Tianling Yang. Between subjectivity and imposition: Power dynamics in data annotation for computer vision. Proceedings of the ACM on Human-Computer Interaction , 4(CSCW2):1-25, 2020. -
  23. [48][48] Jahna Otterbacher. Social cues, social biases: stereotypes in annotations on people images. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing , volume 6, 2018. -
  24. [49][49] Jahna Otterbacher, Pınar Barlas, Styliani Kleanthous, and Kyriakos Kyriakou. How do we talk about other people? group (un) fairness in natural language image descriptions. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing , volume 7, pages 106-114, 2019. -
  25. [52][52] Helen Petrie, Chandra Harrison, and Sundeep Dev. Describing images on the web: a survey of current practice and prospects for the future. Proceedings of Human Computer Interaction International (HCII) , 71:2, 2005. -
  26. [54][54] Jeffrey M Rzeszotarski and Meredith Ringel Morris. Estimating the social costs of friendsourcing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , pages 2735-2744, 2014. -
  27. [55][55] Elliot Salisbury, Ece Kamar, and Meredith Morris. Toward scalable social alt text: Conversational crowdsourcing as a tool for refining vision-to-language technology for the blind. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing , volume 5, 2017. -
  28. [57][57] Morgan Klaus Scheuerman, Kandrea Wade, Caitlin Lustig, and Jed R Brubaker. How we've taught algorithms to see identity: Constructing race and gender in image databases for facial analysis. Proceedings of the ACM on Human-Computer Interaction , 4(CSCW1):1-35, 2020. -
  29. [60][60] Kenneth Tran, Xiaodong He, Lei Zhang, Jian Sun, Cornelia Carapcea, Chris Thrasher, Chris Buehler, and Chris Sienkiewicz. Rich image captioning in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops , pages 49-56, 2016. -
  30. [62][62] Emiel Van Miltenburg. Stereotyping and bias in the flickr30k dataset. arXiv preprint arXiv:1605.06083 , 2016. - · arXiv
  31. [63][63] Emiel van Miltenburg, Desmond Elliott, and Piek Vossen. Talking about other people: an endless range of possibilities. In Proceedings of the 11th International Conference on Natural Language Generation , pages 415-420, 2018. -
  32. [69][69] Shaomei Wu, Jeffrey Wieland, Omid Farivar, and Julie Schiller. Automatic alt-text: Computergenerated image descriptions for blind users on a social network service. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing , pages 1180-1192, 2017. -
  33. [71][71] Yuhang Zhao, Michele Hu, Shafeka Hashash, and Shiri Azenkot. Understanding low vision people's visual perception on commercial augmented reality glasses. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems , pages 4170-4181, 2017. -

Authors
Hanley, M., Barocas, S., Levy, K., Azenkot, S., & Nissenbaum, H.
Affiliations
  • Cornell Tech (US)
  • Cornell University (US)
ORCID
Venue
Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society
Publisher
ACM
Published
2021-07-21
Pages
543-554
ISSN
3065-8365 (ISSN-L)
DOI
10.1145/3461702.3462620
OpenAlex
W3164652469
Cited by
38
References
51
Citations by year
2026: 8 · 2025: 10 · 2024: 7 · 2023: 6 · 2022: 5 · 2021: 2
FWCI
4.52
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • Proceedings of the AAAI/ACM Conference on AI Ethics and Society (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • John D. and Catherine T. MacArthur Foundation
  • National Science Foundation (CHS-1901151)
  • Brown Institute for Media Innovation
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
  • Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
Keywords
picture description, computer vision, Facebook, blind and visually impaired users, accessibility, ethical dilemmas, identity categories, race, gender, age, cultural artifacts, normative dilemmas

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review