CoChat
Sign Up
Cited by 7Ethics and Social Impacts of AIGenerative Adversarial Networks and Image SynthesisHate Speech and Cyberbullying Detection

“I Don’t See Myself Represented Here at All”: User Experiences of Stable Diffusion Outputs Containing Representational Harms across Gender Identities and Nationalities

Open access
Ghosh, Sourojit, Lutz, Nina, Caliskan, Aylin·2024·Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society
DOI 10.1609/aies.v7i1.31650
Open this paper in CoChat — free accountOpen PDF

Abstract

Though research into text-to-image generators (T2Is) such as Stable Diffusion has demonstrated their amplification of societal biases and potentials to cause harm, such research has primarily relied on computational methods instead of seeking information from real users who experience harm, which is a significant knowledge gap. In this paper, we conduct the largest human subjects study of Stable Diffusion, with a combination of crowdsourced data from 133 crowdworkers and 14 semi-structured interviews across diverse countries and genders. Through a mixed-methods approach of intra-set cosine similarity hierarchies (i.e., comparing multiple Stable Diffusion outputs for the same prompt with each other to examine which result is `closest' to the prompt) and qualitative thematic analysis, we first demonstrate a large disconnect between user expectations for Stable Diffusion outputs with those generated, evidenced by a set of Stable Diffusion renditions of `a Person' providing images far away from such expectations. We then extend this finding of general dissatisfaction into highlighting representational harms caused by Stable Diffusion upon our subjects, especially those with traditionally marginalized identities, subjecting them to incorrect and often dehumanizing stereotypes about their identities. We provide recommendations for a harm-aware approach to (re)design future versions of Stable Diffusion and other T2Is.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1609/aies.v7i1.31650

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Existing research predominantly focuses on understanding, defining or evaluating representational 'harms' and often uses poor AI-generated images to elicit feedback from marginalized communities.

    “Our approach to defining and evaluating 'good' representation differs from existing research that predominantly focuses on understanding, defining or evaluating representational 'harms' [24, 28, 38, 44, 67, 77, 91, 116] and often uses poor AI-generated images to elicit feedback from marginalized communities (e.g., [65, 77, 79]).”

    From Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation (2026) · cites this paper as [44]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 48

  1. Badham, R. J.; and Wall, T. 2006. Sociotechnical theory. 2 edition.
  2. Barocas, S.; Crawford, K.; Shapiro, A.; and Wallach, H. 2017. The problem with bias: Allocative versus representa- tional harms in machine learning. In 9th Annual conference of the special interest group for computing, information and society.
  3. Bender, E. M.; and Friedman, B. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6: 587-604.
  4. Bianchi, F.; Kalluri, P.; Durmus, E.; Ladhak, F.; Cheng, M.; Nozza, D.; Hashimoto, T.; Jurafsky, D.; Zou, J.; and Caliskan, A. 2023. Easily accessible text-to-image gener- ation amplifies demographic stereotypes at large scale. In Proceedings of the 2023 ACM Conference on Fairness, Ac- countability, and Transparency, 1493-1504.
  5. Blodgett, S. L.; Barocas, S.; Daumé III, H.; and Wallach, H. 2020. Language (technology) is power: A critical survey of" bias" in nlp. arXiv preprint arXiv:2005.14050. · arXiv
  6. Borji, A. 2022. Generated faces in the wild: Quantitative comparison of stable diffusion, midjourney and dall-e 2. arXiv preprint arXiv:2210.00586. · arXiv
  7. Bowker, G.; Star, S. L.; Gasser, L.; and Turner, W. 1997. So- cial science, technical systems, and cooperative work: Be- yond the great divide. Psychology Press.
  8. Braun, V.; and Clarke, V. 2006. Using Thematic Analysis in Psychology. Qualitative research in psychology, 3(2): 77- 101.
  9. Chancellor, S. 2023. Toward Practices for Human-Centered Machine Learning. Communications of the ACM, 66(3): 78- 85.
  10. Clark, J. 2012. Using diamond ranking as visual cues to engage young people in the research process. Qualitative Research Journal, 12(2): 222-237.
  11. Cooper, R.; and Foster, M. 1971. Sociotechnical systems. American Psychologist, 26(5): 467.
  12. Dev, S.; Sheng, E.; Zhao, J.; Amstutz, A.; Sun, J.; Hou, Y.; Sanseverino, M.; Kim, J.; Nishi, A.; Peng, N.; and Chang, K.-W. 2021. On Measures of Biases and Harms in NLP. arXiv preprint arXiv:2108.03362. · arXiv
  13. Dhariwal, P.; and Nichol, A. 2021. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34: 8780-8794.
  14. D'ignazio, C.; and Klein, L. F. 2020. Data feminism. MIT press.
  15. Fitch, M. A. 2009. Side dishes: Latina American women, sex, and cultural production. Rutgers University Press.
  16. Fraser, K. C.; Kiritchenko, S.; and Nejadgholi, I. 2023. A Friendly Face: Do Text-to-Image Systems Rely on Stereo- types when the Input is Under-Specified? AAAI 2023 Work- shop on Creative AI Across Modalities.
  17. Gadiraju, V.; Kane, S.; Dev, S.; Taylor, A.; Wang, D.; Den- ton, E.; and Brewer, R. 2023. " I wouldn't say offensive but...": Disability-Centered Perspectives on Large Language Models. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, 205-216.
  18. Gasson, S. 2003. Human-centered vs. user-centered ap- proaches to information system design. Journal of Infor- mation Technology Theory and Application (JITTA), 5(2): 5.
  19. Gautam, S.; Venkit, P. N.; and Ghosh, S. 2024. From melting pots to misrepresentations: Exploring harms in generative ai. arXiv preprint arXiv:2403.10776. · arXiv
  20. Ghosh, S.; and Caliskan, A. 2023a. ChatGPT Perpetu- ates Gender Bias in Machine Translation and Ignores Non- Gendered Pronouns: Findings across Bengali and Five other Low-Resource Languages. AAAI/ACM Conference on AI, Ethics, and Society 2023, 901-912.
  21. Pino, J.; and Bali, K., eds., Findings of the Association for Computational Linguistics: EMNLP 2023, 6971-6985. Sin- gapore: Association for Computational Linguistics.
  22. Ghosh, S.; and Chatterjee, S. 2024. Misgendering and As- suming Gender in Machine Translation when Working with Low-Resource Languages. In Monzo-Nebot, E.; and Tasa- Fuster, V., eds., Gendered Technology in Translation and Interpreting: Centering Rights in the Development of Lan- guage Technology. Routledge.
  23. Ghosh, S.; Venkit, P. N.; Gautam, S.; Wilson, S.; and Caliskan, A. 2024. Do Generative AI Models Output Harm while Representing Non-Western Cultures: Evidence from A Community-Centered Approach. arXiv preprint arXiv:2407.14779. · arXiv
  24. Gray, C.; and Hansen, K. 2021. Did COVID-19 lead to an increase in hate crimes toward Chinese people in London? Journal of Contemporary Criminal Justice, 37(4): 569-588.
  25. Grosfoguel, R. 2019. Epistemic Extractivism: A Dialogue with Alberto Acosta, Leanne Betasamosake Simpson, and Silvia Rivera Cusicanqui. In Knowledges born in the strug- gle, 203-218. Routledge.
  26. Guzmán, I. M.; and Valdivia, A. N. 2004. Brain, brow, and booty: Latina iconicity in US popular culture. The commu- nication review, 7(2): 205-221.
  27. Haraway, D. 1988. 'Situated Knowledges: The Science Question in Feminism and the Privilege of Partial Perspec- tive'. In Space, Gender, Knowledge: Feminist Readings, 53- 72. Routledge.
  28. Katzman, J.; Wang, A.; Scheuerman, M.; Blodgett, S. L.; Laird, K.; Wallach, H.; and Barocas, S. 2023. Taxonomizing and Measuring Representational Harms: A Look at Image Tagging. AAAI-23 Special Track on AI for Social Impact.
  29. Laufer, B.; Jain, S.; Cooper, A. F.; Kleinberg, J.; and Heidari, H. 2022. Four years of FAccT: A reflexive, mixed-methods analysis of research contributions, shortcomings, and future prospects. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, 401-426.
  30. Lee, S.; Hoover, B.; Strobelt, H.; Wang, Z. J.; Peng, S.; Wright, A.; Li, K.; Park, H.; Yang, H.; and Chau, D. H. 2023. Diffusion Explainer: Visual Explanation for Text-to-image Stable Diffusion. arXiv preprint arXiv:2305.03509. · arXiv
  31. Liu, M.; and Hu, Y. 2023. Application potential of stable diffusion in different stages of industrial design. In Inter- national Conference on Human-Computer Interaction, 590- 609. Springer.
  32. Luccioni, A. S.; Akiki, C.; Mitchell, M.; and Jernite, Y. 2023. Stable Bias: Analyzing Societal Representations in Diffusion Models. NeurIPS Datasets and Benchmarks.
  33. Lutz, N.; and Aragon, C. 2024. "We're not all construction workers": Algorithmic Compression of Latinidad on Tik- Tok. arXiv preprint arXiv:2407.13927. · arXiv
  34. Mack, A.; Qadri, R.; Denton, R.; Kane, S. K.; and Bennett, C. L. 2024. "They only care to show us the wheelchair": Disability Representation in text-to-image AI models. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems.
  35. Mack, K. A.; McDonnell, E.; Potluri, V.; Xu, M.; Zabala, J.; Bigham, J.; Mankoff, J.; and Bennett, C. 2022. Anticipate and adjust: Cultivating access in human-centered methods. In Proceedings of the 2022 CHI Conference on Human Fac- tors in Computing Systems, 1-18.
  36. Mandal, A.; Leavy, S.; and Little, S. 2023. Multimodal Composite Association Score: Measuring Gender Bias in Generative Multimodal Models. Proceedings of the Fourth International Workshop on Algorithmic Bias in Search and Recommendation held as a part of the 45th European Con- ference on Information Retrieval (ECIR 2023).
  37. Mensa, M.; and Vargas-Bianchi, L. 2022. Portraying of Latin American Women in Falabella's Instagram: Stereo- types and Sexual Objectification Theory. In Communication and Applied Technologies: Proceedings of ICOMTA 2022, 213-225. Springer.
  38. Miceli, M.; Posada, J.; and Yang, T. 2022. Studying up machine learning data: Why talk about bias when we mean power? Proceedings of the ACM on Human-Computer In- teraction, 6(GROUP): 1-14.
  39. Mittelstadt, B. 2019. Principles alone cannot guarantee eth- ical AI. Nature machine intelligence, 1(11): 501-507.
  40. Molina-Guzmán, I. 2010. Dangerous curves: Latina bodies in the media, volume 5. NYU Press.
  41. Qadri, R.; Shelby, R.; Bennett, C. L.; and Denton, E. 2023. AI's Regimes of Representation: A Community-centered Study of Text-to-Image Models in South Asia. In Proceed- ings of the 2023 ACM Conference on Fairness, Accountabil- ity, and Transparency, 506-517.
  42. Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Om- mer, B. 2022. High-Resolution Image Synthesis with Latent Diffusion Models. Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, 10684- 10695.
  43. Roscoe, J. 2024. Review Used By UK to Limit Gender Af- firming Care Uses Images of AI-Generated Kids. 404 Me- dia.
  44. Salaberria, A.; Azkune, G.; de Lacalle, O. L.; Soroa, A.; Agirre, E.; and Keller, F. 2024. Improving Ex- plicit Spatial Relationships in Text-to-Image Generation through an Automatically Derived Dataset. arXiv preprint arXiv:2403.00587. · arXiv
  45. Schramowski, P.; Brack, M.; Deiseroth, B.; and Kersting, K. 2023. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 22522-22531.
  46. Schuhmann, C.; Beaumont, R.; Vencu, R.; Gordon, C.; Wightman, R.; Cherti, M.; Coombes, T.; Katta, A.; Mullis, C.; Wortsman, M.; et al. 2022. LAION-5B: An Open Large-Scale Dataset for Training Next Generation Image- Text Models. Proceedings of the 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
  47. Singhal, A.; et al. 2001. Modern Information Retrieval: A Brief Overview. IEEE Data Engineering Bulletin, 24(4): 35-43.
  48. Wolfe, R.; and Caliskan, A. 2022. Markedness in Visual Semantic AI. Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, 1269-1279.

Cited by

Sorted by citations

Authors
Ghosh, S., Lutz, N., & Caliskan, A.
Affiliations
University of Washington (US)
ORCID
Venue
Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society
Publisher
Association for the Advancement of Artificial Intelligence (AAAI)
Published
2024-10-16
Volume
7
Pages
463-475
ISSN
3065-8365 (ISSN-L)
DOI
10.1609/aies.v7i1.31650
OpenAlex
W4404518375
Cited by
7
References
0
Citations by year
2026: 2 · 2025: 4 · 2024: 1
FWCI
3.19
Type
conference paper
Language
en
Open access
Locations
  • Proceedings of the AAAI/ACM Conference on AI Ethics and Society (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
National Institute of Standards and Technology (60NANB23D194)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
Keywords
Stable Diffusion, text-to-image generation, representational harm, gender identity, nationality, stereotypes, user expectations, marginalized identities, dehumanization, bias amplification

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review