CoChat
Sign Up
PreprintCited by 2Ethics and Social Impacts of AILaw, AI, and Intellectual PropertyTechnology's Impact on Media

Provocation: Who benefits from "inclusion" in Generative AI?

Open access
Dalal, Samantha, Hall, Siobhan Mackenzie, Johnson, Nari·2024·arXiv
DOI 10.48550/arxiv.2411.09102arXiv 2411.09102
Open this paper in CoChat — free accountOpen PDF

Abstract

The demands for accurate and representative generative AI systems means there is an increased demand on participatory evaluation structures. While these participatory structures are paramount to to ensure non-dominant values, knowledge and material culture are also reflected in AI models and the media they generate, we argue that dominant structures of community participation in AI development and evaluation are not explicit enough about the benefits and harms that members of socially marginalized groups may experience as a result of their participation. Without explicit interrogation of these benefits by AI developers, as a community we may remain blind to the immensity of systemic change that is needed as well. To support this provocation, we present a speculative case study, developed from our own collective experiences as AI researchers. We use this speculative context to itemize the barriers that need to be overcome in order for the proposed benefits to marginalized communities to be realized, and harms mitigated.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2411.09102

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Researchers caution that involving marginalized communities in elaborating on their marginalization for red-teaming can become transactional, extractive and exploitative, and warn of emotional and mental health costs of this type of human labor.

    “However, researchers caution that involving marginalized communities in elaborating on their marginalization for red-teaming, can become transactional, extractive and exploitative (cf. [29, 46]); and warn of the emotional and mental health costs of this type of human 'labor'.”

    From Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation (2026) · cites this paper as [29]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 36

  1. [1][1] Federico Bianchi, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng, Debora Nozza, Tatsunori Hashimoto, Dan Jurafsky, James Zou, and Aylin Caliskan. Easily accessible text-to-image generation amplifies demographic stereotypes at large scale. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency , FAccT '23, page 1493-1504, New York, NY, USA, 2023. Association for Computing Machinery. -
  2. [2][2] Melissa Hall, Samuel J. Bell, Candace Ross, Adina Williams, Michal Drozdzal, and Adriana Romero Soriano. Towards geographic inclusion in the evaluation of text-to-image models. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency , FAccT '24, page 585-601, New York, NY, USA, 2024. Association for Computing Machinery. -
  3. [5][5] Kelly Avery Mack, Rida Qadri, Remi Denton, Shaun K. Kane, and Cynthia L. Bennett. 'they only care to show us the wheelchair': disability representation in text-to-image ai models. In Proceedings of the CHI Conference on Human Factors in Computing Systems , CHI '24, New York, NY, USA, 2024. Association for Computing Machinery. -
  4. [7][7] Solon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach. The problem with bias: From allocative to representational harms in machine learning. In SIGCIS conference paper , 2017. -
  5. [8][8] Abeba Birhane, Sepehr Dehdashtian, Vinay Prabhu, and Vishnu Boddeti. The dark side of dataset scaling: Evaluating racial classification in multimodal models. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency , FAccT '24, page 1229-1244, New York, NY, USA, 2024. Association for Computing Machinery. -
  6. [12][12] Yong Cao, Li Zhou, Seolhwa Lee, Laura Cabello, Min Chen, and Daniel Hershcovich. Assessing cross-cultural alignment between ChatGPT and human societies: An empirical study. In Sunipa Dev, Vinodkumar Prabhakaran, David Adelani, Dirk Hovy, and Luciana Benotti, editors, Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP) , pages 53-67, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. -
  7. [13][13] Alice Qian Zhang, Ryland Shaw, Jacy Reese Anthis, Ashlee Milton, Emily Tseng, Jina Suh, Lama Ahmad, Ram Shankar Siva Kumar, Julian Posada, Benjamin Shestakofsky, Sarah T. Roberts, and Mary L. Gray. The human factor in ai red teaming: Perspectives from social and collaborative computing. arXiv preprint arXiv:2407.07786 , 2024. - · arXiv
  8. [14][14] Jennifer Pierre, Roderic Crooks, Morgan Currie, Britt Paris, and Irene Pasquetto. Getting ourselves together: Data-centered participatory design research & epistemic burden. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems , pages 1-11, Yokohama Japan, May 2021. ACM. -
  9. [16][16] Abeba Birhane, William Isaac, Vinodkumar Prabhakaran, Mark Diaz, Madeleine Clare Elish, Iason Gabriel, and Shakir Mohamed. Power to the people? opportunities and challenges for participatory ai. In Proceedings of the 2nd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization , EAAMO '22, New York, NY, USA, 2022. Association for Computing Machinery. -
  10. [19][19] Nina Bozic Yams and Álvaro Aranda Muñoz. Poetics of future work: Blending speculative design with artistic methodology. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems , CHI EA '21, New York, NY, USA, 2021. Association for Computing Machinery. -
  11. [20][20] Shamika Klassen and Casey Fiesler. The stoop: speculation on positive futures of black digital spaces. Proceedings of the ACM on Human-Computer Interaction , 7(GROUP):1-24, 2023. -
  12. [23][23] Ranjita Naik and Besmira Nushi. Social biases through the text-to-image generation lens. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society , AIES '23, page 786-808, New York, NY, USA, 2023. Association for Computing Machinery. -
  13. [25][25] Shahidul Alam. Majority world: Challenging the west's rhetoric of democracy. Amerasia Journal , 34:87-98, 01 2008. -
  14. [27][27] Harini Suresh, Emily Tseng, Meg Young, Mary Gray, Emma Pierson, and Karen Levy. Participation in the age of foundation models. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency , FAccT '24, page 1609-1621, New York, NY, USA, 2024. Association for Computing Machinery. -
  15. [29][29] Maitraye Das, Alexander J. Fiannaca, Meredith Ringel Morris, Shaun K. Kane, and Cynthia L. Bennett. From provenance to aberrations: Image creator and screen reader user perspectives on alt text for aigenerated images. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems , CHI '24, New York, NY, USA, 2024. Association for Computing Machinery. -
  16. [30][30] Tarleton Gillespie. Generative ai and the politics of visibility. Big Data & Society , 11(2):20539517241252131, June 2024. -
  17. [31][31] Safiya Umoja Noble. Algorithms of Oppression: How Search Engines Reinforce Racism . NYU Press, New York, 2018. -
  18. [32][32] Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency , pages 77-91. PMLR, 2018. -
  19. [33][33] Herman Gray. Subject (ed) to recognition. American Quarterly , 65(4):771-798, 2013. -
  20. [34][34] Kristen J. Warner. In the time of plastic representation. Film Quarterly , 71(2):32-37, 2017. -
  21. [35][35] Anamik Saha. Beards, scarves, halal meat, terrorists, forced marriage': television industries and the production of 'race. Media, Culture & Society , 34(4):424-438, 2012. -
  22. [39][39] William Agnew, A. Stevie Bergman, Jennifer Chien, Mark Díaz, Seliem El-Sayed, Jaylen Pittman, Shakir Mohamed, and Kevin R. McKee. The illusion of artificial inclusion. In Proceedings of the CHI Conference on Human Factors in Computing Systems , volume 35 of CHI '24 , page 1-12. ACM, May 2024. -
  23. [40][40] Cedric Deslandes Whitney and Justin Norman. Real risks of fake data: Synthetic data, diversity-washing and consent circumvention. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency , FAccT '24, page 1733-1744, New York, NY, USA, 2024. Association for Computing Machinery. -
  24. [41][41] Felipe Romero Moreno. Generative ai and deepfakes: a human rights approach to tackling harmful content. International Review of Law, Computers & Technology , pages 1-30, 2024. -
  25. [46][46] Shiming Hu and Yifan Li. Policy interventions and regulations on generative artificial intelligence: Key gaps and core challenges. In Proceedings of the 25th Annual International Conference on Digital Government Research , dg.o '24, page 1034-1036, New York, NY, USA, 2024. Association for Computing Machinery. -
  26. [47][47] Philipp Hacker, Andreas Engel, and Marco Mauer. Regulating chatgpt and other large generative ai models. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency , FAccT '23, page 1112-1123, New York, NY, USA, 2023. Association for Computing Machinery. -
  27. [48][48] Kimberly A Christen. Does information really want to be free? indigenous knowledge systems and the question of openness. International Journal of Communication , 6:24, 2012. -
  28. [49][49] Nicholas Vincent, Hanlin Li, Nicole Tilly, Stevie Chancellor, and Brent Hecht. Data leverage: A framework for empowering the public in its relationship with technology companies. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency , pages 215-227, 2021. -
  29. [50][50] Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje F Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, et al. Aya dataset: An open-access collection for multilingual instruction tuning. arXiv preprint arXiv:2402.06619 , 2024. - · arXiv
  30. [53][53] Hellina Hailu Nigatu, Atnafu Lambebo Tonja, Benjamin Rosman, Thamar Solorio, and Monojit Choudhury. The zeno's paradox of 'low-resource' languages. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages 17753-17774, Miami, Florida, USA, November 2024. Association for Computational Linguistics. -
  31. [57][57] Danish Contractor, Daniel McDuff, Julia Katherine Haines, Jenny Lee, Christopher Hines, Brent Hecht, Nicholas Vincent, and Hanlin Li. Behavioral use licensing for responsible ai. In 2022 ACM Conference on Fairness, Accountability, and Transparency , FAccT '22, page 778-788. ACM, June 2022. -
  32. [61][61] Eric P.S. Baumer and M. Six Silberman. When the implication is not to design (technology). In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , CHI '11, page 2271-2274, New York, NY, USA, 2011. Association for Computing Machinery. -
  33. [62][62] Noura Howell, Audrey Desjardins, and Sarah Fox. Cracks in the success narrative: Rethinking failure in design research through a retrospective trioethnography. ACM Trans. Comput.-Hum. Interact. , 28(6), November 2021. -
  34. [64][64] Tzu-Sheng Kuo, Hong Shen, Jisoo Geum, Nev Jones, Jason I. Hong, Haiyi Zhu, and Kenneth Holstein. Understanding frontline workers' and unhoused individuals' perspectives on ai used in homeless services. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems , CHI '23, New York, NY, USA, 2023. Association for Computing Machinery. -
  35. [65][65] Alicia DeVrio, Motahhare Eslami, and Kenneth Holstein. Building, shifting, & employing power: A taxonomy of responses from below to algorithmic harm. In The 2024 ACM Conference on Fairness, Accountability, and Transparency , pages 1093-1106, 2024. -
  36. [66][66] Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. Auditing algorithms: Research methods for detecting discrimination on internet platforms. Data and discrimination: converting critical concerns into productive inquiry , 22(2014):4349-4357, 2014. -

Cited by

Sorted by citations

Authors
Dalal, S., Hall, S. M., & Johnson, N.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2411.09102
arXiv
2411.09102
OpenAlex
W4404450394
Cited by
1
References
0
Citations by year
2026: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Law, AI, and Intellectual Property
  • Social Sciences › Decision Sciences › Information Systems and Management › Technology's Impact on Media
Keywords
generative AI, participatory evaluation, marginalized communities, systemic change, AI researchers

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review