The Illusion of Artificial Inclusion
Open accessThe DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Abstract
Human participants play a central role in the development of modern artificial intelligence (AI) technology, in psychological science, and in user research. Recent advances in generative AI have attracted growing interest to the possibility of replacing human participants in these domains with AI surrogates. We survey several such “substitution proposals” to better understand the arguments for and against substituting human participants with modern generative AI. Our scoping review indicates that the recent wave of these proposals is motivated by goals such as reducing the costs of research and development work and increasing the diversity of collected data. However, these proposals ignore and ultimately conflict with foundational values of work with human participants: representation, inclusion, and understanding. This paper critically examines the principles and goals underlying human participation to help chart out paths for future work that truly centers and empowers participants.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
I.—COMPUTING MACHINERY AND INTELLIGENCE
TURING, A. M. (1950) · Mind · doi:10.1093/mind/lix.236.433 · Cited by 10,290 · cited 1×
“Developing an AI system capable of proficiently conversing with humans has been a longstanding aspiration of AI researchers (e.g., [136]), and NLP systems have existed for decades, in forms including chatbots, text completion, and search engines.”
BackgroundThumbs up?: sentiment classification using machine learning techniques
Pang, B., Lee, L., & Vaithyanathan, S. (2002) · Proceedings of the ACL-02 conference on Empirical methods in natural language processing - EMNLP '02 · doi:10.3115/1118693.1118704 · Cited by 7,024
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 1×
“Critics argue that such claims of understanding and general intelligence generate overconfidence in model abilities, which in turn could deter efforts at oversight and encourage unsafe or inappropriate applications [19, 126, 145].”
BackgroundThe global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0088-2 · Cited by 6,220 · cited 1×
“Non-maleficence and justice are similarly two of the foremost concerns of policymakers and ethicists over AI development [76].”
BackgroundSurvey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Nature · doi:10.1038/466029a · Cited by 3,441 · cited 1×
“As Crockett and Messeri [43] point out, the training data used for LLMs echoes and reinforces the focus of psychology and HCI research on western, educated, industrialized, rich, and democratic people [63, 82].”
BackgroundEkman, P. (1993) · American Psychologist · doi:10.1037/0003-066x.48.4.384 · Cited by 3,441
Research works printed but not linked: 94
- Daron Acemoğlu and James A. Robinson. 2016. Paths to inclusive political institutions. Economic History of Warfare and State Formation (2016), 3–50.
- William Agnew, Kevin R. McKee, Iason Gabriel, Jackie Kay, William Isaac, A. Stevie Bergman, Seliem El-Sayed, and Shakir Mohamed. 2023. Technologies of resistance to AI. In Proceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization. 1–13.
- 10.1177/17456916221105213 · DOI
- 10.1177/0010414017720702 · DOI · Paper page
- 10.1080/01944366908977225 · DOI · Paper page
- Neda Atanasoski and Kalindi Vora. 2019. Surrogate humanity: Race, robots, and the politics of technological futures. Duke University Press.
- Mohammad Atari, Mona J. Xue, Peter S. Park, Damián Blasi, and Joseph Henrich. 2023. Which humans?PsyArXiv (2023).
- Rowland Atkinson and John Flint. 2001. Accessing hidden and hard-to-reach populations: Snowball research strategies. Social Research Update 33, 1 (2001), 1–4.
- Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, 2022. Constitutional AI: Harmlessness from AI feedback. arXiv preprint arXiv:2212.08073 (2022). · arXiv
- 10.3102/0013189X20927363 · DOI · Paper page
- 10.1016/S0020-7373(87)80047-0 · DOI · Paper page
- Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 5185–5198.
- Bernard Berelson. 1952. Content analysis in communication research. Free Press.
- 10.18653/v1/2022.findings-acl.31 · DOI
- 10.1145/3593013.3594019 · DOI
- 10.1080/13600869.1986.9966228 · doi:10.1080/13600869.1986.9966228
- 10.1016/S0953-5438(97)00032-5 · DOI · Paper page
- Allan M. Brandt. 1978. Racism and research: The case of the Tuskegee Syphilis Study. Hastings Center Report (1978), 21–29.
- 10.1145/3404983.3405506 · DOI
- 10.1145/3544549.3582749 · DOI
- 10.1145/3442188.3445872 · DOI
- Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021). · arXiv
- Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, 2022. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 (2022). · arXiv
- Luigina Ciolfi, Myriam Lewkowicz, and Kjeld Schmidt. 2023. Computer-supported cooperative work. In Handbook of Human-Computer Interaction. Springer, 1–26.
- Richard W. Coan. 1979. Psychologists: Personal and theoretical pathways. Irvington Publishers.
- 10.1146/annurev.psych.49.1.559 · DOI · Paper page
- 10.1093/cdj/bsn010 · DOI · Paper page
- Critical Platform Studies Group, Lilly Irani, Niloufar Salehi, Joyojeet Pal, Andrés Monroy-Hernández, Elizabeth Churchill, and Sneha Narayan. 2019. Patron or poison? Industry funding of HCI research. In Conference Companion Publication of the 2019 on Computer Supported Cooperative Work and Social Computing. 111–115.
- Molly Crockett and Lisa Messeri. 2023. Should large language models replace human participants?PsyArXiv (2023).
- Robert A. Dahl. 2008. Polyarchy: Participation and opposition. Yale University Press.
- Kurt Danziger. 1994. Constructing the subject: Historical origins of psychological research. Cambridge University Press.
- Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray. 2023. Can AI language models replace human participants?Trends in Cognitive Sciences (2023).
- 10.1177/1463499610370517 · DOI · Paper page
- Orlando Fals-Borda and Muhammad Anisur Rahman. 1991. Action and knowledge: Breaking the monopoly with participatory action-research. Intermediate Technology Publications.
- 10.1177/20539517221082027 · DOI · Paper page
- Marco A. Gerosa, Bianca Trinkenreich, Igor Steinmacher, and Anita Sarma. 2023. Can AI serve as a substitute for human subjects in software engineering research?arXiv preprint arXiv:2311.11081 (2023). · arXiv
- 10.1007/s10606-021-09422-3 · DOI
- 10.1145/3593013.3594071 · DOI · Paper page
- 10.1609/aimag.v30i4.2271 · DOI · Paper page
- Jonathan Grudin. 2022. From tool to partner: The evolution of human-computer interaction. Springer Nature.
- 10.1145/3544548.3580688 · DOI · Paper page
- David Heckerman, David Maxwell Chickering, Christopher Meek, Robert Rounthwaite, and Carl Kadie. 2000. Dependency networks for inference, collaborative filtering, and data visualization. Journal of Machine Learning Research 1, Oct (2000), 49–75.
- Tom Heyman and Geert Heyman. 2023. The impact of ChatGPT on human data collection: A case study involving typicality norming data. Behavior Research Methods (2023), 1–8.
- Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, 2022. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303 (2022). · arXiv
- John J. Horton. 2023. Large language models as simulated economic agents: What can we learn from homo silicus?Technical Report. National Bureau of Economic Research.
- @infoxiao. 2023. A trend I see at #CHI23 is HCI researchers beginning to use LLMs to create synthetic user studies. E.g. https://dl.acm.org/doi/10.1145/3544548.3580688. While it is convenient I do worry about the potential representational harms posed by such trend – experiences of minorities being erased.https://web.archive.org/web/20240113081958/https://nitter.net/infoxiao/status/1651160544307544064. · DOI · Paper page
- 10.1177/1948550617691100 · DOI · Paper page
- @jacyanthis. 2023. Here’s the latest iteration of the “techniques like RLHF are just putting superficial smiley faces on an opaque, alien, Shoggoth intelligence that we have no real control over” AI safety meme. Artist: Anna Husfeldt, CC-BY SA 3.0. https://web.archive.org/web/20230306110137/https://twitter.com/jacyanthis/status/1631291175381475331.
- Michelle K. Jamieson, Gisela H. Govaart, and Madeleine Pownall. 2023. Reflexivity in quantitative research: A rationale and beginner’s guide. Social and Personality Psychology Compass (2023), e12735.
- 10.1145/3593013.3594060 · DOI
- J. Jones. 2018. Facebook users’ privacy concerns up since 2011. The Gallup Organization.
- 10.1016/j.jretconser.2023.103580 · DOI · Paper page
- 10.1080/00049180500325694 · DOI · Paper page
- Steven Levy. 1992. Artificial life: The quest for a new creation. Random House Inc.
- 10.1177/20539517211047734 · DOI · Paper page
- 10.1162/daed_a_01905 · DOI · Paper page
- 10.1111/j.1467-9280.2009.02453.x · DOI · Paper page
- 10.1353/tech.2010.0009 · DOI · Paper page
- 10.1080/10720537.2019.1635924 · DOI
- Renate Mayntz. 2005. Forschungsmethoden und erkenntnispotential: Natur-und sozialwissenschaften im vergleich. Max-Planck-Institut für Gesellschaftsforschung.
- Kevin R. McKee. 2023. Human participants in AI research: Ethics and transparency in practice. arXiv preprint arXiv:2311.01254 (2023). · arXiv
- 10.1037/0003-066X.38.4.379 · DOI · Paper page
- 10.21236/ADA125318 · DOI
- 10.1002/ejsp.2724 · DOI
- Matt Novak. 2023. That viral image of Pope Francis wearing a white puffer coat is totally fake. Forbes (2023). https://web.archive.org/web/20230914181410/https://www.forbes.com/sites/mattnovak/2023/03/26/that-viral-image-of-pope-francis-wearing-a-white-puffer-coat-is-totally-fake/
- OpenAI. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774 (2023). · arXiv
- 10.18653/v1/2021.acl-long.329 · DOI · Paper page
- Amandalynne Paullada, Inioluwa Deborah Raji, Emily M. Bender, Emily Denton, and Alex Hanna. 2021. Data and its (dis)contents: A survey of dataset development and use in machine learning research. Patterns 2, 11 (2021).
- 10.1177/0739456X11410979 · DOI · Paper page
- Julian Reiss and Jan Sprenger. 2014. Scientific objectivity. In The Stanford Encyclopedia of Philosophy. Zalta, Ed.
- Jennifer Rhee. 2018. The robotic imaginary: The human and the price of dehumanized labor. University of Minnesota Press.
- David K. Robinson. 2001. Reaction-time experiments in Wundt’s institute and beyond. In Wilhelm Wundt in History: The Making of a Scientific Psychology. Springer, 161–204.
- Esther Rolf, Theodora T. Worledge, Benjamin Recht, and Michael Jordan. 2021. Representation matters: Assessing the importance of subgroup allocations in training data. In Proceedings of the 38th International Conference on Machine Learning. Proceedings of Machine Learning Research, 9040–9051.
- Kevin Roose. 2023. A conversation with Bing’s chatbot left me deeply unsettled. The New York Times (2023). https://web.archive.org/web/20230914044532/https://www.nytimes.com/2023/02/16/technology/bing-chatbot-microsoft-chatgpt.html
- 10.1145/3466160 · DOI
- Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect?arXiv preprint arXiv:2303.17548 (2023). · arXiv
- 10.1353/hpu.0.0323 · DOI · Paper page
- 10.1016/0004-3702(78)90012-7 · DOI
- 10.1145/3498366.3505816 · DOI · Paper page
- Murray Shanahan. 2022. Talking about large language models. arXiv preprint arXiv:2212.03551 (2022). · arXiv
- 10.18653/v1/2021.naacl-main.295 · DOI · Paper page
- Åsne Stige, Efpraxia D. Zamani, Patrick Mikalef, and Yuzhen Zhu. 2023. Artificial intelligence (AI) for user experience (UX) design: A systematic literature review and future research agenda. Information Technology & People (2023).
- Derek Thompson. 2022. Breakthroughs of the year. The Atlantic (2022). https://web.archive.org/web/20230421060355/https://www.theatlantic.com/newsletters/archive/2022/12/technology-medicine-law-ai-10-breakthroughs-2022/672390/
- Nitasha Tiku. 2022. The Google engineer who thinks the company’s AI has come to life. The Washington Post (2022). https://web.archive.org/web/20230826183724/https://www.washingtonpost.com/technology/2022/06/11/google-ai-lamda-blake-lemoine/
- 10.1007/BF00430638 · doi:10.1007/bf00430638
- James Vincent. 2022. All these images were generated by Google’s latest text-to-image AI. The Verge (2022). https://web.archive.org/web/20230725102931/https://www.theverge.com/2022/5/24/23139297/google-imagen-text-to-image-ai-system-examples-paper
- 10.1145/2470654.2470716 · DOI · Paper page
- 10.18653/v1/2021.findings-emnlp.354 · DOI · Paper page
- Max Weber, A. M. Henderson, and Talcott Parsons. 1947. The theory of social and economic organization. Free Press.
- Benjamin Weiser. 2023. Here’s what happens when your lawyer uses ChatGPT. The New York Times (2023). https://web.archive.org/web/20230913041650/https://www.nytimes.com/2023/05/27/nyregion/avianca-airline-lawsuit-chatgpt.html
- Blaise Agüera y Arcas. 2022. Do large language models understand us?Daedalus 151, 2 (2022), 183–197.
- 10.1145/2858036.2858523 · DOI · Paper page
- 10.1177/016224399101600106 · DOI · Paper page
- 10.1145/3491101.3516506 · DOI · Paper page
Cited by
Large Language Models Show Amplified Cognitive Biases in Moral Decision-Making
Cheung, V., Maier, M., & Lieder, F. (2025) · doi:10.31234/osf.io/aj46b_v3 · Cited by 53
Take caution in using LLMs as human surrogates
Gao, Y., Lee, D., Burtch, G., & Fazelpour, S. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2501660122 · Cited by 52
Playwriting with Large Language Models: Perceived Features, Interaction Strategies and Outcomes
Grigis, P., & De Angeli, A. (2024) · Proceedings of the 2024 International Conference on Advanced Visual Interfaces · doi:10.1145/3656650.3656688 · Cited by 13
Krauß, V., McGill, M., Kosch, T., Thiel, Y. M., Schön, D., & Gugenheimer, J. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713083 · Cited by 13
Towards interactive evaluations for interaction harms in human-AI systems
Ibrahim, L., Huang, S., Bhatt, U., Ahmad, L., & Anderljung, M. (2024) · arXiv · doi:10.48550/arxiv.2405.10632 · Cited by 11
“In Table 2, we present a non-exhaustive set of these metrics for assessing both human impact and human-AI performance.”
Metadata
- Authors
- Agnew, W., Bergman, A. S., Chien, J., Díaz, M., El-Sayed, S., Pittman, J., Mohamed, S., & McKee, K. R.
- Venue
- Proceedings of the CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2024-05-11
- Pages
- 1-12
- OpenAlex
- W4396833703
- Cited by
- 58
- References
- 43
- FWCI
- 27.19
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The Illusion of Artificial Inclusion. https://app.cochat.ai/papers/10.1145/3613904.3642703
- The paper
- Agnew, W., Bergman, A. S., Chien, J., Díaz, M., El-Sayed, S., Pittman, J., Mohamed, S., & McKee, K. R. (2024). The Illusion of Artificial Inclusion. Proceedings of the CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3613904.3642703
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.