On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Open accessThe DOI matches its Crossref record; 156 of the 159 references it cites were found in a registry or an index, and 3 are not checked yet.
Abstract
The past 3 years of work in NLP have been characterized by the development and deployment of ever larger language models, especially for English. BERT, its variants, GPT-2/3, and others, most recently Switch-C, have pushed the boundaries of the possible both through architectural innovations and through sheer size. Using these pretrained models and the methodology of fine-tuning them for specific tasks, researchers have extended the state of the art on a wide array of tasks as measured by leaderboards on specific benchmarks for English. In this paper, we take a step back and ask: How big is too big? What are the possible risks associated with this technology and what paths are available for mitigating those risks? We provide recommendations including weighing the environmental and financial costs first, investing resources into curating and carefully documenting datasets rather than ingesting everything on the web, carrying out pre-development exercises evaluating how the planned approach fits into research and development goals and supports stakeholder values, and encouraging research directions beyond ever larger language models.
Checked 2026-11-22 · 3 references not checked yet
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 5×
“[39] in particular noted that training on a large dataset and fine-tuning for specific tasks leads to strictly increasing results on the GLUE tasks [138] for English as the hyperparameters of the model were increased.”
BackgroundMethodResultInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 4×
“documents linked to from Reddit [25], plus Wikipedia and a collection”
BackgroundMethodInfluentialRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“[26, 58], quantization [118, 153], factorized embedding parameterization and cross-layer parameter sharing [70], and progressive module replacing [146].”
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Proceedings of the 24th international conference on Machine learning
(2007) · doi:10.1145/1273496 · Cited by 11,630
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
The Mathematical Theory of Communication
Shannon, C. E., Weaver, W., & Wiener, N. (1950) · Physics Today · doi:10.1063/1.3067010 · Cited by 9,801
Research works printed but not linked: 48
- [4] American Psychological Association. 2019. Discrimination: What it is, and how to cope. https://www.apa.org/topics/discrimination (2019). Google Scholar
- 10.18653/v1 · DOI
- 10.18653/v1 · DOI
- [15] Ruha Benjamin. 2019. Race After Technology: Abolitionist Tools for the New Jim Code. Polity Press, Cambridge, UK. Crossref Google Scholar
- [16] Elettra Bietti and Roxana Vatanparast. 2020. Data Waste. Harvard International Law Journal 61 (2020). Google Scholar
- 10.18653/v1 · DOI
- 10.18653/v1 · DOI
- 10.1016/0010-0277(86) · doi:10.1016/0010-0277(86
- [38] Terrance de Vries, Ishan Misra, Changhan Wang, and Laurens van der Maaten. 2019. Does object recognition work for everyone?. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 52--59. Google Scholar
- 10.18653/v1 · DOI
- [42] Ethan Fast, Tina Vachovsky, and Michael Bernstein. 2016. Shirtless and Dangerous: Quantifying Linguistic Signals of Gender Bias in an Online Fiction Writing Community. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 10. Google Scholar
- 10.18653/v1 · DOI
- [45] Darja Fišer, Ruihong Huang, Vinodkumar Prabhakaran, Rob Voigt, Zeerak Waseem, and Jacqueline Wernimont (Eds.). 2018. Proceedings of the 2nd Workshop on Abusive Language Online (ALW2). Association for Computational Linguistics, Brussels, Belgium. https://www.aclweb.org/anthology/W18-5100 Google Scholar
- [50] Batya Friedman, Peter H. Kahn, Jr., and Alan Borning. 2006. Value sensitive design and information systems. In Human-Computer Interaction in Management Information Systems: Foundations, P Zhang and D Galletta (Eds.). M. E. Sharpe, Armonk NY, 348--372. Google Scholar
- 10.18653/v1 · DOI
- 10.18653/v1 · DOI
- 10.18653/v1 · DOI
- [67] Brendan Kennedy, Drew Kogon, Kris Coombs, Joseph Hoover, Christina Park, Gwenyth Portillo-Wightman, Aida Mostafazadeh Davani, Mohammad Atari, and Morteza Dehghani. 2018. A typology and coding manual for the study of hate-based rhetoric. PsyArXiv. July 18 (2018). Google Scholar
- [77] Marianna Martindale and Marine Carpuat. 2018. Fluency Over Adequacy: A Pilot Study in Measuring User Trust in Imperfect MT. In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track). Association for Machine Translation in the Americas, Boston, MA, 13--25. https://www.aclweb.org/anthology/W18-1803 Google Scholar
- [81] Douglas M McLeod. 2007. News coverage and social protest: How the media's protect paradigm exacerbates social conflict. Journal of Dispute Resolution (2007), 185. Google Scholar
- [87] Robert C. Moore and William Lewis. 2010. Intelligent Selection of Language Model Training Data. In Proceedings of the ACL 2010 Conference Short Papers. Association for Computational Linguistics, Uppsala, Sweden, 220--224. https://www.aclweb.org/anthology/P10-2041 Google Scholar
- 10.18653/v1 · DOI
- [92] Maggie Nelson. 2015. The Argonauts. Graywolf Press, Minneapolis. Google Scholar
- 10.18653/v1 · DOI
- [96] David Ortiz, Daniel Myers, Eugene Walls, and Maria-Elena Diaz. 2005. Where do we stand with newspaper data? Mobilization: An International Quarterly 10, 3 (2005), 397--419. Google Scholar
- 10.3115/v1 · doi:10.3115/v1
- 10.18653/v1 · DOI
- 10.18653/v1 · DOI
- [106] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9. Google Scholar
- 10.18653/v1 · DOI
- [109] Sarah T. Roberts, Joel Tetreault, Vinodkumar Prabhakaran, and Zeerak Waseem (Eds.). 2019. Proceedings of the Third Workshop on Abusive Language Online. Association for Computational Linguistics, Florence, Italy. https://www.aclweb.org/anthology/W19-3500 Google Scholar
- [112] Corby Rosset. 2020. Turing-NLG: A 17-billion-parameter language model by Microsoft. Microsoft Blog (2020). Google Scholar
- 10.18653/v1 · DOI
- 10.18653/v1 · DOI
- [121] Joonbo Shin, Yoonhyung Lee, and Kyomin Jung. 2019. Effective Sentence Scoring Method Using BERT for Speech Recognition. In Asian Conference on Machine Learning. 1081--1093. Google Scholar
- [124] Karen Spärck Jones. 2004. Language modelling's generative model: Is it rational? Technical Report. Computer Laboratory, University of Cambridge. Google Scholar
- [127] Katrina Srigley and Lorraine Sutherland. 2019. Decolonizing, Indigenizing, and Learning Biskaaybiiyang in the Field: Our Oral History Journey1. The Oral History Review (2019). Google Scholar
- 10.18653/v1 · DOI
- 10.18653/v1 · DOI
- 10.18653/v1 · DOI
- 10.18653/v1 · DOI
- 10.18653/v1 · DOI
- 10.18653/v1 · DOI
- 10.18653/v1 · DOI
- [150] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems. 5753--5763. Google Scholar
- 10.18653/v1 · DOI
- 10.18653/v1 · DOI
- 10.18653/v1 · DOI
Cited by
Items 1,401–1,500 of 6,931
Large Language Model based Educational Virtual Assistant using RAG Framework
Khan, U. H., Khan, M. H., & Ali, R. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.01.051 · Cited by 25
Security and Privacy Challenges of AIGC in Metaverse: A Comprehensive Survey
Zhang, S., Li, H., Sun, K., Chen, H., Wang, Y., & Li, S. (2025) · ACM Computing Surveys · doi:10.1145/3729419 · Cited by 19
The ROOTS Search Tool: Data Transparency for LLMs
Piktus, A., Akiki, C., Villegas, P., Laurençon, H., Dupont, G., Luccioni, A. S., Jernite, Y., & Rogers, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-demo.29 · Cited by 60
Mohamed, Y. H., Abdelfattah, M., Alhuwaider, S., Li, F., Zhang, X., Church, K., & Elhoseiny, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.600 · Cited by 47
The Gates Foundation’s new AI initiative: attempting to leapfrog global health inequalities?
Shaffer, J. D., Alenichev, A., & Faure, M. C. (2023) · BMJ Global Health · doi:10.1136/bmjgh-2023-013874 · Cited by 20
When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset
Zheng, L., Guha, N., Anderson, B. R., Henderson, P., & Ho, D. E. (2021) · arXiv · doi:10.48550/arxiv.2104.08671
von Struensee, S. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3873097 · Cited by 18
FinBERT—A Deep Learning Approach to Extracting Textual InformationPreprint
Huang, A., Wang, H., & Yang, Y. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3910214 · Cited by 14
Unsupervised and Distributional Detection of Machine-Generated Text
Gallé, M., Rozen, J., Kruszewski, G., & Elsahar, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.02878 · Cited by 14
Algorithmic narrativity: Literary experiments that drive technology
Rettberg, S., & Rettberg, J. W. (2024) · Dialogues on Digital Society · doi:10.1177/29768640241255848 · Cited by 14
“’ The same debate continues today about generative AI (Bender et al., 2021; Hayles, 2022).”
Analyzing Diversity in Healthcare LLM Research: A Scientometric PerspectivePreprint
Restrepo, D. S., Wu, C., Vásquez-Venegas, C., de Matos, J. C. R. G., Gallifant, J., Celi, L. A., Bitterman, D. S., & Nakayama, L. F. (2024) · medRxiv · doi:10.1101/2024.06.18.24309113 · Cited by 22
How Scientists Use Large Language Models to Program
O'Brien, G. (2025) · arXiv · doi:10.48550/arxiv.2502.17348 · Cited by 14
Playwriting with Large Language Models: Perceived Features, Interaction Strategies and Outcomes
Grigis, P., & De Angeli, A. (2024) · Proceedings of the 2024 International Conference on Advanced Visual Interfaces · doi:10.1145/3656650.3656688 · Cited by 13
Sociotechnical Implications of Generative Artificial Intelligence for Information Access
Mitra, B., Cramer, H. S. M., & Gurevich, O. (2024) · The information retrieval series · doi:10.1007/978-3-031-73147-1_7 · Cited by 36
LLMs, Turing tests and Chinese rooms: the prospects for meaning in large language models
Borg, E. (2025) · Inquiry · doi:10.1080/0020174x.2024.2446241 · Cited by 17
“All this leads to a suspicion that LLMs are nothing more than ‘ stochastic parrots ’ (to borrow the now well-known phrase of computational linguist Emily Bender et al., 2021) – that they are probability-based mechanisms which just parrot empty words back to us.”
Paolucci, C. (2025) · Semiotica · doi:10.1515/sem-2024-0204 · Cited by 17
Al-Qudimat, A. R., Fares, Z. E., Elaarag, M., Osman, M., Al‐Zoubi, R. M., & Aboumarzouk, O. M. (2025) · Health Science Reports · doi:10.1002/hsr2.70200 · Cited by 23
Automated Retrosynthesis Planning of Macromolecules Using Large Language Models and Knowledge Graphs
Ma, Q., Zhou, Y., & Li, J. (2025) · Macromolecular Rapid Communications · doi:10.1002/marc.202500065 · Cited by 20
Journalistic Agency and Power in the Era of Artificial Intelligence
Thäsler-Kordonouri, S., & Koliska, M. (2025) · Routledge eBooks · doi:10.1080/17512786.2025.2480238 · Cited by 28
“However, these algorithmically curated environments are often characterized by highly biased and inconsistent discussion points, one-sided and ideologically framed (Bender et al. 2021).”
Li, Z., Liu, Z., Sa, G., Sun, J., Hou, M., Tan, J., Sun, L., & Wei, J. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113147 · Cited by 17
Getto, G., Kelley, S., & Vance, B. (2025) · Journal of Technical Writing and Communication · doi:10.1177/00472816251332208 · Cited by 30
“As we have argued, AI requires oversight due to its fl aws and limitations (Bender et al., 2021).”
"As an Autistic Person Myself:" The Bias Paradox Around Autism in LLMs
Park, S., Min, A., Beltran, J. A., & Hayes, G. R. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713420 · Cited by 29
“Consequently, several emerging studies have focused on revealing what biases GAIs hold regarding people with disabilities [6, 8, 21, 24, 42].”
InfluentialGenerative AI, Quadruple Deception & Trust
Simon, J. (2025) · Social Epistemology · doi:10.1080/02691728.2025.2491087 · Cited by 38
“The core of Generative AI is the capacity to produce new verbal or visual products of increasingly high quality based on patterns discovered in massive amounts of data sources, most of them crawled from the web (cf. Bahrini et al. 2023; Bender et al. 2021).”
From Gaze to Data: Privacy and Societal Challenges of Using Eye-tracking Data to Inform GenAI Models
Abdrabou, Y., Özdel, S., Maquiling, V., Bozkir, E., & Kasneci, E. (2025) · ACM Symposium on Eye Tracking Research and Applications (ETRA) · doi:10.1145/3715669.3726788 · Cited by 19
“We also show the overall structure of the risks in Figure 1.”
InfluentialCalderón, R., & Herrera, F. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-04650-0 · Cited by 24
Behrend, T. S., & Landers, R. N. (2025) · Journal of Business and Psychology · doi:10.1007/s10869-025-10035-6 · Cited by 32
“LLM queries have been estimated to use nearly 100x the energy of a comparable search engine query (Bender et al., 2021).”
Dangol, A., Zhao, R., Wolfe, R., Ramanan, T., Kientz, J. A., & Yip, J. C. (2025) · ACM International Conference on Interaction Design and Children (IDC) · doi:10.1145/3713043.3728836 · Cited by 22
Huang, L. T.-L., & Huang, T.-R. (2025) · AI & Society · doi:10.1007/s00146-025-02533-1 · Cited by 29
A longitudinal analysis of declining medical safety messaging in generative AI models
Sharma, S., Alaa, A. M., & Daneshjou, R. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01943-1 · Cited by 36
Decoding Artificial Sociality: Technologies, Dynamics, Implications
Depounti, I., & Natale, S. (2025) · New Media & Society · doi:10.1177/14614448251359217 · Cited by 20
“…2024; Kotek et al., 2023; Rotaru et al., 2024; Wyer and Black, 2025) in relation to culture, gender, class, race, political views, and religion, and have been critiqued for operating with Whiteness and Western perspectives as their default (Bender et al., 2021; Benjamin, 2019; Broussard, 2023).”
Dual-process theory and decision-making in large language models
Brady, O., Nulty, P., Zhang, L., Ward, T. E., & McGovern, D. P. (2025) · Nature Reviews Psychology · doi:10.1038/s44159-025-00506-1 · Cited by 33
Learning against the machine: the double edged sword of (Gen)AI in STEM education
Wulff, P., & Kubsch, M. (2025) · International Journal of STEM Education · doi:10.1186/s40594-025-00588-6 · Cited by 23
“…to resemble human cognitive processes such as active inference (Pezzulo et al., 2021), which sets the GenAI capabilities into perspective: generating plausible sounding language that humans are adapted to – eventually counterfeiting human semiotics-based communication (Bender et al., 2021).”
Quantifying gender bias towards politicians in cross-lingual language models
Stańczak, K., Ray Choudhury, S., Pimentel, T., Cotterell, R., & Augenstein, I. (2023) · PLOS ONE · doi:10.1371/journal.pone.0277640 · Cited by 13
“However, alongside capturing the linguistic nuances of a language, they also perpetuate (and potentially amplify) biases (Zhao et al., 2017; Bender et al., 2021).”
BackgroundLinder, R., Hunter, C., McLemore, J., Dutta, S., Akbar, F., Grover, T., Breideband, T., Borghouts, J. W., Lu, Y., Mark, G. J., et al. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3492859 · Cited by 16
Classification of Poverty Condition Using Natural Language Processing
Santa, G. M., Escobar-Grisales, D., López-Pabón, F. O., Pérez-Toro, P. A., & Orozco‐Arroyave, J. R. (2022) · Social Indicators Research · doi:10.1007/s11205-022-02883-z · Cited by 18
“Existing language models reproduce particular points of view mainly valid in the rich parts of the world due to possible bias in the Internet users and the selected texts Bender et al. (2021), Salvatore et al. (2020), Prabhakaran et al. (2019).”
BackgroundFoundation Models in Healthcare: Opportunities, Biases and Regulatory Prospects in Europe
Wójcik, M. A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-12673-4_3 · Cited by 26
The Ecological Footprint of Neural Machine Translation Systems
Shterionov, D., & Vanmassenhove, E. (2023) · Machine translation · doi:10.1007/978-3-031-14689-3_10 · Cited by 34
“Another recent, yet heavily debated 11 , paper by Bender et al. (2021), focuses on a broader range of problems related to current trends in A.I. and the risks associated with these trends.”
BackgroundSood, S. C., & Pattinson, H. M. (2023) · International Journal of Innovation and Economic Development · doi:10.18775/ijied.1849-7551-7020.2015.86.2003 · Cited by 16
Honor Ethics: The Challenge of Globalizing Value Alignment in AI
Wu, S., Demetriou, D., & Husain, R. A. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594026 · Cited by 19
“Thus, more recent discourse in AI ethics has pushed beyond implementing assumed ethical stances, and now may disclose an epistemology [28], advocate for tracing connections with human context [2, 4, 49], or adapt to human context [36, 47].”
HyLECA: A Framework for Developing Hybrid Long-term Engaging Controlled Conversational Agents
Başar, E., Balaji, D., He, L., Hendrickx, I., Krahmer, E., de Bruijn, G., & Bosse, T. (2023) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3571884.3604404 · Cited by 22
Sreya, E. K., Sakshi, & Wadhwa, M. (2023) · International Conference on Computing, Communication, and Intelligent Systems (ICCCIS) · doi:10.1109/icccis60361.2023.10425000 · Cited by 16
A Survey of Recent Practice of Artificial Life in Visual Art
Wu, Z., Qu, H., & Kang Zhang, A. (2024) · Artificial Life · doi:10.1162/artl_a_00433 · Cited by 22
“Art, life, and nature always lead the important research topics in many interdisciplinary studies (Bender et al., 2021).”
BackgroundAliferis, C., & Simon, G. (2024) · Health informatics · doi:10.1007/978-3-031-39355-6_12 · Cited by 19
CARMEN: A Cognitively Assistive Robot for Personalized Neurorehabilitation at Home
Bouzida, A., Kubota, A., Sandoval, D. C., Twamley, E. W., & Riek, L. D. (2024) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/3610977.3634971 · Cited by 27
A Hybrid Transformer-LSTM Model With 3D Separable Convolution for Video Prediction
Mathai, M., Liu, Y., & Ling, N. (2024) · IEEE Access · doi:10.1109/access.2024.3375365 · Cited by 16
“Recently, researchers across the NLP and CV communities [30], [31] advocated to shift the focus to Green AI with energy-efficient deep learning solutions, rather than continuously pushing red AI methods to reach state-of-the-art (SOTA) results using massive computational power.”
BackgroundOn-device query intent prediction with lightweight LLMs to support ubiquitous conversations
Dubiel, M., Barghouti, Y., Kudryavtseva, K., & Leiva, L. A. (2024) · Scientific Reports · doi:10.1038/s41598-024-63380-6 · Cited by 29
“Training CAs based on large neural networks is associated with high energy consumption which, in turn, can have a long-term impact on the environment as highlighted in previous work 74,75 .”
Root, D. (2024) · AI & Society · doi:10.1007/s00146-024-01953-9 · Cited by 25
“Even though they process and generate language, the words do not mean anything to them (Bender et al. 2021, 611).”
Leveraging Generative AI Models in Urban Science
Balsa‐Barreiro, J., Cebrián, M., Menéndez, M., & Axhausen, K. W. (2024) · Current topics in behavioral neurosciences · doi:10.1007/7854_2024_482 · Cited by 16
AI as an Ecosystem — Ensuring Generative AI Is Safe and Effective
Coiera, E. W., & Navarro, D. F. (2024) · NEJM AI · doi:10.1056/aip2400611 · Cited by 16
Herman, E., Nicholas, D., Abrizah, A., Revez, J. M. R., Bravo, B. R., Świgoń, M., Clark, D., Xu, J., & Watkinson, A. (2024) · Learned Publishing · doi:10.1002/leap.1623 · Cited by 17
“Thus, as they go on to say, AI is in fact, as Bender et al. (2021) famously dubbed it, a ‘ stochastic parrot ’ , which does not offer a conversation, does not understand, does not communicate and does not really produce knowledge.”
Boggust, A., Sivaraman, V., Assogba, Y., Ren, D., Moritz, D., & Hohman, F. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456371 · Cited by 15
“However, this scale also incurs greater technical, financial, and environmental costs to integrate these models into everyday use [4].”
BackgroundPower and Play: Investigating "License to Critique" in Teams' AI Ethics Discussions
Widder, D. G., Dabbish, L., Herbsleb, J. D., & Martelaro, N. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3686938 · Cited by 25
“This speaks how perhaps team dynamics within companies, by default, do not enable vulnerability nor space to talk beyond work tasks, both of which appeared to be helpful when discussing ethics.”
Eacersall, D. C. F., Pretorius, L., Smirnov, I., Spray, E., Illingworth, S., Chugh, R., Strydom, S. C., Stratton-Maher, D., Simmons, J. S., Jennings, I., et al. (2025) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2025.8.2.9 · Cited by 44
CoPrompter: User-Centric Evaluation of LLM Instruction Alignment for Improved Prompt Engineering
Joshi, I., Shahid, S., Venneti, S., Vasu, M., Zheng, Y., Li, Y., Krishnamurthy, B., & Chan, G. Y.-Y. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712102 · Cited by 37
Artificial Intelligence Project Success Factors: Moral Decision-Making with Algorithms
Miller, G. J. (2021) · Annals of Computer Science and Information Systems · doi:10.15439/2021f26 · Cited by 24
“The energy costs from computing power and electricity consumption (for on-premise or cloud-based services) are relevant for training models [27, 46]; for an incremental increase inaccuracy, the cost of training a single model may be extreme (e.g., 0.1 increase in accuracy for 150,000 USD) [46].”
BackgroundInfluentialGender Bias in Machine Translation Systems
Ullmann, S. (2022) · Social and cultural studies of robots and AI · doi:10.1007/978-3-030-88615-8_7 · Cited by 17
Is love an abstract concept? A view of concepts from an interaction-based perspective
Rączaszek‐Leonardi, J., & Zubek, J. (2022) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2021.0356 · Cited by 20
Myths and legends in high-performance computing
Matsuoka, S., Domke, J., Wahib, M., Валентинович, Д. А., & Hoefler, T. (2023) · The International Journal of High Performance Computing Applications · doi:10.1177/10943420231166608 · Cited by 30
“So, even the smartest of internet parrots (Bender et al. 2021) that was itself created with massive high-performance computation running on a large accelerator system still has a long way to go.”
Backgroundvan Dijk, B., Kouwenhoven, T., Spruit, M. R., & van Duijn, M. J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.779 · Cited by 57
JASMINE: Arabic GPT Models for Few-Shot Learning
Nagoudi, E. M. B., Abdul-Mageed, M., Elmadany, A., Inciarte, A. A., & Khondaker, M. T. I. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.1040 · Cited by 40
On the Effects of Transformer Size on In- and Out-of-Domain Calibration
Dan, S., & Roth, D. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.180 · Cited by 32
“…number of parameters of such transformer based architectures, the high downstream accuracies comes at a large computational cost (Sharir et al., 2020; Bender et al., 2021) during http://cogcomp.org/page/publication_view/953 the pre-training stage and also to a lesser extent, while fine-tuning.”
BackgroundIntersectional Bias in Causal Language ModelsPreprint
Magee, L., Ghahremanlou, L., Soldatić, K., & Robertson, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.07691 · Cited by 51
“Despite acknowledging its importance [27, 34, 49], scholars have paid less attention to intersectional bias in language models [50].”
BackgroundResultInfluentialHi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models
Smith, E. M., & Williams, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.03300 · Cited by 13
Understanding the effects of human-written paraphrases in LLM-generated text detection
Lau, H. T., & Zubiaga, A. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2025.100151 · Cited by 13
Di Paolo, L. D., White, B., Guénin-Carlut, A., Constant, A., & Clark, A. (2024) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2023.0148 · Cited by 14
“Even if the system could be able to recover facts about the world’s structure, the recovered facts are explicitly based on the 4E human experience, hence the depiction of LLMs as ‘stochastic parrots’, due to their pattern-matching approach to the language detailed above [69].”
BackgroundFocus Agent: LLM-Powered Virtual Focus Group
Zhang, T., Zhang, X., Cools, R., & Simeone, A. L. (2024) · ACM International Conference on Intelligent Virtual Agents (IVA) · doi:10.1145/3652988.3673918 · Cited by 24
Madrid, J., Diehl, P., Selig, M., Rolauffs, B., Hans, F. P., Busch, H.-J., Scheef, T., & Benning, L. (2025) · JMIR Medical Education · doi:10.2196/58375 · Cited by 18
Big Data and Machine Learning in ESG Research*
Li, K. (2025) · Asia-Pacific Journal of Financial Studies · doi:10.1111/ajfs.12503 · Cited by 18
Evaluating ChatGPT on Korea's BIM Expertise Exam and improving its performance through RAG
Yu, Y., Kim, S. K., Lee, W., & Koo, B. (2025) · Journal of Computational Design and Engineering · doi:10.1093/jcde/qwaf033 · Cited by 20
Humanism strikes back? A posthumanist reckoning with ‘self-development’ and generative AI
Cadman, S., Tanner, C., & Pang, P. C. (2025) · AI & Society · doi:10.1007/s00146-025-02339-1 · Cited by 22
Elmobark, N. (2025) · International Journal of Environment Engineering and Education · doi:10.55151/ijeedu.v7i1.188 · Cited by 25
“For instance, studies by Bender et al. [11] and Joshi et al. [12] highlight that over 80% of NLP research prioritizes English, resulting in limited benchmarking and optimization for low-resource or morphologically complex languages like Swahili, Bengali, or Basque.”
How Scientists Use Large Language Models to Program
O’Brien, G. E. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713668 · Cited by 43
“Furthermore, Code LLMs are by de-sign statistical models of sequences rather than algebraic engines [10, 64].”
GPT models for text annotation: An empirical exploration in public policy research
Churchill, A., Pichika, S., Xu, C., & Liu, Y. (2025) · Policy Studies Journal · doi:10.1111/psj.70034 · Cited by 29
“Given the source of its training data, GPT models are very likely to be westernized and possess bias (Abid et al., 2021; Bender et al., 2021).”
Navigating the Future: Challenges and Opportunities in Intelligent RPA Implementation
Kumar, M., Dagar, M., Yadav, M., Choudhury, T., & Luu, T. M. N. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-4365-4.ch015 · Cited by 17
A Review of AI and Its Impact on Management Accounting and Society
Kerr, D., Smith, K. T., Smith, L. M., & Xu, T. (2025) · Journal of risk and financial management · doi:10.3390/jrfm18060340 · Cited by 37
Torkestani, M. S., Alameer, A., Shivakumara, P., & Mansouri, T. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11330-7 · Cited by 23
Moving towards critical AI literacy in LIS education: a scoping review and syllabi analysis
Bridges, L. M., Lopezosa, C., Velilla, M. C., & Ferrán-Ferrer, N. (2025) · The Electronic Library · doi:10.1108/el-06-2025-0224 · Cited by 30
Memduhoğlu, A., Fulman, N., Polat, N., & Atas, T. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.130171 · Cited by 17
Revealing emergent human-like conceptual representations from language prediction
Xu, N., Zhang, Q., Du, C., Luo, Q., Qiu, X., Huang, X., & Zhang, M. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2512514122 · Cited by 36
Adaptive Robust Watermarking for Large Language Models via Dynamic Token Embedding Perturbation
Zeng, Z., Lin, H., Zhang, S., & Wang, B. (2026) · IEEE Access · doi:10.1109/access.2026.3653833 · Cited by 33
“…breakthrough has simultaneously raised critical concerns The associate editor coordinating the review of this manuscript and approving it for publication was Ayman El-Baz . regarding copyright protection, content provenance verification, and the potential misuse of AI-generated text [5], [6], [7].”
The silicon gaze: A typology of biases and inequality in LLMs through the lens of place
Kerche, F. W., Zook, M., & Graham, M. (2026) · Platforms & Society · doi:10.1177/29768624251408919 · Cited by 33
A survey on text classification: Practical perspectives on the Italian language
Gasparetto, A., Zangari, A., Marcuzzo, M., & Albarelli, A. (2022) · PLoS ONE · doi:10.1371/journal.pone.0270904 · Cited by 23
A Modeling Approach for Measuring the Performance of a Human-AI Collaborative Process
Sankaran, G., Palomino, M. A., Knahl, M., & Siestrup, G. (2022) · Applied Sciences · doi:10.3390/app122211642 · Cited by 20
Beyond Digital "Echo Chambers": The Role of Viewpoint Diversity in Political Discussion
Hada, R., Fard, A. E., Shugars, S., Bianchi, F., Rossini, P., Hovy, D., Tromble, R., & Tintarev, N. (2023) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3539597.3570487 · Cited by 22
Stakeholder-accountability model for artificial intelligence projects
Maxmetrics, H., & Miller, G. J. (2022) · Journal of Economics and Management · doi:10.22367/jem.2022.44.18 · Cited by 24
“Finally, the model training involved in building AI systems may be ener-gy-intense, using sufficient computing energy to generate carbon emissions with environmental impact (Bender et al., 2021; Ryan & Stahl, 2021).”
BackgroundMethodArtificial intelligence in hybrid and information warfare
Moy, W. R., & Gradoń, K. (2023) · Routledge eBooks · doi:10.4324/9781003284093-4 · Cited by 18
Risk and Harm: Unpacking Ideologies in the AI Discourse
Ferri, G., & Gloerich, I. (2023) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3571884.3603751 · Cited by 18
“3It is also significant that the FLI letter cites a famous paper by the DAIR authors [5], and that the DAIR letter is a direct and critical response to FLI.”
BackgroundInvestigating the Benefits of Free-Form Rationales
Sun, J., Swayamdipta, S., May, J., & Ma, X. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.432 · Cited by 29
Gendered AI: German news media discourse on the future of work
Carstensen, T., & Ganz, K. (2023) · AI & Society · doi:10.1007/s00146-023-01747-5 · Cited by 25
“…et al. 2020), on data crowdwork in the global south (Altenried 2020; Gray and Suri 2019), the exploitation of human data (Iyer et al. 2021), indigenous knowledge (Maitra 2020), or natural resources (Bender et al. 2021; Crawford 2021), is not addressed, and neither is decolonial activism on AI.”
Haase, J., Walker, P. B., Berardi, O., & Karwowski, W. (2023) · Technologies · doi:10.3390/technologies11060165 · Cited by 18
“For example, Bender et al. [4] pointed out that LLMs have been built on foundational work from Natural Language Processing (NLP) [70].”
BackgroundBarea, M. A., Boeren, D., & Gonçalves, J. (2023) · AI & Society · doi:10.1007/s00146-023-01804-z · Cited by 19
“Given the vast number of parameters and size of the training datasets used, these models are increasingly more challenging to curate (Bender et al. 2021).”
InfluentialTowards more sustainable and trustworthy reporting in machine learning
Fischer, R., Liebig, T., & Morik, K. J. (2024) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-024-01020-3 · Cited by 19
“The last aspect is especially noteworthy, as modern ML can act as a tool for making the world more sustainable (van Wynsberghe 2021; Kar et al. 2022), but has also been shown to significantly impact our climate (Strubell et al. 2020; Bender et al. 2021; Patterson et al. 2021).”
Machine learning data practices through a data curation lens: An evaluation framework
Bhardwaj, E., Gujral, H., Wu, S., Zogheib, C., Maharaj, T., & Becker, C. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658955 · Cited by 44
AI Transforming Business and Everyday Life
Bialkova, S. (2024) · Springer eBooks · doi:10.1007/978-3-031-56471-0_9 · Cited by 13
Li, Y. (2024) · Studies in Applied Linguistics and TESOL · doi:10.52214/salt.v24i1.12864 · Cited by 12
Debiasing large language models: research opportunities*
Yogarajan, V., Dobbie, G., & Keegan, T. T. (2024) · Journal of the Royal Society of New Zealand · doi:10.1080/03036758.2024.2398567 · Cited by 30
“LLMs inherit stereotypes and misrepresen-tations of societies from the training data (Bender et al. 2021; Yogarajan et al. 2023a), and can also amplify these biases (Abid et al. 2021; Crutchley 2021).”
Grassini, S. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2422755 · Cited by 25
“Early research has mainly examined the carbon footprint associated with the training of these models (Bender et al., 2021; Schwartz et al., 2020; Wu et al., 2022).”
Assessing Cognitive and Social Awareness among Group Members in AI-assisted Collaboration
de Jong, S., Wester, J., Schrills, T., Secher, K. S., Griggio, C. F., & van Berkel, N. (2024) · International Conference on Mobile and Ubiquitous Multimedia (MUM) · doi:10.1145/3701571.3701582 · Cited by 19
Parity benchmark for measuring bias in LLMs
Simpson, S., Nukpezah, J., Brooks, K., & Pandya, R. (2024) · AI and Ethics · doi:10.1007/s43681-024-00613-4 · Cited by 26
“Bias in LLMs can arise from multiple sources, including biases present in the training data, biases encoded in the model architecture or parameters, and biases introduced during fine-tuning or deployment stages [8 ].”
Tran, H., & Stell, A. (2024) · Australian Review of Applied Linguistics · doi:10.1075/aral.24135.tra · Cited by 20
Metadata
- Authors
- Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S.
- Affiliations
- University of Washington (US)
- Black in AI, Palo Alto, CA, USA
- The Aether
- Venue
- Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2021-03
- Pages
- 610-623
- OpenAlex
- W3133702157
- Cited by
- 6,988
- References
- 117
- Citations by year
- 2026: 2484 · 2025: 1887 · 2024: 1153 · 2023: 842 · 2022: 384 · 2021: 229
- FWCI
- 300.72
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by)
- Creighton Digital Repository (CDR) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language models, model size, AI risks, financial costs, data curation, dataset documentation, stakeholder value, BERT, GPT, fine-tuning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.