On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Open accessThe DOI matches its Crossref record; 156 of the 159 references it cites were found in a registry or an index, and 3 are not checked yet.
Abstract
The past 3 years of work in NLP have been characterized by the development and deployment of ever larger language models, especially for English. BERT, its variants, GPT-2/3, and others, most recently Switch-C, have pushed the boundaries of the possible both through architectural innovations and through sheer size. Using these pretrained models and the methodology of fine-tuning them for specific tasks, researchers have extended the state of the art on a wide array of tasks as measured by leaderboards on specific benchmarks for English. In this paper, we take a step back and ask: How big is too big? What are the possible risks associated with this technology and what paths are available for mitigating those risks? We provide recommendations including weighing the environmental and financial costs first, investing resources into curating and carefully documenting datasets rather than ingesting everything on the web, carrying out pre-development exercises evaluating how the planned approach fits into research and development goals and supports stakeholder values, and encouraging research directions beyond ever larger language models.
Checked 2026-11-22 · 3 references not checked yet
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 5×
“[39] in particular noted that training on a large dataset and fine-tuning for specific tasks leads to strictly increasing results on the GLUE tasks [138] for English as the hyperparameters of the model were increased.”
BackgroundMethodResultInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 4×
“documents linked to from Reddit [25], plus Wikipedia and a collection”
BackgroundMethodInfluentialRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“[26, 58], quantization [118, 153], factorized embedding parameterization and cross-layer parameter sharing [70], and progressive module replacing [146].”
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Proceedings of the 24th international conference on Machine learning
(2007) · doi:10.1145/1273496 · Cited by 11,630
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
The Mathematical Theory of Communication
Shannon, C. E., Weaver, W., & Wiener, N. (1950) · Physics Today · doi:10.1063/1.3067010 · Cited by 9,801
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
The American Psychological Association
Langfeld, H. S. (1919) · Science · doi:10.1126/science.49.1254.51 · Cited by 7,475
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 1×
“0) [108], and the Situations With Adversarial Generations benchmark (SWAG) [155], all datasets designed to test language understanding and/or commonsense reasoning.”
BackgroundCrenshaw, K. (1998) · Feminism And Politics · doi:10.1093/oso/9780198782063.003.0016 · Cited by 5,672
Clark, H. H. (1996) · doi:10.1017/cbo9780511620539 · Cited by 5,123
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653
Colman (2022) · doi:10.1073/pnas · Cited by 4,370
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308 · cited 4×
“…the models retain from the data, we see a trend in reducing the size of these models using various techniques such as knowledge distillation [26, 58], quantization [118, 153], factorized embedding parameterization and cross-layer parameter sharing [70], and progressive module replacing [146].”
InfluentialGLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222
Energy and Policy Considerations for Deep Learning in NLP
Strubell, E., Ganesh, A., & McCallum, A. (2019) · arXiv · doi:10.48550/arxiv.1906.02243 · Cited by 3,735
Energy and Policy Considerations for Deep Learning in NLP
Strubell, E., Ganesh, A., & McCallum, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1355 · Cited by 3,684 · cited 3×
“recently benchmarked model training and development costs in terms of dollars and estimated $2 emissions [129].”
BackgroundInfluentialde Saussure, F., Bally, C., Sechehaye, A., Reidlinger, A., & Baskin, W. (1960) · The Journal of American Folklore · doi:10.2307/538001 · Cited by 3,557
SciBERT: A Pretrained Language Model for Scientific Text
Beltagy, I., Lo, K., & Cohan, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1371 · Cited by 3,218 · cited 2×
“However, in both application areas, the training data has been shown to have problematic characteristics [18, 38, 42, 47, 61] resulting in models that encode stereotypical and derogatory associations along gender, race, ethnicity, and disability status [11, 12, 69, 69, 132, 132, 157].”
BackgroundModel Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · cited 2×
“While documentation allows for potential accountability [13, 52, 86], undocumented training data perpetuates harm without recourse.”
Self-citationGebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Self-citationReferring as a collaborative process
Clark, H. H., & Wilkes-Gibbs, D. (1986) · Cognition · doi:10.1016/0010-0277(86)90010-7 · Cited by 2,401
Nass, C., Steuer, J., & Tauber, E. R. (1994) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/191666.191703 · Cited by 2,148 · cited 1×
“This can seem counter-intuitive given the increasingly fluent qualities of automatically generated text, but we have to account for the fact that our perception of natural language text, regardless of how it was generated, is mediated by our own linguistic competence and our predisposition to interpret communicative a…”
Buciluǎ, C., Caruana, R., & Niculescu-Mizil, A. (2006) · Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1150402.1150464 · Cited by 2,068 · cited 1×
“…the models retain from the data, we see a trend in reducing the size of these models using various techniques such as knowledge distillation [26, 58], quantization [118, 153], factorized embedding parameterization and cross-layer parameter sharing [70], and progressive module replacing…”
Merz, S. (2020) · Ethnic and Racial Studies · doi:10.1080/01419870.2020.1715454 · Cited by 1,870
Schwartz, R., Dodge, J., Smith, N. A., & Etzioni, O. (2020) · Communications of the ACM · doi:10.1145/3381831 · Cited by 1,647
ON THE MATHEMATICAL THEORY OF COMMUNICATION
Takada, Y. (1954) · The Japanese journal of psychology · doi:10.4992/jjpsy.25.110 · Cited by 1,631
Pre-trained models for natural language processing: A survey
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1647-3 · Cited by 1,591 · cited 1×
“For a more in-depth survey of pretrained LMs, see [105].”
BackgroundA primer in BERTology: What we know about how BERT works
Rogers, A., Kovaleva, O., & Rumshisky, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00349 · Cited by 1,552
Conceptual pacts and lexical choice in conversation.
Brennan, S. E., & Clark, H. H. (1996) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037//0278-7393.22.6.1482 · Cited by 1,457
Proceedings of the 26th International Conference on Neural Information Processing Systems
Burges, C. J. C., Bottou, L., Welling, M., Zoubin Ghahramani, A., & Weinberger, K. Q. (2013) · Cited by 1,323
BERT Rediscovers the Classical NLP Pipeline
Tenney, I., Das, D., & Pavlick, E. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1452 · Cited by 1,307
Mariani, J. (2018) · HAL · Cited by 1,151
Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data
Bender, E. M., & Koller, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.463 · Cited by 1,082
Self-citationThe Use of Newspaper Data in the Study of Collective Action
Earl, J., Martin, A., McCarthy, J. D., & Soule, S. A. (2004) · Annual Review of Sociology · doi:10.1146/annurev.soc.30.012703.110603 · Cited by 1,074 · cited 1×
“Media coverage can fail to cover protest events and social movements [41, 96] and can distort events that challenge state power [36].”
BackgroundLanguage (Technology) is Power: A Critical Survey of “Bias” in NLP
Blodgett, S. L., Barocas, S., Daumé III, H., & Wallach, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.485 · Cited by 1,050 · cited 2×
“LM (even for a given protected class) requires engaging with the systems of power that lead to the harmful outcomes such a system would seek to prevent [19].”
BackgroundSwitch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Fedus, W., Zoph, B., & Shazeer, N. (2021) · arXiv · doi:10.48550/arxiv.2101.03961 · Cited by 1,005
Speaker-listener neural coupling underlies successful communication
Stephens, G. J., Silbert, L. J., & Hasson, U. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1008662107 · Cited by 970 · cited 1×
“The fact that human-human communication is a jointly constructed activity [29, 128] is most clearly true in co-situated spoken or signed communication, but we use the same facilities for producing language that is intended for audiences not co-present with us (readers, listeners, watchers at a distance in time or spac…”
BackgroundSpencer, S. J., Logel, C., & Davies, P. G. (2016) · Annual Review of Psychology · doi:10.1146/annurev-psych-073115-103235 · Cited by 920
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019) · arXiv · doi:10.48550/arxiv.1909.08053 · Cited by 893
Bender, E. M., & Friedman, B. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00041 · Cited by 854 · cited 2×
“While documentation allows for potential accountability [13, 52, 86], undocumented training data perpetuates harm without recourse.”
BackgroundSelf-citationValue Sensitive Design: Shaping Technology with Moral Imagination
Friedman, B., & Hendry, D. G. (2019) · The MIT Press eBooks · doi:10.7551/mitpress/7585.001.0001 · Cited by 838
Confronting Environmental Racism: Voices from the Grassroots.
Brown, P., & Bullard, R. (1994) · Contemporary Sociology · doi:10.2307/2076358 · Cited by 831
The design and implementation of Xiaoice, an empathetic social chatbot
Zhou, L., Gao, J., Li, D., & Shum, H.-Y. (2020) · Computational Linguistics · doi:10.1162/coli_a_00368 · Cited by 777
The Design and Implementation of XiaoIce, an Empathetic Social Chatbot
Zhou, L., Gao, J., Li, D., & Shum, H.-Y. (2018) · arXiv · doi:10.48550/arxiv.1812.08989 · Cited by 777
#MeToo and the promise and pitfalls of challenging rape culture through digital feminist activism
Mendes, K., Ringrose, J., & Keller, J. (2018) · European Journal of Women's Studies · doi:10.1177/1350506818765318 · Cited by 772
ERNIE 2.0: A Continual Pre-Training Framework for Language Understanding
Sun, Y., Wang, S., Li, Y., Feng, S., Tian, H., Wu, H., & Wang, H. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6428 · Cited by 766
The State and Fate of Linguistic Diversity and Inclusion in the NLP World
Joshi, P., Santy, S., Budhiraja, A., Bali, K., & Choudhury, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.560 · Cited by 766
Speaking while monitoring addressees for understanding
Clark, H. H., & Krych, M. A. (2004) · Journal of Memory and Language · doi:10.1016/j.jml.2003.08.004 · Cited by 739
Two decades of statistical language modeling: where do we go from here?
Rosenfeld, R. (2000) · Proceedings of the IEEE · doi:10.1109/5.880083 · Cited by 735
Extracting Training Data from Large Language Models
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07805 · Cited by 727 · cited 1×
“For example, [28] demonstrate a methodology for extracting personally identifiable information (PII) from an LM and find that larger LMs are more susceptible to this style of attack than smaller ones.”
ERNIE: Enhanced Representation through Knowledge Integration
Sun, Y., Wang, S., Li, Y., Feng, S., Chen, X., Zhang, H., Tian, X., Zhu, D., Tian, H., & Wu, H. R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.09223 · Cited by 721
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.301 · Cited by 693
SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference
Zellers, R., Bisk, Y., Schwartz, R., & Choi, Y. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1009 · Cited by 690
Koehn, P., & Hoang, H. T. (2007) · The Association for Computational Linguistics · Cited by 673
Flint, Environmental Racism, and Racial Capitalism
Pulido, L. (2016) · Capitalism Nature Socialism · doi:10.1080/10455752.2016.1213013 · Cited by 639
Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior
Founta, A., Djouvas, C., Chatzakou, D., Leontiadis, I., Blackburn, J., Stringhini, G., Vakali, A., Sirivianos, M., & Kourtellis, N. (2018) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v12i1.14991 · Cited by 596
mT5: A massively multilingual pre-trained text-to-text transformer
Xue, L., Constant, N., Roberts, A., Kale, M., Al‐Rfou, R., Siddhant, A., Barua, A., & Raffel, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11934 · Cited by 594
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2018) · arXiv · doi:10.48550/arxiv.1804.07461 · Cited by 562
The Woman Worked as a Babysitter: On Biases in Language Generation
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1339 · Cited by 537
context2vec: Learning Generic Context Embedding with Bidirectional LSTM
Melamud, O., Goldberger, J., & Dagan, I. (2016) · Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1006 · Cited by 513
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.00027 · Cited by 492
Understanding Abuse: A Typology of Abusive Language Detection Subtasks
Waseem, Z., Davidson, T., Warmsley, D., & Weber, I. (2017) · Proceedings of the First Workshop on Abusive Language Online · doi:10.18653/v1/w17-3012 · Cited by 474
Common ground at the understanding of demonstrative reference
Clark, H. H., Schreuder, R., & Buttrick, S. (1983) · Journal of Verbal Learning and Verbal Behavior · doi:10.1016/s0022-5371(83)90189-5 · Cited by 445 · cited 1×
“Our human understanding of coherence derives from our ability to recognize interlocutors’ beliefs [30, 31] and intentions [23, 33] within context [32].”
BackgroundZafrir, O., Boudoukh, G., Izsak, P., & Wasserblat, M. (2019) · 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS Edition (EMC2-NIPS) · doi:10.1109/emc2-nips53020.2019.00016 · Cited by 442
Zafrir, O., Boudoukh, G., Izsak, P., & Wasserblat, M. (2019) · arXiv · doi:10.48550/arxiv.1910.06188 · Cited by 441
Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
Shen, S., Dong, Z., Ye, J., Ma, L., Yao, Z., Gholami, A., Mahoney, M. W., & Keutzer, K. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6409 · Cited by 437
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., & Chen, Z. (2020) · arXiv · doi:10.48550/arxiv.2006.16668 · Cited by 412 · cited 2×
“OpenAI’s GPT-3 [25] and Google’s GShard [73] and Switch-C [43] have increased the definition of large LM by orders of magnitude in terms of parameters at 175B, 600B, and 1.6T parameters, respectively.”
MethodProbing Neural Network Comprehension of Natural Language Arguments
Niven, T., & Kao, H.-Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1459 · Cited by 395 · cited 2×
“However, no actual language understanding is taking place in LM-driven approaches to these tasks, as can be shown by careful manipulation of the test data to remove spurious cues the systems are leveraging [21, 93].”
BackgroundTowards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning
Henderson, P., Hu, J., Romoff, J., Brunskill, E., Jurafsky, D., & Pineau, J. (2020) · arXiv · doi:10.48550/arxiv.2002.05651 · Cited by 393
End-to-End Open-Domain Question Answering with
Yang, W., Xie, Y., Lin, A., Li, X., Tan, L., Xiong, K., Li, M., & Lin, J. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-4013 · Cited by 391
Measuring Bias in Contextualized Word Representations
Kurita, K., Vyas, N., Pareek, A., Black, A. W., & Tsvetkov, Y. (2019) · Proceedings of the First Workshop on Gender Bias in Natural Language Processing · doi:10.18653/v1/w19-3823 · Cited by 348 · cited 2×
“However, in both application areas, the training data has been shown to have problematic characteristics [18, 38, 42, 47, 61] resulting in models that encode stereotypical and derogatory associations along gender, race, ethnicity, and disability status [11, 12, 69, 69, 132, 132, 157].”
BackgroundSocial Bias Frames: Reasoning about Social and Power Implications of Language
Sap, M., Gabriel, S., Qin, L., Jurafsky, D., Smith, N. A., & Choi, Y. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.486 · Cited by 332
Twenty Years of Stereotype Threat Research: A Review of Psychological Mediators
Pennington, C. R., Heim, D., Levy, A. R., & Larkin, D. T. (2016) · PLOS ONE · doi:10.1371/journal.pone.0146487 · Cited by 324 · cited 1×
“Either way, harms ensue: readers subject to the stereotypes may experience the psychological harms of microaggressions [88, 141] and stereotype threat [97, 126].”
BackgroundValues as lived experience: evolving value sensitive design in support of value discovery
Le Dantec, C. A., Poole, E. S., & Wyche, S. P. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518875 · Cited by 320
Gender Bias in Contextualized Word Embeddings
Zhao, J., Wang, T., Yatskar, M., Cotterell, R., Ordonez, V., & Chang, K.-W. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-1064 · Cited by 319
The Values Encoded in Machine Learning Research
Birhane, A., Kalluri, P., Card, D., Agnew, W., Dotan, R., & Bao, M. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533083 · Cited by 317
The Values Encoded in Machine Learning Research
Birhane, A., Kalluri, P., Card, D., Agnew, W., Dotan, R., & Bao, M. (2021) · arXiv · doi:10.48550/arxiv.2106.15590 · Cited by 317
Contending Stories: Narrative in Social Movements
Polletta, F. (1998) · Qualitative Sociology · doi:10.1023/a:1023332410633 · Cited by 309 · cited 1×
“Importantly, articles describing past shootings and incidents of police brutality were created and updated as articles for new events were created, reflecting how social movements make connections between events in time to form cohesive narratives [102].”
BackgroundA Simple Method for Commonsense Reasoning
Trinh, T. H., & Le, Q. V. (2018) · arXiv · doi:10.48550/arxiv.1806.02847 · Cited by 293
Release Strategies and the Social Impacts of Language Models
Solaiman, I., Brundage, M., Clark, J., Askell, A., Herbert-Voss, A., Wu, J., Radford, A., Krueger, G., Kim, J. W., Kreps, S., et al. (2019) · arXiv · doi:10.48550/arxiv.1908.09203 · Cited by 292 · cited 1×
“would be watermarked and thus detectable [7, 66, 123]? Are there policy approaches that could effectively regulate their use?”
BackgroundHutchinson, B., Smart, A., Hanna, A., Denton, R., Greer, C., Kjartansson, O., Barnes, P., & Mitchell, M. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445918 · Cited by 279
Social Biases in NLP Models as Barriers for Persons with Disabilities
Hutchinson, B., Prabhakaran, V., Denton, E., Webster, K., Zhong, Y., & Denuyl, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.487 · Cited by 274
Social Biases in NLP Models as Barriers for Persons with Disabilities
Hutchinson, B., Prabhakaran, V., Denton, E., Webster, K., Zhong, Y., & Denuyl, S. (2020) · arXiv · doi:10.48550/arxiv.2005.00813 · Cited by 274 · cited 4×
“higher levels of toxicity with sentences containing identity markers for marginalized groups or even specific names [61, 103].”
BackgroundInfluential(2008) · Choice Reviews Online · doi:10.5860/choice.46-2349 · Cited by 267
What Makes Wrongful Discrimination Wrong? Biases, Preferences, Stereotypes, and Proxies
Alexander, L. (1992) · University of Pennsylvania Law Review · doi:10.2307/3312397 · Cited by 257
Large image datasets: A pyrrhic win for computer vision?
Birhane, A., & Prabhu, V. U. (2021) · 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv48630.2021.00158 · Cited by 257 · cited 5×
“[62], who cite archival history data collection methods as an example of the amount of resources that should be dedicated to this process, and Birhane and Prabhu [18], who call for a more justice-oriented data collection methodology.”
BackgroundInfluentialLessons from archives: strategies for collecting sociocultural data in machine learning
Jo, E. S., & Gebru, T. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372829 · Cited by 240 · cited 1×
“[62], who cite archival history data collection methods as an example of the amount of resources that should be dedicated to this process, and Birhane and Prabhu [18], who call for a more justice-oriented data collection methodology.”
BackgroundSelf-citationThe envisioning cards: a toolkit for catalyzing humanistic and technical imaginations
Friedman, B., & Hendry, D. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208562 · Cited by 234
Are All Languages Created Equal in Multilingual BERT?
Wu, S., & Dredze, M. (2020) · Proceedings of the 5th Workshop on Representation Learning for NLP · doi:10.18653/v1/2020.repl4nlp-1.16 · Cited by 234
Fiske, S. T. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617708204 · Cited by 227
Gender Stereotypes and the Attribution of Leadership Traits: A Cross-Cultural Comparison
Sczesny, S., Bosak, J., Neff, D., & Schyns, B. (2004) · Sex Roles · doi:10.1007/s11199-004-0715-0 · Cited by 219
A Framework for the Computational Linguistic Analysis of Dehumanization
Mendelsohn, J., Tsvetkov, Y., & Jurafsky, D. (2020) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2020.00055 · Cited by 213
Changing Ideas about Reference
Clark, H. H., & Bangerter, A. (2004) · Experimental Pragmatics · doi:10.1057/9780230524125_2 · Cited by 194
Microaggressions and traumatic stress: Theory, research, and clinical treatment.
Nadal, K. L. Y. (2018) · American Psychological Association eBooks · doi:10.1037/0000073-000 · Cited by 190
Evaluating the Underlying Gender Bias in Contextualized Word Embeddings
Basta, C., Costa-jussà, M. R., & Casas, N. (2019) · Proceedings of the First Workshop on Gender Bias in Natural Language Processing · doi:10.18653/v1/w19-3805 · Cited by 189 · cited 2×
“However, in both application areas, the training data has been shown to have problematic characteristics [18, 38, 42, 47, 61] resulting in models that encode stereotypical and derogatory associations along gender, race, ethnicity, and disability status [11, 12, 69, 69, 132, 132, 157].”
BackgroundPerforming a Project Premortem
Klein, G. (2008) · IEEE Engineering Management Review · doi:10.1109/emr.2008.4534313 · Cited by 183
Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages
Nekoto, W., Marivate, V., Matsila, T., Fasubaa, T., Fagbohungbe, T., Akinola, S. O., Muhammad, S., Kabongo Kabenamualu, S., Osei, S., Sackey, F., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.195 · Cited by 182
Natural Language Watermarking and Tamperproofing
Atallah, M. J., Raskin, V., Hempelmann, C. F., Karahan, M., Sion, R., Topkara, U., & Triezenberg, K. E. (2002) · Lecture notes in computer science · doi:10.1007/3-540-36415-3_13 · Cited by 179
A Review of Text Watermarking: Theory, Methods, and Applications
Kamaruddin, N. S., Kamsin, A., Por, L. Y., & Rahman, H. (2018) · IEEE Access · doi:10.1109/access.2018.2796585 · Cited by 174
BERT-of-Theseus: Compressing BERT by Progressive Module Replacing
Xu, C., Zhou, W., Ge, T., Wei, F., & Zhou, M. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.633 · Cited by 168
Brewer, R., & Piper, A. M. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858379 · Cited by 145
Addressing age-related bias in sentiment analysis,
Diaz, M., Johnson, I., Lazar, A., Piper, A. M., & Gergle, D. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173986 · Cited by 138
Media Bias, Perspective, and State Repression: The Black Panther Party
Davenport, C. (2009) · Cambridge University Press eBooks · doi:10.1017/cbo9780511810985 · Cited by 134
A case study on effectively identifying technical debt
Zazworka, N., Spínola, R. O., Vetro', A., Shull, F., & Seaman, C. (2013) · Proceedings of the 17th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2460999.2461005 · Cited by 133
The economic impact of substantial sea-level rise
Anthoff, D., Nicholls, R. J., & Tol, R. S. J. (2010) · Mitigation and Adaptation Strategies for Global Change · doi:10.1007/s11027-010-9220-7 · Cited by 128
Hurtful words: quantifying biases in clinical contextual word embeddings
Zhang, H., Lu, A. X., Abdalla, M., McDermott, M., & Ghassemi, M. (2020) · Proceedings of the ACM Conference on Health, Inference, and Learning · doi:10.1145/3368555.3384448 · Cited by 122
Field, A., Kliger, D., Wintner, S., Pan, J., Jurafsky, D., & Tsvetkov, Y. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1393 · Cited by 119
Perturbation Sensitivity Analysis to Detect Unintended Model Biases
Prabhakaran, V., Hutchinson, B., & Mitchell, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1578 · Cited by 115 · cited 1×
“higher levels of toxicity with sentences containing identity markers for marginalized groups or even specific names [61, 103].”
BackgroundFinding Microaggressions in the Wild: A Case for Locating Elusive Phenomena in Social Media Posts
Breitfeller, L., Ahn, E., Jurgens, D., & Tsvetkov, Y. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1176 · Cited by 112
Value scenarios: a technique for envisioning systemic effects of new technologies
Nathan, L. P., Klasnja, P. V., & Friedman, B. (2007) · CHI '07 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/1240866.1241046 · Cited by 105
Lessons from Archives: Strategies for Collecting Sociocultural Data in Machine Learning
Jo, E. S., & Gebru, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.10389 · Cited by 100
Self-citationEffectiveness of Self-Supervised Pre-Training for ASR
Baevski, A., & Mohamed, A. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9054224 · Cited by 98
Faces of environmental racism: confronting issues of global justice
(1996) · Choice Reviews Online · doi:10.5860/choice.33-5057 · Cited by 96
Pine, A., & Turin, M. (2017) · Oxford University Press eBooks · doi:10.1093/acrefore/9780199384655.013.8 · Cited by 91
Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets
Solaiman, I., & Dennison, C. (2021) · arXiv · doi:10.48550/arxiv.2106.10328 · Cited by 90
A BERT Baseline for the Natural Questions
Alberti, C., Lee, K., & Collins, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.08634 · Cited by 88
Words Matter: Meaning and Power
McConnell‐Ginet, S. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108641302 · Cited by 87
Psychology Cannot Afford to Ignore the Many Harms Caused by Microaggressions
Williams, M. T. (2020) · Perspectives on Psychological Science · doi:10.1177/1745691619893362 · Cited by 84
What the [MASK]? Making Sense of Language-Specific BERT Models
Nozza, D., Bianchi, F., & Hovy, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.02912 · Cited by 84
Young, M., Magassa, L., & Friedman, B. (2019) · Ethics and Information Technology · doi:10.1007/s10676-019-09497-z · Cited by 71
How to see values in social computing: methods for studying values dimensions
Shilton, K., Koepfler, J. A., & Fleischmann, K. R. (2014) · Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing · doi:10.1145/2531602.2531625 · Cited by 68
Computer Power and Human Reason: From Judgment to Calculation.
Rising, G. R., Weizenbaum, J., & Freeman, W. (1978) · American Mathematical Monthly · doi:10.2307/2321366 · Cited by 63
Computer Power and Human Reason: From Judgment to Calculation
Weizenbaum, J., & McCarthy, J. (1977) · Physics Today · doi:10.1063/1.3037375 · Cited by 62
Going Gray, Failure to Hire, and the Ick Factor: Analyzing How Older Bloggers Talk about Ageism
Lazar, A., Diaz, M., Brewer, R., Kim, C., & Piper, A. M. (2017) · Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/2998181.2998275 · Cited by 61 · cited 1×
“outline how they both individually and collectively articulate anti-ageist frames specifically through blogging [71], which some older adults prefer over more popular social media sites for discussing sensitive topics [24].”
BackgroundAlgorithms of Oppression: How Search Engines Reinforce Racism
Hankwitz, M. (2023) · Leonardo · doi:10.1162/leon_r_02331 · Cited by 59
Assessing Social and Intersectional Biases in Contextualized Word Representations
Tan, Y. C., & Celis, L. E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.01485 · Cited by 59
Predicting Overt Display of Power in Written Dialogs
Prabhakaran, V., Rambow, O., & Diab, M. (2012) · North American Chapter of the Association for Computational Linguistics · Cited by 52
Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTPreprint
Shen, S., Dong, Z., Ye, J., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.05840 · Cited by 52
Black Lives Matter in Wikipedia: Collective Memory and Collaboration around Online Social Movements
Twyman, M., Keegan, B. C., & Shaw, A. (2017) · Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/2998181.2998232 · Cited by 51 · cited 2×
“For instance, the Black Lives Matter movement (BLM) influenced Wikipedia article generation and editing such that, as the BLM movement grew, articles covering shootings of Black people increased in coverage and were generated with reduced latency [135].”
BackgroundClickbait? Sensational Headline Generation with Auto-tuned Reinforcement Learning
Xu, P., Wu, C.-S., Madotto, A., & Fung, P. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1303 · Cited by 49
RtGender: A Corpus for Studying Differential Responses to Gender
Voigt, R., Jurgens, D., Prabhakaran, V., Jurafsky, D., & Tsvetkov, Y. (2018) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3di79eiki43x · Cited by 49
Adapting Pretrained Transformer to Lattices for Spoken Language Understanding
Huang, C.-W., & Chen, Y.-N. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9003825 · Cited by 46
Energy Usage Reports: Environmental awareness as part of algorithmic accountability
Lottick, K., Susai, S., Friedler, S. A., & Wilson, J. P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.08354 · Cited by 43
Read, Attend and Comment: A Deep Architecture for Automatic News Comment Generation
Yang, Z., Xu, C., Wu, W., & Li, Z. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1512 · Cited by 39
The Origins of Sexist Language in Discourse
McConnell‐Ginet, S. (1984) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.1984.tb14764.x · Cited by 38
#BlackLivesMatter: An Analysis of the Movement as Social Drama
Jones, L. K. (2019) · Humanity & Society · doi:10.1177/0160597619832049 · Cited by 38
Xiao, D., Zhang, H., Li, Y., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.11314 · Cited by 28
Subjective Decisionmaking and Unconscious Discrimination
Hart, M. (2005) · bepress Legal Repository · Cited by 23
An Empirical Study of Efficient ASR Rescoring with TransformersPreprint
Huang, H., & Peng, F. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.11450 · Cited by 23
Black Lives Matter in Wikipedia: Collaboration and Collective Memory around Online Social MovementsPreprint
Twyman, M. D., Keegan, B., & Shaw, A. (2016) · doi:10.31235/osf.io/ytpu2 · Cited by 23
Media Bias, Perspective, and State Repression: The Black Panther Party
Brooks, D. E. (2011) · Journal of Broadcasting & Electronic Media · doi:10.1080/08838151.2011.572494 · Cited by 17
Dialog Structure Through the Lens of Gender, Gender Environment, and Power
Prabhakaran, V., & Rambow, O. (2017) · Dialogue & Discourse · doi:10.5087/dad.2017.202 · Cited by 12
Indigenous Peoples, Racism and the Environment
Barsh, R. L. (1990) · Meanjin · Cited by 10
Modelling the substitutability of discourse connectives
Hutchinson, B. (2005) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.3115/1219840.1219859 · Cited by 7
Gender bias in natural language processing
Basta, C. (2022) · UPCommons institutional repository (Universitat Politècnica de Catalunya) · doi:10.5821/dissertation-2117-379361 · Cited by 7
Lundsfryd, M. E. (2017) · Middle East Journal of Refugee Studies · doi:10.12738/mejrs.2017.2.1.0110 · Cited by 5
Large Scale Crowdsourcing and Characterization of Twitter Abusive BehaviorPreprint
Founta, A.-M., Djouvas, C., Chatzakou, D., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.00393 · Cited by 5
Interpreting Social Respect: A Normative Lens for ML ModelsPreprint
Hutchinson, B., Pittl, K. J., & Mitchell, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.07336 · Cited by 4
Mind the Gap: Addressing Structural Equity and Inclusion on Wikipedia
Barera, M. (2020) · Cited by 4
The Five Hundred Year Rebellion: Indigenous Movements and the Decolonization of History in Bolivia
Solíz, C. (2021) · Hispanic American Historical Review · doi:10.1215/00182168-9366975 · Cited by 1
Pre-trained Models for Natural Language Processing: A Survey
Qiu, X., Sun, T., Xu, Y., et al. (2020) · arXiv · doi:10.48550/arxiv.2003.08271
Guo, W., & Caliskan, A. (2020) · arXiv · doi:10.48550/arxiv.2006.03955
Research works printed but not linked: 48
- [4] American Psychological Association. 2019. Discrimination: What it is, and how to cope. https://www.apa.org/topics/discrimination (2019). Google Scholar
- 10.18653/v1 · DOI
- 10.18653/v1 · DOI
- [15] Ruha Benjamin. 2019. Race After Technology: Abolitionist Tools for the New Jim Code. Polity Press, Cambridge, UK. Crossref Google Scholar
- [16] Elettra Bietti and Roxana Vatanparast. 2020. Data Waste. Harvard International Law Journal 61 (2020). Google Scholar
- 10.18653/v1 · DOI
- 10.18653/v1 · DOI
- 10.1016/0010-0277(86) · doi:10.1016/0010-0277(86
- [38] Terrance de Vries, Ishan Misra, Changhan Wang, and Laurens van der Maaten. 2019. Does object recognition work for everyone?. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 52--59. Google Scholar
- 10.18653/v1 · DOI
- [42] Ethan Fast, Tina Vachovsky, and Michael Bernstein. 2016. Shirtless and Dangerous: Quantifying Linguistic Signals of Gender Bias in an Online Fiction Writing Community. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 10. Google Scholar
- 10.18653/v1 · DOI
- [45] Darja Fišer, Ruihong Huang, Vinodkumar Prabhakaran, Rob Voigt, Zeerak Waseem, and Jacqueline Wernimont (Eds.). 2018. Proceedings of the 2nd Workshop on Abusive Language Online (ALW2). Association for Computational Linguistics, Brussels, Belgium. https://www.aclweb.org/anthology/W18-5100 Google Scholar
- [50] Batya Friedman, Peter H. Kahn, Jr., and Alan Borning. 2006. Value sensitive design and information systems. In Human-Computer Interaction in Management Information Systems: Foundations, P Zhang and D Galletta (Eds.). M. E. Sharpe, Armonk NY, 348--372. Google Scholar
- 10.18653/v1 · DOI
- 10.18653/v1 · DOI
- 10.18653/v1 · DOI
- [67] Brendan Kennedy, Drew Kogon, Kris Coombs, Joseph Hoover, Christina Park, Gwenyth Portillo-Wightman, Aida Mostafazadeh Davani, Mohammad Atari, and Morteza Dehghani. 2018. A typology and coding manual for the study of hate-based rhetoric. PsyArXiv. July 18 (2018). Google Scholar
- [77] Marianna Martindale and Marine Carpuat. 2018. Fluency Over Adequacy: A Pilot Study in Measuring User Trust in Imperfect MT. In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track). Association for Machine Translation in the Americas, Boston, MA, 13--25. https://www.aclweb.org/anthology/W18-1803 Google Scholar
- [81] Douglas M McLeod. 2007. News coverage and social protest: How the media's protect paradigm exacerbates social conflict. Journal of Dispute Resolution (2007), 185. Google Scholar
- [87] Robert C. Moore and William Lewis. 2010. Intelligent Selection of Language Model Training Data. In Proceedings of the ACL 2010 Conference Short Papers. Association for Computational Linguistics, Uppsala, Sweden, 220--224. https://www.aclweb.org/anthology/P10-2041 Google Scholar
- 10.18653/v1 · DOI
- [92] Maggie Nelson. 2015. The Argonauts. Graywolf Press, Minneapolis. Google Scholar
- 10.18653/v1 · DOI
- [96] David Ortiz, Daniel Myers, Eugene Walls, and Maria-Elena Diaz. 2005. Where do we stand with newspaper data? Mobilization: An International Quarterly 10, 3 (2005), 397--419. Google Scholar
- 10.3115/v1 · doi:10.3115/v1
- 10.18653/v1 · DOI
- 10.18653/v1 · DOI
- [106] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9. Google Scholar
- 10.18653/v1 · DOI
- [109] Sarah T. Roberts, Joel Tetreault, Vinodkumar Prabhakaran, and Zeerak Waseem (Eds.). 2019. Proceedings of the Third Workshop on Abusive Language Online. Association for Computational Linguistics, Florence, Italy. https://www.aclweb.org/anthology/W19-3500 Google Scholar
- [112] Corby Rosset. 2020. Turing-NLG: A 17-billion-parameter language model by Microsoft. Microsoft Blog (2020). Google Scholar
- 10.18653/v1 · DOI
- 10.18653/v1 · DOI
- [121] Joonbo Shin, Yoonhyung Lee, and Kyomin Jung. 2019. Effective Sentence Scoring Method Using BERT for Speech Recognition. In Asian Conference on Machine Learning. 1081--1093. Google Scholar
- [124] Karen Spärck Jones. 2004. Language modelling's generative model: Is it rational? Technical Report. Computer Laboratory, University of Cambridge. Google Scholar
- [127] Katrina Srigley and Lorraine Sutherland. 2019. Decolonizing, Indigenizing, and Learning Biskaaybiiyang in the Field: Our Oral History Journey1. The Oral History Review (2019). Google Scholar
- 10.18653/v1 · DOI
- 10.18653/v1 · DOI
- 10.18653/v1 · DOI
- 10.18653/v1 · DOI
- 10.18653/v1 · DOI
- 10.18653/v1 · DOI
- 10.18653/v1 · DOI
- [150] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems. 5753--5763. Google Scholar
- 10.18653/v1 · DOI
- 10.18653/v1 · DOI
- 10.18653/v1 · DOI
Cited by
Dwivedi, Y. K., Kshetri, N., Hughes, L., Slade, E. L., Jeyaraj, A., Kar, A. K., Baabdullah, A. M., Koohang, A., Raghavan, V., Ahuja, M., et al. (2023) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2023.102642 · Cited by 4,584
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944
“Where LLMs are used as tools, issues of responsibility and credit must be addressed 96,101–103 .”
BackgroundMethodOn the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
“However, recent research has pointed out various potential risks associated with such large-scale generalpurpose language models trained on web text (Bender et al., 2021; Bommasani & et. al., 2021), for instance, perpetuating or exacerbating social stereotypes and disparities that are reflected in training data (Sheng…”
Background
Metadata
- Authors
- Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S.
- Affiliations
- University of Washington (US)
- Black in AI, Palo Alto, CA, USA
- The Aether
- Venue
- Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2021-03
- Pages
- 610-623
- OpenAlex
- W3133702157
- Cited by
- 6,988
- References
- 117
- Citations by year
- 2026: 2484 · 2025: 1887 · 2024: 1153 · 2023: 842 · 2022: 384 · 2021: 229
- FWCI
- 300.72
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by)
- Creighton Digital Repository (CDR) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language models, model size, AI risks, financial costs, data curation, dataset documentation, stakeholder value, BERT, GPT, fine-tuning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.