Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
Open accessAbstract
Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, Matt Gardner. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Large language models inherit stereotypes, misrepresentations, derogatory and exclusionary language, and other denigrating behaviors that disproportionately affect already-vulnerable and marginalized communities.
“Typically trained on an enormous scale of uncurated Internet-based data, LLMs inherit stereotypes, misrepresentations, derogatory and exclusionary language, and other denigrating behaviors that disproportionately affect already-vulnerable and marginalized communities (Bender et al. 2021; Dodge et al. 2021; Sheng et al. 2021b).”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Dodge et al. 2021
Section 4.3, “Demographic Biases in C4.EN” (page not provided), reports ethnicity bias in a model and a corresponding sentiment bias in its pretraining corpus. The “Social biases and representational harms” section explicitly says the paper has not shown a causal link between the corpus statistics and downstream bias. It also reports disproportionate exclusion of minority-related text, but does not establish the broader claim that LLMs inherit derogatory or exclusionary language and other denigrating behaviors. (LLM, conf=0.87)
Experimental — accuracy not yet measuredLarge language models are typically trained on an enormous scale of uncurated Internet-based data.
“Typically trained on an enormous scale of uncurated Internet-based data, LLMs inherit stereotypes, misrepresentations, derogatory and exclusionary language, and other denigrating behaviors that disproportionately affect already-vulnerable and marginalized communities (Bender et al. 2021; Dodge et al. 2021; Sheng et al. 2021b).”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Dodge et al. 2021
The Introduction says the largest models train on substantial portions of the public internet, and §7 Related Work gives GPT-3 as an example trained on large amounts of web text. But the paper describes that text as filtered, including C4’s multiple cleaning steps, so it does not support calling the training data wholly uncurated. (LLM, conf=0.86)
Experimental — accuracy not yet measuredVery recent research has begun to analyze dataset values, question assumptions around dataset use, unpack what is represented in ML datasets, and establish basics of how an organized dataset lifecycle might proceed.
“Very recent research - most published within the last year - has begun to analyze dataset values [18, 40], question assumptions around dataset use [100], unpack what is represented in ML datasets [43, 88, 124] and establish basics of how an organized dataset lifecycle might proceed [74].”
From Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022) · cites this paper as [43]
LLM datasets are poorly documented with little understanding of their ultimate makeup.
“LLM datasets are poorly documented with little understanding of their ultimate makeup62.”
From Medical large language models are vulnerable to data-poisoning attacks (2025) · cites this paper as [62]
The pre-release increase in specific terms could occur naturally and may already be reflected in the pretraining corpora of later LLMs, which were built from time-specific web snapshots (e.g., C4, April 2019).
“The pre-release increase in specific terms could occur naturally and may already be reflected in the pretraining corpora of later LLMs, which were built from time-specific web snapshots (e.g., C4, April 2019 [31]).”
From Delving Into PubMed Records: How AI-Influenced Vocabulary has Transformed Medical Writing since ChatGPT (2025) · cites this paper as [31]
Access to pre-training data enables leakage detection through overlap analysis.
“Access to pre-training data enables leakage detection through overlap analysis, as highlighted in previous research (Dodge et al., 2021;Deng et al., 2023); even without direct access, details like data sources and time frames can reveal leakages (Li, 2023a;b).”
From Benchmarking Benchmark Leakage in Large Language Models (2024) · cites this paper as Dodge et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741 · see citing works
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · see citing works
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · see citing works
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · see citing works
VADER: a parsimonious rule-based model for sentiment analysis of social media text
Hutto, C., & Gilbert, E. (2014) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v8i1.14550 · Cited by 6,087 · see citing works
Learning Word Vectors for Sentiment Analysis
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., & Potts, C. (2011) · Cited by 3,710 · see citing works
Word association norms, mutual information, and lexicography
Church, K., & Hanks, P. (1990) · doi:10.5555/89086.89095 · Cited by 3,644 · see citing works
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342 · see citing works
Teaching Machines to Read and ComprehendPreprint
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,779 · see citing works
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469 · see citing works
Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond
Nallapati, R., Zhou, B., dos Santos, C., Gulcehre, C., & Xiang, B. (2016) · Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1028 · Cited by 2,361 · see citing works
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.11 · Cited by 2,264 · see citing works
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.740 · Cited by 2,103 · see citing works
Self-citationScaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 4× · see citing works
“This paradigm incentivizes the use of ever larger corpora (Kaplan et al., 2020; Henighan et al., 2020), with the biggest models now training on a substantial fraction of the publicly-available internet (Raffel et al.”
MethodInfluentialThe PASCAL Recognising Textual Entailment Challenge
Dagan, I., Glickman, O., & Magnini, B. (2006) · Lecture notes in computer science · doi:10.1007/11736790_9 · Cited by 1,674 · see citing works
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer
Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., & Raffel, C. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.41 · Cited by 1,629 · see citing works
(2018) · International Conference on Learning Representations · Cited by 1,621 · see citing works
SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation
Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., & Specia, L. (2017) · Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) · doi:10.18653/v1/s17-2001 · Cited by 1,562 · see citing works
Making Pre-trained Language Models Better Few-shot Learners
Gao, T., Fisch, A., & Chen, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.295 · Cited by 1,380 · see citing works
Narayan, S., Cohen, S. B., & Lapata, M. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1206 · Cited by 1,207 · see citing works
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., et al. (2020) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.5347031 · Cited by 1,152 · see citing works
Automatically Constructing a Corpus of Sentential Paraphrases.
Dolan, W. B., & Brockett, C. (2005) · Cited by 1,101 · see citing works
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., D., M., Châtel, G., Chaumond, J., Rault, T., Voss, C., Wang, F., et al. (2019) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.3385997 · Cited by 1,054 · see citing works
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051 · see citing works
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Blodgett, S. L., Barocas, S., Daumé III, H., & Wallach, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.485 · Cited by 1,050 · cited 1× · see citing works
“This exclusion is a form of allocational harms (Barocas et al., 2017; Blodgett et al., 2020) and exacerbates existing (language-based) racial inequality (Rosa, 2019) as well as stigmatization of LGBTQ+ identities (Pinsof and Haselton, 2017).”
BackgroundSwitch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Fedus, W., Zoph, B., & Shazeer, N. (2021) · arXiv · doi:10.48550/arxiv.2101.03961 · Cited by 1,005 · see citing works
Neural Network Acceptability Judgments
Warstadt, A., Singh, A., & Bowman, S. R. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00290 · Cited by 984 · see citing works
Levesque, H. J., Davis, E., & Morgenstern, L. (2011) · Cited by 879 · see citing works
Bender, E. M., & Friedman, B. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00041 · Cited by 854 · cited 1× · see citing works
“EN dataset. structured, task-specific NLP datasets, best practices have emerged around documenting the collection process, composition, intended uses, and other characteristics (Bender and Friedman, 2018; Gebru et al., 2018; Hutchinson et al., 2021).”
BackgroundmT5: A massively multilingual pre-trained text-to-text transformerPreprint
Xue, L., Constant, N., Roberts, A., Kale, M., Al‐Rfou, R., Siddhant, A., Barua, A., & Raffel, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11934 · Cited by 852 · see citing works
Extracting Training Data from Large Language Models
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07805 · Cited by 727 · see citing works
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.301 · Cited by 693 · see citing works
Self-citationAdversarial NLI: A New Benchmark for Natural Language Understanding
Nie, Y., Williams, A., Dinan, E., Bansal, M., Weston, J., & Kiela, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.441 · Cited by 656 · see citing works
Neural Text Generation from Structured Data with Application to the Biography Domain
Lebret, R., Grangier, D., & Auli, M. (2016) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1128 · Cited by 595 · see citing works
The pascal recognising textual entailment challenge
Dagan, I., & Glickman, O. (2005) · Cited by 579 · see citing works
Data and its (dis)contents: A survey of dataset development and use in machine learning research
Paullada, A., Raji, I. D., Bender, E. M., Denton, E., & Hanna, A. (2021) · Patterns · doi:10.1016/j.patter.2021.100336 · Cited by 548 · cited 1× · see citing works
“Since analyzing pretraining corpora is challenging due to their size, their documentation is often missing (Bender et al., 2021; Paullada et al., 2020).”
BackgroundThe Seventh PASCAL Recognizing Textual Entailment Challenge.
Bentivogli, L., Clark, P. E., Dagan, I., & Giampiccolo, D. (2008) · Cited by 533 · see citing works
UNIFIEDQA: Crossing Format Boundaries with a Single QA System
Khashabi, D., Min, S., Khot, T., Sabharwal, A., Tafjord, O., Clark, P., & Hajishirzi, H. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.171 · Cited by 514 · see citing works
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.00027 · Cited by 492 · see citing works
The Woman Worked as a Babysitter: On Biases in Language Generation
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1339 · Cited by 431 · cited 1× · see citing works
“Much recent work has found various kinds of bias in fine-tuned models (e.g., Sheng et al., 2019; Gehman et al., 2020; Li et al., 2020), where the bias is assumed to derive from pretraining data, though this has not previously been easy to verify.”
BackgroundBoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., & Toutanova, K. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1300 · Cited by 391 · see citing works
CCNet: Extracting High Quality Monolingual Datasets from Web Crawl DataPreprint
Wenzek, G., Lachaux, M.-A., Conneau, A., Chaudhary, V., Guzmán, F., Joulin, A., & Grave, É. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.00359 · Cited by 362 · see citing works
Inducing Domain-Specific Sentiment Lexicons from Unlabeled Corpora
Hamilton, W. L., Clark, K., Leskovec, J., & Jurafsky, D. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1057 · Cited by 361 · see citing works
WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia
Schwenk, H., Chaudhary, V., Sun, S., Gong, H., & Guzmán, F. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.115 · Cited by 317 · see citing works
A Simple Method for Commonsense Reasoning
Trinh, T. H., & Le, Q. V. (2018) · arXiv · doi:10.48550/arxiv.1806.02847 · Cited by 293 · see citing works
Demographic dialectal variation in social media: a case study of African-American English
Blodgett, S. L., Green, L., & O'Connor, B. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1120 · Cited by 291 · cited 1× · see citing works
“We use a dialect-aware topic model from Blodgett et al. (2016), which was trained on 60M geolocated tweets and relies on US census race/ethnicity data as topics.”
MethodHutchinson, B., Smart, A., Hanna, A., Denton, R., Greer, C., Kjartansson, O., Barnes, P., & Mitchell, M. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445918 · Cited by 279 · see citing works
Self-citationDynabench: Rethinking Benchmarking in NLP
Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., et al. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.324 · Cited by 254 · see citing works
How to do things with slurs: Studies in the way of derogatory words
Croom, A. M. (2013) · Language & Communication · doi:10.1016/j.langcom.2013.03.008 · Cited by 240 · see citing works
The Reappropriation of Stigmatizing Labels
Galinsky, A. D., Wang, C. S., Whitson, J., Anicich, E. M., Hugenberg, K., & Bodenhausen, G. V. (2013) · Psychological Science · doi:10.1177/0956797613482943 · Cited by 170 · see citing works
A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages
Ortiz Suárez, P. J., Romary, L., & Sagot, B. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.156 · Cited by 166 · see citing works
Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack
Dinan, E., Humeau, S., Chintagunta, B., & Weston, J. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1461 · Cited by 161 · see citing works
Scaling Laws for Autoregressive Generative Modeling
Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.14701 · Cited by 150 · see citing works
ParaCrawl: Web-Scale Acquisition of Parallel Corpora
Bañón, M., Chen, P., Haddow, B., Heafield, K., Hoang, H. T., Esplà-Gomis, M., Forcada, M. L., Kamran, A., Kirefu, F., Koehn, P., et al. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.417 · Cited by 139 · see citing works
CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs
El-Kishky, A., Chaudhary, V., Guzmán, F., & Koehn, P. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.480 · Cited by 119 · see citing works
Finding Microaggressions in the Wild: A Case for Locating Elusive Phenomena in Social Media Posts
Breitfeller, L., Ahn, E. P., Jurgens, D., & Tsvetkov, Y. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1176 · Cited by 115 · see citing works
Lessons from Archives: Strategies for Collecting Sociocultural Data in Machine LearningPreprint
Jo, E. S., & Gebru, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.10389 · Cited by 102 · see citing works
Groenwold, S., Ou, L.-H., Parekh, A., Honnavalli, S., Levy, S., Mirza, D., & Wang, W. Y. (2020) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2020.emnlp-main.473 · Cited by 100 · see citing works
Abstractive Summarization of Reddit Posts with Multi-level Memory Networks
Kim, B., Kim, H., & Kim, G. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1260 · Cited by 96 · see citing works
UNQOVERing Stereotyping Biases via Underspecified Questions
Li, T., Khashabi, D., Khot, T., Sabharwal, A., & Srikumar, V. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.311 · Cited by 72 · see citing works
What’s in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus
Luccioni, A., & Viviano, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) · doi:10.18653/v1/2021.acl-short.24 · Cited by 69 · see citing works
The effect of the promiscuity stereotype on opposition to gay rights
Pinsof, D., & Haselton, M. G. (2017) · PLoS ONE · doi:10.1371/journal.pone.0178534 · Cited by 58 · see citing works
C4Corpus: Multilingual Web-size Corpus with Free License
Habernal, I., Zayed, O., & Gurevych, I. (2016) · Proceedings of the Language Resources and Evaluation Conference · doi:10.63317/3iti4i8msr33 · Cited by 30 · see citing works
A Non-Parametric Test to Detect Data-Copying in Generative ModelsPreprint
Meehan, C., Chaudhuri, K., & Dasgupta, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.05675 · Cited by 27 · see citing works
Schäfer, R. (2016) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/39e9qei8rcoe · Cited by 21 · see citing works
Wang, W. Y., Finkelstein, S. L., Ogan, A., Black, A. W., & Cassell, J. (2018) · Figshare · doi:10.1184/r1/6469757 · Cited by 19 · see citing works
Flagged first, then by citations
Research works printed but not linked: 8
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901.
- Isaac Caswell, Julia Kreutzer, Lisa Wang, Ahsan Wa- hab, D. V. Esch, Nasanbayar Ulzii-Orshikh, Al- lahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, S. Sarin, Sokhar Samb, B. Sagot, C. Rivera, Annette Rios Gonzales, Isabel Papadimitriou, S. Osei, Pedro Javier Ortiz Suárez, Iroro Orife, Kelechi Ogueji, Rubungo Andre Niyongabo, Toan Q. Nguyen, Math- ias Muller, A. Muller, S. Muhammad, N. Muham- mad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, M. Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, N. D. Silva, Sakine cCabuk Balli, Stella Rose Biderman, Alessia Battisti, A. Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayo- dele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi. 2021. Quality at a glance: An audit of web-crawled multilingual datasets. In Proceedings of the AfricanNLP Workshop.
- Kenneth Church and Patrick Hanks. 1990. Word as- sociation norms, mutual information, and lexicogra- phy. Computational linguistics, 16(1):22-29.
- Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third PASCAL recogniz- ing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 1-9, Prague. Association for Computational Linguistics.
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. Journal of Machine Learning Research.
- Jonathan Rosa. 2019. Looking like a language, sound- ing like a race. Oxford University Press. · DOI
- Rachael Tatman. 2020. What i won't build. WiNLP Workshop at ACL.
- Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In NeurIPS.
Cited by
Sorted by citations
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · see citing works
“Removing these associations, however, is non-trivial; for instance, filtering off content that is deemed toxic by an automated tool may disproportionately exclude content about or authored by marginalized subgroups in the training data (Dodge et al., 2021).”
BackgroundTowards a general-purpose foundation model for computational pathology
Chen, R. J., Ding, T., Lu, M. Y., Williamson, D. F. K., Jaume, G., Song, A. H., Chen, B., Zhang, A., Shao, D., Shaban, M., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02857-3 · Cited by 1,560 · see citing works
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey
Min, B., Ross, H., Sulem, E., Veyseh, A. P. B., Nguyen, T. H., Sainz, O., Agirre, E., Heintz, I., & Roth, D. (2024) · ACM Computing Surveys · doi:10.1145/3605943 · Cited by 1,387 · see citing works
Bias and Fairness in Large Language Models: A Survey
Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., & Ahmed, N. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00524 · Cited by 665 · see citing works
Advances, challenges and opportunities in creating data for trustworthy AI
Liang, W., Tadesse, G. A., Ho, D. E., Fei-Fei, L., Zaharia, M., Zhang, C., & Zou, J. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00516-1 · Cited by 582 · see citing works
Hirosawa, T., Harada, Y., Yokose, M., Sakamoto, T., Kawamura, R., & Shimizu, T. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20043378 · Cited by 345 · see citing works
Auditing large language models: a three-layered approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2024) · AI and Ethics · doi:10.1007/s43681-023-00289-2 · Cited by 289 · see citing works
AI generates covertly racist decisions about people based on their dialect
Hofmann, V., Kalluri, P. R., Jurafsky, D., & King, S. (2024) · Nature · doi:10.1038/s41586-024-07856-5 · Cited by 233 · see citing works
Alawida, M., Mejri, S., Mehmood, A., Chikhaoui, B., & Abiodun, O. I. (2023) · Information · doi:10.3390/info14080462 · Cited by 211 · see citing works
Medical large language models are vulnerable to data-poisoning attacks
Alber, D. A., Yang, Z., Alyakin, A., Yang, E., Rai, S., Valliani, A. A., Zhang, J., Rosenbaum, G. R., Amend-Thomas, A. K., Kurland, D. B., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03445-1 · Cited by 195 · see citing works
OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantization
Guo, C., Tang, J., Hu, W., Leng, J., Zhang, C., Yang, F., Liu, Y., Guo, M., & Zhu, Y. (2023) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3579371.3589038 · Cited by 162 · see citing works
Fields, J., Chovanec, K., & Madiraju, P. (2024) · IEEE Access · doi:10.1109/access.2024.3349952 · Cited by 165 · see citing works
Feng, S., Park, C. Y., Liu, Y., & Tsvetkov, Y. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.656 · Cited by 149 · see citing works
Hirosawa, T., Kawamura, R., Harada, Y., Mizuta, K., Tokumasu, K., Kaji, Y., Suzuki, T., & Shimizu, T. (2023) · JMIR Medical Informatics · doi:10.2196/48808 · Cited by 130 · see citing works
Gibson, D., Jackson, S., Shanmugasundaram, R., Seth, I., Siu, A., Ahmadi, N., Kam, J., Mehan, N., Thanigasalam, R., Jeffery, N., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/55939 · Cited by 122 · see citing works
Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs
Sap, M., Le Bras, R., Fried, D., & Choi, Y. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.248 · Cited by 118 · see citing works
Self-citationHistory, development, and principles of large language models: an introductory survey
Wang, Z., Chu, Z., Doan, T. V., Ni, S., Yang, M., & Zhang, W. (2025) · AI and Ethics · doi:10.1007/s43681-024-00583-7 · Cited by 109 · see citing works
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text
Gehrmann, S., Clark, E., & Sellam, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13715 · Cited by 123 · see citing works
Evolution and Prospects of Foundation Models: From Large Language Models to Large Multimodal Models
Chen, Z., Xu, L., Zheng, H., Chen, L., Tolba, A. K. R., Zhao, L., Yu, K., & Feng, H. (2024) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2024.052618 · Cited by 112 · see citing works
Pellert, M., Lechner, C. M., Wagner, C., Rammstedt, B., & Strohmaier, M. (2024) · Perspectives on Psychological Science · doi:10.1177/17456916231214460 · Cited by 97 · see citing works
Designing Responsible AI: Adaptations of UX Practice to Meet Responsible AI Challenges
Wang, Q., Madaio, M., Kane, S. K., Kapania, S., Terry, M. A., & Wilcox, L. G. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581278 · Cited by 94 · see citing works
A large-scale audit of dataset licensing and attribution in AI
Longpre, S., Mahari, R., Chen, A., Obeng-Marnu, N., Sileo, D., Brannon, W., Muennighoff, N., Khazam, N., Kabbara, J., Perisetla, K., et al. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00878-8 · Cited by 150 · see citing works
Impact of Pretraining Term Frequencies on Few-Shot Numerical Reasoning
Razeghi, Y., Logan IV, R. L., Gardner, M., & Singh, S. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.59 · Cited by 89 · see citing works
Self-citationData Contamination: From Memorization to Exploitation
Magar, I., & Schwartz, R. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/2022.acl-short.18 · Cited by 88 · see citing works
NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark
Sainz, O., Campos, J., García-Ferrero, I., Etxaniz, J., de Lacalle, O. L., & Agirre, E. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.722 · Cited by 129 · see citing works
Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4
Chang, K. K., Cramer, M. H., Soni, S., & Bamman, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.453 · Cited by 110 · see citing works
Markl, N. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533117 · Cited by 86 · see citing works
Contrasting Linguistic Patterns in Human and LLM-Generated News Text
Muñoz-Ortiz, A., Gómez-Rodríguez, C., & Vilares, D. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10903-2 · Cited by 94 · see citing works
Cultural Bias in Large Language Models: A Comprehensive Analysis and Mitigation Strategies
Liu, Z. (2025) · Journal of Transcultural Communication · doi:10.1515/jtc-2023-0019 · Cited by 77 · see citing works
Can we trust the evaluation on ChatGPT?
Aiyappa, R., An, J., Kwak, H., & Ahn, Y.-Y. (2023) · Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023) · doi:10.18653/v1/2023.trustnlp-1.5 · Cited by 72 · see citing works
Designing Word Filter Tools for Creator-led Comment Moderation
Jhaver, S., Chen, Q. Z., Knauss, D., & Zhang, A. X. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517505 · Cited by 72 · see citing works
Queer In AI: A Case Study in Community-Led Participatory AI
QueerInAI, O. O., Ovalle, A., Subramonian, A., Singh, A., Voelcker, C., Sutherland, D. J., Locatelli, D., Breznik, E., Klubička, F., Yuan, H., et al. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594134 · Cited by 84 · see citing works
Self-citationInitial policy considerations for generative artificial intelligence
Lorenz, P., Perset, K., & Berryhill, J. (2023) · OECD artificial intelligence papers · doi:10.1787/fae2d1e6-en · Cited by 105 · see citing works
Multilingualism and AI: The Regimentation of Language in the Age of Digital Capitalism
Schneider, B. (2022) · Signs and Society · doi:10.1086/721757 · Cited by 81 · see citing works
Schramowski, P., Tauchmann, C., & Kersting, K. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533192 · Cited by 63 · see citing works
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
Laurençon, H., Saulnier, L., Wang, T., Akiki, C., del Moral, A. V., Scao, T. L., Von Werra, L., Mou, C., Ponferrada, E. G., Nguyen, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.03915 · Cited by 61 · see citing works
“Other LM pre-training datasets that have been documented and analyzed include C4 (Dodge et al., 2021; Luccioni and Viviano, 2021; Kreutzer et al., 2022), OSCAR (Kreutzer et al.”
BackgroundMethodSelf-citationYou reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings
Talat, Z., Névéol, A., Biderman, S., Clinciu, M., Dey, M., Longpre, S., Luccioni, A. S., Masoud, M., Mitchell, M., Radev, D., et al. (2022) · doi:10.18653/v1/2022.bigscience-1.3 · Cited by 61 · see citing works
Self-citationRevisiting Transformer-based Models for Long Document Classification
Dai, X., Chalkidis, I., Darkner, S., & Elliott, D. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.534 · Cited by 76 · see citing works
Based on billions of words on the internet, people = men
Bailey, A. H., Williams, A., & Cimpian, A. (2022) · Science Advances · doi:10.1126/sciadv.abm2463 · Cited by 55 · see citing works
gSASRec: Reducing Overconfidence in Sequential Recommendation Trained with Negative Sampling
Petrov, A. V., & Macdonald, C. (2023) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3604915.3608783 · Cited by 53 · see citing works
Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debate
Wang, B., Yue, X., & Sun, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.795 · Cited by 71 · see citing works
Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models
Arora, D., Singh, H., & Mausam, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.468 · Cited by 63 · see citing works
A Critical Analysis of the Largest Source for Generative AI Training Data: Common Crawl
Baack, S. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3659033 · Cited by 52 · see citing works
Large language models: an overview of foundational architectures, recent trends, and a new taxonomy
Mienye, I. D., Jere, N., Obaido, G., Ogunruku, O. O., Esenogho, E., & Modisane, C. (2025) · Discover Applied Sciences · doi:10.1007/s42452-025-07668-w · Cited by 46 · see citing works
Dong, Y., Jiang, X., Liu, H., Jin, Z., Gu, B., Yang, M., & Li, G. (2024) · arXiv · doi:10.48550/arxiv.2402.15938 · Cited by 52
“Some work (Pan et al., 2020; Zhou et al., 2023; Jacovi et al., 2023; Dodge et al., 2021) exposed the serious consequences of data contamination and urged attention to this problem.”
The Foundation Model Transparency Index
Bommasani, R., Klyman, K., Longpre, S., Kapoor, S., Maslej, N., Xiong, B., Zhang, D., & Liang, P. (2023) · arXiv · doi:10.48550/arxiv.2310.12941 · Cited by 46 · see citing works
Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation
Fernandes, P., Madaan, A., Liu, E., Farinhas, A., Martins, P. H., Bertsch, A., de Souza, J. G. C., Zhou, S., Wu, T., Neubig, G., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00626 · Cited by 57 · see citing works
The OMG dataset: An Open MetaGenomic corpus for mixed-modality genomic language modelingPreprint
Cornman, A., West-Roberts, J. A., Camargo, A. P., Roux, S., Beracochea, M., Mirdita, M., Ovchinnikov, S., & Hwang, Y. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.08.14.607850 · Cited by 45 · see citing works
Open Science at the generative AI turn: An exploratory analysis of challenges and opportunities
Hosseini, M., Horbach, S. P. J. M., Holmes, K., & Ross-Hellauer, T. (2025) · Quantitative Science Studies · doi:10.1162/qss_a_00337 · Cited by 40 · see citing works
Time Waits for No One! Analysis and Challenges of Temporal Misalignment
Luu, K., Khashabi, D., Gururangan, S., Mandyam, K., & Smith, N. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.435 · Cited by 41 · see citing works
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey
Kumar, S., Balachandran, V., Njoo, L., Anastasopoulos, A., & Tsvetkov, Y. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.241 · Cited by 38 · see citing works
Data Representativeness in Accessibility Datasets: A Meta-Analysis
Kamikubo, R., Wang, L., Marte, C., Mahmood, A., & Kacorri, H. (2022) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3517428.3544826 · Cited by 41 · see citing works
Chorus: Foundation Models for Unified Data Discovery and Exploration
Kayali, M., Lykov, A., Fountalis, I., Vasiloglou, N., Olteanu, D., & Suciu, D. M. (2024) · Proceedings of the VLDB Endowment · doi:10.14778/3659437.3659461 · Cited by 52 · see citing works
Investigating machine moral judgement through the Delphi experiment
Jiang, L., Hwang, J. D., Bhagavatula, C., Bras, R. J. L., Liang, J. T., Levine, S., Dodge, J., Sakaguchi, K., Forbes, M., Hessel, J., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-024-00969-6 · Cited by 40 · see citing works
Self-citationFairness in Language Models Beyond English: Gaps and Challenges
Ramesh, K., Sitaram, S., & Choudhury, M. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.157 · Cited by 46 · see citing works
Jacovi, A., Caciularu, A., Goldman, O., & Goldberg, Y. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.308 · Cited by 36 · see citing works
Hype, Sustainability, and the Price of the Bigger-is-Better Paradigm in AI
Varoquaux, G. P., Luccioni, A. S., & Whittaker, M. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732006 · Cited by 49 · see citing works
Jacovi, A., Caciularu, A., Goldman, O., & Goldberg, Y. (2023) · arXiv · doi:10.48550/arxiv.2305.10160 · Cited by 36 · see citing works
Potential Societal Biases of ChatGPT in Higher Education: A Scoping Review
Li, M., Enkhtur, A., Yamamoto, B. A., Cheng, F., & Chen, L. (2025) · Open Praxis · doi:10.55982/openpraxis.17.1.750 · Cited by 47 · see citing works
Future Shock: Generative AI and the International AI Policy and Governance Crisis
Leslie, D., & Perini, A. M. (2024) · Harvard Data Science Review · doi:10.1162/99608f92.88b4cc98 · Cited by 33 · see citing works
NLPositionality: Characterizing Design Biases of Datasets and Models
Santy, S., Liang, J. T., Bras, R. J. L., Reinecke, K., & Sap, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.505 · Cited by 45 · see citing works
Self-citationWill Large-scale Generative Models Corrupt Future Datasets?
Hataya, R., Bao, H., & Arai, H. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01879 · Cited by 32 · see citing works
AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Models
Chronopoulou, A., Peters, M., Fraser, A., & Dodge, J. (2023) · Findings of the Association for Computational Linguistics: EACL 2023 · doi:10.18653/v1/2023.findings-eacl.153 · Cited by 30 · see citing works
Self-citationOrr, W., & Crawford, K. (2024) · New Media & Society · doi:10.1177/14614448241251797 · Cited by 35 · see citing works
Efficient Hierarchical Domain Adaptation for Pretrained Language Models
Chronopoulou, A., Peters, M. E., & Dodge, J. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.96 · Cited by 34 · see citing works
Self-citationBitMoD: Bit-serial Mixture-of-Datatype LLM Acceleration
Chen, Y., AbouElhamayed, A. F., Dai, X., Wang, Y., Andronic, M., Constantinides, G. A., & Abdelfattah, M. S. (2025) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00084 · Cited by 26 · see citing works
Bridging large language model disparities: skill tagging of multilingual educational content
Kwak, Y., & Pardos, Z. A. (2024) · British Journal of Educational Technology · doi:10.1111/bjet.13465 · Cited by 25 · see citing works
BBS: Bi-Directional Bit-Level Sparsity for Deep Learning Acceleration
Chen, Y., Meng, J., Seo, J.-S., & Abdelfattah, M. S. (2024) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro61859.2024.00048 · Cited by 25 · see citing works
SwahBERT: Language Model of Swahili
Martin, G. L., Mswahili, M. E., Jeong, Y.-S., & Young-Seob, J. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.23 · Cited by 25 · see citing works
Language Contamination Helps Explains the Cross-lingual Capabilities of English Pretrained Models
Blevins, T., & Zettlemoyer, L. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.233 · Cited by 27 · see citing works
LLM-Driven Robots Risk Enacting Discrimination, Violence, and Unlawful Actions
Hundt, A., Azeem, R., Mansouri, M. I., & Brandão, M. (2025) · International Journal of Social Robotics · doi:10.1007/s12369-025-01301-x · Cited by 24 · see citing works
The chat-chamber effect: Trusting the AI hallucination
Jacob, C., Kerrigan, P., & Bastos, M. (2025) · Big Data & Society · doi:10.1177/20539517241306345 · Cited by 22 · see citing works
Extensive evaluation of transformer-based architectures for adverse drug events extraction
Scaboro, S., Portelli, B., Chersoni, E., Santus, E., & Serra, G. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110675 · Cited by 25 · see citing works
Evaluation of African American Language Bias in Natural Language Generation
Deas, N., Grieser, J. A., Kleiner, S., Patton, D. U., Turcan, E., & McKeown, K. R. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.421 · Cited by 29 · see citing works
Lookahead Bias in Pretrained Language Models
Sarkar, S., & Vafa, K. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4754678 · Cited by 23 · see citing works
A Framework for Deprecating Datasets: Standardizing Documentation, Identification, and Communication
Luccioni, A. S., Corry, F., Sridharan, H., Ananny, M., Schultz, J., & Crawford, K. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533086 · Cited by 22 · see citing works
Data Authenticity, Consent, and Provenance for AI Are All Broken: What Will It Take to Fix Them?
Longpre, S., Mahari, R., Obeng-Marnu, N., Brannon, W., South, T., Kabbara, J., & Pentland, A. (2024) · doi:10.21428/e4baedd9.a650f77d · Cited by 23 · see citing works
Benchmark Data Contamination of Large Language Models: A Survey
Xu, C., Guan, S., Greene, D., & Kechadi, M.-T. (2024) · arXiv · doi:10.48550/arxiv.2406.04244 · Cited by 22 · see citing works
FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference
de Jong, M., Zemlyanskiy, Y., Ainslie, J., FitzGerald, N., Sanghai, S. K., Sha, F., & Cohen, W. W. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.732 · Cited by 26 · see citing works
An archival perspective on pretraining data
Desai, M. A., Pasquetto, I. V., Jacobs, A. Z., & Card, D. (2024) · Patterns · doi:10.1016/j.patter.2024.100966 · Cited by 21 · see citing works
A Primer on Seq2Seq Models for Generative Chatbots
Scotti, V., Sbattella, L., & Tedesco, R. (2023) · ACM Computing Surveys · doi:10.1145/3604281 · Cited by 21 · see citing works
Prompting Language Models for Linguistic Structure
Blevins, T., Gonen, H., & Zettlemoyer, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.367 · Cited by 25 · see citing works
Aligning generalization between humans and machines
Ilievski, F., Hammer, B., van Harmelen, F., Paaßen, B., Saralajew, S., Schmid, U., Biehl, M. L., Bolognesi, M., Dong, X., Gashteovski, K., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01109-4 · Cited by 23 · see citing works
Caution Ahead: Numerical Reasoning and Look‐Ahead Bias in AI Models
LEVY, B. (2026) · Journal of Accounting Research · doi:10.1111/1475-679x.70058 · Cited by 20 · see citing works
Farrell, H. J. (2025) · Annual Review of Political Science · doi:10.1146/annurev-polisci-040723-013245 · Cited by 21 · see citing works
A survey on large language models with multilingualism: recent advances and new frontiers
Huang, K., Mo, F., Zhang, X., Li, H., Li, Y., Zhang, Y., yi, W., Mao, Y., Liu, J., Xu, Y., et al. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11534-5 · Cited by 73 · see citing works
On the Limitations of Dataset Balancing: The Lost Battle Against Spurious Correlations
Schwartz, R., & Stanovsky, G. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.168 · Cited by 19 · see citing works
Hate Speech Classifiers are Culturally Insensitive
Lee, N., Jung, C., & Oh, A. H. (2023) · Workshop on Cross-Cultural Considerations in NLP (C3NLP) · doi:10.18653/v1/2023.c3nlp-1.5 · Cited by 18 · see citing works
Demystifying Data Management for Large Language Models
Miao, X., Jia, Z., & Cui, B. (2024) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3626246.3654683 · Cited by 18 · see citing works
Fan, X., Xiao, Q., Zhou, X., Pei, J., Sap, M., Lu, Z., & Shen, H. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713477 · Cited by 23 · see citing works
Self-citationCreative Writers' Attitudes on Writing as Training Data for Large Language Models
Gero, K. I., Desai, M., Schnitzler, C., Eom, N., Cushman, J., & Glassman, E. L. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713287 · Cited by 16 · see citing works
Un-Straightening Generative AI: How Queer Artists Surface and Challenge Model Normativity
Taylor, J., Mire, J., Spektor, F., DeVrio, A., Sap, M., Zhu, H., & Fox, S. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732061 · Cited by 16 · see citing works
Self-citationRettberg, J. W., & Wigers, H. (2025) · Open Research Europe · doi:10.12688/openreseurope.20576.1 · Cited by 16 · see citing works
Jain, C., Anish, P. R., Singh, A. D., & Ghaisas, S. S. (2023) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re57278.2023.00025 · Cited by 17 · see citing works
Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications
Zhou, K., Blodgett, S. L., Trischler, A., Daumé, H., Suleman, K., & Olteanu, A. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.24 · Cited by 16 · see citing works
Pellert, M., Lechner, C. M., Wagner, C., Rammstedt, B., & Strohmaier, M. (2022) · doi:10.31234/osf.io/jv5dt · Cited by 16 · see citing works
Fewer Errors, but More Stereotypes? The Effect of Model Size on Gender Bias
Tal, Y., Magar, I., & Schwartz, R. (2022) · Workshop on Gender Bias in Natural Language Processing (GeBNLP) · doi:10.18653/v1/2022.gebnlp-1.13 · Cited by 17 · see citing works
Bound by the Bounty: Collaboratively Shaping Evaluation Processes for Queer AI Harms
Dennler, N. S., Ovalle, A., Singh, A., Soldaini, L., Subramonian, A., Tu, H., Agnew, W. S., Ghosh, A., Yee, K., Peradejordi, I. F., et al. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604682 · Cited by 20 · see citing works
Self-citationFairPrism: Evaluating Fairness-Related Harms in Text Generation
Fleisig, E., Amstutz, A., Atalla, C., Blodgett, S. L., Daumé III, H., Olteanu, A., Sheng, E., Vann, D., & Wallach, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.343 · Cited by 14 · see citing works
“This allows us to prevent simple discussion of sensitive topics from being conflated with the perpetuation of stereotyping or demeaning harms, an issue that erases marginalized voices and necessary discussion of real-world harms (Dodge et al., 2021; Schlesinger et al., 2018).”
BackgroundReproducibility in NLP: What Have We Learned from the Checklist?
Magnusson, I., Smith, N. A., & Dodge, J. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.809 · Cited by 14 · see citing works
Self-citation
- Authors
- Dodge, J., Sap, M., Marasović, A., Agnew, W., Ilharco, G., Groeneveld, D., Mitchell, M., & Gardner, M.
- Affiliations
- University of Washington (US)
- Venue
- Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
- Published
- 2021
- Pages
- 1286-1305
- OpenAlex
- W3204096139
- Cited by
- 306
- References
- 65
- Citations by year
- 2026: 76 · 2025: 77 · 2024: 54 · 2023: 65 · 2022: 33 · 2021: 1
- FWCI
- 17.67
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Allen Institute for Artificial Intelligence
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- web crawling, corpus annotation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
