The DOI matches its Crossref record; 67 of the 70 references it cites were found in a registry or an index.
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
Open accessAbstract
Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, Matt Gardner. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Typically trained on an enormous scale of uncurated Internet-based data, LLMs inherit stereotypes, misrepresentations, derogatory and exclusionary language, and other denigrating behaviors that disproportionately affect already-vulnerable and marginalized communities (Bender et al. 2021; Dodge et al. 2021; Sheng et al. 2021b).”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Dodge et al. 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Typically trained on an enormous scale of uncurated Internet-based data, LLMs inherit stereotypes, misrepresentations, derogatory and exclusionary language, and other denigrating behaviors that disproportionately affect already-vulnerable and marginalized communities (Bender et al. 2021; Dodge et al. 2021; Sheng et al. 2021b).”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Dodge et al. 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Very recent research - most published within the last year - has begun to analyze dataset values [18, 40], question assumptions around dataset use [100], unpack what is represented in ML datasets [43, 88, 124] and establish basics of how an organized dataset lifecycle might proceed [74].”
From Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022) · cites this paper as [43]
The cited source says what the claim attributes to it.
In the citing paper
“LLM datasets are poorly documented with little understanding of their ultimate makeup62.”
From Medical large language models are vulnerable to data-poisoning attacks (2025) · cites this paper as [62]
The cited source says what the claim attributes to it.
In the citing paper
“The pre-release increase in specific terms could occur naturally and may already be reflected in the pretraining corpora of later LLMs, which were built from time-specific web snapshots (e.g., C4, April 2019 [31]).”
From Delving Into PubMed Records: How AI-Influenced Vocabulary has Transformed Medical Writing since ChatGPT (2025) · cites this paper as [31]
The cited source says what the claim attributes to it.
In the citing paper
“Access to pre-training data enables leakage detection through overlap analysis, as highlighted in previous research (Dodge et al., 2021;Deng et al., 2023); even without direct access, details like data sources and time frames can reveal leakages (Li, 2023a;b).”
From Benchmarking Benchmark Leakage in Large Language Models (2024) · cites this paper as Dodge et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 4×
“This paradigm incentivizes the use of ever larger corpora (Kaplan et al., 2020; Henighan et al., 2020), with the biggest models now training on a substantial fraction of the publicly-available internet (Raffel et al.”
MethodInfluentialTransformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
VADER: a parsimonious rule-based model for sentiment analysis of social media text
Hutto, C., & Gilbert, E. (2014) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v8i1.14550 · Cited by 6,087
Word association norms, mutual information, and lexicography
Church, K., & Hanks, P. (1990) · doi:10.5555/89086.89095 · Cited by 3,737
Research works printed but not linked: 7
- Isaac Caswell, Julia Kreutzer, Lisa Wang, Ahsan Wa- hab, D. V. Esch, Nasanbayar Ulzii-Orshikh, Al- lahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, S. Sarin, Sokhar Samb, B. Sagot, C. Rivera, Annette Rios Gonzales, Isabel Papadimitriou, S. Osei, Pedro Javier Ortiz Suárez, Iroro Orife, Kelechi Ogueji, Rubungo Andre Niyongabo, Toan Q. Nguyen, Math- ias Muller, A. Muller, S. Muhammad, N. Muham- mad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, M. Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, N. D. Silva, Sakine cCabuk Balli, Stella Rose Biderman, Alessia Battisti, A. Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayo- dele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi. 2021. Quality at a glance: An audit of web-crawled multilingual datasets. In Proceedings of the AfricanNLP Workshop.
- Kenneth Church and Patrick Hanks. 1990. Word as- sociation norms, mutual information, and lexicogra- phy. Computational linguistics, 16(1):22-29.
- Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third PASCAL recogniz- ing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 1-9, Prague. Association for Computational Linguistics.
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. Journal of Machine Learning Research.
- Jonathan Rosa. 2019. Looking like a language, sound- ing like a race. Oxford University Press. · DOI
- Rachael Tatman. 2020. What i won't build. WiNLP Workshop at ACL.
- Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In NeurIPS.
Cited by
Sorted by citations
Items 301–317 of 317
219Chapter 11 Means of Discrimination. A Feminist Materialist Analysis of Making and Collecting Data
Savić, S. (2026) · De Gruyter eBooks · doi:10.1515/9783111373096-012
Benchmark-Task Heterogeneity in Misinformation-Related and Human–AI Text Classification
Swierk, G., & Olszowski, R. (2026) · Information · doi:10.3390/info17090866
Kim, J., Son, K., Yeom, C., Kim, W., Lee, S. H., Shin, J. S., Yang, H., Yoo, D., & Hahm, B. (2026) · Journal of Medical Internet Research · doi:10.2196/91405
The Core Levers of LLM Evaluation: Sets, Templates, and Raters
Robsky, A., Lavitas, L., & Wang, Y. (2026) · Springer eBooks · doi:10.1007/978-3-032-26749-8_3
Generative AI Technologies, Techniques and Tensions: A Primer
Behrens, J. T. (2026) · Springer eBooks · doi:10.1007/978-3-032-24819-0_2 · Cited by 2
A Computational Framework for Zipfian Analysis in Large-Scale NLP Corpus Engineering
Zhang, J., & Shen, L. (2026) · International Conference on Software Engineering and Computer Applications · doi:10.1145/3829641.3829739
iBRAHIMOĞLU, N., Demirezen, M. U., & Yıldız, F. (2026) · Cluster Computing · doi:10.1007/s10586-026-06537-4 · Cited by 2
Suresh, S., Hunter, R. B., Periard, B., Espinoza, J., Lourie, E. M., Suresh, S., Chaparro, J. D., Cheek, K., Dufendach, K. R., Goldstein, R. L., et al. (2026) · PEDIATRICS · doi:10.1542/peds.2026-079037
Charles, K. (2026) · Cogent Education · doi:10.1080/2331186x.2026.2740865
Training datasets tailored for domain-specific large language models: challenges, methods and trends
Qiu, X., Ren, K., Cui, X., Liu, W., Gao, Z., Hu, J., & Liang, Q. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-01150-2
Made and Real: Neomythism and the Life of Myth Beyond Its Makers
de Lune, R. (2026) · Global Philosophy · doi:10.1007/s10516-026-09804-6
A Unified Framework of Extractability in Large Language Models
Santos, S. M., Carrasco, M., & Yovine, S. (2026) · Machine Learning and Knowledge Extraction · doi:10.3390/make8100335
Artemis: Accelerating Mixture-Of-Experts Training Via Dynamic Expert Parallelism
Meng, Z., Diao, L., Qian, Z., Lin, W., & Wu, C. (2026) · IEEE International Conference on Cluster Computing (CLUSTER) · doi:10.1109/cluster68337.2026.00018
Position Paper: Just as Humans Need Vaccines, So Do Models - Immunization to Combat Falsehoods
Raza, S., Qureshi, R., Farooq, A., Lotif, M., Chadha, A., Pandya, D. A., & Emmanouilidis, C. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11728502
Chen, L., Wang, Z., Li, Z., Li, W., Zhu, H., & Han, D. (2026) · Applied Sciences · doi:10.3390/app162010430
FuseDot: A Multiplication-Fused Dot Product Accelerator for Efficient LLM Inference
Zhao, W., Yue, J., Yao, P., Cui, Y., Wang, Q., Sun, Y., Zhai, J., Jin, H., & Liao, X. (2026) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1145/3770743.3804105
Usha, C. R. (2026) · SDMIMD Journal of Management · doi:10.18311/sdmimd/2026/56257
Metadata
- Authors
- Dodge, J., Sap, M., Marasović, A., Agnew, W., Ilharco, G., Groeneveld, D., Mitchell, M., & Gardner, M.
- Affiliations
- University of Washington (US)
- Venue
- Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
- Published
- 2021
- Pages
- 1286-1305
- OpenAlex
- W3204096139
- Cited by
- 306
- References
- 65
- Citations by year
- 2026: 76 · 2025: 77 · 2024: 54 · 2023: 65 · 2022: 33 · 2021: 1
- FWCI
- 17.67
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Allen Institute for Artificial Intelligence
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- web crawling, corpus annotation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
