The Pile: An 800GB Dataset of Diverse Text for Language Modeling
Open accessAbstract
Recent work has demonstrated that increased training dataset diversity improves general cross-domain knowledge and downstream generalization capability for large-scale language models. With this in mind, we present \textit{the Pile}: an 825 GiB English text corpus targeted at training large-scale language models. The Pile is constructed from 22 diverse high-quality subsets -- both existing and newly constructed -- many of which derive from academic or professional sources. Our evaluation of the untuned performance of GPT-2 and GPT-3 on the Pile shows that these models struggle on many of its components, such as academic writing. Conversely, models trained on the Pile improve significantly over both Raw CC and CC-100 on all components of the Pile, while improving performance on downstream evaluations. Through an in-depth exploratory analysis, we document potentially concerning aspects of the data for prospective users. We make publicly available the code used in its construction.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
High-quality books provide formal long-form texts, which are particularly beneficial for modeling long-term dependencies and generating coherent outputs.
“Additionally, high-quality books (e.g., the Books3 and Bookcorpus2 datasets from the Pile [35]) provide formal long-form texts, which are particularly beneficial for modeling long-term dependencies and generating coherent outputs.”
From A Survey of Large Language Models (2026) · cites this paper as [35]
Section 2.3 (p. 3) says books are invaluable for long-range context modeling research and coherent storytelling, closely supporting the claimed benefits. Sections 2.3 and 2.15 (pp. 3, 5) identify Books3 and BookCorpus2 as book datasets, but do not characterize their texts as formal; the source's coherence point is specifically about storytelling, not outputs generally. (LLM, conf=0.82)
Experimental — accuracy not yet measuredThe Pile is a 400-billion token compilation of 22 individual datasets, including Pile-CC, PubMed Central, and Wikipedia.
“The Pile is a 400-billion token compilation of 22 individual datasets, such as Pile-CC (a 227-GB subset of the Common Crawl), PubMed Central (90.27 GB of peer-reviewed medical articles) and Wikipedia (40 GB).”
From Medical large language models are vulnerable to data-poisoning attacks (2025) · cites this paper as [18]
Section 2 (pp. 2–3) confirms that the Pile has 22 constituent datasets, including Pile-CC, PubMed Central, and Wikipedia; its table gives Pile-CC as 227.12 GiB and PubMed Central as 90.27 GiB before weighting. The paper does not state a 400-billion-token total, and its table lists Wikipedia as 6.38 GiB before weighting or 19.13 GiB effective size—not 40 GB. Section 3.1 (p. 6) gives a token-per-byte ratio, but does not attach a 400-billion-token figure to the Pile. (LLM, conf=0.88)
Experimental — accuracy not yet measuredDue to licensing restrictions, existing LLMs are legally constrained to training on corpora that are publicly licensed or otherwise available for use without infringing copyright laws.
“Due to licensing restrictions [262], existing LLMs are legally constrained to training on corpora that are publicly licensed [63, 93] or otherwise available for use without infringing copyright laws [10, 115].”
From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [93]
Section 7.1 (p. 15) acknowledges that copyright law varies by country and that some Pile components may need to be excluded to comply with local law. But the paper does not establish a blanket licensing restriction on existing LLMs: it argues that its noncommercial use of copyrighted works is fair use, and Section 6.5 (p. 13) says author permission is typically not legally required for NLP training. (LLM, conf=0.84)
Experimental — accuracy not yet measuredThe Pile documentation specifies that raw HTML inputs from the Common Crawl are used to construct the dataset.
“The Pile documentation specifies that raw HTML inputs from the Common Crawl are used to construct the dataset.”
From Medical large language models are vulnerable to data-poisoning attacks (2025) · cites this paper as [18]
The pretrained checkpoint is fine-tuned on The Pile, reaching a final validation perplexity of 6.3 on the Pile validation split.
“The pretrained checkpoint is fine-tuned on The Pile [32], reaching a final validation perplexity of 6 . 3 on the Pile validation split.”
From Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection (2026) · cites this paper as [32]
Human-written ingroup and outgroup sentences were obtained from large-scale web corpora commonly used to pretrain LLMs, including C4, The Pile, OpenWebText and OLM.
“To juxtapose social identity biases measured in LLMs against human-level biases, we obtained human-written ingroup and outgroup sentences from large-scale web corpora commonly used to pretrain LLMs, including C448, The Pile55, OpenWebText56 and the November–December 2022 edition of OLM57.”
From Generative language models exhibit social identity biases (2024) · cites this paper as [55]
LLMs are predominantly trained on extensive general domain corpora, and may exhibit deficits in domain-specific knowledge.
“Furthermore, given that LLMs are predominantly trained on extensive general domain corpora [93, 243, 254], they may exhibit deficits in domain-specific knowledge.”
From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [93]
InCoder is a 6.7B parameter language model trained using a causal masking objective.
“InCoder [4] is a 6.7B parameter language model trained using a causal masking objective [31].”
From MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023) · cites this paper as [31]
Citation [31] is attached to the causal masking objective. The Pile’s Evaluation section (p. 7) describes training 1.3-billion-parameter models on different datasets, but neither it nor the passages returned for searches on “causal masking objective” and “InCoder 6.7B” addresses InCoder’s training objective. (LLM, conf=0.77)
Experimental — accuracy not yet measuredThe Wikimedia projects offer a wealth of experience in highly collaborative and largely self-regulated data curation, similar to the goals in the proposed governance structure.
“The Wikimedia projects offer a wealth of experience in highly collaborative and largely self-regulated data curation [56], similar to the goals in the proposed governance structure.”
From Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022) · cites this paper as [56]
Section 2.10 (p. 4) describes Wikipedia as a source of high-quality text for language modeling, but does not discuss how Wikimedia projects curate data. Searches for collaborative curation, community editors, and self-regulation found no passage addressing the claim. (LLM, conf=0.76)
Experimental — accuracy not yet measuredGPT-J-6B is pretrained on the Pile with a cutoff in 2020.
“adopt GPT-J6B as the base model, which is pretrained on the Pile [51] with a cutoff in 2020.”
From WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models (2024) · cites this paper as [51]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
References 50 · Cited by 492
Research works printed but not linked: 53
- Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. 2016. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565 . · arXiv
- Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani. 2010. Sentiwordnet 3.0: An enhanced lexical resource for sentiment analysis and opinion mining. In LREC . European Language Resources Association.
- Stella Biderman. 2021. Data statement for the Pile. arXiv preprint arXiv .
- Stella Biderman, Kieran Bicheno, and Leo Gao. 2021. Datasheet for the Pile. arXiv preprint arXiv .
- Stella Biderman and Walter J. Scheirer. 2020. Pitfalls in machine learning research: Reexamining the development cycle. NeurIPS 'I Can't Believe It's Not Better!' Workshop .
- David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research , 3(Jan):993-1022.
- Nick Bostrom. 2014. Superintelligence: Paths, Dangers, Strategies . Oxford University Press, Inc.
- Nick Bostrom and Eliezer Yudkowsky. 2014. The ethics of artificial intelligence. The Cambridge handbook of artificial intelligence , 1:316-334.
- Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 . · arXiv
- Isaac Caswell, Theresa Breiner, Daan van Esch, and Ankur Bapna. 2020. Language id in the wild: Unexpected challenges on the path to a thousand-language web text corpus. arXiv preprint arXiv:2010.14571 . · arXiv
- Alina Maria Ciobanu, Liviu P Dinu, and Andrea Sgarro. 2017. Towards a map of the syntactic similarity of languages. In International Conference on Computational Linguistics and Intelligent Text Processing , pages 576590. Springer.
- Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised crosslingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 8440-8451, Online. Association for Computational Linguistics.
- Andrew Critch and David Krueger. 2020. AI Research Considerations for Human Existential Safety (ARCHES). Preprint at acritch.com/arches .
- Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860 . · arXiv
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) . Association for Computational Linguistics.
- Niels Ferguson and Bruce Schneier. 2003. Practical Cryptography . John Wiley & Sons.
- Casey Fiesler, Nathan Beard, and Brian C Keegan. 2020. No robots, spiders, or scrapers: Legal and ethical regulation of data collection methods in social media terms of service. In Proceedings of the International AAAI Conference on Web and Social Media , volume 14, pages 187-196.
- Katja Grace, John Salvatier, Allan Dafoe, Baobao Zhang, and Owain Evans. 2018. When will AI exceed human performance? evidence from AI experts. Journal of Artificial Intelligence Research , 62:729-754.
- David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2003. English gigaword. Linguistic Data Consortium, Philadelphia , 4(1):34.
- Declan Groves and Andy Way. 2006. Hybridity in mt: Experiments on the Europarl corpus. In Proceeedings of the 11th Annual conference of the European Association for Machine Translation (EAMT 2006) .
- Alexander Halavais. 2019. Overcoming terms of service: a proposal for ethical distributed research. Information, Communication & Society , 22(11):1567-1581.
- Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. 2020. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701 . · arXiv
- Matthew Hoffman, Francis Bach, and David Blei. 2010. Online learning for latent dirichlet allocation. advances in neural information processing systems , 23:856-864.
- Eun Seo Jo and Timnit Gebru. 2020. Lessons from archives: Strategies for collecting sociocultural data in machine learning. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency , pages 306-316.
- Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 . · arXiv
- Bryan Klimt and Yiming Yang. 2004. The Enron corpus: A new dataset for email classification research. In European Conference on Machine Learning , pages 217-226. Springer.
- Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In MT summit , volume 5, pages 79-86. Citeseer.
- Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summarizing long sequences. arXiv preprint arXiv:1801.10198 . · arXiv
- Edward Loper and Steven Bird. 2002. NLTK: The Natural Language Toolkit. In Proceedings of the ACL Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics , pages 62-69. Somerset, NJ: Association for Computational Linguistics. http://arXiv. org/abs/cs/0205028 .
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems , volume 26, pages 3111-3119. Curran Associates, Inc.
- Jonathan A Obar. 2020. Sunlight alone is not a disinfectant: Consent and the futility of opening big data black boxes (without assistance). Big Data & Society , 7(1):2053951720935615.
- Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP) , pages 1532-1543.
- Vinay Uday Prabhu and Abeba Birhane. 2020. Large image datasets: A pyrrhic win for computer vision? arXiv preprint arXiv:2006.16923 . · arXiv
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog , 1(8):9.
- Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap. 2019. Compressive transformers for long-range sequence modelling. arXiv preprint .
- Inioluwa Deborah Raji and Jingying Yang. 2019. ABOUT ML: Annotation and benchmarking on understanding and transparency of machine learning lifecycles. arXiv preprint arXiv:1912.06166 . · arXiv
- C. Radhakrishna Rao. 1961. Generation of random permutations of given number of elements using random sampling numbers. Sankhy¯ a: The Indian Journal of Statistics, Series A (1961-2002) , 23(3):305-307.
- S. Russell. 2019. Human Compatible: Artificial Intelligence and the Problem of Control . Penguin Publishing Group.
- David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. 2019. Analysing mathematical reasoning abilities of neural models. arXiv preprint arXiv:1904.01557 . · arXiv
- Ismaïla Seck, Khouloud Dahmane, Pierre Duthon, and Gaëlle Loosli. 2018. Baselines and a datasheet for the Cerema AWP dataset. arXiv preprint arXiv:1806.04016 . · arXiv
- Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-LM: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053 . · arXiv
- Carl Shulman and Nick Bostrom. 2020. Sharing the world with digital minds. preprint .
- Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. Advances in Neural Information Processing Systems , 33.
- Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019a. Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures. In 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7) . Leibniz-Institut für Deutsche Sprache.
- Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019b. Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures. In 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7) . Leibniz-Institut für Deutsche Sprache.
- Anja Thieme, Danielle Belgrave, and Gavin Doherty. 2020. Machine learning in mental health: A systematic review of the HCI literature to support the development of effective and implementable ML systems. ACM Transactions on Computer-Human Interaction (TOCHI) , 27(5):1-53.
- J. Tiedemann. 2016. Finding alternative translations in a large corpus of movie subtitles. Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016) .
- Trieu H. Trinh and Quoc V. Le. 2018. A simple method for commonsense reasoning. CoRR , abs/1806.02847. · arXiv
- Hans Van Halteren. 2008. Source language markers in Europarl translations. In Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008) , pages 937-944.
- Jessica Vitak, Katie Shilton, and Zahra Ashktorab. 2016. Beyond the Belmont principles: Ethical challenges, practices, and beliefs in the online data research community. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing , pages 941-953.
- Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2019. CCNet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.00359 . · arXiv
- Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-ofthe-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 38-45, Online. Association for Computational Linguistics.
- Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dÁlché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32 , pages 9054-9065. Curran Associates, Inc.
- Authors
- Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., & Leahy, C.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2101.00027
- OpenAlex
- W3118781290
- Cited by
- 492
- References
- 50
- Citations by year
- 2026: 25 · 2025: 40 · 2024: 98 · 2023: 209 · 2022: 89 · 2021: 31
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- large language models, text corpus construction, academic writing, GPT-2, GPT-3, domain generalization, data quality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
