Datasheet for the Pile
Open accessAbstract
This datasheet describes the Pile, a 825 GiB dataset of human-authored text compiled by EleutherAI for use in large-scale language modeling. The Pile is comprised of 22 different text sources, ranging from original scrapes done for this project, to text data made available by the data owners, to third-party scrapes available online.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The Pile authors analyzed and documented syntactic and semantic properties of their dataset and released a datasheet.
“The most notable example of this is the Pile, for which the authors themselves analyze and document a variety of syntactic and semantic properties of the dataset including structural statistics (n-gram counts, language, document sizes), topical distributions across its components, social bias and sentiment co-occurrence, pejorative content, and information about licensing and authorial consent, in”
From The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023) · cites this paper as Biderman et al., 2022
The Abstract (p. 1) identifies this work as a datasheet for the Pile. The dataset-composition section (p. 8) gives document counts and sizes by component, and the legal and ethical sections (pp. 14–17) discuss licensing, potentially offensive content, and consent. The passages do not substantiate the more specific attribution of n-gram counts, topical distributions, sentiment co-occurrence, or pejorative-content analyses to this datasheet. (LLM, conf=0.88)
Experimental — accuracy not yet measuredThere has been an increased focus on analyzing other dimensions of data quality and stewardship, with several initiatives aiming to document existing and newly developed resources.
“There has also been an increased focus on analyzing other dimensions of data quality and stewardship [102, 107, 121, 123], with several noteworthy initiatives aiming to document both existing [9, 20, 30, 43], and newly developed [16, 60, 136] resources.”
From Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022) · cites this paper as [16]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Part of this paper’s reference list could not be split into single entries.
Rethinking Attention with Performers
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.14794 · Cited by 695 · cited 1× · see citing works
“Project Gutenberg (PG-19): Project Gutenberg 1919 has been used extensively, including by [Zaheer et al., 2020, Choromanski et al., 2020, Roller et al., 2020, Huang and Yang, 2020, Irie et al., 2020].”
Method
Flagged first, then by citations
Research works printed but not linked: 11
- [2]2. Junshu Wang. Information Extraction from TV Series Scripts for Uptake Prediction . PhD thesis, Auckland University of Technology, 2017.
- [4]4. Leonardo Zilio, Rodrigo Wilkens, and C´ edrick Fairon. Passport: A dependency parsing model for portuguese. In Aline Villavicencio, Viviane Moreira, Alberto Abad, Helena Caseli, Pablo Gamallo, Carlos Ramisch, Hugo Gon¸ calo Oliveira, and Gustavo Henrique Paetzold, editors, Computational Processing of the Portuguese Language , pages 479-489, Cham, 2018. Springer International Publishing. ISBN 978-3319-99722-3.
- [6]6. Rihards Kriˇ slauks and M¯ arcis Pinnis. Tilde at wmt 2020: News task systems. arXiv preprint arXiv:2010.15423 , 2020. · arXiv
- [7]7. Sungjae Cho, Jaeseo Lim, Chris Hickey, Jung Park, and Byoung-Tak Zhang. Simulating problem difficulty in arithmetic cognition through dynamic connectionist models. arXiv preprint arXiv:1905.03617 , 2019. · arXiv
- [9]9. Alon Talmor, Oyvind Tafjord, Peter Clark, Yoav Goldberg, and Jonathan Berant. Leap-Of-Thought: Teaching pre-trained models to systematically reason over implicit knowledge. Advances in Neural Information Processing Systems , 33, 2020.
- [11]11. Chaz Firestone. Performance vs. competence in human-machine comparisons. Proceedings of the National Academy of Sciences , 117(43):26562-26571, 2020. ISSN 0027-8424. doi: 10.1073/pnas.1905334117. URL https://www.pnas.org/content/117/43/26562 . · DOI · Paper page
- [12]12. Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3128-3137, 2015.
- [13]13. Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 28, pages 3294-3302. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper/2015/file/f442d33fa06832082290ad8544a8da27-Paper.pdf .
- [14]14. Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. arXiv preprint arXiv:1605.05396 , 2016. · arXiv
- [16]16. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018. · arXiv
- [17]17. Jack Bandy and Nicholas Vincent. Addressing' documentation debt' in machine learning research: A retrospective datasheet for bookcorpus. arXiv preprint arXiv:2105.05241 , 2021. · arXiv
Cited by
Sorted by citations
RWKV: Reinventing RNNs for the Transformer Era
Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Derczynski, L., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.936 · Cited by 390 · see citing works
Self-citationShould ChatGPT be Biased? Challenges and Risks of Bias in Large Language Models
Ferrara, E. (2023) · arXiv · doi:10.48550/arxiv.2304.03738 · Cited by 282
Should ChatGPT be biased? Challenges and risks of bias in large language models
Ferrara, E. (2023) · First Monday · doi:10.5210/fm.v28i11.13346 · Cited by 282 · see citing works
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
Laurençon, H., Saulnier, L., Wang, T., Akiki, C., del Moral, A. V., Scao, T. L., Von Werra, L., Mou, C., Ponferrada, E. G., Nguyen, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.03915 · Cited by 61 · see citing works
Self-citationData Governance in the Age of Large-Scale Data-Driven Language Technology
Jernite, Y., Nguyen, H., Biderman, S., Rogers, A., Masoud, M., Danchev, V., Tan, S., Luccioni, A. S., Subramani, N., Johnson, I., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3534637 · Cited by 45 · see citing works
Self-citationEliciting Latent Predictions from Transformers with the Tuned Lens
Belrose, N., Ostrovsky, I., McKinney, L., Furman, Z., Smith, L., Halawi, D., Biderman, S., & Steinhardt, J. (2023) · arXiv · doi:10.48550/arxiv.2303.08112 · Cited by 27 · see citing works
“Since BLOOM and GPT-2 do not have publicly available validation sets, we use the Pile validation set (Gao et al., 2020; Biderman et al., 2022).”
MethodSelf-citation
- Authors
- Biderman, S., Bicheno, K., & Gao, L.
- ORCID
- Biderman, Stella
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2201.07311
- OpenAlex
- W4226163008
- Cited by
- 12
- References
- 1
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 2 · 2023: 6 · 2022: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- text corpora, web scraping, training data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
