MathPile: A Billion-Token-Scale Pretraining Corpus for Math
Open accessAbstract
High-quality, large-scale corpora are the cornerstone of building foundation models. In this work, we introduce MathPile, a diverse and high-quality math-centric corpus comprising about 9.5 billion tokens. Throughout its creation, we adhered to the principle of "less is more", firmly believing in the supremacy of data quality over quantity, even in the pre-training phase. Our meticulous data collection and processing efforts included a complex suite of preprocessing, prefiltering, language identification, cleaning, filtering, and deduplication, ensuring the high quality of our corpus. Furthermore, we performed data contamination detection on downstream benchmark test sets to eliminate duplicates and conducted continual pre-training experiments, booting the performance on common mathematical reasoning benchmarks. We aim for our MathPile to boost language models' mathematical reasoning abilities and open-source its different versions and processing scripts to advance the field.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
The publisher hasn’t deposited this paper’s reference list.
Cited by
Sorted by citations
LIMO: Less is More for Reasoning
Ye, Y., Huang, Z., Xiao, Y., Chern, E., Xia, S., & Liu, P. (2025) · arXiv · doi:10.48550/arxiv.2502.03387 · Cited by 4 · see citing works
Self-citationAutoMathKG: The Automated Mathematical Knowledge Graph Based on LLM and Vector Database
Bian, R., Geng, Y., Yang, Z., & Cheng, B. (2025) · Computational Intelligence · doi:10.1111/coin.70096 · Cited by 5 · see citing works
Humanity Amplified: The Fusion of Deep Learning and Human Insight to Shape the Future of InnovationPreprint
Bauschard, S., & Quidwai, S. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4621210 · Cited by 2 · see citing works
Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination
Wu, M., Zhang, Z., Dong, Q., Xi, Z., Zhao, J., Jin, S., Fan, X., Zhou, Y., Lv, H., Zhang, M., et al. (2025) · arXiv · doi:10.48550/arxiv.2507.10532 · Cited by 1 · see citing works
barfLM: A Small Language Model
Shreyas, V., Ekbote, A., Reddy, C. T. K., & Kumar, N. N. (2025) · International Conference on IoT Based Control Networks and Intelligent Systems (ICICNIS) · doi:10.1109/icicnis66685.2025.11315709
- Authors
- Wang, Z., Li, X., Xia, R., & Liu, P.
- ORCID
- Wang, Zengzhi
- Venue
- Advances in Neural Information Processing Systems 37
- Published
- 2024
- Pages
- 25426-25468
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W4415798467
- Cited by
- 3
- References
- 0
- Citations by year
- 2025: 2 · 2023: 1
- FWCI
- 0.58
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- cc-by-nc-sa
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Mathematics, Computing, and Information Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- mathematical texts, language model pretraining
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
