Scaling Data-Constrained Language Models
Open accessThe DOI matches its Crossref record; 149 of the 149 references it cites were found in a registry or an index; 37 of the 48 sources its claims cite were checked for support.
Abstract
The current trend of scaling language models involves increasing both parameter count and training dataset size. Extrapolating this trend suggests that training dataset size may soon be limited by the amount of text data available on the internet. Motivated by this limit, we investigate scaling language models in data-constrained regimes. Specifically, we run a large set of experiments varying the extent of data repetition and compute budget, ranging up to 900 billion training tokens and 9 billion parameter models. We find that with constrained data for a fixed compute budget, training with up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data. However, with more repetition, the value of adding compute eventually decays to zero. We propose and empirically validate a scaling law for compute optimality that accounts for the decreasing value of repeated tokens and excess parameters. Finally, we experiment with approaches mitigating data scarcity, including augmenting the training dataset with code data or removing commonly used filters. Models and datasets from our 400 training runs are freely available at https://github.com/huggingface/datablations.
Overview
CoChat Summary
TL;DR
The paper studies scaling language models when unique training data is limited, training more than 400 models up to 9B parameters and 900B tokens. It finds that up to 4 epochs of repeated data is nearly as good as unique data, but returns diminish sharply around 16 epochs, and proposes a data-constrained scaling law.
Key findings
- With constrained data and fixed compute, training up to 4 epochs yields negligible loss change versus unique data; an 8.7B model trained for 4 epochs on 44B unique tokens finished with only 0.5% higher validation loss than the single-epoch model trained on 178B unique tokens. More repetition eventually decays to zero, with \(R^*_D \approx 15\) (about 16 epochs) as the learned “half-life.”
- Data-constrained allocation should scale epochs faster than parameters: the fit gives \(R^*_N = 5.309743\) and \(R^*_D = 15.387756\), so excess parameters decay faster than repeated data. A 6.3B data-constrained compute-optimal model with 25B unique tokens achieved better loss and downstream performance than the 8.7B Chinchilla-style model despite having 27% fewer parameters.
- Complementary strategies can extend data: mixing in Python code provides a 2× increase in effective tokens, and filling up to 50% of data with code (42B tokens) showed no deterioration on natural language tasks. Perplexity filtering was effective, while deduplication did not help on C4 but was effective on noisier OSCAR.
- Double descent was observed: on 100M tokens, loss increased at 200 epochs before decreasing again; most such examples were removed from the scaling-law fitting.
Methods
- Trained more than 400 GPT-2 architecture/tokenizer transformer language models from 10M to 9B parameters, for up to 900B total tokens and up to 1500 epochs, on subsets of C4, OSCAR, and Python data from The Stack, with data constraints designed for maximal overlap.
- Used three protocols: fixed unique data (varying epochs and parameters), fixed FLOPs (varying data budget), and parametric fit. The parametric fit generalizes the Chinchilla law by replacing data and parameter terms with effective data and effective parameters via exponential decay constants \(R^*_D\) and \(R^*_N\), fitted on 182 samples.
- Evaluated held-out test loss (210M C4 validation tokens) and 19 downstream natural language tasks with 0–5 few-shot exemplars, normalizing scores; training used up to 256 AMD Instinct MI250X GPUs on LUMI.
Limitations
- The paper states it focuses on repeating the entire unique dataset, not fractions; returns from additional epochs may depend heavily on hyperparameters such as learning rate, dropout, or optimizer; findings are dataset-dependent (C4 and OSCAR) and limited to text data and GPT-style architectures; other data strategies and modalities are left to future work.
- Evident from the text: the fitting removed double-descent and outlier samples where excess epochs or parameters hurt, and the alternative alpha-beta decay formulation lacked rigorous mathematical justification and mispredicted returns. The downstream evaluation is described as not comprehensive, and the scaling laws fitted on C4 were applied to Galactica’s predominantly scientific dataset.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
29 Valid · 2 Not valid · 8 Not checked · 138 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“however, most prior large language models have been trained for a single epoch (Komatsuzaki, 2019;Brown et al., 2020)”
§ Scaling Data-Constrained Language Models > 1. Introduction
What the sources say
- Disputedcites Komatsuzaki, 2019 sourceExperimental — accuracy not yet measured
The Introduction (p. 1) describes multi-epoch training as the prevailing practice, not single-epoch training. Table 1 (p. 2) lists GPT at 100 epochs and BERT at 40; it does not report a verified epoch count for GPT-2. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
- Could not verifycites Brown et al., 2020 sourceExperimental — accuracy not yet measured
Section 2.2 (p. 8) describes how often Brown et al. sampled their own training datasets, and Appendix B (p. 43) describes their training procedure. Neither passage says whether most prior large language models were trained for a single epoch; searches for “single epoch” and equivalent descriptions found no such statement in our copy. (LLM, conf=0.73)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Villalobos et al. (2022) estimate that even high-quality English language data will be exhausted by the year 2024 given the Chinchilla scaling laws and the trend of training ever-larger models.”
§ Scaling Data-Constrained Language Models > 1. Introduction
What the sources say
- Partly supportedcites Villalobos et al., 2022 sourceExperimental — accuracy not yet measured
The Introduction (p. 1) links growing training-data demand to scaling laws, and Section 2.5 (p. 6) projects a data bottleneck if dataset growth continues. But our copy does not estimate exhaustion of high-quality English data by 2024; it gives a median exhaustion year of 2028 for public human text data, with a projected range of 2026–2032 in the Conclusion (p. 9). (LLM, conf=0.86)
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“However, the experiments of Galactica do not compare this setup to an alternative non-data-constrained model trained for one epoch on unique data.”
§ Scaling Data-Constrained Language Models > 1. Introduction
What the sources say
- Supportedcites Taylor et al., 2022 source
Section 5.1 (pp. 11–12) describes training Galactica for approximately 4.25 epochs and evaluates performance across repeated passes over its corpus. Section 5.5 (p. 19) compares Galactica with other language models, but does not describe the proposed one-epoch, unique-data alternative. The claim accurately identifies a comparison absent from the reported experiments. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Results from (Kaplan et al., 2020) also show that loss can increase when too many parameters are used, even with early stopping.”
§ Scaling Data-Constrained Language Models > 5. Results: Resource Allocation for Data-Constrained Scaling
What the sources say
- Partly supportedcites Kaplan et al., 2020Experimental — accuracy not yet measured
Figure 9 in Section 4.1 (p. 10) says that, for a smaller fixed dataset, early-stopped test performance stops improving as parameter count increases and the model begins to overfit. This supports the early-stopping and overfitting parts of the claim, but it does not show that loss increases with additional parameters. (LLM, conf=0.86)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“For 100M tokens, the corresponding one-epoch compute-optimal model according to scaling laws from (Hoffmann et al., 2022) has U N of approximately 7M parameters (see §C for the scaling coefficients we use).”
§ Scaling Data-Constrained Language Models > 5. Results: Resource Allocation for Data-Constrained Scaling
What the sources say
- Partly supportedcites Hoffmann et al., 2022 sourceExperimental — accuracy not yet measured
Section 3.4 (p. 8) supports using Hoffmann et al.’s scaling laws to relate optimal model size to training tokens. It does not give a 100M-token, one-epoch estimate: its Table 3 pairs 400M parameters with 8.0B tokens, which extrapolates to roughly 5M—not 7M—parameters at 100M tokens. That extrapolation is also below the model sizes used to fit the laws (Section 3, p. 4). (LLM, conf=0.82)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
89 more claims · 7 with their cited sources checked
Create a free account to see all 177 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 49
- It may state two separate claims as one. 45
- The quote only partly states this claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“They found that up to roughly four epochs, repetition is nearly as effective as new data, after which returns decay quickly”
From Training Compute-Optimal Transformer Models: A Review of Scaling Laws, Allocation, and Practice (2026) · cites this paper as Muennighoff et al., 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Growing concerns suggest that LLM pretraining may be approaching a ceiling as available high-quality data sources near exhaustion [58].”
From A Survey of Large Language Models (2026) · cites this paper as [58]
The cited source says what the claim attributes to it.
In the citing paper
“However, these studies find that repeating the same corpus only partially compensates for data scarcity; as the number of epochs increases, marginal gains diminish, and the risks of memorization and overfitting rise.”
From Scaling Data-Constrained Language Models with Synthetic Data (2026) · cites this paper as Muennighoff et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“The authors estimate the maximum increase in the effective dataset size that can be gained from multiple epochs at between 3x and 15x, and we anchor to this estimate in adjusting our model.”
From Will we run out of data? Limits of LLM scaling based on human-generated data (2022) · cites this paper as Muennighoff et al., 2023
In the citing paper
“Our theory will only rely upon the general form of the equations, specifically, that the dependence is inverse polynomial in N, D. So it applies to other frameworks of training (e.g., overtrained models Muennighoff et al. [2023]) where scaling laws have also been found.”
From A Theory for Emergence of Complex Skills in Language Models (2023) · cites this paper as Muennighoff et al. [2023]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“We find the best loss to be at around 20-60× more parameters and epochs, which corresponds to spending around 7000× more FLOPs.”
§ Scaling Data-Constrained Language Models > 5. Results: Resource Allocation for Data-Constrained Scaling
Reported 7000 · recomputed 3600
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Scaling Data-Constrained Language Models > Training data
“Like the equations in §B this formulation also reduces to the Chinchilla scaling laws in the base case of R D = 0 or R N = 0.”
§ Scaling Data-Constrained Language Models > Training data
“L(N, D, R N , R D ) = E + A N α * max(0,1-(R N /R * N )) + B D β * max(0,1-(R D /R * D )) (7”
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Self-citationUnsupervised Cross-lingual Representation Learning at Scale
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F. (., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2019) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.1911.02116 · Cited by 9,924
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550
Unsupervised Cross-lingual Representation Learning at Scale
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.747 · Cited by 5,637
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Research works printed but not linked: 9
- Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference on machine learning, pages 1691-1703. PMLR, 2020.
- Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, pages 5547-5569. PMLR, 2022.
- Kenneth Heafield. KenLM: Faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 187-197, Edinburgh, Scotland, July 2011. Association for Computational Linguistics. URL https://aclanthology.org/W11-2123.
- Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
- Pedro Javier Ortiz Su'arez, Laurent Romary, and Benoit Sagot. A monolingual approach to contextualized word embeddings for mid-resource languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1703-1714, Online, July 2020. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/2020.acl-main.156.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
- Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI spring symposium: logical formalizations of commonsense reasoning, pages 90-95, 2011.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
- Ge Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neural networks via zero-shot hyperparameter transfer. Advances in Neural Information Processing Systems, 34:17084-17097, 2021.
Cited by
Items 101–200 of 253
A Bitter Lesson for Data FilteringPreprint
Mohri, C., Duchi, J. C., & Hashimoto, T. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.19407 · Cited by 9
Scaling Retrieval-Based Language Models with a Trillion-Token Datastore
Shao, R., He, J., Asai, A., Shi, W., Dettmers, T., Min, S., Zettlemoyer, L., & Koh, P. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-2896 · Cited by 8
Towards EnergyGPT: A Large Language Model Specialized for the Energy Sector
Chebbi, A., & Kolade, B. (2025) · IEEE Access · doi:10.1109/access.2026.3693610 · Cited by 9
Provenance Tracking in Large-Scale Machine Learning Systems
Padovani, G., Anantharaj, V. G., & Fiore, S. (2025) · International Conference on Parallel Processing Workshops (ICPPW) · doi:10.1145/3750720.3757295 · Cited by 18
Evolution of ESG-focused DLT research: An NLP analysis of the literature
Cruz, W. H., Tylinski, K., Moore, A. P., Roche, N., Vadgama, N., Treiblmaier, H., Shangguan, J., Tasca, P., & Xu, J. (2025) · Quantitative Science Studies · doi:10.1162/qss.a.7 · Cited by 8
The Reasoning Under Uncertainty Trap: A Structural AI RiskPreprint
Pilditch, T. D. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.01743 · Cited by 8
Ghosh, S., Jain, N., & Sunoj, R. B. (2025) · ACS Catalysis · doi:10.1021/acscatal.5c07050 · Cited by 7
Scaling Laws for Discriminative Classification in Large Language Models
Wyatte, D., Tahmasbi, F., Li, M., & Markovich, T. (2025) · Applied AI Letters · doi:10.1002/ail2.109 · Cited by 9
Data+AI: LLM4Data and Data4LLM
Li, G., Wang, J., Zhang, C., & Wang, J. (2025) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3722212.3725641 · Cited by 12
Dresden, T. U., Kretzschmar, M., Dammann, M. P., Schwoch, S., Berger, E., Saske, B., & Paetzold-Byhain, K. (2024) · Entwerfen Entwickeln Erleben in Produktentwicklung und Design (EEE) · doi:10.25368/2024.eee.031 · Cited by 6
Domain-adaptative Continual Learning for Low-resource Tasks: Evaluation on NepaliPreprint
Duwal, S., Prasai, S., & Manandhar, S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.13860 · Cited by 6
Language models scale reliably with over-training and on downstream tasks
Gadre, S. Y., Smyrnis, G., Shankar, V., Gururangan, S., Wortsman, M., Shao, R., Mercat, J., Fang, A., Li, J., Keh, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.08540 · Cited by 5
Self-citationAthena: Intermediate Representations for Iterative Scaffolded App Generation with an LLM
Beason, J.-T., Cheng, R., Schoop, E., & Nichols, J. W. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789133 · Cited by 6
BabyLlama-2: Ensemble-Distilled Models Consistently Outperform Teachers With Limited Data
Tastet, J., & Timiryasov, I. (2024) · Research at the University of Copenhagen (University of Copenhagen) · doi:10.48550/arxiv.2409.17312 · Cited by 10
Stable Language Model Pre-training by Reducing Embedding Variability
Chung, W., Hong, J., An, N., Thorne, J., & Yun, S.-Y. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.606 · Cited by 10
Big Data Versus Big GPU: Evolving Requirements and Governance Dynamics of AI Training Data
Cheng, L., Gong, X., & Zhao, Y. (2025) · International Journal of Digital Law and Governance · doi:10.1515/ijdlg-2025-0004 · Cited by 5
Bringing Two Worlds Together: The Convergence of Large Language Models and Knowledge Graphs
Cai, L., Kang, Y., Yu, C., Fu, Y., Zhang, H., & Zhao, Y. (2024) · International Conference on Automation, Robotics and Computer Engineering (ICARCE) · doi:10.1109/icarce63054.2024.00046 · Cited by 9
Enhancing Data Quality in Federated Fine-Tuning of Foundation ModelsPreprint
Zhao, W., Du, Y., Lane, N. D., Chen, S., & Wang, Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.04529 · Cited by 9
LLMs can Design Sustainable Concrete – a Systematic BenchmarkPreprint
Völker, C., Rug, T., Jablonka, K. M., & Kruschwitz, S. (2024) · Research Square · doi:10.21203/rs.3.rs-3913272/v1 · Cited by 6
Scaling Law Phenomena Across Regression Paradigms: Multiple and Kernel ApproachesPreprint
Chen, Y., Guo, X., Li, X., Liang, Y., Shi, Z., & Song, Z. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.01314 · Cited by 5
Chain and Causal Attention for Efficient Entity Tracking
Fagnou, E., Caillon, P., Delattre, B., & Allauzen, A. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.731 · Cited by 12
Evaluation of Document Deduplication Algorithms for Large Text Corpora
Leveling, J., Helmer, L., Stein, B. J., Wegener, D., Sheikh, Z., Fernandes, E., & Abdelwahab, H. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-82481-4_27 · Cited by 6
Burns, T. F., Pârcălăbescu, L., Wäldchen, S., Barlow, M., Ziegltrum, G., Stampa, V., Harren, B., & Deiseroth, B. (2026) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2026.eacl-long.58 · Cited by 8
Bakaev, M., Gorovaia, S., & Mitrofanova, O. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9020046 · Cited by 6
Data Mixture Optimization: A Multi-fidelity Multi-scale Bayesian Framework
Yen, T., Siah, A. W. T., Chen, H., Guetta, D., Peng, T., & Namkoong, H. (2025) · neural information processing systems · doi:10.52202/085713-3300 · Cited by 8
Position: Enough of Scaling LLMs! Lets Focus on DownscalingPreprint
Goel, Y., Sengupta, A., & Chakraborty, T. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.00985 · Cited by 4
CLUES: Collaborative High-Quality Data Selection for LLMs via Training DynamicsPreprint
Zhao, W., Fan, H., Hu, S. X., Zhou, W., Chen, B., & Lane, N. D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.03004 · Cited by 4
ToReMi: Topic-Aware Data Reweighting for Dynamic Pre-Training Data SelectionPreprint
Zhu, X., Gu, Z., Wu, B., Zheng, S., Wang, T., Li, T., Feng, H., & Xiao, Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.00695 · Cited by 6
Scale Dependent Data DuplicationPreprint
Kazdan, J., Levi, N., Schaeffer, R., Chudnovsky, J., Puri, A., He, B., Donmez, M., Koyejo, S., & Donoho, D. L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.06603 · Cited by 5
q0: Primitives for Hyper-Epoch PretrainingPreprint
Mandal, B., Berman, S., Vegesna, A., & Dahal, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.03938 · Cited by 4
Bridging Compute- and Data-Optimal PretrainingPreprint
Qin, T., Hamidieh, K., & Alvarez-Melis, D. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.25271 · Cited by 4
Abra: Scaling Diffusion Image TrainingPreprint
Chickering, K. R., Lin, W.-A., Bhanded, S., Saunders, D., Tripathi, A., Song, J., Buch, S., & Yan, X. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.17286 · Cited by 4
To Code, or Not To Code? Exploring Impact of Code in Pre-training
Aryabumi, V., Su, Y., Ma, R., Morisot, A., Zhang, I., Locatelli, A., Fadaee, M., Üstün, A., & Hooker, S. (2024) · arXiv · doi:10.48550/arxiv.2408.10914 · Cited by 3
Tree-Instruct: A Preliminary Study of the Intrinsic Relationship between Complexity and Alignment
Zhao, Y., Yu, B., Hui, B., Yu, H., Li, M., Huang, F., Zhang, N. L., & Li, Y. (2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3cft4npm9kpq · Cited by 27
A theory of inference compute scaling: reasoning through directed stochastic skill search
Ellis-Mohr, A. R., Nayak, A. K., & Varshney, L. R. (2026) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2024.0510 · Cited by 4
Belay, T. D., Nahin, S. K., Azime, I. A., Monjur, O., Rei, M., Biemann, C., Muhammad, S. H., Yimam, S. M., & Chhabra, A. (2026) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/2026.findings-emnlp.546 · Cited by 3
XtremeLLMs: Towards Extremely Large Language ModelsPreprint
Mienye, I. D., Swart, T. G., & Obaido, G. (2024) · Preprints.org · doi:10.20944/preprints202408.1483.v1 · Cited by 3
Dorkin, A., Purason, T., Kalbaliyev, E., Kuulmets, H.-A., Ojastu, M., Fishel, M., Alumäe, T., Aedmaa, E., Kruusmaa, K., & Sirts, K. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.02041 · Cited by 4
Structured Distillation of Web Agent Capabilities Enables GeneralizationPreprint
Lù, X. H., & Reddy, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.07776 · Cited by 3
Better language models better model the N400, but not reading time
Michaelov, J. A., & Bergen, B. K. (2026) · Journal of Memory and Language · doi:10.1016/j.jml.2026.104762 · Cited by 4
Chronologically consistent large language models
He, S., Lv, L., Manela, A., & Wu, J. (2026) · Journal of Financial Economics · doi:10.1016/j.jfineco.2026.104374 · Cited by 4
Observational Scaling Laws and the Predictability of Language Model Performance
Ruan, Y., Maddison, C. J., & Hashimoto, T. (2024) · arXiv · doi:10.48550/arxiv.2405.10938 · Cited by 3
Kagan, B. J., & Kitchen, A. C. (2025) · Springer eBooks · doi:10.1007/978-3-031-93623-4_12 · Cited by 2
Large language model scaling laws for neural quantum states in quantum chemistry
Knitter, O., Zhao, D., Leichenauer, S., & Veerapaneni, S. (2026) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ae538f · Cited by 4
Zhou, Y. P., Qiu, J., Zhang, W., & Ni, F. (2026) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2026.3678742 · Cited by 4
Incentivizing inclusive contributions in model sharing markets
Zhang, E., Chai, J., Ye, R., Wang, Y., & Chen, S. (2025) · Nature Communications · doi:10.1038/s41467-025-62959-5 · Cited by 6
LinkQA: Synthesizing Diverse QA from Multiple Seeds Strongly Linked by Knowledge Points
Zhang, X., Ren, C., Tu, C. W., Weng, R., Yan, H., Wang, J., & Cai, X. (2026) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2026.acl-long.1036 · Cited by 2
Zhou, W., Bitterman, D. S., Afshar, M., & Miller, T. A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.12339 · Cited by 3
Dzierzanowski, R., Watson, T., & Stevenson, C. (2024) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/bvyjt · Cited by 2
Dobler, K., & de Melo, G. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.15793 · Cited by 3
Yang, Y., Bean, A. M., McCraith, R., & Mahdi, A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.07888 · Cited by 8
Problems with Chinchilla Approach 2: Systematic Biases in IsoFLOP Parabola FitsPreprint
Czech, E., Xu, Z., Elmatad, Y. S., Wang, Y., & Held, W. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.22339 · Cited by 2
Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-ExpertsPreprint
Dwivedi, C., Huang, B., Gupta, H., Jayarao, P., Varshney, N., & Yin, B. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.19835 · Cited by 2
Unified Neural Scaling LawsPreprint
Caballero, E., Jaini, P., Krueger, D., & Rish, I. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.26248 · Cited by 2
Predict before you train: Scaling Laws for particle physics foundation modelsPreprint
Uslu, J.-L., Nachman, B. P., & Ré, C. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.23377 · Cited by 2
Optimizer Memory Schedules for Outscaling the Overtraining AxisPreprint
Everett, K., & Qiu, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.04577 · Cited by 2
Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated DataPreprint
Jha, A., Li, M., Leskovec, J., Liang, P., & Zettlemoyer, L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.11917 · Cited by 3
LoopVL: Recurrent Visual IntelligencePreprint
Qian, Z., Gong, Z., Xu, Z., Li, H., Wang, Z., Liu, H., Qian, L., Pei, X., Xu, P., Luo, F., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.38426 · Cited by 2
A Comparative Analysis of the Lossless Data Compression Methods for Unsparsed Tabular Data
Erkuş, E. C., & Bursali, A. (2024) · International Conference on Electrical, Computer and Energy Technologies (ICECET) · doi:10.1109/icecet61485.2024.10698440 · Cited by 1
Zhao, N., Duan, X., & Cai, Z. G. (2025) · Open Mind · doi:10.1162/opmi.a.33 · Cited by 2
Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe
Jiang, A. Q., Ziarko, A., Piotrowski, B., Li, W., Jamnik, M., & Miłoś, P. (2024) · neural information processing systems · doi:10.52202/079017-1953 · Cited by 7
Beyond Repetition: Text Simplification and Curriculum Learning for Data-Constrained Pretraining
Roque, M. T., & Velasco, D. J. (2025) · BabyLM Workshop · doi:10.18653/v1/2025.babylm-main.19 · Cited by 4
Is More Data Worth the Cost? Dataset Scaling Laws in a Tiny Attention-Only Decoder
Wiegand, G.-H., Raichle, L., Staedeli, R., Hrycej, T., Bermeitinger, B., & Handschuh, S. (2026) · IEEE Swiss Conference on Data Science (SDS) · doi:10.1109/sds70563.2026.00019 · Cited by 2
Calculus of intelligence: A topos-monadic framework for agentic workflows
Yuan, Y., & Yao, A. C.-C. (2026) · iFuture · doi:10.26599/if.2026.9710001 · Cited by 2
Lukáč, M., Duchoň, F., Ivan, J., Dekan, M., Zelenay, E., Kocúr, M., Trepáčová, I., Jurov, T., & Pšenka, R. (2026) · Applied Sciences · doi:10.3390/app16157849 · Cited by 1
Efficient and scalable masked word prediction using concept formation
Lian, X., Wang, Z., & MacLellan, C. J. (2025) · Cognitive Systems Research · doi:10.1016/j.cogsys.2025.101371 · Cited by 6
How Hyper-Datafication Impacts the Sustainability Costs in Frontier AI
Wilson, S. N., Mair, S., Okinyi, M., Dam, E. B., Koch, J., & Selvan, R. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812393 · Cited by 2
Generalization error bounds for iterative learning algorithms with bounded updatesPreprint
Fu, J., & Zheng, N. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.05077 · Cited by 1
Technological Perspective on Digital Sovereignty
Stuermer, M. (2024) · ARBOR - Bern University of Applied Sciences Repository · doi:10.24451/arbor.22222 · Cited by 1
Retrieval Capabilities of Large Language Models Scale with Pretraining FLOPsPreprint
Portes, J., Jennings, C., Yuen, E. J., Doubov, S., & Carbin, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.17400 · Cited by 2
Natural Ungrokking: Asymmetric Control of Which Rules Survive PretrainingPreprint
Li, J., & Sreedhar, D. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.26050 · Cited by 1
Moving Alphabet: A Controlled Study of Training Data for Text-to-Video GenerationPreprint
Zheng, A. Y., Liu, L., Yeh, R. A., & Yin, X. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.18789 · Cited by 1
Chen, W., Chen, J., Lin, Z., & Vong, C. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.25052 · Cited by 1
A Scaling Study for fMRI Foundation ModelsPreprint
Ye, W., Pan, X., Xia, J., Zhang, J., Wang, M., & Liu, Q. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.27232 · Cited by 1
How to scale your HEP ML models: A recipe for robust architecture comparisons at scalePreprint
Vigl, M., Pond, N. I., Barr, J., Froch, A., Guest, D., Hartman, N. M., Kagan, M., & Heinrich, L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.06784 · Cited by 1
Data Predictability Shapes Weibull Weight-Scale Growth in Transformer TrainingPreprint
Ding, T. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.23573 · Cited by 1
Cen, Y., Wang, H., Zhu, X., Miao, C., & Luo, J. (2026) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2026.3655736
Unifying Data, Memory, and Compute Efficiency in LLM Training: A Survey
Schmidt, V., Nguyen, H. H., Jung, C., Salehi, S., & Schmeink, A. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2026.3702625
RADS-PDD: Reproducibility-aware dynamic similarity-based pre-training data detection against LLMs
Fan, X., Miyamoto, R., Mo, F., Chen, C., Matsumoto, T., Kido, F., & Yamana, H. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.109393 · Cited by 1
Bilingual Augmentation for Low- and Medium-Resource Languages
Daðason, J. F., & Loftsson, H. (2025) · IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT) · doi:10.1109/wi-iat67162.2025.00029
Training Compute-Optimal Transformer Models: A Review of Scaling Laws, Allocation, and Practice
Memming, A. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7142719
Scaling Data-Constrained Language Models with Synthetic Data
Kiyomaru, H., Oda, Y., Kodama, T., Liu, C., & Kawahara, D. (2026) · Findings of the Association for Computational Linguistics: EACL 2026 · doi:10.18653/v1/2026.findings-eacl.52
Scaling Training Data with Lossy Image Compression
Mentzer, K. L., & Montanari, A. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671904
Knitter, O. (2024) · Deep Blue (University of Michigan) · doi:10.7302/25065 · Cited by 1
Imran, A., & Gopalakrishnan, K. (2025) · Apress eBooks · doi:10.1007/979-8-8688-0989-7_4 · Cited by 1
It is time to talk about inorganic lifePreprint
Tureček, P., & Sobička, I. (2025) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/nwg26_v1
Data Heterogeneity Limits the Scaling Effect of Pretraining in Neural Data TransformersPreprint
Jiang, L. P., Chen, S., Tanumihardja, E., Han, X., Shi, W., Shea‐Brown, E., & Rao, R. P. N. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.05.12.653551
Relative Overfitting and Accept-Reject FrameworkPreprint
Liu, Y., & Zhang, Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.07783 · Cited by 1
Prompt Relativity Theory: A Relativistic Framework for AI CommunicationPreprint
Salem, M. F. A. (2025) · doi:10.22541/au.175207801.11250158/v2
Supernova: Achieving More with Less in Transformer ArchitecturesPreprint
Tanase, A.-V., & Pelican, E. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.15773
Data-Centric Methods for Decentralizing Large Language Models
Gururangan, S. (2024) · ResearchWorks at the University of Washington (University of Washington)
Improving large language model reasoning: On neurosymbolic training utilizing family tree problem
Leuze, F. (2025) · Aaltodoc (Aalto University)
Towards Data-efficient Machine Learning Systems
Sachdeva, N. (2024) · eScholarship (California Digital Library)
Training with Subword Variants Delays Overfitting in Repeated-Data Language ModelingPreprint
Cosma, A., Ruşeţi, Ş., Rǎdoi, E., & Dascălu, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.02867
Unsupervised Outlier Detection for Language-Independent Text Quality Filtering
Daðason, J. F., & Loftsson, H. (2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/4mxxg2kxppsy · Cited by 4
Text Filtering Classifiers for Medium-Resource Languages
Daðason, J. F., & Loftsson, H. (2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2xnr42qjhz9m · Cited by 3
Cognitively Layered Data Synthesis for Domain Adaptation of LLMs to Space Situational AwarenessPreprint
Linghu, D., Wang, C., Fan, D., Shi, W., Yin, K., Xue, X., Yang, F., Ren, H., & Zhang, C. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.09231
Language Representation Models for Low- and Medium-Resource Languages
Daðason, J. F. (2025) · Opin vísindi (Opin vísindi) · Cited by 1
Carpe Datum: Scaling behavior of transformers for heavy hadron flavor identification
The ATLAS collaboration (2026) · CERN Document Server (European Organization for Nuclear Research) · Cited by 11
Metadata
- Authors
- Muennighoff, N., Rush, A., Barak, B., Scao, T. L., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., & Raffel, C.
- ORCID
- Venue
- Advances in Neural Information Processing Systems 36
- Published
- 2023
- Pages
- 50358-50376
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W7133188041
- Cited by
- 47
- References
- 91
- Citations by year
- 2026: 7 · 2025: 21 · 2024: 15 · 2023: 4
- FWCI
- 8.42
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- other-oa
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, other-oa)
- UTUPub (University of Turku) (submittedVersion, other-oa)
- IRIS Research product catalog (Sapienza University of Rome) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2134157)
- U.S. Department of Energy (DESC0022199)
- European Commission (101070350)
- China Scholarship Council
- Defense Advanced Research Projects Agency (W911NF2010021, DE-SC0022199)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- language model scaling, training data efficiency, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.