Scaling Data-Constrained Language Models
Open accessThe DOI matches its Crossref record; 149 of the 149 references it cites were found in a registry or an index; 37 of the 48 sources its claims cite were checked for support.
Abstract
The current trend of scaling language models involves increasing both parameter count and training dataset size. Extrapolating this trend suggests that training dataset size may soon be limited by the amount of text data available on the internet. Motivated by this limit, we investigate scaling language models in data-constrained regimes. Specifically, we run a large set of experiments varying the extent of data repetition and compute budget, ranging up to 900 billion training tokens and 9 billion parameter models. We find that with constrained data for a fixed compute budget, training with up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data. However, with more repetition, the value of adding compute eventually decays to zero. We propose and empirically validate a scaling law for compute optimality that accounts for the decreasing value of repeated tokens and excess parameters. Finally, we experiment with approaches mitigating data scarcity, including augmenting the training dataset with code data or removing commonly used filters. Models and datasets from our 400 training runs are freely available at https://github.com/huggingface/datablations.
Overview
CoChat Summary
TL;DR
The paper studies scaling language models when unique training data is limited, training more than 400 models up to 9B parameters and 900B tokens. It finds that up to 4 epochs of repeated data is nearly as good as unique data, but returns diminish sharply around 16 epochs, and proposes a data-constrained scaling law.
Key findings
- With constrained data and fixed compute, training up to 4 epochs yields negligible loss change versus unique data; an 8.7B model trained for 4 epochs on 44B unique tokens finished with only 0.5% higher validation loss than the single-epoch model trained on 178B unique tokens. More repetition eventually decays to zero, with \(R^*_D \approx 15\) (about 16 epochs) as the learned “half-life.”
- Data-constrained allocation should scale epochs faster than parameters: the fit gives \(R^*_N = 5.309743\) and \(R^*_D = 15.387756\), so excess parameters decay faster than repeated data. A 6.3B data-constrained compute-optimal model with 25B unique tokens achieved better loss and downstream performance than the 8.7B Chinchilla-style model despite having 27% fewer parameters.
- Complementary strategies can extend data: mixing in Python code provides a 2× increase in effective tokens, and filling up to 50% of data with code (42B tokens) showed no deterioration on natural language tasks. Perplexity filtering was effective, while deduplication did not help on C4 but was effective on noisier OSCAR.
- Double descent was observed: on 100M tokens, loss increased at 200 epochs before decreasing again; most such examples were removed from the scaling-law fitting.
Methods
- Trained more than 400 GPT-2 architecture/tokenizer transformer language models from 10M to 9B parameters, for up to 900B total tokens and up to 1500 epochs, on subsets of C4, OSCAR, and Python data from The Stack, with data constraints designed for maximal overlap.
- Used three protocols: fixed unique data (varying epochs and parameters), fixed FLOPs (varying data budget), and parametric fit. The parametric fit generalizes the Chinchilla law by replacing data and parameter terms with effective data and effective parameters via exponential decay constants \(R^*_D\) and \(R^*_N\), fitted on 182 samples.
- Evaluated held-out test loss (210M C4 validation tokens) and 19 downstream natural language tasks with 0–5 few-shot exemplars, normalizing scores; training used up to 256 AMD Instinct MI250X GPUs on LUMI.
Limitations
- The paper states it focuses on repeating the entire unique dataset, not fractions; returns from additional epochs may depend heavily on hyperparameters such as learning rate, dropout, or optimizer; findings are dataset-dependent (C4 and OSCAR) and limited to text data and GPT-style architectures; other data strategies and modalities are left to future work.
- Evident from the text: the fitting removed double-descent and outlier samples where excess epochs or parameters hurt, and the alternative alpha-beta decay formulation lacked rigorous mathematical justification and mispredicted returns. The downstream evaluation is described as not comprehensive, and the scaling laws fitted on C4 were applied to Galactica’s predominantly scientific dataset.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
29 Valid · 2 Not valid · 8 Not checked · 138 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“however, most prior large language models have been trained for a single epoch (Komatsuzaki, 2019;Brown et al., 2020)”
§ Scaling Data-Constrained Language Models > 1. Introduction
What the sources say
- Disputedcites Komatsuzaki, 2019 sourceExperimental — accuracy not yet measured
The Introduction (p. 1) describes multi-epoch training as the prevailing practice, not single-epoch training. Table 1 (p. 2) lists GPT at 100 epochs and BERT at 40; it does not report a verified epoch count for GPT-2. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
- Could not verifycites Brown et al., 2020 sourceExperimental — accuracy not yet measured
Section 2.2 (p. 8) describes how often Brown et al. sampled their own training datasets, and Appendix B (p. 43) describes their training procedure. Neither passage says whether most prior large language models were trained for a single epoch; searches for “single epoch” and equivalent descriptions found no such statement in our copy. (LLM, conf=0.73)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Villalobos et al. (2022) estimate that even high-quality English language data will be exhausted by the year 2024 given the Chinchilla scaling laws and the trend of training ever-larger models.”
§ Scaling Data-Constrained Language Models > 1. Introduction
What the sources say
- Partly supportedcites Villalobos et al., 2022 sourceExperimental — accuracy not yet measured
The Introduction (p. 1) links growing training-data demand to scaling laws, and Section 2.5 (p. 6) projects a data bottleneck if dataset growth continues. But our copy does not estimate exhaustion of high-quality English data by 2024; it gives a median exhaustion year of 2028 for public human text data, with a projected range of 2026–2032 in the Conclusion (p. 9). (LLM, conf=0.86)
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“However, the experiments of Galactica do not compare this setup to an alternative non-data-constrained model trained for one epoch on unique data.”
§ Scaling Data-Constrained Language Models > 1. Introduction
What the sources say
- Supportedcites Taylor et al., 2022 source
Section 5.1 (pp. 11–12) describes training Galactica for approximately 4.25 epochs and evaluates performance across repeated passes over its corpus. Section 5.5 (p. 19) compares Galactica with other language models, but does not describe the proposed one-epoch, unique-data alternative. The claim accurately identifies a comparison absent from the reported experiments. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Results from (Kaplan et al., 2020) also show that loss can increase when too many parameters are used, even with early stopping.”
§ Scaling Data-Constrained Language Models > 5. Results: Resource Allocation for Data-Constrained Scaling
What the sources say
- Partly supportedcites Kaplan et al., 2020Experimental — accuracy not yet measured
Figure 9 in Section 4.1 (p. 10) says that, for a smaller fixed dataset, early-stopped test performance stops improving as parameter count increases and the model begins to overfit. This supports the early-stopping and overfitting parts of the claim, but it does not show that loss increases with additional parameters. (LLM, conf=0.86)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“For 100M tokens, the corresponding one-epoch compute-optimal model according to scaling laws from (Hoffmann et al., 2022) has U N of approximately 7M parameters (see §C for the scaling coefficients we use).”
§ Scaling Data-Constrained Language Models > 5. Results: Resource Allocation for Data-Constrained Scaling
What the sources say
- Partly supportedcites Hoffmann et al., 2022 sourceExperimental — accuracy not yet measured
Section 3.4 (p. 8) supports using Hoffmann et al.’s scaling laws to relate optimal model size to training tokens. It does not give a 100M-token, one-epoch estimate: its Table 3 pairs 400M parameters with 8.0B tokens, which extrapolates to roughly 5M—not 7M—parameters at 100M tokens. That extrapolation is also below the model sizes used to fit the laws (Section 3, p. 4). (LLM, conf=0.82)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
89 more claims · 7 with their cited sources checked
Create a free account to see all 177 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 49
- It may state two separate claims as one. 45
- The quote only partly states this claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“They found that up to roughly four epochs, repetition is nearly as effective as new data, after which returns decay quickly”
From Training Compute-Optimal Transformer Models: A Review of Scaling Laws, Allocation, and Practice (2026) · cites this paper as Muennighoff et al., 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Growing concerns suggest that LLM pretraining may be approaching a ceiling as available high-quality data sources near exhaustion [58].”
From A Survey of Large Language Models (2026) · cites this paper as [58]
The cited source says what the claim attributes to it.
In the citing paper
“However, these studies find that repeating the same corpus only partially compensates for data scarcity; as the number of epochs increases, marginal gains diminish, and the risks of memorization and overfitting rise.”
From Scaling Data-Constrained Language Models with Synthetic Data (2026) · cites this paper as Muennighoff et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“The authors estimate the maximum increase in the effective dataset size that can be gained from multiple epochs at between 3x and 15x, and we anchor to this estimate in adjusting our model.”
From Will we run out of data? Limits of LLM scaling based on human-generated data (2022) · cites this paper as Muennighoff et al., 2023
In the citing paper
“Our theory will only rely upon the general form of the equations, specifically, that the dependence is inverse polynomial in N, D. So it applies to other frameworks of training (e.g., overtrained models Muennighoff et al. [2023]) where scaling laws have also been found.”
From A Theory for Emergence of Complex Skills in Language Models (2023) · cites this paper as Muennighoff et al. [2023]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“We find the best loss to be at around 20-60× more parameters and epochs, which corresponds to spending around 7000× more FLOPs.”
§ Scaling Data-Constrained Language Models > 5. Results: Resource Allocation for Data-Constrained Scaling
Reported 7000 · recomputed 3600
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Scaling Data-Constrained Language Models > Training data
“Like the equations in §B this formulation also reduces to the Chinchilla scaling laws in the base case of R D = 0 or R N = 0.”
§ Scaling Data-Constrained Language Models > Training data
“L(N, D, R N , R D ) = E + A N α * max(0,1-(R N /R * N )) + B D β * max(0,1-(R D /R * D )) (7”
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Self-citationUnsupervised Cross-lingual Representation Learning at Scale
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F. (., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2019) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.1911.02116 · Cited by 9,924
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550
Unsupervised Cross-lingual Representation Learning at Scale
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.747 · Cited by 5,637
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Research works printed but not linked: 9
- Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference on machine learning, pages 1691-1703. PMLR, 2020.
- Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, pages 5547-5569. PMLR, 2022.
- Kenneth Heafield. KenLM: Faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 187-197, Edinburgh, Scotland, July 2011. Association for Computational Linguistics. URL https://aclanthology.org/W11-2123.
- Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
- Pedro Javier Ortiz Su'arez, Laurent Romary, and Benoit Sagot. A monolingual approach to contextualized word embeddings for mid-resource languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1703-1714, Online, July 2020. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/2020.acl-main.156.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
- Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI spring symposium: logical formalizations of commonsense reasoning, pages 90-95, 2011.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
- Ge Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neural networks via zero-shot hyperparameter transfer. Advances in Neural Information Processing Systems, 34:17084-17097, 2021.
Cited by
Items 201–253 of 253
Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
Luukkonen, R., Ville, K., Jouni, L., Anni, E., Jenna, K., Kupari, H.-M., Filip, G., Veronika, L., Muennighoff, N., Aleksandra, P., et al. (2025) · UTUPub (University of Turku)
Self-citationHesslow, D. (2024) · theses.fr (ABES) · doi:10.70675/f705b608z49b0z490cz88dez9f4819f93f11 · Cited by 2
DOSE: Data Selection for Multi-Modal LLMs via Off-the-Shelf ModelsPreprint
Wu, B., Zhong, Y., Fang, M., & Chen, L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.16979
Chen, Z., Wang, S., Chen, L., Du, W., & Wang, Y. (2026) · Science China Information Sciences · doi:10.1007/s11432-025-4913-5
Predicting Large Model Test Losses with a Noisy Quadratic SystemPreprint
Li, C., & Maddison, C. J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.09154
Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law ExtrapolationPreprint
Kricheli, J. S., Reid, A. L., Sarkar, S., Gandikota, V., & Shakarian, P. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.08541
Understanding and Improving Optimization Algorithms in Deep Learning
Morwani, D. (2026) · Digital Access to Scholarship at Harvard (DASH) (Harvard University)
Where Does the Signal Live? A Web Data Recipe for Medical Encoder Pretraining
Huang, B., Sun, J., Bouchacourt, D., Barascud, N., & Fogel, F. (2026) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/2026.findings-emnlp.273
DOSE: Data Selection for Multi-Modal LLMs via Off-the-Shelf Models
Wu, B., Zhong, Y., Fang, M., & Chen, L. (2026) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2026.findings-acl.45 · Cited by 1
Memming, A. (2026) · Research Square · doi:10.21203/rs.3.rs-10434935/v1
Memming, A. (2026) · Research Square · doi:10.21203/rs.3.rs-10486789/v1
Safety, Alignment, and Robustness of Large Language Models: A Review
Moradi, M. (2026) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2026.086229
Scalable Frequency- and Length-Aware Subdocument Deduplication for Large Language Model PretrainingPreprint
Wang, H., Wang, C., Cai, Q., Liu, Y., Peng, M., Chen, N., Tu, Y., Xu, C., & Zhang, F. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.03089
DMA Hypothesis: An Exploratory Framework for Data-model-task Interactions in Language ModelsPreprint
Wang, D. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7246840
Daedalus-150M: A Convolution-Attention Hybrid Designed for CPU InferencePreprint
Koutsiaris, C. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.20210
Menezes, B. L. S., Cardoso, C. L. S., & Porto, F. A. M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.30114 · Cited by 1
Evaluating Tiny Recursive Models Across Training for Code GenerationPreprint
Sirivella, A., Newaz, A., & Melo, G. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.29376
From Zero to Hero: An Open LLM Ecosystem for ArmenianPreprint
Arakelyan, E., Avetisyan, K., Davtyan, M., Grigoryan, H., Khachatryan, N., Shahsuvaryan, H., Sergoyan, H., & Martirosyan, V. A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.03350
OmniTable: A Unified Wide-Table System for Petabyte-Scale LLM Data Curation and Exploration
Fu, Y., Wang, X., Huang, C., Wang, L., Zeng, B., Wang, Y., Nie, T., Dongke, H., Hong, W., Wang, J., et al. (2026) · Proceedings of the VLDB Endowment · doi:10.14778/3827998.3828032
Emadi, S. M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.14500
Pretraining, Quantization, and Evaluation of Romanian Small Language Models
DIAC, T. A. (2026) · Lund University Publications Student Papers (Lund University)
Lin, L. (2026) · eScholarship (California Digital Library)
Gharbi, W. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7479298
Schiffer, C., Bode, M., Lippert, T., Amunts, K., & Dickscheid, T. (2026) · arXiv (Cornell University) · doi:10.34734/fzj-2026-04629
Improving Generative Model Self-Training with Geometrically Modified OutputsPreprint
Batsell, P., Walker, T., & Baraniuk, R. G. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.35512
In-Context Binding Capacity in Language ModelsPreprint
Ravulapalli, M. V. S., & Chadha, S. S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.30634
Context scaling: A new frontier of AI scaling
Qiu, X. (2026) · AI Plus · doi:10.59717/ipj.aiplus.2026.100010
Does Learning Protein Folding Generalize to Broader Reasoning?Preprint
Liu, Y., Shi, Z., Dang, Y., Zhang, Z., Shi, X., Wei, Z., & Zheng, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.38879
Learning Under Forgetting: Statistical Support-Selective Retention in Stochastic Training DynamicsPreprint
Gao, F., Zhang, Z., & Sun, G. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.38768
Training and Scaling Compute-Optimal Physiological Waveform Foundation ModelsPreprint
Li, P., Peng, J., Luo, S., Plotkin, Z., & Chen, T. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.05649
Selecting Repetition Counts Across Model Scales in Data-Constrained PretrainingPreprint
Ziyue, W. A. N. G., & Kanamori, T. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.05126
Cai, H. (2026) · arXiv (Cornell University)
HAVEN: A unified monocular RGB vision-and-language navigation framework for wheeled-legged robots
Dai, M., Huang, C., Zheng, B., Cheng, Z., Dong, W., & Wu, L. (2026) · Displays · doi:10.1016/j.displa.2026.103724
Neurograf v1: An End-to-End Romanian-First Language Model FamilyPreprint
Rotaru, I. A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.13134
Scaling Laws for Multimodal Data MixturesPreprint
Khandelwal, A., Tarun, A. K., Xu, Y., Schlag, I., Farnadi, G., Reddy, S., Zettlemoyer, L., & Bosselut, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.14423
Ozelim, L., Uthamacumaran, A., & Zenil, H. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.14007
Shen, Y., Liu, Y., Dong, C., Pal, S., Yang, Y., & Shen, J. (2026) · CCF Transactions on Pervasive Computing and Interaction · doi:10.1007/s42486-026-00277-5
Li, S., Pan, L., Sun, S., Zhou, H., & Fan, S. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11729092
Shin, W., & Kim, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.27590
Foundations of the Scaling LawPreprint
Zhao, T., & Ren, Z. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.27789 · Cited by 1
On Multilingual Text Summarization: A Review of Model Architectures, Techniques, and Research Gaps
Muia, M., Oirere, A. M., & Wanjau, S. K. (2026) · International Journal of Data Informatics and Intelligent Computing · doi:10.59461/ijdiic.v5i4.311
Rewriting Scaling Laws for Synthetic Data with Compute and Data ConstraintsPreprint
Roberts, N. P., Ahmed, A., Kotha, S., Liang, P., Wang, Y., Renda, A., & Mani, N. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.31867
Yang, H. (2026) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.23283821
Chaudhary, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.33016
Training with Subword Variants Delays Overfitting in Repeated-Data Language Modeling
Cosma, A., Ruşeţi, Ş., Rǎdoi, E., & Dascălu, M. (2026) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/2026.findings-emnlp.2157
ParaRater: Enhancing Cross-Lingual Transfer in LLMs with Meta-Learning
Han, W., Liu, B., Chen, Z., Zhang, Y., Pechenizkiy, M., Fang, M., & Zheng, Y. (2026) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/2026.emnlp-main.1760
Halved CLM Exposure Mitigates Late-Training Degradation in Small Recurrent Language Models
Maheshwari, Y. (2026) · BabyLM Workshop · doi:10.18653/2026.babylm-main.1 · Cited by 1
Homebrewed SLMs: Autoregressive and Masked Diffusion Language Models Built at Home for PortuguesePreprint
Brom, P. C. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.40533
Training Foundation Models with a Single AlgorithmPreprint
Song, Y., Zhang, A., Tajwar, F., Arora, D., Schneider, J., Khashabi, D., Salakhutdinov, R., Bagnell, J. A., Singh, A., & Zanette, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.40564
Zhang, Y., & Zhang, C. (2026) · Scientometrics · doi:10.48550/arxiv.2610.41552
Sudhi, V., Billi, G., Scrob, A., Stodden, R., & Cimiano, P. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2611.02889
Olivetti, J. F. H., dos Santos, J. H., Gonçalves, M. A., Melo, P. D. F., Reis, J. C. S., Benevenuto, F., Silva, T. H., Borges, K., Guimaraes, M. G., de Souza, G., et al. (2026) · Brazilian Symposium on Multimedia and the Web · doi:10.5753/webmedia.2026.30857
Xin, E. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2611.02529
Metadata
- Authors
- Muennighoff, N., Rush, A., Barak, B., Scao, T. L., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., & Raffel, C.
- ORCID
- Venue
- Advances in Neural Information Processing Systems 36
- Published
- 2023
- Pages
- 50358-50376
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W7133188041
- Cited by
- 47
- References
- 91
- Citations by year
- 2026: 7 · 2025: 21 · 2024: 15 · 2023: 4
- FWCI
- 8.42
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- other-oa
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, other-oa)
- UTUPub (University of Turku) (submittedVersion, other-oa)
- IRIS Research product catalog (Sapienza University of Rome) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2134157)
- U.S. Department of Energy (DESC0022199)
- European Commission (101070350)
- China Scholarship Council
- Defense Advanced Research Projects Agency (W911NF2010021, DE-SC0022199)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- language model scaling, training data efficiency, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.