Scaling Data-Constrained Language Models
Open accessThe DOI matches its Crossref record; 149 of the 149 references it cites were found in a registry or an index; 37 of the 48 sources its claims cite were checked for support.
Abstract
The current trend of scaling language models involves increasing both parameter count and training dataset size. Extrapolating this trend suggests that training dataset size may soon be limited by the amount of text data available on the internet. Motivated by this limit, we investigate scaling language models in data-constrained regimes. Specifically, we run a large set of experiments varying the extent of data repetition and compute budget, ranging up to 900 billion training tokens and 9 billion parameter models. We find that with constrained data for a fixed compute budget, training with up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data. However, with more repetition, the value of adding compute eventually decays to zero. We propose and empirically validate a scaling law for compute optimality that accounts for the decreasing value of repeated tokens and excess parameters. Finally, we experiment with approaches mitigating data scarcity, including augmenting the training dataset with code data or removing commonly used filters. Models and datasets from our 400 training runs are freely available at https://github.com/huggingface/datablations.
Overview
CoChat Summary
TL;DR
The paper studies scaling language models when unique training data is limited, training more than 400 models up to 9B parameters and 900B tokens. It finds that up to 4 epochs of repeated data is nearly as good as unique data, but returns diminish sharply around 16 epochs, and proposes a data-constrained scaling law.
Key findings
- With constrained data and fixed compute, training up to 4 epochs yields negligible loss change versus unique data; an 8.7B model trained for 4 epochs on 44B unique tokens finished with only 0.5% higher validation loss than the single-epoch model trained on 178B unique tokens. More repetition eventually decays to zero, with \(R^*_D \approx 15\) (about 16 epochs) as the learned “half-life.”
- Data-constrained allocation should scale epochs faster than parameters: the fit gives \(R^*_N = 5.309743\) and \(R^*_D = 15.387756\), so excess parameters decay faster than repeated data. A 6.3B data-constrained compute-optimal model with 25B unique tokens achieved better loss and downstream performance than the 8.7B Chinchilla-style model despite having 27% fewer parameters.
- Complementary strategies can extend data: mixing in Python code provides a 2× increase in effective tokens, and filling up to 50% of data with code (42B tokens) showed no deterioration on natural language tasks. Perplexity filtering was effective, while deduplication did not help on C4 but was effective on noisier OSCAR.
- Double descent was observed: on 100M tokens, loss increased at 200 epochs before decreasing again; most such examples were removed from the scaling-law fitting.
Methods
- Trained more than 400 GPT-2 architecture/tokenizer transformer language models from 10M to 9B parameters, for up to 900B total tokens and up to 1500 epochs, on subsets of C4, OSCAR, and Python data from The Stack, with data constraints designed for maximal overlap.
- Used three protocols: fixed unique data (varying epochs and parameters), fixed FLOPs (varying data budget), and parametric fit. The parametric fit generalizes the Chinchilla law by replacing data and parameter terms with effective data and effective parameters via exponential decay constants \(R^*_D\) and \(R^*_N\), fitted on 182 samples.
- Evaluated held-out test loss (210M C4 validation tokens) and 19 downstream natural language tasks with 0–5 few-shot exemplars, normalizing scores; training used up to 256 AMD Instinct MI250X GPUs on LUMI.
Limitations
- The paper states it focuses on repeating the entire unique dataset, not fractions; returns from additional epochs may depend heavily on hyperparameters such as learning rate, dropout, or optimizer; findings are dataset-dependent (C4 and OSCAR) and limited to text data and GPT-style architectures; other data strategies and modalities are left to future work.
- Evident from the text: the fitting removed double-descent and outlier samples where excess epochs or parameters hurt, and the alternative alpha-beta decay formulation lacked rigorous mathematical justification and mispredicted returns. The downstream evaluation is described as not comprehensive, and the scaling laws fitted on C4 were applied to Galactica’s predominantly scientific dataset.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
29 Valid · 2 Not valid · 8 Not checked · 138 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“however, most prior large language models have been trained for a single epoch (Komatsuzaki, 2019;Brown et al., 2020)”
§ Scaling Data-Constrained Language Models > 1. Introduction
What the sources say
- Disputedcites Komatsuzaki, 2019 sourceExperimental — accuracy not yet measured
The Introduction (p. 1) describes multi-epoch training as the prevailing practice, not single-epoch training. Table 1 (p. 2) lists GPT at 100 epochs and BERT at 40; it does not report a verified epoch count for GPT-2. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
- Could not verifycites Brown et al., 2020 sourceExperimental — accuracy not yet measured
Section 2.2 (p. 8) describes how often Brown et al. sampled their own training datasets, and Appendix B (p. 43) describes their training procedure. Neither passage says whether most prior large language models were trained for a single epoch; searches for “single epoch” and equivalent descriptions found no such statement in our copy. (LLM, conf=0.73)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Villalobos et al. (2022) estimate that even high-quality English language data will be exhausted by the year 2024 given the Chinchilla scaling laws and the trend of training ever-larger models.”
§ Scaling Data-Constrained Language Models > 1. Introduction
What the sources say
- Partly supportedcites Villalobos et al., 2022 sourceExperimental — accuracy not yet measured
The Introduction (p. 1) links growing training-data demand to scaling laws, and Section 2.5 (p. 6) projects a data bottleneck if dataset growth continues. But our copy does not estimate exhaustion of high-quality English data by 2024; it gives a median exhaustion year of 2028 for public human text data, with a projected range of 2026–2032 in the Conclusion (p. 9). (LLM, conf=0.86)
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“However, the experiments of Galactica do not compare this setup to an alternative non-data-constrained model trained for one epoch on unique data.”
§ Scaling Data-Constrained Language Models > 1. Introduction
What the sources say
- Supportedcites Taylor et al., 2022 source
Section 5.1 (pp. 11–12) describes training Galactica for approximately 4.25 epochs and evaluates performance across repeated passes over its corpus. Section 5.5 (p. 19) compares Galactica with other language models, but does not describe the proposed one-epoch, unique-data alternative. The claim accurately identifies a comparison absent from the reported experiments. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Results from (Kaplan et al., 2020) also show that loss can increase when too many parameters are used, even with early stopping.”
§ Scaling Data-Constrained Language Models > 5. Results: Resource Allocation for Data-Constrained Scaling
What the sources say
- Partly supportedcites Kaplan et al., 2020Experimental — accuracy not yet measured
Figure 9 in Section 4.1 (p. 10) says that, for a smaller fixed dataset, early-stopped test performance stops improving as parameter count increases and the model begins to overfit. This supports the early-stopping and overfitting parts of the claim, but it does not show that loss increases with additional parameters. (LLM, conf=0.86)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“For 100M tokens, the corresponding one-epoch compute-optimal model according to scaling laws from (Hoffmann et al., 2022) has U N of approximately 7M parameters (see §C for the scaling coefficients we use).”
§ Scaling Data-Constrained Language Models > 5. Results: Resource Allocation for Data-Constrained Scaling
What the sources say
- Partly supportedcites Hoffmann et al., 2022 sourceExperimental — accuracy not yet measured
Section 3.4 (p. 8) supports using Hoffmann et al.’s scaling laws to relate optimal model size to training tokens. It does not give a 100M-token, one-epoch estimate: its Table 3 pairs 400M parameters with 8.0B tokens, which extrapolates to roughly 5M—not 7M—parameters at 100M tokens. That extrapolation is also below the model sizes used to fit the laws (Section 3, p. 4). (LLM, conf=0.82)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
89 more claims · 7 with their cited sources checked
Create a free account to see all 177 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 49
- It may state two separate claims as one. 45
- The quote only partly states this claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“They found that up to roughly four epochs, repetition is nearly as effective as new data, after which returns decay quickly”
From Training Compute-Optimal Transformer Models: A Review of Scaling Laws, Allocation, and Practice (2026) · cites this paper as Muennighoff et al., 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Growing concerns suggest that LLM pretraining may be approaching a ceiling as available high-quality data sources near exhaustion [58].”
From A Survey of Large Language Models (2026) · cites this paper as [58]
The cited source says what the claim attributes to it.
In the citing paper
“However, these studies find that repeating the same corpus only partially compensates for data scarcity; as the number of epochs increases, marginal gains diminish, and the risks of memorization and overfitting rise.”
From Scaling Data-Constrained Language Models with Synthetic Data (2026) · cites this paper as Muennighoff et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“The authors estimate the maximum increase in the effective dataset size that can be gained from multiple epochs at between 3x and 15x, and we anchor to this estimate in adjusting our model.”
From Will we run out of data? Limits of LLM scaling based on human-generated data (2022) · cites this paper as Muennighoff et al., 2023
In the citing paper
“Our theory will only rely upon the general form of the equations, specifically, that the dependence is inverse polynomial in N, D. So it applies to other frameworks of training (e.g., overtrained models Muennighoff et al. [2023]) where scaling laws have also been found.”
From A Theory for Emergence of Complex Skills in Language Models (2023) · cites this paper as Muennighoff et al. [2023]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“We find the best loss to be at around 20-60× more parameters and epochs, which corresponds to spending around 7000× more FLOPs.”
§ Scaling Data-Constrained Language Models > 5. Results: Resource Allocation for Data-Constrained Scaling
Reported 7000 · recomputed 3600
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Scaling Data-Constrained Language Models > Training data
“Like the equations in §B this formulation also reduces to the Chinchilla scaling laws in the base case of R D = 0 or R N = 0.”
§ Scaling Data-Constrained Language Models > Training data
“L(N, D, R N , R D ) = E + A N α * max(0,1-(R N /R * N )) + B D β * max(0,1-(R D /R * D )) (7”
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Self-citationUnsupervised Cross-lingual Representation Learning at Scale
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F. (., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2019) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.1911.02116 · Cited by 9,924
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550
Unsupervised Cross-lingual Representation Learning at Scale
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.747 · Cited by 5,637
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Research works printed but not linked: 9
- Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference on machine learning, pages 1691-1703. PMLR, 2020.
- Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, pages 5547-5569. PMLR, 2022.
- Kenneth Heafield. KenLM: Faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 187-197, Edinburgh, Scotland, July 2011. Association for Computational Linguistics. URL https://aclanthology.org/W11-2123.
- Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
- Pedro Javier Ortiz Su'arez, Laurent Romary, and Benoit Sagot. A monolingual approach to contextualized word embeddings for mid-resource languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1703-1714, Online, July 2020. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/2020.acl-main.156.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
- Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI spring symposium: logical formalizations of commonsense reasoning, pages 90-95, 2011.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
- Ge Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neural networks via zero-shot hyperparameter transfer. Advances in Neural Information Processing Systems, 34:17084-17097, 2021.
Cited by
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.12948 · Cited by 18,918
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093
The Falcon Series of Open Language Models
Almazrouei, E., Alobeidli, H., Alshamsi, A., Cappelli, A., Cojocaru, R., Debbah, M. A., Goffinet, É., Hesslow, D., Launay, J., Malartic, Q., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.16867 · Cited by 113
Olmo 3Preprint
Team OLMo, :, Ettinger, A., Bertsch, A., Kuehl, B., Graham, D., Heineman, D., Groeneveld, D., Brahman, F., Timbers, F., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.13961 · Cited by 609
C-Pack: Packed Resources For General Chinese Embeddings
Xiao, S., Liu, Z., Zhang, P., Muennighoff, N., Lian, D., & Nie, J.-Y. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3657878 · Cited by 523
Self-citationRWKV: Reinventing RNNs for the Transformer Era
Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Derczynski, L., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.936 · Cited by 390
Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model
Üstün, A., Aryabumi, V., Yong, Z., Ko, W.-Y., D’souza, D., Onilude, G., Bhandari, N., Singh, S., Ooi, H.-L., Kayid, A., et al. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.845 · Cited by 576
Self-citationA large-scale audit of dataset licensing and attribution in AI
Longpre, S., Mahari, R., Chen, A., Obeng-Marnu, N., Sileo, D., Brannon, W., Muennighoff, N., Khazam, N., Kabbara, J., Perisetla, K., et al. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00878-8 · Cited by 86
Self-citationOctoPack: Instruction Tuning Code Large Language Models
Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., & Longpre, S. (2023) · arXiv · doi:10.48550/arxiv.2308.07124 · Cited by 245
Self-citationWhen Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method
Zhang, B., Liu, Z., Cherry, C., & Fırat, O. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2402.17193 · Cited by 251
DataComp-LM: In search of the next generation of training sets for language models
Li, J. J.-P., Fang, A., Smyrnis, G., Ivgi, M., Jordan, M., Gadre, S. Y., Bansal, H., Guha, E., Keh, S. S., Arora, K., et al. (2024) · neural information processing systems · doi:10.52202/079017-0455 · Cited by 629
Self-citationFields, J., Chovanec, K., & Madiraju, P. (2024) · IEEE Access · doi:10.1109/access.2024.3349952 · Cited by 246
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A. A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.14219 · Cited by 166
Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning
Singh, S., Vargus, F., D’souza, D., Karlsson, B. F., Mahendiran, A., Ko, W.-Y., Shandilya, H., Patel, J., Mataciunas, D., O'Mahony, L., et al. (2024) · VBN Forskningsportal (Aalborg Universitet) · doi:10.48550/arxiv.2402.06619 · Cited by 180
Self-citationHow to Train Data-Efficient LLMsPreprint
Sachdeva, N., Coleman, B., Kang, W.-C., Ni, J., Hong, L., Ed, H., Caverlee, J., McAuley, J., & Cheng, D. Z. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.09668 · Cited by 161
A Theory for Emergence of Complex Skills in Language Models
Arora, S., & Goyal, A. (2023) · arXiv · doi:10.48550/arxiv.2307.15936 · Cited by 129
Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling
Maini, P., Seto, S., Bai, R., Grangier, D., Zhang, Y., & Jaitly, N. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.757 · Cited by 203
Consent in Crisis: The Rapid Decline of the AI Data Commons
Longpre, S., Mahari, R., Lee, A., Lund, C., Oderinwale, H., Brannon, W., Saxena, N., Obeng-Marnu, N., South, T., Hunter, C., et al. (2024) · neural information processing systems · doi:10.52202/079017-3431 · Cited by 200
Self-citationMultiLegalPile: A 689GB Multilingual Legal Corpus
Niklaus, J., Matoshi, V., Stuermer, M., Chalkidis, I., & Ho, D. E. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.805 · Cited by 134
Scalable Extraction of Training Data from (Production) Language Models
Nasr, M., Carlini, N., Hayase, J., Jagielski, M., Cooper, A. F., Ippolito, D., Choquette-Choo, C. A., Wallace, E., Tramèr, F., & Lee, K. (2023) · arXiv · doi:10.48550/arxiv.2311.17035 · Cited by 96
Diffusion Language Models Are Versatile Protein LearnersPreprint
Wang, X., Zheng, Z., Ye, F., Xue, D., Huang, S., & Gu, Q. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.18567 · Cited by 111
ICE-Score: Instructing Large Language Models to Evaluate Code
Zhuo, T. Y. (2024) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2024.findings-eacl.148 · Cited by 149
Will we run out of data? Limits of LLM scaling based on human-generated data
Villalobos, P., Ho, A., Sevilla, J., Besiroglu, T., Heim, L., & Hobbhahn, M. (2022) · arXiv · doi:10.48550/arxiv.2211.04325 · Cited by 80
A Dynamical Model of Neural Scaling LawsPreprint
Bordelon, B., Atanasov, A., & Pehlevan, C. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.01092 · Cited by 141
Salamandra Technical ReportPreprint
González-Agirre, A., Pàmies, M., Palao, J. L., Baucells, I., Dalt, S. D., Tamayo, D. C. L., Saiz, J. J., Espuña, F., Prats, J., Aula-Blasco, J., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.08489 · Cited by 100
Javaid, S., Khalil, R. A., Saeed, N., He, B., & Alouini, M. (2024) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2024.3522103 · Cited by 92
OLMo: Accelerating the Science of Language Models
Groeneveld, D., Beltagy, I., Walsh, E., Bhagia, A., Kinney, R., Tafjord, O., Jha, A., Ivison, H., Magnusson, I., Wang, Y., et al. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.841 · Cited by 70
Self-citationAligning Large Language Models with Human: A Survey
Wang, Y., Zhong, W., Li, L., Mi, F., Zeng, X., Huang, W., Shang, L., Jiang, X., & Liu, Q. (2023) · arXiv · doi:10.48550/arxiv.2307.12966 · Cited by 69
Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning
Singh, S., Vargus, F., D’souza, D., Karlsson, B. F., Mahendiran, A., Ko, W.-Y., Shandilya, H., Patel, J., Mataciunas, D., O'Mahony, L., et al. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.620 · Cited by 247
Self-citationSILO Language Models: Isolating Legal Risk In a Nonparametric Datastore
Min, S., Gururangan, S., Wallace, E., Shi, W., Hajishirzi, H., Smith, N. A., & Zettlemoyer, L. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2308.04430 · Cited by 89
Data Management For Training Large Language Models: A SurveyPreprint
Wang, Z., Zhong, W., Wang, Y., Zhu, Q., Mi, F., Wang, B., Shang, L., Jiang, X., & Liu, Q. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.01700 · Cited by 85
Chronologically Consistent Large Language ModelsPreprint
He, S., Lv, L., Manela, A., & Wu, J. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5159615 · Cited by 106
Busbridge, D., Shidani, A., Weers, F., Ramapuram, J., Littwin, E., & Webb, R. (2025) · arXiv · doi:10.48550/arxiv.2502.08606
The Efficiency Spectrum of Large Language Models: An Algorithmic SurveyPreprint
Ding, T., Chen, T., Zhu, H., Jiang, J., Zhong, Y., Zhou, J., Wang, G., Zhu, Z., Zharkov, I., & Liang, L. (2023) · arXiv (Cornell University) · doi:10.13140/rg.2.2.32142.54087/1 · Cited by 72
Dolma: an open corpus of three trillion tokens for language model pretraining research
Soldaini, L., Kinney, R., Bhagia, A., Schwenk, D., Atkinson, D., Authur, R., Bogin, B., Chandu, K., Dumas, J., Elazar, Y., et al. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.840 · Cited by 56
Self-citationScaling Law of Large Sequential Recommendation Models
Zhang, G., Hou, Y., Lu, H., Chen, Y., Zhao, W. X., & Wen, J.-R. (2024) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3640457.3688129 · Cited by 75
Astraios: Parameter-Efficient Instruction Tuning Code Large Language ModelsPreprint
Zhuo, T. Y., Zebaze, A. R., Suppattarachai, N., von Werra, L., de Vries, H., Liu, Q., & Muennighoff, N. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.00788 · Cited by 64
Self-citationFinGPT: Large Generative Models for a Small Language
Luukkonen, R., Komulainen, V., Luoma, J., Eskelinen, A., Kanerva, J., Kupari, H.-M., Ginter, F., Laippala, V., Muennighoff, N., Piktus, A., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.164 · Cited by 111
Self-citationAsymptotic theory of in-context learning by linear attention
Lu, Y. M., Letey, M. I., Zavatone-Veth, J. A., Maiti, A., & Pehlevan, C. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2502599122 · Cited by 80
Empirical Guidelines for Deploying LLMs onto Resource-constrained Edge Devices
Qin, R., Liu, D., Xu, C., Yan, Z., Tan, Z., Jia, Z., Nassereldine, A., Li, J., Jiang, M., Abbasi, A., et al. (2025) · ACM Transactions on Design Automation of Electronic Systems · doi:10.1145/3736721 · Cited by 62
Overtrained Language Models Are Harder to Fine-TunePreprint
Springer, J. M., Goyal, S., Wen, K., Kumar, T., Yue, X., Malladi, S., Neubig, G., & Raghunathan, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.19206 · Cited by 87
How Does Critical Batch Size Scale in Pre-training?
Zhang, H., Morwani, D., Vyas, N., Wu, J., Zou, D., Ghai, U., Foster, D. P., & Kakade, S. M. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2410.21676 · Cited by 49
Scaling Laws for Data Filtering—Data Curation Cannot be Compute Agnostic
Goyal, S., Maini, P., Lipton, Z. C., Raghunathan, A., & Kolter, J. Z. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02142 · Cited by 119
Tiny Aya: Bridging Scale and Multilingual DepthPreprint
Salamanca, A., Abagyan, D., D’souza, D., Khairi, A., Mora, D., Dash, S., Aryabumi, V., Rajaee, S., Mofakhami, M., Sahu, A., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.11510 · Cited by 55
DeepGate3: Towards Scalable Circuit Representation Learning
Shi, Z., Zheng, Z., Khan, S., Zhong, J., Li, M., & Xu, Q. (2024) · International Conference on Computer Aided Design · doi:10.1145/3676536.3676791 · Cited by 50
The 2025 Foundation Model Transparency Index
Wan, A., Klyman, K., Kapoor, S., Maslej, N., Longpre, S., Xiong, B., Liang, P., & Bommasani, R. (2026) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.70777/si.v2i4.17165 · Cited by 42
How feature learning can improve neural scaling laws *
Bordelon, B., Atanasov, A., & Pehlevan, C. (2025) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/adefb1 · Cited by 56
Tucano: Advancing neural text generation for Portuguese
Corrêa, N. K., Sen, A., Falk, S., & Fatimah, S. (2025) · Patterns · doi:10.1016/j.patter.2025.101325 · Cited by 50
An Overview of Large Language Models for Statisticians
Ji, W., Yuan, W., Getzen, E. J., Cho, K., Jordan, M. I., Mei, S., Weston, J., Su, W., Xu, J., & Zhang, L. (2026) · The American Statistician · doi:10.1080/00031305.2026.2657480 · Cited by 51
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?
Tay, Y., Dehghani, M., Abnar, S., Chung, H., Fedus, W., Rao, J., Narang, S., Tran, V., Yogatama, D., & Metzler, D. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.825 · Cited by 34
Scaling Laws in Linear Regression: Compute, Parameters, and Data
Lin, L., Wu, J., Kakade, S. M., Bartlett, P. L., & Lee, J. D. (2024) · neural information processing systems · doi:10.52202/079017-1937 · Cited by 97
A Preliminary Study of the Intrinsic Relationship between Complexity and AlignmentPreprint
Zhao, Y., Yu, B., Hui, B., Yu, H., Huang, F., Li, Y., & Zhang, N. L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.05696 · Cited by 42
BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale PretrainingPreprint
DatologyAI, :, Maini, P., Dorna, V., Doshi, P., Carranza, A., Pan, F., Urbanek, J., Burstein, P., Fang, A., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.10975 · Cited by 38
The AI revolution is running out of data. What can researchers do?
Jones, N. (2024) · Nature · doi:10.1038/d41586-024-03990-2 · Cited by 51
Ziya2: Data-centric Learning is All LLMs NeedPreprint
Gan, R., Wu, Z., Sun, R., Lu, J., Wu, X., Zhang, D., Pan, K., He, J., Tian, Y., Yang, P., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.03301 · Cited by 34
Adversarial Robustness Limits via Scaling-Law and Human-Alignment StudiesPreprint
Bartoldson, B. R., Diffenderfer, J., Parasyris, K., & Kailkhura, B. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.09349 · Cited by 67
TeenyTinyLlama: Open-source tiny language models trained in Brazilian Portuguese
Corrêa, N. K., Falk, S., Fatimah, S., Sen, A., & Oliveira, N. F. D. (2024) · Machine Learning with Applications · doi:10.1016/j.mlwa.2024.100558 · Cited by 41
TURNA: A Turkish Encoder-Decoder Language Model for Enhanced Understanding and Generation
Uludoğan, G., Yirmibeşoğlu, Z., Akkurt, F., Turker, M., Güngör, O., & Üsküdarlı, S. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.600 · Cited by 38
Training Compute-Optimal Protein Language ModelsPreprint
Cheng, X., Chen, B., Li, P., Gong, J., Tang, J., & Song, L. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.06.06.597716 · Cited by 32
Scaling Laws of Motion Forecasting and Planning -- Technical ReportPreprint
Baniodeh, M., Goel, K., Ettinger, S. M., Fuertes, C., Seff, A., Shen, T., Gulino, C., Yang, C., Jerfel, G., Choe, D., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.08228 · Cited by 30
Scaling Laws for Sparsely-Connected Foundation Models
Frantar, E., Riquelme, C., Houlsby, N., Alistarh, D., & Evci, U. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2309.08520 · Cited by 37
The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resources
Longpre, S., Biderman, S., Albalak, A., Schoelkopf, H., McDuff, D., Kapoor, S., Klyman, K., Lo, K., Ilharco, G., San, N., et al. (2024) · Transactions on machine learning research · doi:10.48550/arxiv.2406.16746 · Cited by 37
Linguistic Collapse: Neural Collapse in (Large) Language Models
Wu, R., & Papyan, V. (2024) · neural information processing systems · doi:10.52202/079017-4366 · Cited by 56
Scaling laws for learning with real and surrogate data
Jain, A., Andrea Montanari, A., & Şaşoğlu, E. (2024) · neural information processing systems · doi:10.52202/079017-3499 · Cited by 44
Toward Inference-optimal Mixture-of-Expert Large Language ModelsPreprint
Yun, L., Zhuang, Y., Fu, Y., Xing, E. P., & Zhang, H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.02852 · Cited by 29
Sloth: scaling laws for LLM skills to predict multi-benchmark performance across families
Polo, F. M., Somerstep, S., Choshen, L., Sun, Y., & Yurochkin, M. (2025) · neural information processing systems · doi:10.52202/085713-1318 · Cited by 32
Demystifying Data Management for Large Language Models
Miao, X., Jia, Z., & Cui, B. (2024) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3626246.3654683 · Cited by 33
Efficient Training and Inference: Techniques for Large Language Models Using LlamaPreprint
Cunningham, S. R., Archambault, D., & Kung, A. (2024) · doi:10.36227/techrxiv.171651876.65094225/v1 · Cited by 32
Scaling Dense Representations for Single Cell with Transcriptome-Scale ContextPreprint
Ho, N., Ellington, C. N., Hou, J., Addagudi, S., Mo, S., Tao, T., Li, D., Zhuang, Y., Wang, H., Cheng, X., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.11.28.625303 · Cited by 25
Siudek, M., Smith, M. J., Martínez-Solaeche, G., Lanusse, F., Ho, S., Angeloudi, E., Cunha, P. A. C., Sánchez, H. D., Dunn, M., Fu, Y., et al. (2025) · Astronomy and Astrophysics · doi:10.1051/0004-6361/202554611 · Cited by 22
Too Long, Do Re-weighting for Efficient LLM Reasoning Compression
Li, Z.-Z., Liang, X., Tang, Z., Ji, L., Wang, P., Xu, H., Xing, W., Huang, H., Deng, W., Gong, Y., et al. (2026) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2026.acl-long.1856 · Cited by 22
Tan, X., Yao, T., Qu, C., Li, B., Yang, M., Lu, D., Wang, H., Xu, Y., & Qiu, X. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770854.3780168 · Cited by 26
Position: The Most Expensive Part of an LLM should be its Training DataPreprint
Kandpal, N., & Raffel, C. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.12427 · Cited by 29
Self-citationQin, R., Yang, K., Abbasi, A., Dobolyi, D. G., Seyedi, S. S., Griner, E. L., Kwon, H., Cotes, R. O., Jiang, Z., Clifford, G. D., et al. (2025) · ACM Transactions on Management Information Systems · doi:10.1145/3673906 · Cited by 33
Predicting Emergent Abilities with Infinite Resolution Evaluation
Hu, S., Liu, X., Han, X., Zhang, X., He, C., Zhao, W., Lin, Y., Ding, N., Ou, Z., Zeng, G., et al. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2310.03262 · Cited by 31
Foundation Models for Discovery and Exploration in Chemical SpacePreprint
Wadell, A., Bhutani, A., Azumah, V., Ellis-Mohr, A. R., Stier, A. J., Hegazy, K., Brace, A., Zhao, H., Kelly, C., Nayak, A. K., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.18900 · Cited by 18
RWKV: Reinventing RNNs for the Transformer Era
Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Grella, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.13048 · Cited by 17
Physics of language models: Part 3.3, knowledge capacity scaling laws
Allen-Zhu, Z., & Li, Y. (2025) · doi:10.2139/ssrn.5250617 · Cited by 17
AlGhafa Evaluation Benchmark for Arabic Language Models
Almazrouei, E., Cojocaru, R., Baldo, M., Malartic, Q., Alobeidli, H., Mazzotta, D., Penedo, G., Campesan, G., Farooq, M., Alhammadi, M., et al. (2023) · Arabic Natural Language Processing Conference (ArabicNLP) · doi:10.18653/v1/2023.arabicnlp-1.21 · Cited by 61
Lucy, L., Gururangan, S., Soldaini, L., Strubell, E., Bamman, D., Klein, L. F., & Dodge, J. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.400 · Cited by 34
CITING: Large Language Models Create Curriculum for Instruction TuningPreprint
Feng, T., Wang, Z., & Sun, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.02527 · Cited by 19
Small Language Models for Application Interactions: A Case StudyPreprint
Li, B., Zhang, Y., Bubeck, S., Pathuri, J., & Menache, I. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.20347 · Cited by 20
Improved Scaling Laws in Linear Regression via Data Reuse
Lin, L., Wu, J., & Bartlett, P. L. (2025) · neural information processing systems · doi:10.52202/085713-3034 · Cited by 23
Neural Scaling Laws for Boosted Jet TaggingPreprint
Vigl, M., Hartman, N. M., Kagan, M., & Heinrich, L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.15781 · Cited by 22
Shaping capabilities with token-level data filteringPreprint
Rathi, N., & Radford, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.21571 · Cited by 20
Olmix: A Framework for Data Mixing Throughout LM DevelopmentPreprint
Chen, M. F., Murray, T., Heineman, D., Jordan, M., Hajishirzi, H., Ré, C., Soldaini, L., & Lo, K. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.12237 · Cited by 20
Song, M. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104368 · Cited by 16
Multi-view fusion for instruction mining of large language model
Huang, H., Xu, B., Liang, X., Chen, K., Yang, M., Zhao, T., & Zhu, C. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102480 · Cited by 21
Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling
Grangier, D., Fan, S., Seto, S., & Ablin, P. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2410.03735 · Cited by 15
Kimi k1.5: Scaling Reinforcement Learning with LLMs
Kimi Team, Du, A., Gao, B., Xing, B., Jiang, C., Chen, C., Li, C., Xiao, C., Du, C., Liao, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.12599 · Cited by 12
Scaling Data Diversity for Fine-Tuning Language Models in Human Alignment
Song, F., Yu, B., Lang, H., Yu, H., Huang, F., Wang, H., & Li, Y. (2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2do58nv6r7wc · Cited by 36
Diverse AI Feedback For Large Language Model Alignment
Yu, T., Lin, T.-E., Wu, Y., Yang, M., Huang, F., & Li, Y. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00746 · Cited by 21
The Optimization Landscape of SGD Across the Feature Learning StrengthPreprint
Atanasov, A., Meterez, A., Simon, J. B., & Pehlevan, C. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.04642 · Cited by 15
Towards Foundational Models for Single-Chip Radar
Huang, T., Prabhakara, A., Chen, C., Karhade, J., Ramanan, D., O’Toole, M., & Rowe, A. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02286 · Cited by 22
On the origin of neural scaling laws: from random graphs to natural languagePreprint
Barkeshli, M., Alfarano, A., & Gromov, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.10684 · Cited by 13
Neural scaling laws from large-N field theory: solvable model beyond the ridgeless limit
Zhang, Z. (2025) · Machine Learning Science and Technology · doi:10.1088/2632-2153/adc872 · Cited by 11
SambaLingo: Teaching Large Language Models New Languages
Csaki, Z., Li, B., Li, J. L., Xu, Q., Pawakapan, P., Zhang, L., Du, Y., Zhao, H., Hu, C., & Thakker, U. (2024) · Workshop on Multilingual Representation Learning (MRL) · doi:10.18653/v1/2024.mrl-1.1 · Cited by 26
Bigger is not always better: The importance of human-scale language modeling for psycholinguistics
Wilcox, E. G., Hu, M. Y., Mueller, A., Warstadt, A., Choshen, L., Zhuang, C., Williams, A., Cotterell, R., & Linzen, T. (2025) · Journal of Memory and Language · doi:10.1016/j.jml.2025.104650 · Cited by 9
Breaking Language Barriers: Cross-Lingual Continual Pre-Training at Scale
Zheng, W., Pan, W., Xu, X., Qin, L., Li, Y., & Zhou, M. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.441 · Cited by 27
How to Upscale Neural Networks with Scaling Law? A Survey and Practical Guidelines
Sengupta, A., Goel, Y., & Chakraborty, T. (2025) · Transactions on machine learning research · doi:10.48550/arxiv.2502.12051 · Cited by 10
Metadata
- Authors
- Muennighoff, N., Rush, A., Barak, B., Scao, T. L., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., & Raffel, C.
- ORCID
- Venue
- Advances in Neural Information Processing Systems 36
- Published
- 2023
- Pages
- 50358-50376
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W7133188041
- Cited by
- 47
- References
- 91
- Citations by year
- 2026: 7 · 2025: 21 · 2024: 15 · 2023: 4
- FWCI
- 8.42
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- other-oa
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, other-oa)
- UTUPub (University of Turku) (submittedVersion, other-oa)
- IRIS Research product catalog (Sapienza University of Rome) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2134157)
- U.S. Department of Energy (DESC0022199)
- European Commission (101070350)
- China Scholarship Council
- Defense Advanced Research Projects Agency (W911NF2010021, DE-SC0022199)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- language model scaling, training data efficiency, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.