The DOI matches its Crossref record; 149 of the 149 references it cites were found in a registry or an index; 37 of the 48 sources its claims cite were checked for support.
Scaling Data-Constrained Language Models
Open accessAbstract
The current trend of scaling language models involves increasing both parameter count and training dataset size. Extrapolating this trend suggests that training dataset size may soon be limited by the amount of text data available on the internet. Motivated by this limit, we investigate scaling language models in data-constrained regimes. Specifically, we run a large set of experiments varying the extent of data repetition and compute budget, ranging up to 900 billion training tokens and 9 billion parameter models. We find that with constrained data for a fixed compute budget, training with up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data. However, with more repetition, the value of adding compute eventually decays to zero. We propose and empirically validate a scaling law for compute optimality that accounts for the decreasing value of repeated tokens and excess parameters. Finally, we experiment with approaches mitigating data scarcity, including augmenting the training dataset with code data or removing commonly used filters. Models and datasets from our 400 training runs are freely available at https://github.com/huggingface/datablations.
Overview
CoChat Summary
TL;DR
The paper studies scaling language models when unique training data is limited, training more than 400 models up to 9B parameters and 900B tokens. It finds that up to 4 epochs of repeated data is nearly as good as unique data, but returns diminish sharply around 16 epochs, and proposes a data-constrained scaling law.
Key findings
- With constrained data and fixed compute, training up to 4 epochs yields negligible loss change versus unique data; an 8.7B model trained for 4 epochs on 44B unique tokens finished with only 0.5% higher validation loss than the single-epoch model trained on 178B unique tokens. More repetition eventually decays to zero, with \(R^*_D \approx 15\) (about 16 epochs) as the learned “half-life.”
- Data-constrained allocation should scale epochs faster than parameters: the fit gives \(R^*_N = 5.309743\) and \(R^*_D = 15.387756\), so excess parameters decay faster than repeated data. A 6.3B data-constrained compute-optimal model with 25B unique tokens achieved better loss and downstream performance than the 8.7B Chinchilla-style model despite having 27% fewer parameters.
- Complementary strategies can extend data: mixing in Python code provides a 2× increase in effective tokens, and filling up to 50% of data with code (42B tokens) showed no deterioration on natural language tasks. Perplexity filtering was effective, while deduplication did not help on C4 but was effective on noisier OSCAR.
- Double descent was observed: on 100M tokens, loss increased at 200 epochs before decreasing again; most such examples were removed from the scaling-law fitting.
Methods
- Trained more than 400 GPT-2 architecture/tokenizer transformer language models from 10M to 9B parameters, for up to 900B total tokens and up to 1500 epochs, on subsets of C4, OSCAR, and Python data from The Stack, with data constraints designed for maximal overlap.
- Used three protocols: fixed unique data (varying epochs and parameters), fixed FLOPs (varying data budget), and parametric fit. The parametric fit generalizes the Chinchilla law by replacing data and parameter terms with effective data and effective parameters via exponential decay constants \(R^*_D\) and \(R^*_N\), fitted on 182 samples.
- Evaluated held-out test loss (210M C4 validation tokens) and 19 downstream natural language tasks with 0–5 few-shot exemplars, normalizing scores; training used up to 256 AMD Instinct MI250X GPUs on LUMI.
Limitations
- The paper states it focuses on repeating the entire unique dataset, not fractions; returns from additional epochs may depend heavily on hyperparameters such as learning rate, dropout, or optimizer; findings are dataset-dependent (C4 and OSCAR) and limited to text data and GPT-style architectures; other data strategies and modalities are left to future work.
- Evident from the text: the fitting removed double-descent and outlier samples where excess epochs or parameters hurt, and the alternative alpha-beta decay formulation lacked rigorous mathematical justification and mispredicted returns. The downstream evaluation is described as not comprehensive, and the scaling laws fitted on C4 were applied to Galactica’s predominantly scientific dataset.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
29 Valid · 2 Not valid · 8 Not checked · 138 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“however, most prior large language models have been trained for a single epoch (Komatsuzaki, 2019;Brown et al., 2020)”
§ Scaling Data-Constrained Language Models > 1. Introduction
What the sources say
- Disputedcites Komatsuzaki, 2019 sourceExperimental — accuracy not yet measured
The Introduction (p. 1) describes multi-epoch training as the prevailing practice, not single-epoch training. Table 1 (p. 2) lists GPT at 100 epochs and BERT at 40; it does not report a verified epoch count for GPT-2. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
- Could not verifycites Brown et al., 2020 sourceExperimental — accuracy not yet measured
Section 2.2 (p. 8) describes how often Brown et al. sampled their own training datasets, and Appendix B (p. 43) describes their training procedure. Neither passage says whether most prior large language models were trained for a single epoch; searches for “single epoch” and equivalent descriptions found no such statement in our copy. (LLM, conf=0.73)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Villalobos et al. (2022) estimate that even high-quality English language data will be exhausted by the year 2024 given the Chinchilla scaling laws and the trend of training ever-larger models.”
§ Scaling Data-Constrained Language Models > 1. Introduction
What the sources say
- Partly supportedcites Villalobos et al., 2022 sourceExperimental — accuracy not yet measured
The Introduction (p. 1) links growing training-data demand to scaling laws, and Section 2.5 (p. 6) projects a data bottleneck if dataset growth continues. But our copy does not estimate exhaustion of high-quality English data by 2024; it gives a median exhaustion year of 2028 for public human text data, with a projected range of 2026–2032 in the Conclusion (p. 9). (LLM, conf=0.86)
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“However, the experiments of Galactica do not compare this setup to an alternative non-data-constrained model trained for one epoch on unique data.”
§ Scaling Data-Constrained Language Models > 1. Introduction
What the sources say
- Supportedcites Taylor et al., 2022 source
Section 5.1 (pp. 11–12) describes training Galactica for approximately 4.25 epochs and evaluates performance across repeated passes over its corpus. Section 5.5 (p. 19) compares Galactica with other language models, but does not describe the proposed one-epoch, unique-data alternative. The claim accurately identifies a comparison absent from the reported experiments. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Results from (Kaplan et al., 2020) also show that loss can increase when too many parameters are used, even with early stopping.”
§ Scaling Data-Constrained Language Models > 5. Results: Resource Allocation for Data-Constrained Scaling
What the sources say
- Partly supportedcites Kaplan et al., 2020Experimental — accuracy not yet measured
Figure 9 in Section 4.1 (p. 10) says that, for a smaller fixed dataset, early-stopped test performance stops improving as parameter count increases and the model begins to overfit. This supports the early-stopping and overfitting parts of the claim, but it does not show that loss increases with additional parameters. (LLM, conf=0.86)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“For 100M tokens, the corresponding one-epoch compute-optimal model according to scaling laws from (Hoffmann et al., 2022) has U N of approximately 7M parameters (see §C for the scaling coefficients we use).”
§ Scaling Data-Constrained Language Models > 5. Results: Resource Allocation for Data-Constrained Scaling
What the sources say
- Partly supportedcites Hoffmann et al., 2022 sourceExperimental — accuracy not yet measured
Section 3.4 (p. 8) supports using Hoffmann et al.’s scaling laws to relate optimal model size to training tokens. It does not give a 100M-token, one-epoch estimate: its Table 3 pairs 400M parameters with 8.0B tokens, which extrapolates to roughly 5M—not 7M—parameters at 100M tokens. That extrapolation is also below the model sizes used to fit the laws (Section 3, p. 4). (LLM, conf=0.82)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
89 more claims · 7 with their cited sources checked
Create a free account to see all 177 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 49
- It may state two separate claims as one. 45
- The quote only partly states this claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“They found that up to roughly four epochs, repetition is nearly as effective as new data, after which returns decay quickly”
From Training Compute-Optimal Transformer Models: A Review of Scaling Laws, Allocation, and Practice (2026) · cites this paper as Muennighoff et al., 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Growing concerns suggest that LLM pretraining may be approaching a ceiling as available high-quality data sources near exhaustion [58].”
From A Survey of Large Language Models (2026) · cites this paper as [58]
The cited source says what the claim attributes to it.
In the citing paper
“However, these studies find that repeating the same corpus only partially compensates for data scarcity; as the number of epochs increases, marginal gains diminish, and the risks of memorization and overfitting rise.”
From Scaling Data-Constrained Language Models with Synthetic Data (2026) · cites this paper as Muennighoff et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“The authors estimate the maximum increase in the effective dataset size that can be gained from multiple epochs at between 3x and 15x, and we anchor to this estimate in adjusting our model.”
From Will we run out of data? Limits of LLM scaling based on human-generated data (2022) · cites this paper as Muennighoff et al., 2023
In the citing paper
“Our theory will only rely upon the general form of the equations, specifically, that the dependence is inverse polynomial in N, D. So it applies to other frameworks of training (e.g., overtrained models Muennighoff et al. [2023]) where scaling laws have also been found.”
From A Theory for Emergence of Complex Skills in Language Models (2023) · cites this paper as Muennighoff et al. [2023]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“We find the best loss to be at around 20-60× more parameters and epochs, which corresponds to spending around 7000× more FLOPs.”
§ Scaling Data-Constrained Language Models > 5. Results: Resource Allocation for Data-Constrained Scaling
Reported 7000 · recomputed 3600
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Scaling Data-Constrained Language Models > Training data
“Like the equations in §B this formulation also reduces to the Chinchilla scaling laws in the base case of R D = 0 or R N = 0.”
§ Scaling Data-Constrained Language Models > Training data
“L(N, D, R N , R D ) = E + A N α * max(0,1-(R N /R * N )) + B D β * max(0,1-(R D /R * D )) (7”
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Self-citationUnsupervised Cross-lingual Representation Learning at Scale
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F. (., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2019) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.1911.02116 · Cited by 9,880
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550
Unsupervised Cross-lingual Representation Learning at Scale
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.747 · Cited by 5,637
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., & Toutanova, K. (2019) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n19-1300 · Cited by 2,979
Kudo, T., & Richardson, J. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/d18-2012 · Cited by 2,869
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.18223 · Cited by 2,090
Large Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013
The PASCAL Recognising Textual Entailment Challenge
Dagan, I., Glickman, O., & Magnini, B. (2006) · Lecture Notes in Computer Science · doi:10.1007/11736790_9 · Cited by 1,628
WizardLM: Empowering large pre-trained language models to follow complex instructionsPreprint
Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., Lin, Q., & Jiang, D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.12244 · Cited by 1,531
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441
Self-Instruct: Aligning Language Models with Self-Generated Instructions
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., & Hajishirzi, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.754 · Cited by 1,243
Narayan, S., Cohen, S. B., & Lapata, M. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1206 · Cited by 1,207
Finetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Fedus, W., Zoph, B., & Shazeer, N. (2021) · arXiv · doi:10.48550/arxiv.2101.03961 · Cited by 1,005
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2176 · Cited by 932
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.15556 · Cited by 929
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019) · arXiv · doi:10.48550/arxiv.1909.08053 · Cited by 893
HellaSwag: Can a Machine Really Finish Your Sentence?
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1472 · Cited by 830
Rasley, J., Rajbhandari, S., Ruwase, O., & He, Y. (2020) · Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3394486.3406703 · Cited by 823
LaMDA: Language Models for Dialog Applications
Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.08239 · Cited by 778
ST-MoE: Designing Stable and Transferable Sparse Expert ModelsPreprint
Zoph, B., Bello, I., Kumar, S., Du, N., Huang, Y., Dean, J., Shazeer, N., & Fedus, W. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.08906 · Cited by 734
Summarizing Source Code using a Neural Attention Model
Iyer, S., Konstas, I., Cheung, A., & Zettlemoyer, L. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1195 · Cited by 701
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.301 · Cited by 693
Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks
Weston, J., Bordes, A., Chopra, S., Rush, A. M., van Merriënboer, B., Joulin, A., & Mikolov, T. (2015) · arXiv · doi:10.48550/arxiv.1502.05698 · Cited by 670
Self-citationMusic TransformerPreprint
Huang, C.-Z. A., Vaswani, A., Uszkoreit, J., Shazeer, N., Simon, I., Hawthorne, C., Dai, A. M., Hoffman, M. D., Dinculescu, M., & Eck, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.04281 · Cited by 669
Adversarial NLI: A New Benchmark for Natural Language Understanding
Nie, Y., Williams, A., Dinan, E., Bansal, M., Weston, J., & Kiela, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.441 · Cited by 656
Efficient large-scale language model training on GPU clusters using megatron-LM
Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., et al. (2021) · Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.1145/3458817.3476209 · Cited by 639
Multitask Prompted Training Enables Zero-Shot Task Generalization
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.08207 · Cited by 612
Self-citationGalactica: A Large Language Model for Science
Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., & Stojnic, R. (2022) · arXiv · doi:10.48550/arxiv.2211.09085 · Cited by 605
mT5: A massively multilingual pre-trained text-to-text transformer
Xue, L., Constant, N., Roberts, A., Kale, M., Al‐Rfou, R., Siddhant, A., Barua, A., & Raffel, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11934 · Cited by 594
Self-citationUL2: Unifying Language Learning ParadigmsPreprint
Tay, Y., Dehghani, M., Tran, V. Q., Garcia, X., Wei, J., Wang, X., Chung, H. W., Shakeri, S., Bahri, D., Schuster, T., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.05131 · Cited by 591
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569
Self-citationDeep double descent: where bigger models and more data hurt*
Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., & Sutskever, I. (2021) · Journal of Statistical Mechanics: Theory and Experiment · doi:10.1088/1742-5468/ac3a74 · Cited by 565
Self-citationMTEB: Massive Text Embedding Benchmark
Muennighoff, N., Tazi, N., Magne, L., & Reimers, N. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.148 · Cited by 548
Self-citationHow Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources
Wang, Y., Ivison, H., Dasigi, P., Hessel, J., Khot, T., Chandu, K., Wadden, D., MacMillan, K., Smith, N., Beltagy, I., et al. (2023) · neural information processing systems · doi:10.52202/075280-3268 · Cited by 536
How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources
Wang, Y., Ivison, H., Dasigi, P., Hessel, J., Khot, T., Chandu, K. R., Wadden, D., MacMillan, K., Smith, N. A., Beltagy, I., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2306.04751 · Cited by 534
Multilingual is not enough: BERT for FinnishPreprint
Virtanen, A., Kanerva, J., Ilo, R., Luoma, J., Luotolahti, J., Salakoski, T., Ginter, F., & Pyysalo, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.07076 · Cited by 517
Self-citationThe Pile: An 800GB Dataset of Diverse Text for Language Modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.00027 · Cited by 492
Ethical and social risks of harm from Language Models
Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.04359 · Cited by 477
Dehghani, M., Gouws, S., Vinyals, O., Uszkoreit, J., & Kaiser, Ł. (2018) · arXiv · doi:10.48550/arxiv.1807.03819 · Cited by 467
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., & Tafjord, O. (2018) · arXiv · doi:10.48550/arxiv.1803.05457 · Cited by 462
The Natural Language Decathlon: Multitask Learning as Question Answering
McCann, B., Keskar, N. S., Xiong, C., & Socher, R. (2018) · arXiv · doi:10.48550/arxiv.1806.08730 · Cited by 448
SGPT: GPT Sentence Embeddings for Semantic SearchPreprint
Muennighoff, N. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.08904 · Cited by 386
Self-citationSuggesting accurate method and class names
Allamanis, M., Barr, E. T., Bird, C., & Sutton, C. (2015) · Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering · doi:10.1145/2786805.2786849 · Cited by 369
Crosslingual Generalization through Multitask Finetuning
Muennighoff, N., Wang, T., Sutawika, L., Roberts, A., Biderman, S., Le Scao, T., Bari, M. S., Shen, S., Yong, Z. X., Schoelkopf, H., et al. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.891 · Cited by 361
Self-citationWinoGrande: an adversarial winograd schema challenge at scale
Sakaguchi, K., Bras, R. L., Bhagavatula, C., & Choi, Y. (2021) · Communications of the ACM · doi:10.1145/3474381 · Cited by 356
Choosing Transfer Languages for Cross-Lingual Learning
Lin, Y.-H., Chen, C.-Y., Lee, J., Li, Z., Zhang, Y., Xia, M., Rijhwani, S., He, J., Zhang, Z., Ma, X., et al. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1301 · Cited by 344
Few-shot Learning with Multilingual Language ModelsPreprint
Lin, X. V., Mihaylov, T., Artetxe, M., Wang, T., Chen, S., Simig, D., Ott, M., Goyal, N., Bhosale, S., Du, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.10668 · Cited by 340
CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., & Xiong, C. (2022) · arXiv · doi:10.48550/arxiv.2203.13474 · Cited by 330
GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow
Black, S., Leo, G., Wang, P., Leahy, C., & Biderman, S. (2021) · Zenodo · doi:10.5281/zenodo.5297715 · Cited by 324
A large-scale audit of dataset licensing and attribution in AI
Longpre, S., Mahari, R., Chen, A., Obeng-Marnu, N., Sileo, D., Brannon, W., Muennighoff, N., Khazam, N., Kabbara, J., Perisetla, K., et al. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00878-8 · Cited by 307
Self-citationLongpre, S., Yauney, G., Reif, E., Lee, K., Roberts, A., Zoph, B., Zhou, D., Wei, J., Robinson, K., Mimno, D., et al. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.179 · Cited by 292
Smith, S., Patwary, M., Norick, B., LeGresley, P., Rajbhandari, S., Casper, J., Liu, Z., Prabhumoye, S., Zerveas, G., Korthikanti, V., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.11990 · Cited by 286
GLM-130B: An Open Bilingual Pre-trained Model
Zeng, A., Liu, X., Du, Z., Wang, Z., Lai, H., Ding, M., Yang, Z., Xu, Y., Zheng, W., Xia, X., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.02414 · Cited by 281
LIMA: Less Is More for Alignment
Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.11206 · Cited by 275
LIMA: Less Is More for Alignment
Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2400 · Cited by 275
StarCoder: may the source be with you!
Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.06161 · Cited by 261
Self-citationHolistic Evaluation of Language Models
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.09110 · Cited by 255
OctoPack: Instruction Tuning Code Large Language Models
Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., & Longpre, S. (2023) · arXiv · doi:10.48550/arxiv.2308.07124 · Cited by 245
Self-citationWikiLingua: A New Benchmark Dataset for Multilingual Abstractive Summarization
Ladhak, F., Durmus, E., Cardie, C., & McKeown, K. (2020) · Findings · doi:10.48550/arxiv.2010.03093 · Cited by 245
Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training
Li, S., Liu, H., Bian, Z., Fang, J., Huang, H., Liu, Y., Wang, B., & You, Y. (2023) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3605573.3605613 · Cited by 244
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.11446 · Cited by 235
CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data
Wenzek, G., Lachaux, M.-A., Conneau, A., Chaudhary, V., Guzmán, F., Joulin, A., & Grave, É. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.00359 · Cited by 231
Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures
Suárez, P. O., Sagot, B., & Romary, L. (2019) · Publication Server of the Institute for German Language (Institute for German Language) · doi:10.14618/ids-pub-9021 · Cited by 227
ProGen: Language Modeling for Protein Generation
Madani, A., McCann, B., Naik, N., Keskar, N. S., Anand, N., Eguchi, R. R., Huang, P.-S., & Socher, R. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.03.07.982272 · Cited by 212
Emergent and Predictable Memorization in Large Language ModelsPreprint
Biderman, S., Prashanth, U. S., Sutawika, L., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2304.11158 · Cited by 205
Quantifying Memorization Across Neural Language Models
Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramer, F., & Zhang, C. (2022) · arXiv · doi:10.48550/arxiv.2202.07646 · Cited by 204
Evaluating the state-of-the-art of End-to-End Natural Language Generation: The E2E NLG challenge
Dušek, O., Novikova, J., & Rieser, V. (2020) · Computer Speech & Language · doi:10.1016/j.csl.2019.06.009 · Cited by 192
Crowdsourcing Multiple Choice Science Questions
Welbl, J., Liu, N. F., & Gardner, M. (2017) · Proceedings of the 3rd Workshop on Noisy User-generated Text · doi:10.18653/v1/w17-4413 · Cited by 188
Explaining Neural Scaling Laws
Bahri, Y., Dyer, E., Kaplan, J., Lee, J., & Sharma, U. (2021) · arXiv · doi:10.48550/arxiv.2102.06701 · Cited by 182
WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarization
Ladhak, F., Durmus, E., Cardie, C., & McKeown, K. R. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.360 · Cited by 182
Explaining neural scaling laws
Bahri, Y., Dyer, E., Kaplan, J., Lee, J., & Sharma, U. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2311878121 · Cited by 181
PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts
Bach, S., Sanh, V., Yong, Z. X., Webson, A., Raffel, C., Nayak, N. V., Sharma, A., Kim, T., Bari, M. S., Fevry, T., et al. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2022.acl-demo.9 · Cited by 178
Self-citationThe CommitmentBank: Investigating projection in naturally occurring discourse
de Marneffe, M.-C., Simons, M., & Tonhauser, J. (2019) · Proceedings of Sinn und Bedeutung · doi:10.18148/sub/2019.v23i2.601 · Cited by 176
Zeng, W., Ren, X., Su, T., Wang, H., Liao, Y., Wang, Z., Jiang, X., Yang, Z., Wang, K., Zhang, X., et al. (2021) · arXiv · doi:10.48550/arxiv.2104.12369 · Cited by 172
Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Cappelli, A., Alobeidli, H., Pannier, B., Almazrouei, E., & Launay, J. (2023) · arXiv · doi:10.48550/arxiv.2306.01116 · Cited by 164
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
Biderman, S., Schoelkopf, H., Anthony, Q., Bradley, H., O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.01373 · Cited by 160
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
Gehrmann, S., Adewumi, T., Aggarwal, K., Ammanamanchi, P. S., Aremu, A., Bosselut, A., Chandu, K. R., Clinciu, M.-A., Das, D., Dhole, K., et al. (2021) · Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021) · doi:10.18653/v1/2021.gem-1.10 · Cited by 159
Scaling Laws for Autoregressive Generative Modeling
Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.14701 · Cited by 150
Beyond neural scaling laws: beating power law scaling via data pruning
Sorscher, B., Geirhos, R., Shekhar, S., Ganguli, S., & Morcos, A. S. (2022) · arXiv · doi:10.48550/arxiv.2206.14486 · Cited by 142
Beyond Neural Scaling Laws: Beating Power Law Scaling Via Data Pruning
Sorscher, B., Geirhos, R., Shekhar, S., Ganguli, S., & Morcos, A. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1419 · Cited by 142
LSDSem 2017 Shared Task: The Story Cloze Test
Mostafazadeh, N., Roth, M., Louis, A. P., Chambers, N., & Allen, J. F. (2017) · LSDSem@EACL · doi:10.18653/v1/w17-0906 · Cited by 140
AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq ModelPreprint
Soltan, S., Ananthakrishnan, S., FitzGerald, J., Gupta, R., Hamza, W., Khan, H., Peris, C., Rawls, S., Rosenbaum, A., Rumshisky, A., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.01448 · Cited by 133
Scaling Laws for Neural Machine TranslationPreprint
Ghorbani, B., Fırat, O., Freitag, M., Bapna, A., Krikun, M., Garcia, X., Chelba, C. I., & Cherry, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.07740 · Cited by 132
A framework for few-shot language model evaluation
Sutawika, L., Gao, L., Schoelkopf, H., Biderman, S., Tow, J., Abbasi, B., fattori, B., Lovering, C., farzanehnakhaee70, Phang, J., et al. (2021) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.5371628 · Cited by 124
Self-citationSelf-Instruct: Aligning Language Models with Self-Generated Instructions
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., & Hajishirzi, H. (2022) · arXiv · doi:10.48550/arxiv.2212.10560 · Cited by 118
Scaling vision transformers to 22 billion parameters
Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heek, J., Gilmer, J., Steiner, A., Caron, M., Geirhos, R., Alabdulmohsin, I., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.05442 · Cited by 112
FinGPT: Large Generative Models for a Small Language
Luukkonen, R., Komulainen, V., Luoma, J., Eskelinen, A., Kanerva, J., Kupari, H.-M., Ginter, F., Laippala, V., Muennighoff, N., Piktus, A., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.164 · Cited by 111
Self-citationFerreira, T. C., Gardent, C., Ilinykh, N., van der Lee, C., Mille, S., Moussallem, D., & Shimorina, A. (2020) · Tilburg University Research Portal · doi:10.5281/zenodo.6552784 · Cited by 109
Transcending Scaling Laws with 0.1% Extra Compute
Tay, Y., Wei, J., Chung, H. W., Tran, V., So, D. R., Shakeri, S., Garcia, X., Zheng, S., Rao, J., Chowdhery, A., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.91 · Cited by 106
Vilio: State-of-the-art Visio-Linguistic Models applied to Hateful MemesPreprint
Muennighoff, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.07788 · Cited by 105
Self-citationScaling Laws for Downstream Task Performance of Large Language ModelsPreprint
Isik, B., Ponomareva, N., Hazimeh, H., Paparas, D., Vassilvitskii, S., & Koyejo, S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.04177 · Cited by 93
Training Trajectories of Language Models Across ScalesPreprint
Xia, M., Artetxe, M., Zhou, C., et al. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2212.09803 · Cited by 86
Training Trajectories of Language Models Across Scales
Xia, M., Artetxe, M., Zhou, C., Lin, X. V., Pasunuru, R., Chen, D., Zettlemoyer, L., & Stoyanov, V. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.767 · Cited by 83
Will we run out of data? Limits of LLM scaling based on human-generated data
Villalobos, P., Ho, A., Sevilla, J., Besiroglu, T., Heim, L., & Hobbhahn, M. (2022) · arXiv · doi:10.48550/arxiv.2211.04325 · Cited by 80
StarCoder 2 and The Stack v2: The Next Generation
Lozhkov, A., Li, R., Allal, L. B., Cassano, F., Lamy-Poirier, J., Tazi, N., Tang, A., Pykhtar, D., Liu, J., Wei, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2402.19173 · Cited by 74
Self-citationDeduplicating Training Data Mitigates Privacy Risks in Language Models
Kandpal, N., Wallace, E., & Raffel, C. (2022) · arXiv · doi:10.48550/arxiv.2202.06539 · Cited by 68
Self-citationGPT-NeoX-20B: An Open-Source Autoregressive Language Model
Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.06745 · Cited by 67
The Deep Bootstrap Framework: Good Online Learners are Good Offline GeneralizersPreprint
Nakkiran, P., Neyshabur, B., & Sedghi, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.08127 · Cited by 64
Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
Tay, Y., Dehghani, M., Rao, J., Fedus, W., Abnar, S., Chung, H. W., Narang, S., Yogatama, D., Vaswani, A., & Metzler, D. (2021) · arXiv · doi:10.48550/arxiv.2109.10686 · Cited by 63
Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset
Henderson, P., Krass, M., Zheng, L., Guha, N., Manning, C. D., Jurafsky, D., & Ho, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2119 · Cited by 62
SantaCoder: don't reach for the stars!
Allal, L. B., Li, R., Kocetkov, D., Mou, C., Akiki, C., Ferrandis, C. M., Muennighoff, N., Mishra, M., Gu, A., Dey, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.03988 · Cited by 61
Self-citationMetaICL: Learning to Learn In Context
Min, S., Lewis, M., Zettlemoyer, L., & Hajishirzi, H. (2021) · arXiv · doi:10.48550/arxiv.2110.15943 · Cited by 61
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language ModelPreprint
Shin, S., Lee, S.-W., Ahn, H., Kim, S., Kim, H., Kim, B., Cho, K., Lee, G., Park, W., Ha, J.-W., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.13509 · Cited by 54
Hernandez, D., Kaplan, J., Henighan, T., & McCandlish, S. (2021) · arXiv · doi:10.48550/arxiv.2102.01293 · Cited by 51
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
Gehrmann, S., Adewumi, T., Aggarwal, K., Ammanamanchi, P. S., Anuoluwapo, A., Bosselut, A., Chandu, K. R., Clinciu, M., Das, D., Dhole, K. D., et al. (2021) · arXiv · doi:10.48550/arxiv.2102.01672 · Cited by 49
Instruction Tuned Models are Quick LearnersPreprint
Gupta, H., Sawant, S. A., Mishra, S. R., Nakamura, M., Mitra, A., Mashetty, S., & Baral, C. R. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.05539 · Cited by 47
ProGen: Language Modeling for Protein GenerationPreprint
Madani, A., McCann, B., Naik, N., Keskar, N. S., Anand, N., Eguchi, R. R., Huang, P., & Socher, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.03497 · Cited by 42
The Stack: 3 TB of permissively licensed source code
Kocetkov, D., Li, R., Allal, L. B., Li, J., Mou, C., Ferrandis, C. M., Jernite, Y., Mitchell, M., Hughes, S., Wolf, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.15533 · Cited by 42
Self-citationRevisiting neural scaling laws in language and vision
Alabdulmohsin, I., Neyshabur, B., & Zhai, X. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1620 · Cited by 41
MetaXL: Meta Representation Transformation for Low-resource Cross-lingual Learning
Xia, M., Zheng, G., Mukherjee, S., Shokouhi, M., Neubig, G., & Awadallah, A. H. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.42 · Cited by 39
PIQA: Reasoning about Physical Commonsense in Natural Language
Bisk, Y., Zellers, R., Le bras, R., Gao, J., & Choi, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6239 · Cited by 37
Choosing Transfer Languages for Cross-Lingual LearningPreprint
Lin, Y.-H., Chen, C.-Y., Lee, J., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.12688 · Cited by 33
Data Scaling Laws in NMT: The Effect of Noise and ArchitecturePreprint
Bansal, Y., Ghorbani, B., Garg, A., Zhang, B., Krikun, M., Cherry, C., Neyshabur, B., & Fırat, O. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.01994 · Cited by 33
(2023) · doi:10.52202/075280-1219 · Cited by 33
WeLM: A Well-Read Pre-trained Language Model for ChinesePreprint
Su, H., Zhou, X., Yu, H., Shen, X., Chen, Y., Zhu, Z., Yu, Y., & Zhou, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.10372 · Cited by 31
Adding Instructions during Pretraining: Effective way of Controlling Toxicity in Language Models
Prabhumoye, S., Patwary, M. A., Shoeybi, M., & Catanzaro, B. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.193 · Cited by 31
BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting
Yong, Z. X., Schoelkopf, H., Muennighoff, N., Aji, A. F., Adelani, D. I., Almubarak, K., Bari, M. S., Sutawika, L., Kasai, J., Baruwa, A., et al. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.653 · Cited by 30
Self-citationCrosslingual Generalization through Multitask Finetuning
Muennighoff, N., Wang, T., Sutawika, L., Roberts, A., Biderman, S., Scao, T. L., Bari, M. S., Shen, S., Yong, Z.-X., Schoelkopf, H., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.01786 · Cited by 29
Self-citationScaling Laws and Interpretability of Learning from Repeated Data
Hernandez, D., Brown, T., Conerly, T., DasSarma, N., Drain, D., El-Showk, S., Elhage, N., Hatfield-Dodds, Z., Henighan, T., Hume, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.10487 · Cited by 24
Measuring The Impact Of Programming Language Distribution
Orlanski, G., Xiao, K., Garcia, X., Hui, J., Howland, J., Malmaud, J., Austin, J., Singh, R., & Catasta, M. (2023) · arXiv · doi:10.48550/arxiv.2302.01973 · Cited by 21
Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel TrainingPreprint
Li, S., Liu, H., Bian, Z., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.14883 · Cited by 21
Komatsuzaki, A. (2019) · arXiv · doi:10.48550/arxiv.1906.06669 · Cited by 20
Deduplicating Training Data Makes Language Models Better
Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., & Carlini, N. (2021) · arXiv · doi:10.48550/arxiv.2107.06499 · Cited by 11
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?
Tay, Y., Dehghani, M., Abnar, S., Chung, H. W., Fedus, W., Rao, J., Narang, S., Tran, V. Q., Yogatama, D., & Metzler, D. (2022) · arXiv · doi:10.48550/arxiv.2207.10551 · Cited by 11
Scaling Laws for Generative Mixed-Modal Language Models
Aghajanyan, A., Yu, L., Conneau, A., Hsu, W.-N., Hambardzumyan, K., Zhang, S., Roller, S., Goyal, N., Levy, O., & Zettlemoyer, L. (2023) · arXiv · doi:10.48550/arxiv.2301.03728 · Cited by 10
What Language Model to Train if You Have One Million GPU Hours?
Scao, T. L., Wang, T., Hesslow, D., Saulnier, L., Bekman, S., Bari, M. S., Biderman, S., Elsahar, H., Muennighoff, N., Phang, J., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.15424 · Cited by 9
Self-citationTranscending Scaling Laws with 0.1% Extra ComputePreprint
Tay, Y., Wei, J., Chung, H. W., Tran, V. Q., So, D. R., Shakeri, S., Garcia, X., Zheng, H., Rao, J., Chowdhery, A., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.11399 · Cited by 6
Language models scale reliably with over-training and on downstream tasks
Gadre, S. Y., Smyrnis, G., Shankar, V., Gururangan, S., Wortsman, M., Shao, R., Mercat, J., Fang, A., Li, J., Keh, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.08540 · Cited by 5
Self-citationBLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting
Yong, Z.-X., Schoelkopf, H., Muennighoff, N., Aji, A. F., Adelani, D. I., Almubarak, K., Bari, M. S., Sutawika, L., Kasai, J., Baruwa, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.09535 · Cited by 2
Self-citationMetaXL: Meta Representation Transformation for Low-resource Cross-lingual LearningPreprint
Xia, M., Zheng, G., Mukherjee, S., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.07908 · Cited by 2
Adding Instructions during Pretraining: Effective Way of Controlling Toxicity in Language ModelsPreprint
Prabhumoye, S., Patwary, M. A., Shoeybi, M., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.07388 · Cited by 2
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Workshop, B., :, Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.05100
Self-citationOPT: Open Pre-trained Transformer Language Models
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.01068
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · arXiv · doi:10.48550/arxiv.2009.11462
The Flan Collection: Designing Data and Methods for Effective Instruction Tuning
Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H. W., Tay, Y., Zhou, D., Le, Q. V., Zoph, B., Wei, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.13688
Crowdsourcing Multiple Choice Science Questions
Welbl, J., Liu, N. F., & Gardner, M. (2017) · arXiv · doi:10.48550/arxiv.1707.06209
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
Laurençon, H., Saulnier, L., Wang, T., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2306
Research works printed but not linked: 9
- Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference on machine learning, pages 1691-1703. PMLR, 2020.
- Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, pages 5547-5569. PMLR, 2022.
- Kenneth Heafield. KenLM: Faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 187-197, Edinburgh, Scotland, July 2011. Association for Computational Linguistics. URL https://aclanthology.org/W11-2123.
- Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
- Pedro Javier Ortiz Su'arez, Laurent Romary, and Benoit Sagot. A monolingual approach to contextualized word embeddings for mid-resource languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1703-1714, Online, July 2020. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/2020.acl-main.156.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
- Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI spring symposium: logical formalizations of commonsense reasoning, pages 90-95, 2011.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
- Ge Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neural networks via zero-shot hyperparameter transfer. Advances in Neural Information Processing Systems, 34:17084-17097, 2021.
Cited by
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.12948 · Cited by 18,918
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093
The Falcon Series of Open Language ModelsPreprint
Almazrouei, E., Alobeidli, H., Alshamsi, A., Cappelli, A., Cojocaru, R., Debbah, M. A., Goffinet, É., Hesslow, D., Launay, J., Malartic, Q., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.16867 · Cited by 1,327
Olmo 3Preprint
Team OLMo, :, Ettinger, A., Bertsch, A., Kuehl, B., Graham, D., Heineman, D., Groeneveld, D., Brahman, F., Timbers, F., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.13961 · Cited by 601
C-Pack: Packed Resources For General Chinese Embeddings
Xiao, S., Liu, Z., Zhang, P., Muennighoff, N., Lian, D., & Nie, J.-Y. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3657878 · Cited by 523
Self-citation
Metadata
- Authors
- Muennighoff, N., Rush, A., Barak, B., Scao, T. L., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., & Raffel, C.
- ORCID
- Venue
- Advances in Neural Information Processing Systems 36
- Published
- 2023
- Pages
- 50358-50376
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W7133188041
- Cited by
- 47
- References
- 91
- Citations by year
- 2026: 7 · 2025: 21 · 2024: 15 · 2023: 4
- FWCI
- 8.42
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- other-oa
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, other-oa)
- UTUPub (University of Turku) (submittedVersion, other-oa)
- IRIS Research product catalog (Sapienza University of Rome) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2134157)
- U.S. Department of Energy (DESC0022199)
- European Commission (101070350)
- China Scholarship Council
- Defense Advanced Research Projects Agency (W911NF2010021, DE-SC0022199)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- language model scaling, training data efficiency, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.