CoChat
Sign Up
PreprintCited by 49Topic ModelingNatural Language Processing TechniquesDomain Adaptation and Few-Shot Learning

Scaling Data-Constrained Language Models

Open access
Muennighoff, Niklas, Rush, Alexander M., Barak, Boaz, Scao, Teven Le, Piktus, Aleksandra, Tazi, Nouamane, Pyysalo, Sampo, Wolf, Thomas +1 more·2023·arXiv
DOI 10.48550/arxiv.2305.16264arXiv 2305.16264
Open this paper in CoChat — free accountOpen PDF

Abstract

The current trend of scaling language models involves increasing both parameter count and training dataset size. Extrapolating this trend suggests that training dataset size may soon be limited by the amount of text data available on the internet. Motivated by this limit, we investigate scaling language models in data-constrained regimes. Specifically, we run a large set of experiments varying the extent of data repetition and compute budget, ranging up to 900 billion training tokens and 9 billion parameter models. We find that with constrained data for a fixed compute budget, training with up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data. However, with more repetition, the value of adding compute eventually decays to zero. We propose and empirically validate a scaling law for compute optimality that accounts for the decreasing value of repeated tokens and excess parameters. Finally, we experiment with approaches mitigating data scarcity, including augmenting the training dataset with code data or removing commonly used filters. Models and datasets from our 400 training runs are freely available at https://github.com/huggingface/datablations.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2305.16264

Cites

Statistics

Nothing to recompute.

Claims the paper makes

0 of 59 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Two benchmarked tasks, WebNLG and bAbI, see jumps in performance as soon as code is added.

    “Two of the tasks benchmarked, WebNLG [17,34], a generation task, and bAbI [122,57], a reasoning task, see jumps in performance as soon as code is added”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [17] 17. Thiago Castro Ferreira, Claire Gardent, Nikolai Ilinykh, Chris van der Lee, Simon Mille, Diego Moussallem, and Anastasia Shimorina. 2020. The 2020 Bilingual, Bi-Directional WebNLG+ Shared Task Overview and Evaluation Results (WebNLG+ 2020). In Proceedings of the 3rd WebNLG Workshop on Natural Language Generation from the Semantic Web (WebNLG+ 2020), pages 55-76, Dublin, Ireland (Virtual). Association for Computational Linguistics.

      Not checked in the free run.

    • cites [34] 34. Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Aremu Anuoluwapo, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna Clinciu, Dipanjan Das, Kaustubh D Dhole, et al. 2021. The gem benchmark: Natural language generation, its evaluation and metrics. arXiv preprint arXiv:2102.01672.

      Not checked in the free run.

    • cites [122] 122. Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart Van Merriënboer, Armand Joulin, and Tomas Mikolov. 2015. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698.

      Not checked in the free run.

    • cites [57] 57. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110.

      Not checked in the free run.

  • Findingcitation

    Models trained on less unique data (and correspondingly more epochs) have consistently higher loss.

    “Conforming to intuition and prior work on deduplication [55], repeated data is worth less, thus models trained on less unique data (and, correspondingly, more epochs) have consistently higher loss.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [55] 55. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2021. Deduplicating Training Data Makes Language Models Better. arXiv preprint arXiv:2107.06499.

      Not checked in the free run.

  • Findingown data

    The N=8.7 billion parameter model trained for four epochs (D_C=44 billion unique tokens) finishes training with only 0.5% higher validation loss than the single-epoch model (D_C=178 billion unique tokens).

    “For example, the N = 8.7 billion parameter model trained for four epochs (D C = 44 billion unique tokens) finishes training with only 0.5% higher validation loss than the single-epoch model (D C = 178 billion unique tokens).”
    Quote in the paperSubstantial claim
  • Findingown data

    With more than 4 epochs of repetition, the value of adding compute eventually decays to zero.

    “However, with more repetition, the value of adding compute eventually decays to zero.”
    • Also stated in: § 1 Introduction · plain
      “Additional epochs continue to be beneficial, but returns eventually diminish to zero.”
    Quote in the paperSubstantial claim
  • Findingown data

    As data is repeated for more than a single epoch, the fit predicts that excess parameters decay faster in value than repeated data (R*N < R*D).

    “As data is repeated for more than a single epoch, our fit predicts that excess parameters decay faster in value than repeated data (R * N < R * D ).”
    Quote in the paperSubstantial claim
  • May combine two claims. 57
  • Classified as probably not a substantial claim. 32
  • The quote only partly states this claim. 1

  • May combine two claims.

    Findingown data

    The best loss is at around 20-60× more parameters and epochs, corresponding to spending around 7000× more FLOPs.

    “We find the best loss to be at around 20-60× more parameters and epochs, which corresponds to spending around 7000× more FLOPs.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The new data-constrained scaling law generalizes the Chinchilla scaling law to the repeated data regime and yields a better prediction of loss in this setting.

    “We use these results to fit a new data-constrained scaling law that generalizes the Chinchilla scaling law [42] to the repeated data regime and yields a better prediction of loss in this setting.”
    • Also stated in: § 6 Results: Resource Return for Data-Constrained Scaling · plain
      “The data-constrained scaling laws can accurately measure the decay in the value of repeated data as seen by the proximity of empirical results (dots) and parametric fit (lines).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [42] 42. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training Compute-Optimal Large Language Models. arXiv preprint arXiv:2203.15556.

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    Training large language models for multiple epochs by repeating data is beneficial and scaling laws continue to hold in the multi-epoch regime, albeit with diminishing returns.

    “We find that despite recommendations of earlier work, training large language models for multiple epochs by repeating data is beneficial and that scaling laws continue to hold in the multi-epoch regime, albeit with diminishing returns.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Through fitting, R*D ≈ 15, corresponding to 15 repetitions (or 16 epochs).

    “Through our fitting in Appendix A, we found R * D ≈ 15, corresponding to 15 repetitions (or 16 epochs).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    More than a 50% reduction in loss can be attained by training for several epochs and increasing model size beyond what would be compute-optimal for 100M tokens.

    “Results show that more than a 50% reduction in loss can be attained by training for several epochs (R D > 0) and increasing model size beyond what would be compute-optimal for 100M tokens (R N > 0).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Perplexity-filtering is effective, while deduplication does not help.

    “Of the filtering approaches, we find perplexity-filtering to be effective, while deduplication does not help.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    In the concrete example with δ=0.25, U=1, and R_D=4, Equation 10 yields D'=3.05.

    “In that case Equation 10 yields D ′ = U + (1 -δ)U (1-(1-δ) R D ) δ = 1 + (0.75) * 4 * (1 -0.75 4 ) = 3.05.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: δ is small.

  • May combine two claims.

    Findingown data

    They fit R*_N = 5.309743 and R*_D = 15.387756.

    “Nevertheless, we are able to get a fairly stable fit resulting in R * N = 5.309743 and R * D = 15.387756.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Excess parameters or epochs only cause performance to plateau but never worsen.

  • May combine two claims.

    Findingown data

    The learned R*_D yields δ ≈ 6e-2 (0.19 for R*_N), which respects the assumption that δ is small.

    “This value of R * D yields δ ≈ 6 * 10 -2 (0.19 for R * N ), which respects the assumption that δ is small.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: δ is small.

  • May combine two claims.

    Findingown data

    Explicit decay, Equation 10, and Equation 14 all yield similar results with R^2 around 80.

    “This explicit decay, Equation 10, and Equation 14 all yield similar results with R 2 around 80.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Excess parameters or epochs only cause performance to plateau but never worsen.

  • May combine two claims.

    Findingown data

    This results in R*_D = 26530.611 and R*_N = 2040.8163.

    “This results in R * D = 26530.611and R * N = 2040.8163.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    No decay or decaying only D or N of Equation 14 leads to worse loss and R^2 than Equation 14.

    “No decay or decaying only D or N of Equation 14 leads to worse loss and R 2 than Equation 14.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Excess parameters or epochs only cause performance to plateau but never worsen.

  • May combine two claims.

    Findingown data

    Their fit results in a = 6.255414, b = 7.3049974, e = 0.6254804, α = β = 0.3526596.

    “Our fit results in a = 6.255414, b = 7.3049974, e = 0.6254804, α = β = 0.3526596.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Regardless of deduplication, they find 59 epochs to be optimal and overall trend very similar.

    “Regardless of deduplication we find 59 epochs to be optimal and the overall trend to be very similar.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning

    Equation 14 fits the data slightly worse than Equation 10, likely due to approximations.

    “Equation 14 fits the data slightly worse than Equation 10, likely due to our approximations.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Excess parameters or epochs only cause performance to plateau but never worsen.

  • May combine two claims.

    Findingreasoning

    While predicted optimal allocation (efficient frontier) is similar to Figure 3, predicted return from repeated data differs significantly.

    “While the predicted optimal allocation (efficient frontier) is similar to Figure 3, the predicted return from repeated data differs significantly.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Training on up to 50% of Python data maintains performance on all natural language tasks while enabling huge performance gains on state-tracking (bAbI) for C4.

    “We find that training on up to 50% of Python data maintains performance on all natural language tasks while enabling huge performance gains on state-tracking (bAbI) for C4.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The model pre-trained on 90% of Python code data and 10% of C4 has the highest C4 validation loss, but the lowest Python validation loss.

    “The model pre-trained on 90% of Python code data and 10% of C4 has the highest C4 validation loss, but the lowest Python validation loss.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    The model trained on deduplicated data has the highest validation and training loss.

    “The model trained on deduplicated data has the highest validation and training loss.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    They find the trend to be the same as for models trained on C4: While models with fewer repeats have better loss, differences for a few repeats are insignificant.

    “We find the trend to be the same as for models trained on C4: While models with fewer repeats have better loss, differences for a few repeats are insignificant.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Models trained on data that is repeated for multiple epochs have consistently worse loss and diverge if too many epochs are used.

    “Models trained on data that is repeated for multiple epochs have consistently worse loss and diverge if too many epochs are used.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Notably, models trained on C4 completely fail on bAbI [122], while OSCAR models are able to perform better than random.

    “Notably, models trained on C4 completely fail on bAbI [122], while OSCAR models are able to perform better than random.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [122] 122. Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart Van Merriënboer, Armand Joulin, and Tomas Mikolov. 2015. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698.

      Not checked in the free run.

  • May combine two claims.

    Findingnone given

    Deduplicating the selected 25% OSCAR subset is very strict and preserves less than 5% of the full OSCAR.

    “This is very strict and preserves less than 5% of the full OSCAR.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The data-constrained compute-optimal variant (6.3 billion parameters) performs better by using fewer parameters and repeating more data.

    “The data-constrained compute-optimal variant (6.3 billion parameters) performs better by using fewer parameters and repeating more data.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    There are temporary spikes for 8.7 billion parameter models, commonly at the start of a new epoch.

    “There are temporary spikes for 8.7 billion parameter models, commonly at the start of a new epoch.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    Excess parameters empirically hurt performance.

    “Excess parameters empirically hurt performance, but this may be due to a lack of regularization.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Using the 25% OSCAR suffix array to deduplicate the full OSCAR is more permissive and allows preserving 31% of the original dataset.

    “This is more permissive and allows us to preserve 31% of the original dataset.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Models trained on The Pile showed similar trends as for OSCAR and C4.

    “We have also trained several models on The Pile [31] and found similar trends as for OSCAR and C4.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [31] 31. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv preprint arXiv:2101.00027.

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    Deduplication does not improve downstream performance for C4 while being effective for OSCAR.

    “Deduplication does not improve downstream performance for C4 while being effective for OSCAR which has significantly more noise.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: OSCAR is noisier than C4.

  • May combine two claims.

    Findingown data

    However, these are temporary and loss continues to go down smoothly thereafter.

    “However, these are temporary and loss continues to go down smoothly thereafter.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Our findings on both datasets were overall in agreement with each other.

    “Our findings on both datasets were overall in agreement with each other.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Loss progresses smoothly throughout training.

    “Loss progresses smoothly throughout training.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    OSCAR is a noisier dataset than C4 due to less filtering, thus models trained on C4 generally perform better.

    “OSCAR is a noisier dataset than C4 due to less filtering, thus models trained on C4 generally perform better.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The Galactica models were trained for 4.25 epochs and exhibit continually decreasing validation loss and improving downstream performance throughout training.

    “An exception is the recent Galactica models [108] that were trained for 4.25 epochs and exhibit continually decreasing validation loss and improving downstream performance throughout training.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [108] 108. Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085.

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    One-epoch models significantly under-utilize their training data and more signal can be extracted by repeating data and adding parameters at the cost of sub-optimal compute utilization.

    “These results suggest that one-epoch models significantly under-utilize their training data and more signal can be extracted by repeating data and adding parameters at the cost of sub-optimal compute utilization.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Resources are divided roughly equally between scaling of parameters and data.

    “Allocation is balanced: resources are divided roughly equally between scaling of parameters and data [42].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [42] 42. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training Compute-Optimal Large Language Models. arXiv preprint arXiv:2203.15556.

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Extrapolating Chinchilla scaling laws to a 530 billion parameter model such as MT-NLG would require training on 11 trillion tokens, corresponding to more than 30 terabytes of text data.

    “Extrapolating these laws for compute allocation (hereafter”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [99] 99. Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. 2022. Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model. arXiv preprint arXiv:2201.11990.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The formula presented in §3 and its predicted isoLoss contours do not model the possibility that excess epochs or parameters could hurt performance.

    “Thus, our formula presented in §3 and its predicted isoLoss contours in Figure 3 do not model the possibility that excess epochs or parameters could hurt performance.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Loss scales as a power-law with the amount of compute used for training.

    “Return follows a power-law: loss scales as a power-law with the amount of compute used for training [39,46,6,35,7,41].”
    Quote in the paperSubstantial claim0 of 6 sources checked
    • cites [39] 39. Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. 2020. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701.

      Not checked in the free run.

    • cites [46] 46. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.

      Not checked in the free run.

    • cites [6] 6. Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. 2021. Explain- ing neural scaling laws. arXiv preprint arXiv:2102.06701.

      Not checked in the free run.

    • cites [35] 35. Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. 2021. Scaling laws for neural machine translation. arXiv preprint arXiv:2109.07740.

      Not checked in the free run.

    • cites [7] 7. Yamini Bansal, Behrooz Ghorbani, Ankush Garg, Biao Zhang, Colin Cherry, Behnam Neyshabur, and Orhan Firat. 2022. Data scaling laws in NMT: The effect of noise and architecture. In International Conference on Machine Learning, pages 1466-1482. PMLR.

      Not checked in the free run.

    • cites [41] 41. Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. 2021. Scaling laws for transfer. arXiv preprint arXiv:2102.01293.

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationnone given

    The work indicates there are limits on the scaling horizon.

    “However, our work also indicates that there are limits on the scaling horizon.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    In the data-constrained regime, allocating new compute to both more parameters and epochs is necessary, and epochs should be scaled slightly faster.

    “We find that, in the data-constrained regime, allocating new compute to both more parameters and epochs is necessary, and that epochs should be scaled slightly faster.”
    • Also stated in: § 5 Results: Resource Allocation for Data-Constrained Scaling · hedged
      “For all three data budgets, our results suggest that Allocation is optimized by scaling epochs faster than parameters.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Villalobos et al. estimate that even high-quality English language data will be exhausted by the year 2024 given the Chinchilla scaling laws and the trend of training ever-larger models.

    “Villalobos et al. [112] estimate that even high-quality English language data will be exhausted by the year 2024 given the Chinchilla scaling laws and the trend of training ever-larger models.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [112] 112. Pablo Villalobos, Jaime Sevilla, Lennart Heim, Tamay Besiroglu, Marius Hobbhahn, and Anson Ho. 2022. Will we run out of data? An analysis of the limits of scaling datasets in Machine Learning. arXiv preprint arXiv:2211.04325.

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Chinchilla methods lead to the conclusion that α≈β and hence N and D should be scaled proportionally for compute-optimal training.

    “These methods lead to the conclusion that α ≈ β and hence N and D should be scaled proportionally for compute-optimal training.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [42] 42. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training Compute-Optimal Large Language Models. arXiv preprint arXiv:2203.15556.

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Most prior large language models have been trained for a single epoch and some work explicitly advocates against reusing data.

    “however, most prior large language models have been trained for a single epoch [51,15] and some work explicitly advocates against reusing data [40].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [51] 51. Aran Komatsuzaki. 2019. One epoch is all you need. arXiv preprint arXiv:1906.06669.

      Not checked in the free run.

    • cites [15] 15. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877- 1901.

      Not checked in the free run.

    • cites [40] 40. Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, et al. 2022. Scaling Laws and Interpretability of Learning from Repeated Data. arXiv preprint arXiv:2205.10487.

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    They do not have a rigorous mathematical justification for this alpha-beta decay formulation which could be cause of incorrect return predictions.

    “Beyond its potentially useful properties, we do not have a rigorous mathematical justification for this alpha-beta decay formulation which could be the cause of the incorrect return predictions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Equation 13 allows interpreting R*_D as the number of repetitions beyond which repeating yields sharply diminishing returns and flattens out soon after.

    “Equation 13 has the benefit that we can interpret R * D as the number of repetitions beyond which repeating yields sharply diminishing returns and flattens out soon after.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: δ is small.

  • Classified as probably not a substantial claim.

    Premisecitation

    They trained their models on 106 billion unique tokens for 4.25 epochs.

    “They trained their models on 106 billion unique tokens for 4.25 epochs.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [108] 108. Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Hoffmann et al. use training loss as their core metric.

    “Hoffmann et al. [42] use training loss as their core metric.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Hoffmann et al. [42] 42. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training Compute-Optimal Large Language Models. arXiv preprint arXiv:2203.15556.

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    They settle on exponential decay formulation from Appendix A that does not allow excess parameters or epochs to hurt, as preventing such behavior is trivial by stopping training or removing excess parameters.

    “Ultimately, we settle on our exponential decay formulation from Appendix A that does not allow excess parameters or epochs to hurt, as preventing such behavior is trivial by stopping training (in the case of epochs hurting) or removing excess parameters (in the case of model parameters hurting).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    This decision was likely driven by intuition that repeated data is worth less, thus one should spend more compute on parameters.

    “This decision was likely driven by the intuition that repeated data is worth less, thus one should spend more compute on parameters.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Prior work found deduplication was able to improve perplexity.

    “Prior work found deduplication was able to improve perplexity [55]; however, it did not evaluate on downstream tasks.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [55] 55. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2021. Deduplicating Training Data Makes Language Models Better. arXiv preprint arXiv:2107.06499.

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    Even more epochs can be beneficial and a small spike in validation loss does not justify stopping training.

    “Our findings on Return from repeated data agree with their conclusion that multiple epochs are beneficial, however, we find that even more epochs can be beneficial and a small spike in validation loss does not justify stopping training (Appendix J).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Prior work has reported double descent phenomena when repeating data, where loss initially increases then decreases as model trained for more epochs.

    “Prior work has reported double descent phenomena when repeating data, where the loss initially increases and then decreases again as the model is trained for more epochs [75,40].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [75] 75. Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. 2021. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021(12):124003.

      Not checked in the free run.

    • cites [40] 40. Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, et al. 2022. Scaling Laws and Interpretability of Learning from Repeated Data. arXiv preprint arXiv:2205.10487.

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    Parameters learning from repeated data are worth even less than repeated data, thus one should overallocate to epochs, not parameters.

    “Parameters learning from repeated data are worth even less than repeated data, thus one should overallocate to epochs, not parameters.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Hoffmann et al. have shown that equal scaling of model parameters and training tokens holds across different training datasets, but precise ratios vary considerably across datasets and approaches.

    “While Hoffmann et al. [42] have shown that the equal scaling of model parameters and training tokens holds across different training datasets, the precise ratios vary considerably across datasets and approaches.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Hoffmann et al. [42] 42. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training Compute-Optimal Large Language Models. arXiv preprint arXiv:2203.15556.

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    Default Chinchilla scaling law [42] predicts loss to continue decreasing as parameters are added, which is in stark contrast to empirical data.

    “Meanwhile, the default Chinchilla scaling law [42] predicts loss to continue decreasing as parameters are added, which is in stark contrast to the empirical data.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [42] 42. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training Compute-Optimal Large Language Models. arXiv preprint arXiv:2203.15556.

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Based on IsoFLOP training runs on C4, Hoffmann et al. provide that for C4, compute allocated to data and parameters should be scaled exactly equally for optimality, i.e. a=b=0.5.

    “Based on their IsoFLOP training runs on C4, they only provide the information that for C4, compute (C) allocated to data (D) and parameters (N ) should be scaled exactly equally for optimality, i.e. a = b = 0.5 in the relationship N opt ∝ C a and D opt ∝ C b .”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [42] 42. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training Compute-Optimal Large Language Models. arXiv preprint arXiv:2203.15556.

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Their proposed scaling equations predict loss to plateau, as seen in the straight line.

    “Thus, our proposed scaling equations predict loss to plateau, as seen in the straight line.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    R*_N is significantly lower resulting in excess parameters hurting faster than excess epochs, which is in line with empirical data from Figure 3.

    “R * N is significantly lower resulting in excess parameters hurting faster than excess epochs, which is in line with empirical data from Figure 3.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Specifically, 40B parameters trained for 1.35T tokens (12.75 epochs) would have been optimal according to data-constrained scaling laws.

    “Specifically, 40 billion parameters trained for 1.35 trillion tokens (12.75 epochs) would have been optimal according to data-constrained scaling laws.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Excess parameters learn the same features and do not add any value in the extreme.

    “Symmetric to seeing the same data, excess parameters learn the same features and do not add any value in the extreme.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Equation 14 generalizes Equation 7: it provides the same estimates for optimal model and data size in the single epoch case, but allows for decay in the value of parameters and tokens, thus generalizing to training for multiple epochs and with excess parameters.

    “Equation 14 is a generalization of Equation 7: It provides the same estimates for optimal model and data size in the single epoch case, but allows for decay in the value of parameters and tokens, thus generalizing to training for multiple epochs and with excess parameters.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    For small positive R_D, R_N, the curve agrees with Chinchilla and one needs to increase R_N, R_D by the same amount to maintain proportionality.

    “For small positive R D , R N , our curve agrees with Chinchilla and so we need to increase R N , R D by the same amount to maintain the proportionality.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: In the analytical properties, they assume they are in the multi-epoch regime and hence R_D > 0.

  • May combine two claims.

    Interpretationreasoning

    This is very close to the 73B parameters and 1.3T tokens predicted by IsoFLOP curves on C4 from [42], and thus they consider it a good fit.

    “This is very close to the 73 billion parameters and 1.3 trillion tokens predicted by the IsoFLOP curves on C4 from [42] and thus we consider it a good fit.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [42] 42. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training Compute-Optimal Large Language Models. arXiv preprint arXiv:2203.15556.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    The Galactica models [108] are the only publicly known LLMs that explicitly trained for a significant number of epochs prior to this work.

    “The Galactica models [108] are the only publicly known LLMs that explicitly trained for a significant number of epochs prior to this work.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [108] 108. Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085.

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Processing more tokens will be more effective than increasing the number of parameters, and the optimal compute curve is expected to break away from proportionality.

    “Hence processing more tokens will be more effective than increasing the number of parameters, and we expect the optimal compute curve to break away from proportionality.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: In the analytical properties, they assume they are in the multi-epoch regime and hence R_D > 0.

  • May combine two claims.

    Premisecitation

    Results from [42] show that there are differences in scaling coefficients when training on C4 as compared to GitHub code, however, the overall allocation trend is the same.

    “Results from [42] show that there are differences in the scaling coefficients when training on C4 as compared to GitHub code, however, the overall allocation trend is the same.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [42] 42. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training Compute-Optimal Large Language Models. arXiv preprint arXiv:2203.15556.

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    While they expect a smaller model trained for more epochs to be better than the 120B parameter model, the optimal allocation is unlikely to be exactly 40B parameters and 1.35T tokens.

    “Thus, while we expect a smaller model trained for more epochs to be better than the 120 billion parameter model, the optimal allocation is unlikely to be exactly 40 billion parameters and 1.35 trillion tokens.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Results from [46] show that loss can increase when too many parameters are used, even with early stopping.

    “Results from [46] also show that loss can increase when too many parameters are used, even with early stopping.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [46] 46. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    Loss is reported on a held-out test set unless otherwise specified.

    “As repeating data can result in extreme overfitting (see Appendix H), we report loss on a held-out test set unless otherwise specified (see Appendix K).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Decaying exponents to 0 has effect of more repetitions eventually hurting performance as lim α→0 D^α = 1 and same for β.

    “Decaying the exponents to 0 has the effect of more repetitions eventually hurting performance as lim α→0 D α = 1 and the same for β.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Initially the model size and tokens processed grow proportionally (R N = R D ) but since R * N < R * D , at some point adding parameters offers worse returns compared to increasing the number of tokens processed, and hence we deviate from the Chinchilla curve.

    “Initially the model size and tokens processed grow proportionally (R N = R D ) but since R * N < R * D , at some point adding parameters offers worse returns compared to increasing the number of tokens processed, and hence we deviate from the Chinchilla curve.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationreasoning

    The authors' hyperparameter choices are not exotic, so their setup is expected to generalize to many other setups.

    “As none of our hyperparameter choices is particularly exotic, we expect our setup to generalize to many other setups.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Study design and conductnone given

    By training five seeds for most models benchmarked, always averaging 0-5 fewshots, and ensuring maximum data overlap for repeated data (§4) they significantly reduce uncertainty.

    “However, by training five seeds for most models benchmarked, always averaging 0-5 fewshots, and ensuring maximum data overlap for repeated data ( §4) we significantly reduce uncertainty.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The small number of evaluation tokens for 8.7B parameter models likely contributes to loss spikes for 8.7B parameter models seen in Figure 4.

    “The small number of evaluation tokens for the 8.7 billion parameter models likely contributes to the loss spikes for 8.7 billion parameter models seen in Figure 4.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Extrapolating from the proposed data-constrained scaling law shows that at small numbers epochs are benign, but at large number of epochs loss stops improving.

    “Extrapolating from the proposed data-constrained scaling law shows that at small numbers epochs are benign, but at large number of epochs loss stops improving.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The perplexity-filtered model's training data has been filtered to the top 25% of examples with the lowest perplexity, thus high loss examples have been explicitly filtered out from the training data resulting in low training loss.

    “Its training data has been filtered to the top 25% of examples with the lowest perplexity (Appendix N) thus high loss examples have been explicitly filtered out from the training data resulting in low training loss.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationcitation

    The less clear gains for OSCAR are likely due to OSCAR containing code, while code data was explicitly filtered out for C4.

    “For OSCAR gains are less clear, which is likely due to OSCAR containing code [83], while code data was explicitly filtered out for C4 [90].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [83] 83. Pedro Javier Ortiz Su'arez, Laurent Romary, and Benoit Sagot. 2020. A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1703-1714, Online. Association for Computational Linguistics.

      Not checked in the free run.

    • cites [90] 90. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1-67.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    OSCAR is considered noisier than C4 due to its less stringent duplication.

    “OSCAR is considered noisier than C4 [90] due to its less stringent duplication.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [83] 83. Pedro Javier Ortiz Su'arez, Laurent Romary, and Benoit Sagot. 2020. A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1703-1714, Online. Association for Computational Linguistics.

      Not checked in the free run.

    • cites [90] 90. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1-67.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The compute cost of training a model can be approximated as FLOPs(N, D) ≈ 6ND, where N is the number of model parameters and D is the number of tokens processed.

    “In this work we use[46]'s approximation for the compute cost: FLOPs(N, D) ≈ 6N D, where N denotes the number of model parameters and D denotes the number of tokens processed.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [46] 46. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationnone given

    The optimal data strategy is dependent on the dataset at hand and we cannot give universally applicable filtering recommendations.

    “The optimal data strategy is dependent on the dataset at hand and we cannot give universally applicable filtering recommendations.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Taylor et al. [108] decided to early-stop pre-training of Galactica models due to a small increase in validation loss at start of fifth epoch.

    “Taylor et al. [108] decided to early-stop pre-training of the Galactica models due to a small increase in validation loss at the start of the fifth epoch.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Taylor et al. [108] 108. Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085.

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Our scaling formula predicts loss to plateau, while Chinchilla predicts loss to improve.

    “Thus, our scaling formula predicts loss to plateau, while Chinchilla predicts loss to improve.”
    Quote in the paperSubstantial claim
83 of 83

  • The paper relies on this without support.

    For most languages, available data is several orders of magnitude smaller, meaning that LLMs in those languages are already data-constrained.

    “For most languages, available data is several orders of magnitude smaller, meaning that LLMs in those languages are already data-constrained.”

  • The authors state this as an assumption.

    Each time a model trains on a token, it learns a 1-δ fraction of the information in it for some constant 0≤δ≤1.

    “Assume that each time a model trains on a token, it learns a 1 -δ fraction of the information in it for some constant 0 ≤ δ ≤ 1.”

  • The authors state this as an assumption.

    The decrease in value of each repetition is proportional to the value of the prior repetition, equivalent to exponential decay.

    “In other words, we expect the decrease in value of each repetition to be proportional to the value of the prior repetition, which is equivalent to exponential decay.”

  • The paper relies on this without support.

    The effective data D' should be smaller or equal to D because repeated tokens provide less useful information than new ones.

    “Intuitively, D ′ should be smaller or equal to D where D is the total number of processed tokens since repeated tokens provide less useful information to the model than new ones.”

  • The paper relies on this without support.

    Models with sizes that vastly outstrip the available data offer diminishing returns per parameter.

    “Just as processing repeated tokens yields a diminishing return, both intuitively and empirically, models with sizes that vastly outstrip the available data also offer diminishing returns per parameter.”

  • The paper relies on this without support.

    The held-out data stems from the same underlying dataset.

    “This contrasts training loss used in [42], but should not alter our findings as the held-out data stems from the same underlying dataset.”

  • The authors state this as an assumption.

    δ is small.

    “If we assume δ to be small, 1 -δ tends to one and we can approximate 1/R * D = δ 1-δ ≈ δ.”

  • The authors state this as an assumption.

    Excess parameters or epochs only cause performance to plateau but never worsen.

    “We assume excess parameters or epochs only cause performance to plateau but never to worsen.”

  • The authors state this as an assumption.

    In the analytical properties, they assume they are in the multi-epoch regime and hence R_D > 0.

    “we assume that we are in the multi-epoch regime and hence R D > 0”

  • The paper relies on this without support.

    OSCAR is noisier than C4.

    “OSCAR, which is noisier than C4”
    “OSCAR which has significantly more noise”

  • The paper relies on this without support.

    The unfiltered validation set contains commonly repeated sequences.

    “when encountering these common sequences in the unfiltered validation set”
11 of 11

The citation lists haven’t been fetched yet

Cited by 49

Research works printed but not linked: 41

  1. [2]2. Ibrahim M Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. 2022. Revisiting neural scaling laws in language and vision. Advances in Neural Information Processing Systems, 35:22300-22312.
  2. [4]4. Miltiadis Allamanis, Earl T Barr, Christian Bird, and Charles Sutton. 2015. Suggesting accurate method and class names. In Proceedings of the 2015 10th joint meeting on foundations of software engineering, pages 38-49.
  3. [6]6. Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. 2021. Explain- ing neural scaling laws. arXiv preprint arXiv:2102.06701. · arXiv
  4. [7]7. Yamini Bansal, Behrooz Ghorbani, Ankush Garg, Biao Zhang, Colin Cherry, Behnam Neyshabur, and Orhan Firat. 2022. Data scaling laws in NMT: The effect of noise and architecture. In International Conference on Machine Learning, pages 1466-1482. PMLR.
  5. [8]8. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610-623.
  6. [12]12. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. PIQA: Rea- soning about Physical Commonsense in Natural Language. In Thirty-Fourth AAAI Conference on Artificial Intelligence.
  7. [15]15. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877- 1901.
  8. [17]17. Thiago Castro Ferreira, Claire Gardent, Nikolai Ilinykh, Chris van der Lee, Simon Mille, Diego Moussallem, and Anastasia Shimorina. 2020. The 2020 Bilingual, Bi-Directional WebNLG+ Shared Task Overview and Evaluation Results (WebNLG+ 2020). In Proceedings of the 3rd WebNLG Workshop on Natural Language Generation from the Semantic Web (WebNLG+ 2020), pages 55-76, Dublin, Ireland (Virtual). Association for Computational Linguistics.
  9. [18]18. Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. 2020. Generative pretraining from pixels. In International conference on machine learning, pages 1691-1703. PMLR.
  10. [21]21. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. In NAACL.
  11. [23]23. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wen- zek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoy- anov. 2019. Unsupervised Cross-lingual Representation Learning at Scale. arXiv preprint arXiv:1911.02116. · arXiv
  12. [24]24. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. In Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment: First PASCAL Machine Learning Challenges Workshop, MLCW 2005, Southampton, UK, April 11-13, 2005, Revised Selected Papers, pages 177-190. Springer.
  13. [25]25. Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. 2019. The commitment- bank: Investigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pages 107-124.
  14. [28]28. Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. 2022. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, pages 5547-5569. PMLR.
  15. [37]37. Kenneth Heafield. 2011. KenLM: Faster and Smaller Language Model Queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 187-197, Edinburgh, Scotland. Association for Computational Linguistics.
  16. [38]38. Peter Henderson, Mark Krass, Lucia Zheng, Neel Guha, Christopher D Manning, Dan Jurafsky, and Daniel Ho. 2022. Pile of law: Learning responsible data filtering from the law and a 256gb open-source legal dataset. Advances in Neural Information Processing Systems, 35:29217- 29234.
  17. [44]44. Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016. Summarizing source code using a neural attention model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2073-2083.
  18. [52]52. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
  19. [53]53. Faisal Ladhak, Esin Durmus, Claire Cardie, and Kathleen McKeown. 2020. WikiLin- gua: A new benchmark dataset for cross-lingual abstractive summarization. arXiv preprint arXiv:2010.03093. · arXiv
  20. [54]54. Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, et al. 2022. The BigScience ROOTS Corpus: A 1.6 TB Composite Multilingual Dataset. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
  21. [59]59. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
  22. [65]65. Risto Luukkonen, Ville Komulainen, Jouni Luoma, Anni Eskelinen, Jenna Kanerva, Hanna- Mari Kupari, Filip Ginter, Veronika Laippala, Niklas Muennighoff, Aleksandra Piktus, Thomas Wang, Nouamane Tazi, Teven Le Scao, Thomas Wolf, Osma Suominen, Samuli Sairanen, Mikko Merioksa, Jyrki Heinonen, Aija Vahtola, Samuel Antao, and Sampo Pyysalo. 2023. FinGPT: Large Generative Models for a Small Language. In The 2023 Conference on Empirical Methods in Natural Language Processing.
  23. [69]69. Nasrin Mostafazadeh, Michael Roth, Annie Louis, Nathanael Chambers, and James Allen. 2017. Lsdsem 2017 shared task: The story cloze test. In Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics, pages 46-51.
  24. [75]75. Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. 2021. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021(12):124003.
  25. [76]76. Preetum Nakkiran, Behnam Neyshabur, and Hanie Sedghi. 2021. The Deep Bootstrap Frame- work: Good Online Learners are Good Offline Generalizers. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
  26. [77]77. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.
  27. [78]78. Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. 2021. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-15.
  28. [79]79. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A New Benchmark for Natural Language Understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics.
  29. [83]83. Pedro Javier Ortiz Su'arez, Laurent Romary, and Benoit Sagot. 2020. A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1703-1714, Online. Association for Computational Linguistics.
  30. [84]84. Pedro Javier Ortiz Su'arez, Benoit Sagot, and Laurent Romary. 2019. Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures. Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, pages 9 -16, Mannheim. Leibniz-Institut f"ur Deutsche Sprache.
  31. [88]88. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
  32. [90]90. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1-67.
  33. [91]91. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505-3506.
  34. [92]92. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011. Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning. In AAAI spring symposium: logical formalizations of commonsense reasoning, pages 90-95.
  35. [93]93. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99- 106.
  36. [94]94. Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2022. Multitask Prompted Training Enables Zero-Shot Task Generalization. In The Tenth International Conference on Learning Representations.
  37. [111]111. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  38. [120]120. Johannes Welbl, Nelson F Liu, and Matt Gardner. 2017. Crowdsourcing multiple choice science questions. arXiv preprint arXiv:1707.06209. · arXiv
  39. [122]122. Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart Van Merriënboer, Armand Joulin, and Tomas Mikolov. 2015. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698. · arXiv
  40. [127]127. Ge Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. 2021. Tuning large neural networks via zero-shot hyperparameter transfer. Advances in Neural Information Processing Systems, 34:17084-17097.
  41. [129]129. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a Machine Really Finish Your Sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

Authors
Muennighoff, N., Rush, A. M., Barak, B., Scao, T. L., Piktus, A., Tazi, N., Pyysalo, S., Wolf, T., & Raffel, C.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Pages
50358-50376
ISSN
1049-5258 (ISSN-L)
DOI
10.48550/arxiv.2305.16264
arXiv
2305.16264
OpenAlex
W7133188041
Cited by
49
References
0
Citations by year
2026: 9 · 2025: 21 · 2024: 15 · 2023: 4
FWCI
8.68
Type
conference paper
Language
en
Open access
Licence
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, other-oa)
  • UTUPub (University of Turku) (submittedVersion, other-oa)
  • IRIS Research product catalog (Sapienza University of Rome) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (2134157)
  • U.S. Department of Energy (DESC0022199)
  • European Commission (101070350)
  • China Scholarship Council
  • Defense Advanced Research Projects Agency (W911NF2010021, DE-SC0022199)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
language model scaling, training data efficiency, large language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-11.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-11.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review