CoChat
Sign Up
PreprintCited by 16Topic ModelingNatural Language Processing TechniquesSpeech Recognition and Synthesis

Algorithmic progress in language models

Open access
Ho, Anson, Besiroglu, Tamay, Erdil, Ege, Owen, David, Rahman, Robi, Guo, Zifan Carl, Atkinson, David, Thompson, Neil +1 more·2024·arXiv
DOI 10.48550/arxiv.2403.05812arXiv 2403.05812
Open this paper in CoChat — free accountOpen PDF

Abstract

We investigate the rate at which algorithms for pre-training language models have improved since the advent of deep learning. Using a dataset of over 200 language model evaluations on Wikitext and Penn Treebank spanning 2012-2023, we find that the compute required to reach a set performance threshold has halved approximately every 8 months, with a 95% confidence interval of around 5 to 14 months, substantially faster than hardware gains per Moore's Law. We estimate augmented scaling laws, which enable us to quantify algorithmic progress and determine the relative contributions of scaling models versus innovations in training algorithms. Despite the rapid pace of algorithmic progress and the development of new architectures such as the transformer, our analysis reveals that the increase in compute made an even larger contribution to overall performance improvements over this time period. Though limited by noisy benchmark data, our analysis quantifies the rapid progress in language modeling, shedding light on the relative contributions from compute and algorithms.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2403.05812

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Algorithmic improvements have historically decreased the compute requirements for a fixed performance level and can substitute for compute limitations.

    “However, algorithmic improvements, which have historically decreased the compute requirements for a fixed performance level [48] [49], can substitute for compute limitations.”

    From Measuring AI Ability to Complete Long Software Tasks (2025) · cites this paper as [49]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

  • Scaling Laws for Neural Language Models

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 10× · see citing works

    “For one, measured perplexity on benchmarks can depend on the context length (Kaplan et al. 2020; Xiong et al. 2023).”
    Influential

Flagged first, then by citations

Research works printed but not linked: 40

  1. Alon, Uri et al. (2022). 'Neuro-symbolic language modeling with automaton-augmented retrieval'. In: International Conference on Machine Learning . PMLR, pp. 468-485.
  2. Baevski, Alexei and Michael Auli (2018). 'Adaptive Input Representations for Neural Language Modeling'. In: DOI: 10.48550/ARXIV.1809.10853 . URL: https://arxiv.org/abs/1809.10853 . · DOI
  3. Bixby, Robert E (2012). 'A brief history of linear and mixed-integer programming computation'. In: Documenta Mathematica 2012, pp. 107-121.
  4. Brown, Tom B. et al. (2020). 'Language Models are Few-Shot Learners'. In: DOI: 10.48550/ARXIV.2005.14165 . URL: https://arxiv.org/abs/2005.14165 . · DOI · Paper page
  5. Chowdhery, Aakanksha et al. (2023). 'Palm: Scaling language modeling with pathways'. In: Journal of Machine Learning Research 24.240, pp. 1-113.
  6. Cobbe, Karl et al. (2021). 'Training Verifiers to Solve Math Word Problems'. In: DOI: 10.48550/ARXIV.2110.14168 . URL: https://arxiv.org/abs/2110.14168 . · DOI · Paper page
  7. Dao, Tri et al. (2022). 'FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness'. In: ArXiv abs/2205.14135. URL: https://api.semanticscholar.org/CorpusID:249151871 . · arXiv
  8. Davidson, Tom et al. (2023). 'AI capabilities can be significantly improved without expensive retraining'. In: arXiv preprint arXiv:2312.07413 . · arXiv
  9. Dorner, Florian E (2021). 'Measuring progress in deep reinforcement learning sample efficiency'. In: arXiv preprint arXiv:2102.04881 . · arXiv
  10. Droppo, Jasha and Oguz H. Elibol (2021). 'Scaling Laws for Acoustic Models'. In: Interspeech . URL: https: //api.semanticscholar.org/CorpusID:235458551 .
  11. Erdil, Ege and Tamay Besiroglu (2022). 'Algorithmic progress in computer vision'. In: DOI: 10.48550/ARXIV.2212. 05153 . URL: https://arxiv.org/abs/2212.05153 . · DOI
  12. Fichte, Johannes K, Markus Hecher, and Stefan Szeider (2020). 'A time leap challenge for SAT-solving'. In: International Conference on Principles and Practice of Constraint Programming . Springer, pp. 267-285.
  13. Gao, Leo, Stella Biderman, et al. (2020). 'The Pile: An 800GB dataset of diverse text for language modeling'. In: arXiv preprint arXiv:2101.00027 . · arXiv
  14. Gu, Albert and Tri Dao (2023). 'Mamba: Linear-Time Sequence Modeling with Selective State Spaces'. In: ArXiv abs/2312.00752. URL: https://api.semanticscholar.org/CorpusID:265551773 . · arXiv
  15. Gunasekar, Suriya et al. (2023). 'Textbooks Are All You Need'. In: arXiv preprint arXiv:2306.11644 . · arXiv
  16. Hernandez, Danny and Tom Brown (2020). 'Measuring the algorithmic efficiency of neural networks'. In: arXiv preprint arXiv:2005.04305 . · arXiv
  17. Hoffmann, Jordan et al. (2022). 'Training Compute-Optimal Large Language Models'. In: arXiv preprint arXiv:2203.15556 . · arXiv
  18. Huang, Jie and Kevin Chen-Chuan Chang (2022). 'Towards Reasoning in Large Language Models: A Survey'. In: arXiv preprint arXiv:2212.10403 . · arXiv
  19. Jing, Kun and Jungang Xu (2019). 'A survey on neural network language models'. In: arXiv preprint arXiv:1906.03591 . · arXiv
  20. Koch, Thorsten et al. (2022). 'Progress in mathematical programming solvers from 2001 to 2020'. In: EURO Journal on Computational Optimization 10, p. 100031.
  21. LeCun, Yann et al. (1989). 'Backpropagation applied to handwritten zip code recognition'. In: Neural computation 1.4, pp. 541-551.
  22. Li, Yujia et al. (2022). 'Competition-level code generation with alphacode'. In: arXiv preprint arXiv:2203.07814 . · arXiv
  23. Liu, Hao et al. (2024). 'World Model on Million-Length Video And Language With RingAttention'. In: ArXiv abs/2402.08268. URL: https://api.semanticscholar.org/CorpusID:267637090 . · arXiv
  24. Marion, Max et al. (2023). 'When less is more: Investigating data pruning for pretraining llms at scale'. In: arXiv preprint arXiv:2309.04564 . · arXiv
  25. Merity, Stephen et al. (2016). 'Pointer sentinel mixture models'. In: arXiv preprint arXiv:1609.07843 . · arXiv
  26. Mialon, Grégoire et al. (2023). 'Augmented language models: a survey'. In: arXiv preprint arXiv:2302.07842 . · arXiv
  27. Muennighoff, Niklas et al. (2023). 'Scaling Data-Constrained Language Models'. In: ArXiv abs/2305.16264. URL: https://api.semanticscholar.org/CorpusID:258888192 . · arXiv
  28. Rae, Jack W. et al. (2021). 'Scaling Language Models: Methods, Analysis & Insights from Training Gopher'. In: DOI: 10.48550/ARXIV.2112.11446 . URL: https://arxiv.org/abs/2112.11446 . · DOI · Paper page
  29. Shazeer, Noam (2019). 'Fast transformer decoding: One write-head is all you need'. In: arXiv preprint arXiv:1911.02150 . · arXiv
  30. Sherry, Yash and Neil C Thompson (2021). 'How fast do algorithms improve?[point of view]'. In: Proceedings of the IEEE 109.11, pp. 1768-1777.
  31. Sorscher, Ben et al. (2022). 'Beyond neural scaling laws: beating power law scaling via data pruning'. In: Advances in Neural Information Processing Systems 35, pp. 19523-19536.
  32. Sun, Kaili, Xudong Luo, and Michael Y Luo (2022). 'A survey of pretrained language models'. In: Knowledge Science, Engineering and Management: 15th International Conference, KSEM 2022, Singapore, August 6-8, 2022, Proceedings, Part II . Springer, pp. 442-456.
  33. Takase, Sho, Jun Suzuki, and Masaaki Nagata (2018). 'Direct Output Connection for a High-Rank Language Model'. In: ArXiv abs/1808.10143. URL: https://api.semanticscholar.org/CorpusID:52138320 . · arXiv
  34. Tay, Yi et al. (2022). 'Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?' In: ArXiv abs/2207.10551. URL: https://api.semanticscholar.org/CorpusID:250920512 . · arXiv
  35. Taylor, Ann, Mitchell Marcus, and Beatrice Santorini (2003). 'The Penn treebank: an overview'. In: Treebanks: Building and using parsed corpora , pp. 5-22.
  36. Trinh, Trieu H et al. (2024). 'Solving olympiad geometry without human demonstrations'. In: Nature 625.7995, pp. 476-482.
  37. Vaswani, Ashish et al. (2017). 'Attention is All you Need'. In: ArXiv abs/1706.03762. · arXiv
  38. Wang, Chenguang, Mu Li, and Alexander J Smola (2019). 'Language models with transformers'. In: arXiv preprint arXiv:1904.09408 . · arXiv
  39. Xiong, Wenhan et al. (2023). 'Effective Long-Context Scaling of Foundation Models'. In: ArXiv abs/2309.16039. URL: https://api.semanticscholar.org/CorpusID:263134982 . · arXiv
  40. Zhao, Wayne Xin et al. (2023). 'A survey of large language models'. In: arXiv preprint arXiv:2303.18223 . · arXiv

Cited by

Sorted by citations

Authors
Ho, A., Besiroglu, T., Erdil, E., Owen, D., Rahman, R., Guo, Z. C., Atkinson, D., Thompson, N., & Sevilla, J.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
Pages
58245-58283
ISSN
1049-5258 (ISSN-L)
DOI
10.48550/arxiv.2403.05812
arXiv
2403.05812
OpenAlex
W7103757841
Cited by
16
References
1
Citations by year
2026: 5 · 2025: 6 · 2024: 5
FWCI
4.72
Type
conference paper
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Keywords
language models, large language models, machine translation, neural machine translation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review