CoChat
Sign Up
PreprintCited by 25Stochastic Gradient Optimization TechniquesDomain Adaptation and Few-Shot LearningTopic Modeling

The DOI matches its DataCite record; 2 of the 2 references it cites were found in a registry or an index.

Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scaling

Open access
Xiao, Lechao·2024·arXiv
DOI 10.48550/arxiv.2409.15156arXiv 2409.15156
Open this paper in CoChat — free accountOpen PDF

Abstract

The remarkable success of large language pretraining and the discovery of scaling laws signify a paradigm shift in machine learning. Notably, the primary objective has evolved from minimizing generalization error to reducing approximation error, and the most effective strategy has transitioned from regularization (in a broad sense) to scaling up models. This raises a critical question: Do the established principles that proved successful in the generalization-centric era remain valid in this new era of scaling? This paper examines several influential regularization-based principles that may no longer hold true in the scaling-centric, large language model (LLM) era. These principles include explicit L2 regularization and implicit regularization through small batch sizes and large learning rates. Additionally, we identify a new phenomenon termed ``scaling law crossover,'' where two scaling curves intersect at a certain scale, implying that methods effective at smaller scales may not generalize to larger ones. Together, these observations highlight two fundamental questions within this new paradigm: $\bullet$ Guiding Principles for Scaling: If regularization is no longer the primary guiding principle for model design, what new principles are emerging to guide scaling? $\bullet$ Model Comparison at Scale: How to reliably and effectively compare models at the scale where only a single experiment is feasible?

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2409.15156

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 A proposal to reconcile 'Classical' Machine Learning (U-shape), ImageNet-scale Deep Learning (Second-descent) and Internet-scale Deep Learning (Skydiving).
Figure 1 A proposal to reconcile 'Classical' Machine Learning (U-shape), ImageNet-scale Deep Learning (Second-descent) and Internet-scale Deep Learning (Skydiving).CC-BY · From the PDF
Figure 2 Generalization vs Scaling Paradigms.
Figure 2 Generalization vs Scaling Paradigms.CC-BY · From the PDF
Figure 3 U-Shaped (a) and double-descent curves (b). Figure is from Belkin et al. (2019)
Figure 3 U-Shaped (a) and double-descent curves (b). Figure is from Belkin et al. (2019)CC-BY · From the PDF
Figure 4 Learning Dynamics: Generalization (Image Classification) vs. Scaling (Language Model Pretraining). (a) ResNet-18 on CIFAR-10. Training and test error curves initially overlap, then diverge, forming a generalization gap. Minimizing this gap is the central objective as the network easily interpolates training data. (b) . Decoder-only transformer on C4. Evaluation curves consistently remain within training curves throughout training, even when the model size and compute is scaled up by a factor of 500 and 250,000, respectively.
Figure 4 Learning Dynamics: Generalization (Image Classification) vs. Scaling (Language Model Pretraining). (a) ResNet-18 on CIFAR-10. Training and test error curves initially overlap, then diverge, forming a generalization gap. Minimizing this gap is the central objective as the network easily interpolates training data. (b) . Decoder-only transformer on C4. Evaluation curves consistently remain within training curves throughout training, even when the model size and compute is scaled up by a factor of 500 and 250,000, respectively.CC-BY · From the PDF
Figure 6 ResNet-50 on ImageNet. L2 regularization reduces overfitting and improves generalization.
Figure 6 ResNet-50 on ImageNet. L2 regularization reduces overfitting and improves generalization.CC-BY · From the PDF
Figure 5 Training dynamics of four transformer models. From left to right: no L2 and no weight decay, small L2 and no weight decay, no L2 but with weight decay, with both L2 and weight decay.
Figure 5 Training dynamics of four transformer models. From left to right: no L2 and no weight decay, small L2 and no weight decay, no L2 but with weight decay, with both L2 and weight decay.CC-BY · From the PDF
Figure 7 Large Learning Rate Improves Generalization. (a) Networks trained with large or small learning rates can achieve perfect training accuracy, but large learning rates generalize better. (b) The optimal learning rate is often near the maximum stable rate.
Figure 7 Large Learning Rate Improves Generalization. (a) Networks trained with large or small learning rates can achieve perfect training accuracy, but large learning rates generalize better. (b) The optimal learning rate is often near the maximum stable rate.CC-BY · From the PDF
Figure 8 Optimal Learning Rates Are Significantly Lower Than Maximal Stable Learning Rates. (a) Loss vs. learning rate curves reveal U-shaped relationships, with optimal learning rates far below stability limits. (b) Despite smooth training curves, maximal stable rates consistently underperform.
Figure 8 Optimal Learning Rates Are Significantly Lower Than Maximal Stable Learning Rates. (a) Loss vs. learning rate curves reveal U-shaped relationships, with optimal learning rates far below stability limits. (b) Despite smooth training curves, maximal stable rates consistently underperform.CC-BY · From the PDF
Figure 9 Small Batch Size performs better for ImageNet. Small batch size achieves better Test Error (left) but worse Training Error (right), vice versa for large batch size.
Figure 9 Small Batch Size performs better for ImageNet. Small batch size achieves better Test Error (left) but worse Training Error (right), vice versa for large batch size.CC-BY · From the PDF
Figure 10 Optimal Batch Size. Learning rate search for various batch sizes. (a) 19M model, (b) 151M model. (c) Performance as a function of batch size is U-shaped and is sensitive to the choice of batch size. Both large and small batch sizes lead to sub-optimal performance.
Figure 10 Optimal Batch Size. Learning rate search for various batch sizes. (a) 19M model, (b) 151M model. (c) Performance as a function of batch size is U-shaped and is sensitive to the choice of batch size. Both large and small batch sizes lead to sub-optimal performance.CC-BY · From the PDF
Figure 11 Training Instability. (a) The model with D = 3072 deviates from the expected scaling trend. (b) Closer examination of the training dynamics reveals that the D = 3072 model exhibits instability. This instability is mitigated by the application of QK-Norm. (c) With QK-Norm, the scaling law exhibits a normal and smooth behavior.
Figure 11 Training Instability. (a) The model with D = 3072 deviates from the expected scaling trend. (b) Closer examination of the training dynamics reveals that the D = 3072 model exhibits instability. This instability is mitigated by the application of QK-Norm. (c) With QK-Norm, the scaling law exhibits a normal and smooth behavior.CC-BY · From the PDF
Figure 12 Constant Learning Rate vs LR = 2 /D . A constant learning rate LR = 2 / 1024 and a learning rate of LR = 2 /D yield nearly identical performance at small scales ( D ≤ 1024). However, their performance diverges at larger scales ( D > 1024), with LR = 2 /D demonstrating superior performance both (b) with and (a) without QK-Norm.
Figure 12 Constant Learning Rate vs LR = 2 /D . A constant learning rate LR = 2 / 1024 and a learning rate of LR = 2 /D yield nearly identical performance at small scales ( D ≤ 1024). However, their performance diverges at larger scales ( D > 1024), with LR = 2 /D demonstrating superior performance both (b) with and (a) without QK-Norm.CC-BY · From the PDF
Figure 13 Crossover Phenomenon due to Sub-optimal Weight Decay. (a) A constant weight decay strategy initially outperforms a decaying weight decay approach at smaller scales. However, this advantage diminishes and reverses at larger scales, demonstrating a crossover phenomenon. (b) With constant weight decay, parameter norms are significantly suppressed throughout training. (c) Decaying the weight decay alongside the learning rate results in less suppressed parameter norms.
Figure 13 Crossover Phenomenon due to Sub-optimal Weight Decay. (a) A constant weight decay strategy initially outperforms a decaying weight decay approach at smaller scales. However, this advantage diminishes and reverses at larger scales, demonstrating a crossover phenomenon. (b) With constant weight decay, parameter norms are significantly suppressed throughout training. (c) Decaying the weight decay alongside the learning rate results in less suppressed parameter norms.CC-BY · From the PDF
Figure 14 Gradient Normalization Leads to Scaling Law Crossover? (a) An ablation study showing Proposal Blue performs better at small scales. In particular, gradient normalization not only improves performance but also reduces learning rate sensitivity in the scale we are testing. (b) Performance gain from Proposal Blue does not transfer to large scale. A crossover occurs around 2 -3 × 10 2 exaflops.
Figure 14 Gradient Normalization Leads to Scaling Law Crossover? (a) An ablation study showing Proposal Blue performs better at small scales. In particular, gradient normalization not only improves performance but also reduces learning rate sensitivity in the scale we are testing. (b) Performance gain from Proposal Blue does not transfer to large scale. A crossover occurs around 2 -3 × 10 2 exaflops.CC-BY · From the PDF
Figure 15 Scaling Law Extrapolation : Naively fitting a power law to observed data points can lead to inaccurate extrapolations. For example, our 'Blue' and 'Red' proposals showed significant deviations after 10x and 30x compute extrapolation, respectively.
Figure 15 Scaling Law Extrapolation : Naively fitting a power law to observed data points can lead to inaccurate extrapolations. For example, our 'Blue' and 'Red' proposals showed significant deviations after 10x and 30x compute extrapolation, respectively.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 89

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. · arXiv
  2. Ben Adlam and Jeffrey Pennington. Understanding double descent requires a fine-grained bias-variance decomposition. Advances in neural information processing systems, 33:11022-11032, 2020.
  3. Atish Agarwala, Fabian Pedregosa, and Jeffrey Pennington. Second-order regression models exhibit pro- gressive sharpening to the edge of stability. arXiv preprint arXiv:2210.04860, 2022. · arXiv
  4. Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sang- hai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245, 2023. · arXiv
  5. Maksym Andriushchenko, Francesco D'Angelo, Aditya Varre, and Nicolas Flammarion. Why do we need weight decay in modern deep learning? arXiv preprint arXiv:2310.04415, 2023. · arXiv
  6. Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: A family of highly capable mul- timodal models. arXiv preprint arXiv:2312.11805, 1, 2023. · arXiv
  7. Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, 121(27):e2311878121, 2024.
  8. Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regres- sion. Proceedings of the National Academy of Sciences, 117(48):30063-30070, 2020.
  9. Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short- term memory. arXiv preprint arXiv:2405.04517, 2024. · arXiv
  10. Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning prac- tice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, 116(32): 15849-15854, 2019.
  11. BIG bench authors. Beyond the imitation game: Quantifying and extrapolating the capabilities of lan- guage models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https: //openreview.net/forum?id=uyTL5Bvosj.
  12. Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024. · arXiv
  13. Charlie Blake, Constantin Eichenberg, Josef Dean, Lukas Balles, Luke Y Prince, Björn Deiseroth, An- dres Felipe Cruz-Salinas, Carlo Luschi, Samuel Weinbach, and Douglas Orr. u-µp: The unit-scaled max- imal update parametrization. arXiv preprint arXiv:2407.17465, 2024. · arXiv
  14. Blake Bordelon, Lorenzo Noci, Mufan Bill Li, Boris Hanin, and Cengiz Pehlevan. Depthwise hyperparame- ter transfer in residual networks: Dynamics and scaling limit. arXiv preprint arXiv:2309.16620, 2023. · arXiv
  15. Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. A dynamical model of neural scaling laws. arXiv preprint arXiv:2402.01092, 2024. · arXiv
  16. Aleksandar Botev, Soham De, Samuel L Smith, Anushan Fernando, George-Cristian Muraru, Ruba Haroun, Leonard Berrada, Razvan Pascanu, Pier Giuseppe Sessa, Robert Dadashi, et al. Recurrent- gemma: Moving past transformers for efficient open language models. arXiv preprint arXiv:2404.07839, 2024. · arXiv
  17. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learn- ers. Advances in neural information processing systems, 33:1877-1901, 2020.
  18. Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, et al. Symbolic discovery of optimization algorithms. Advances in neural information processing systems, 36, 2024.
  19. Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Ad- vances in neural information processing systems, 32, 2019.
  20. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
  21. Jeremy Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. In International Conference on Learning Representa- tions, 2020.
  22. Jeremy M Cohen, Behrooz Ghorbani, Shankar Krishnan, Naman Agarwal, Sourabh Medapati, Michal Badura, Daniel Suo, David Cardoze, Zachary Nado, George E Dahl, et al. Adaptive gradient methods at the edge of stability. arXiv preprint arXiv:2207.14484, 2022. · arXiv
  23. Alex Damian, Eshaan Nichani, and Jason D Lee. Self-stabilization: The implicit bias of gradient descent at the edge of stability. arXiv preprint arXiv:2209.15594, 2022. · arXiv
  24. DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Ji- aqi Ni, Jiashi Li, Jin Chen, Jingyang Yuan, Junjie Qiu, Junxiao Song, Kai Dong, Kaige Gao, Kang Guan, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruizhe Pan, Runxin Xu, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Size Zheng, T. Wang, Tian Pei, Tian Yuan, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wei An, Wen Liu, Wen- feng Liang, Wenjun Gao, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxi- ang Wang, Xin Liu, Xin Xie, Xingkai Yu, Xinnan Song, Xinyi Zhou, Xinyu Yang, Xuan Lu, Xuecheng Su, Y. Wu, Y. K. Li, Y. X. Wei, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Zheng, Yichao Zhang, Yiliang Xiong, Yilong Zhao, Ying He, Ying Tang, Yishi Piao, Yixin Dong, Yixuan Tan, Yiyuan Liu, Yongji Wang, Yongqiang Guo, Yuchen Zhu, Yuduan Wang, Yuheng Zou, Yukun Zha, Yunxian Ma, Yuting Yan, Yuxiang You, Yuxuan Liu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhewen Hao, Zhihong Shao, Zhiniu Wen, Zhipeng Xu, Zhongyu Zhang, Zhuoshu Li, Zihan Wang, Zihui Gu, Zilin Li, and Ziwei Xie. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024. URL https://arxiv.org/abs/2405.04434. · arXiv
  25. Aaron Defazio, Harsh Mehta, Konstantin Mishchenko, Ahmed Khaled, Ashok Cutkosky, et al. The road less scheduled. arXiv preprint arXiv:2405.15682, 2024. · arXiv
  26. Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, An- dreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, Rodolphe Jenatton, Lu- cas Beyer, Michael Tschannen, Anurag Arnab, Xiao Wang, Carlos Riquelme Ruiz, Matthias Minderer, Joan Puigcerver, Utku Evci, Manoj Kumar, Sjoerd Van Steenkiste, Gamaleldin Fathy Elsayed, Aravindh Mahendran, Fisher Yu, Avital Oliver, Fantine Huot, Jasmijn Bastings, Mark Collier, Alexey A. Grit- senko, Vighnesh Birodkar, Cristina Nader Vasconcelos, Yi Tay, Thomas Mensink, Alexander Kolesnikov, Filip Pavetic, Dustin Tran, Thomas Kipf, Mario Lucic, Xiaohua Zhai, Daniel Keysers, Jeremiah J. Harmsen, and Neil Houlsby. Scaling vision transformers to 22 billion parameters. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 7480-7512. PMLR, 23-29 Jul 2023. URL https://proceedings.mlr. press/v202/dehghani23a.html.
  27. Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In International Conference on Machine Learning, pp. 1019-1028. PMLR, 2017.
  28. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. · arXiv
  29. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Let- man, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. · arXiv
  30. Katie E Everett, Lechao Xiao, Mitchell Wortsman, Alexander A Alemi, Roman Novak, Peter J Liu, Izzed- din Gur, Jascha Sohl-Dickstein, Leslie Pack Kaelbling, Jaehoon Lee, and Jeffrey Pennington. Scaling exponents across parameterizations and optimizers. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=0ksNeD1SJT.
  31. Alessandro Favero, Francesco Cagnetta, and Matthieu Wyart. Locality defeats the curse of dimensionality in convolutional teacher-student scenarios. Advances in Neural Information Processing Systems, 34: 9456-9467, 2021.
  32. Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2020. · arXiv
  33. Gabriel Goh. Why momentum really works. Distill, 2017. doi: 10.23915/distill.00006. URL http:// distill.pub/2017/momentum. · DOI
  34. Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. · arXiv
  35. Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning, pp. 1842-1850. PMLR, 2018.
  36. Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statisti- cal learning: data mining, inference, and prediction, volume 2. Springer, 2009.
  37. Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high-dimensional ridgeless least squares interpolation. Annals of statistics, 50(2):949, 2022.
  38. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
  39. Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016. · arXiv
  40. Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017. · arXiv
  41. Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation, 9(1):1-42, 1997.
  42. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Ruther- ford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute- optimal large language models. arXiv preprint arXiv:2203.15556, 2022. · arXiv
  43. Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generaliza- tion in neural networks. Advances in neural information processing systems, 31, 2018.
  44. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. · arXiv
  45. Dhruva Karkada. The lazy (ntk) and rich (µ-p) regimes: a gentle tutorial. arXiv preprint arXiv:2404.19719, 2024. · arXiv
  46. Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016. · arXiv
  47. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. · arXiv
  48. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
  49. Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems, 32, 2019.
  50. Jaehoon Lee, Samuel Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstein. Finite versus infinite neural networks: an empirical study. Advances in Neural Information Processing Systems, 33:15156-15172, 2020.
  51. Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. The large learn- ing rate phase of deep learning: the catapult mechanism. arXiv preprint arXiv:2003.02218, 2020. · arXiv
  52. Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learn- ing rate in training neural networks. Advances in neural information processing systems, 32, 2019.
  53. Licong Lin, Jingfeng Wu, Sham M Kakade, Peter L Bartlett, and Jason D Lee. Scaling laws in linear re- gression: Compute, parameters, and data. arXiv preprint arXiv:2406.08466, 2024. · arXiv
  54. Lucas Lingle. A large-scale exploration of µ-transfer. arXiv preprint arXiv:2404.05728, 2024. · arXiv
  55. Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. Understanding the difficulty of training transformers. arXiv preprint arXiv:2004.08249, 2020. · arXiv
  56. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. · arXiv
  57. Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. An empirical model of large- batch training. arXiv preprint arXiv:1812.06162, 2018. · arXiv
  58. Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymp- totics and the double descent curve. Communications on Pure and Applied Mathematics, 75(4):667-766, 2022.
  59. Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Learning with invariances in random features and kernel models. In Conference on Learning Theory, pp. 3351-3418. PMLR, 2021.
  60. Theodor Misiakiewicz and Song Mei. Learning with convolution and pooling operations in kernel methods. Advances in Neural Information Processing Systems, 35:29014-29025, 2022.
  61. Igor Molybog, Peter Albert, Moya Chen, Zachary DeVito, David Esiobu, Naman Goyal, Punit Singh Koura, Sharan Narang, Andrew Poulton, Ruan Silva, et al. A theory on adam instability in large-scale machine learning. arXiv preprint arXiv:2304.09871, 2023. · arXiv
  62. Behnam Neyshabur. Implicit regularization in deep learning. arXiv preprint arXiv:1709.01953, 2017. · arXiv
  63. Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. Towards un- derstanding the role of over-parametrization in generalization of neural networks. arXiv preprint arXiv:1805.12076, 2018. · arXiv
  64. R OpenAI et al. Gpt-4 technical report. ArXiv, 2303:08774, 2023.
  65. Elliot Paquette, Courtney Paquette, Lechao Xiao, and Jeffrey Pennington. 4+ 3 phases of compute- optimal neural scaling laws. arXiv preprint arXiv:2405.15074, 2024. · arXiv
  66. Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale, 2024. URL https://arxiv.org/abs/2406.17557. · arXiv
  67. Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, and Yair Carmon. Resolving discrepan- cies in compute-optimal scaling of language models. arXiv preprint arXiv:2406.19146, 2024. · arXiv
  68. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language mod- els are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  69. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text trans- former. Journal of machine learning research, 21(140):1-67, 2020.
  70. Christopher J Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E Dahl. Measuring the effects of data parallelism on neural network training. Journal of Machine Learning Research, 20(112):1-49, 2019.
  71. Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150, 2019. · arXiv
  72. Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020. · arXiv
  73. Samuel L Smith and Quoc V Le. A bayesian perspective on generalization and stochastic gradient descent. arXiv preprint arXiv:1710.06451, 2017. · arXiv
  74. Jascha Sohl-Dickstein, Roman Novak, Samuel S Schoenholz, and Jaehoon Lee. On the infinite width limit of neural networks with a standard parameterization. arXiv preprint arXiv:2001.07301, 2020. · arXiv
  75. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. · arXiv
  76. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing sys- tems, 30, 2017.
  77. Nikhil Vyas, Depen Morwani, Rosie Zhao, Gal Kaplun, Sham Kakade, and Boaz Barak. Beyond implicit bias: The insignificance of sgd noise in online learning. arXiv preprint arXiv:2306.08590, 2023. · arXiv
  78. Xi Wang and Laurence Aitchison. How to set adamw's weight decay as you scale model and dataset size. arXiv preprint arXiv:2405.13698, 2024. · arXiv
  79. Blake Woodworth, Suriya Gunasekar, Jason D. Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Jacob Abernethy and Shivani Agarwal (eds.), Proceedings of Thirty Third Conference on Learning Theory, vol- ume 125 of Proceedings of Machine Learning Research, pp. 3635-3673. PMLR, 09-12 Jul 2020. URL https://proceedings.mlr.press/v125/woodworth20a.html.
  80. Mitchell Wortsman, Peter J Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, et al. Small-scale proxies for large-scale transformer training instabilities. arXiv preprint arXiv:2309.14322, 2023. · arXiv
  81. Lechao Xiao. Eigenspace restructuring: a principle of space and frequency in neural networks. In Confer- ence on Learning Theory, pp. 4888-4944. PMLR, 2022.
  82. Greg Yang and Edward J Hu. Tensor programs iv: Feature learning in infinite-width neural networks. In International Conference on Machine Learning, pp. 11727-11737. PMLR, 2021.
  83. Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero- shot hyperparameter transfer. arXiv preprint arXiv:2203.03466, 2022. · arXiv
  84. Greg Yang, Dingli Yu, Chen Zhu, and Soufiane Hayou. Tensor programs vi: Feature learning in infinite- depth neural networks. arXiv preprint arXiv:2310.02244, 2023. · arXiv
  85. Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
  86. Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger Grosse. Three mechanisms of weight decay regular- ization. arXiv preprint arXiv:1810.12281, 2018. · arXiv
  87. Shuo Zhang, Boci Peng, Xinping Zhao, Boren Hu, Yun Zhu, Yanjia Zeng, and Xuming Hu. Llasa: Large language and e-commerce shopping assistant. arXiv preprint arXiv:2408.02006, 2024. · arXiv
  88. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models, 2022. URL https://arxiv. org/abs/2205.01068, 3:19-0, 2023.
  89. Rosie Zhao, Depen Morwani, David Brandfonbrener, Nikhil Vyas, and Sham Kakade. Deconstructing what makes a good optimizer for language models. arXiv preprint arXiv:2407.07972, 2024. · arXiv

Cited by

See all 25

Metadata

Authors
Xiao, L.
Corresponding author
Xiao, Lechao
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2409.15156
arXiv
2409.15156
OpenAlex
W4403781043
Cited by
25
References
2
Citations by year
2026: 8 · 2025: 14 · 2024: 3
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
DeepMind
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
large language models, scaling laws, approximation error, generalization error, regularization, Tikhonov regularization, implicit regularization, small batch sizes, learning rate, model scaling

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review