CoChat
Sign Up
PreprintCited by 77Stochastic Gradient Optimization TechniquesDomain Adaptation and Few-Shot LearningNeural Networks and Applications

The DOI matches its DataCite record; 2 of the 2 references it cites were found in a registry or an index.

On Warm-Starting Neural Network Training

Open access
Ash, Jordan T., Adams, Ryan P.·2019·arXiv
DOI 10.48550/arxiv.1910.08475arXiv 1910.08475
Open this paper in CoChat — free accountOpen PDF

Abstract

In many real-world deployments of machine learning systems, data arrive piecemeal. These learning scenarios may be passive, where data arrive incrementally due to structural properties of the problem (e.g., daily financial data) or active, where samples are selected according to a measure of their quality (e.g., experimental design). In both of these cases, we are building a sequence of models that incorporate an increasing amount of data. We would like each of these models in the sequence to be performant and take advantage of all the data that are available to that point. Conventional intuition suggests that when solving a sequence of related optimization problems of this form, it should be possible to initialize using the solution of the previous iterate -- to "warm start" the optimization rather than initialize from scratch -- and see reductions in wall-clock time. However, in practice this warm-starting seems to yield poorer generalization performance than models that have fresh random initializations, even though the final training losses are similar. While it appears that some hyperparameter settings allow a practitioner to close this generalization gap, they seem to only do so in regimes that damage the wall-clock gains of the warm start. Nevertheless, it is highly desirable to be able to warm-start neural network training, as it would dramatically reduce the resource usage associated with the construction of performant deep learning systems. In this work, we take a closer look at this empirical phenomenon and try to understand when and how it occurs. We also provide a surprisingly simple trick that overcomes this pathology in several important situations, and present experiments that elucidate some of its properties.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1910.08475

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The Shrink and Perturb algorithm performed well in Continual ImageNet.

    “The Shrink and Perturb algorithm11, which includes L2 regularization, also performed well.”

    From Loss of plasticity in deep continual learning (2024) · cites this paper as [11]

    Section 4 (p. 6) reports that shrink and perturb closed the generalization gap and sped up training in a sequential CIFAR-10 experiment. Our copy does not establish that it performed well in Continual ImageNet. Section 4.1 (p. 7) describes repeated shrink and perturb as similar to noisy L2 regularization, rather than stating that the algorithm includes L2 regularization. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • This type of loss of plasticity is different from the type in which the system can keep optimizing new objectives but lose the ability to generalize.

    “This is different from the type of loss of plasticity in which the system can keep optimizing new objectives but lose the ability to generalize11,12.”

    From Loss of plasticity in deep continual learning (2024) · cites this paper as [11]

  • Shrink and Perturb reduced loss of plasticity.

    “Another existing method that reduced loss of plasticity is Shrink and Perturb11 (orange line in Figs. 1 and 2).”

    From Loss of plasticity in deep continual learning (2024) · cites this paper as [11]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 44

  1. [1][1] Xinran He, Junfeng Pan, Ou Jin, Tianbing Xu, Bo Liu, Tao Xu, Yanxin Shi, Antoine Atallah, Ralf Herbrich, Stuart Bowers, et al. Practical lessons from predicting clicks on ads at Facebook. In Workshop on Data Mining for Online Advertising , pages 1-9. ACM, 2014. -
  2. [2][2] Badrish Chandramouli, Justin J Levandoski, Ahmed Eldawy, and Mohamed F Mokbel. StreamRec: a real-time recommender system. In International Conference on Management of Data , pages 1243-1246. ACM, 2011. -
  3. [3][3] Ludmila I Kuncheva. Classifier ensembles for detecting concept change in streaming data: Overview and perspectives. In 2nd Workshop SUEMA , volume 2008, pages 5-10, 2008. -
  4. [5][5] Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. Green AI. arXiv preprint arXiv:1907.10597 , 2019. - · arXiv
  5. [6][6] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations , 2018. -
  6. [7][7] Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. arXiv preprint arXiv:1906.03671 , 2019. - · arXiv
  7. [10][10] Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio. Why does unsupervised pre-training help deep learning? Journal of Machine Learning Research , 11(Feb):625-660, 2010. -
  8. [11][11] Yoshua Bengio. Deep learning of representations for unsupervised and transfer learning. In International Conference on Unsupervised and Transfer Learning Workshop , pages 17-37. JMLR, 2011. -
  9. [12][12] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, and Daan Wierstra. Matching networks for one shot learning. In Advances in Neural Information Processing Systems , pages 3630-3638, 2016. -
  10. [13][13] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems , pages 4077-4087, 2017. -
  11. [14][14] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning , pages 1126-1135. JMLR. org, 2017. -
  12. [15][15] Jordan T Ash, Robert E Schapire, and Barbara E Engelhardt. Unsupervised domain adaptation using approximate label matching. arXiv preprint arXiv:1602.04889 , 2016. - · arXiv
  13. [17][17] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations , 2014. -
  14. [18][18] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 , 2015. - · arXiv
  15. [19][19] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. International Conference on Learning Representations , 2016. -
  16. [20][20] Yuanzhi Li, Tengyu Ma, and Hongyang Zhang. Algorithmic regularization in over-parameterized matrix sensing and neural networks with quadratic activations. pages 75:2-47, 2018. -
  17. [21][21] Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. In Advances in Neural Information Processing Systems , pages 6151-6159, 2017. -
  18. [22][22] Anders Krogh and John A Hertz. A simple weight decay can improve generalization. In Advances in Neural Information Processing Systems , pages 950-957, 1992. -
  19. [23][23] Gabriel Pereyra, George Tucker, Jan Chorowski, Łukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.06548 , 2017. - · arXiv
  20. [24][24] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations , 2014. -
  21. [25][25] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. arXiv preprint arXiv:2001.06782 , 2020. - · arXiv
  22. [26][26] Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep networks. In International Conference on Learning Representations , 2018. -
  23. [27][27] Alex Nichol and John Schulman. Reptile: a scalable metalearning algorithm. arXiv preprint arXiv:1803.02999 , 2, 2018. - · arXiv
  24. [28][28] Lili Mou, Zhao Meng, Rui Yan, Ge Li, Yan Xu, Lu Zhang, and Zhi Jin. How transferable are neural networks in NLP applications? In Empirical Methods in Natural Language Processing , pages 478-489, 2016. -
  25. [29][29] Bo-Yu Chu, Chia-Hua Ho, Cheng-Hao Tsai, Chieh-Yen Lin, and Chih-Jen Lin. Warm start for parameter selection of linear classifiers. In International Conference on Knowledge Discovery and Data Mining , pages 149-158. ACM, 2015. -
  26. [30][30] Dennis DeCoste and Kiri Wagstaff. Alpha seeding for support vector machines. In International Conference on Knowledge Discovery and Data Mining , pages 345-349. ACM, 2000. -
  27. [31][31] Zeyi Wen, Bin Li, Ramamohanarao Kotagiri, Jian Chen, Yawen Chen, and Rui Zhang. Improving efficiency of SVM k -fold cross-validation by alpha seeding. In Thirty-First AAAI Conference on Artificial Intelligence , 2017. -
  28. [32][32] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 , 2016. - · arXiv
  29. [33][33] Pranav Shyam, Wojciech Ja´ skowski, and Faustino Gomez. Model-based active exploration. In International Conference on Machine Learning , pages 5779-5788, 2018. -
  30. [34][34] Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International Conference on Machine Learning , pages 1139-1147, 2013. -
  31. [35][35] Rupesh K Srivastava, Klaus Greff, and J¨ urgen Schmidhuber. Training very deep networks. In Advances in Neural Information Processing Systems , pages 2377-2385, 2015. -
  32. [36][36] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In International Conference on Artificial Intelligence and Statistics , pages 249-256, 2010. -
  33. [37][37] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In IEEE International Conference on Computer Vision , pages 1026-1034, 2015. -
  34. [38][38] Vaishnavh Nagarajan and J Zico Kolter. Generalization in deep networks: The role of distance from initialization. In Neural Information Processing Systems , 2017. -
  35. [39][39] Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds for neural networks. In Advances in Neural Information Processing Systems , pages 6240-6249, 2017. -
  36. [40][40] Colin Wei, Jason D Lee, Qiang Liu, and Tengyu Ma. On the margin theory of feedforward neural networks. arXiv preprint arXiv:1810.05369 , 2018. - · arXiv
  37. [41][41] Sepp Hochreiter and J Schmidhuber. Flat minima. Neural Computation , 9(1):1-42, 1997. -
  38. [42][42] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems , pages 6389-6399, 2018. -
  39. [43][43] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations , 2017. -
  40. [44][44] Tengyuan Liang, Tomaso Poggio, Alexander Rakhlin, and James Stokes. Fisher-Rao metric, geometry, and complexity of neural networks. pages 888-896, 2017. -
  41. [45][45] Roman Novak, Yasaman Bahri, Daniel A Abolafia, Jeffrey Pennington, and Jascha SohlDickstein. Sensitivity and generalization in neural networks: an empirical study. In International Conference on Learning Representations , 2018. -
  42. [46][46] Wenda Zhou, Victor Veitch, Morgane Austern, Ryan P Adams, and Peter Orbanz. Non-vacuous generalization bounds at the ImageNet scale: a PAC-Bayesian compression approach. In International Conference on Learning Representations , 2019. -
  43. [47][47] Hengduo Li, Bharat Singh, Mahyar Najibi, Zuxuan Wu, and Larry S Davis. An analysis of pre-training on object detection. arXiv preprint arXiv:1904.05871 , 2019. - · arXiv
  44. [48][48] Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks. In Advances in neural information processing systems , pages 153-160, 2007.

Cited by

Sorted by citations

Metadata

Authors
Ash, J. T., & Adams, R. P.
Affiliations
  • Microsoft (United States) (US)
  • Princeton University (US)
ORCID
Adams, Ryan P.
Venue
arXiv
Publisher
arXiv
Published
2019
Volume
33
Pages
3884-3894
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1910.08475
arXiv
1910.08475
OpenAlex
W3048000637
Cited by
77
References
2
Citations by year
2025: 15 · 2024: 20 · 2023: 15 · 2022: 11 · 2021: 8 · 2020: 8
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • neural information processing systems
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
Keywords
warm start, neural network training, incremental learning, active learning (machine learning), generalization gap, random initialization, hyperparameters, training efficiency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review