CoChat
Sign Up
PreprintCited by 5

Gradient-based Hyperparameter Optimization through Reversible Learning

Maclaurin, Dougal, Duvenaud, David, Adams, Ryan P.·2015·arXiv
DOI 10.48550/arxiv.1502.03492arXiv 1502.03492
Open this paper in CoChat — free account

Abstract

Tuning hyperparameters of learning algorithms is hard because gradients are usually unavailable. We compute exact gradients of cross-validation performance with respect to all hyperparameters by chaining derivatives backwards through the entire training procedure. These gradients allow us to optimize thousands of hyperparameters, including step-size and momentum schedules, weight initialization distributions, richly parameterized regularization schemes, and neural network architectures. We compute hyperparameter gradients by exactly reversing the dynamics of stochastic gradient descent with momentum.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1502.03492

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Examples of hyper-parameter selection methods include architecture search guided by hyper-gradients and differential hyper-parameters, as well as genetic algorithm or particle swarm style optimization.

    “Examples include architecture search guided by hyper-gradients and differential hyper-parameters [59] as well as genetic algorithm or particle swarm style optimization [60].”

    From An Introduction to Deep Learning for the Physical Layer (2017) · cites this paper as [59]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 35

  1. Bastien, Frédéric, Lamblin, Pascal, Pascanu, Razvan, Bergstra, James, Goodfellow, Ian J., Bergeron, Arnaud, Bouchard, Nicolas, and Bengio, Yoshua. Theano: new features and speed improvements. Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop, 2012.
  2. Baydin, A. G. and Pearlmutter, B. A. Automatic differenti- ation of algorithms for machine learning. In Proceedings of the AutoML Workshop at the International Conference on Machine Learning (ICML), 2014.
  3. Bengio, Yoshua. Gradient-based optimization of hyperpa- rameters. Neural computation, 12(8):1889-1900, 2000.
  4. Bengio, Yoshua, Simard, Patrice, and Frasconi, Paolo. Learning long-term dependencies with gradient descent is difficult. Neural Networks, IEEE Transactions on, 5 (2):157-166, 1994.
  5. Bergstra, James, Breuleux, Olivier, Bastien, Frédéric, Lamblin, Pascal, Pascanu, Razvan, Desjardins, Guil- laume, Turian, Joseph, Warde-Farley, David, and Ben- gio, Yoshua. Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy), June 2010. Oral Pre- sentation.
  6. Bergstra, James, Bardenet, Rémi, Bengio, Yoshua, Kégl, Balázs, et al. Algorithms for hyper-parameter optimiza- tion. In Advances in Neural Information Processing Sys- tems, 2011.
  7. Bergstra, James, Yamins, Daniel, and Cox, David. Making a science of model search: Hyperparameter optimiza- tion in hundreds of dimensions for vision architectures. In International Conference on Machine Learning, pp. 115-123, 2013.
  8. Chapelle, Olivier, Vapnik, Vladimir, Bousquet, Olivier, and Mukherjee, Sayan. Choosing multiple parameters for support vector machines. Machine learning, 46(1-3): 131-159, 2002.
  9. Chen, Dingding and Hagan, Martin T. Optimal use of reg- ularization and cross-validation in neural network mod- eling. In International Joint Conference on Neural Net- works, volume 2, pp. 1275-1280. IEEE, 1999.
  10. Courbariaux, Matthieu, Bengio, Yoshua, and David, Jean- Pierre. Low precision arithmetic for deep learning. arXiv preprint arXiv:1412.7024, 2014. · arXiv
  11. Dahl, George E, Jaitly, Navdeep, and Salakhutdinov, Rus- lan. Multi-task neural networks for QSAR predictions. arXiv preprint arXiv:1406.1231, 2014. · arXiv
  12. Domke, Justin. Generic methods for optimization-based modeling. In International Conference on Artificial In- telligence and Statistics, pp. 318-326, 2012.
  13. Eigenmann, Robert and Nossek, Josef A. Gradient based adaptive regularization. In Proceedings of the 1999 IEEE Signal Processing Society Workshop on Neural Networks, pp. 87-94. IEEE, 1999.
  14. Erhan, Dumitru, Bengio, Yoshua, Courville, Aaron, Man- zagol, Pierre-Antoine, Vincent, Pascal, and Bengio, Samy. Why does unsupervised pre-training help deep learning? The Journal of Machine Learning Research, 11:625-660, 2010.
  15. Foo, Chuan-sheng, Do, Chuong B., and Ng, Andrew Y. Ef- ficient multiple hyperparameter learning for log-linear models. In Advances in neural information processing systems, pp. 377-384, 2008.
  16. Graves, Alex, Wayne, Greg, and Danihelka, Ivo. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014. · arXiv
  17. Hensman, James and Lawrence, Neil D. Nested variational compression in deep Gaussian processes. arXiv preprint arXiv:1412.1370, 2014. · arXiv
  18. Hochreiter, Sepp and Schmidhuber, Jürgen. Long short- term memory. Neural computation, 9(8):1735-1780, 1997.
  19. Hut, P., Makino, J., and McMillan, S. Building a better leapfrog. Astrophysical Journal, Part 2 -Letters, 443: L93-L96, April 1995. doi: 10.1086/187844. · DOI
  20. Hutter, Frank, Hoos, Holger H, and Leyton-Brown, Kevin. Sequential model-based optimization for general algo- rithm configuration. In Proceedings of LION-5, volume 6683, pp. 507-523. Springer, 2011.
  21. Lake, Brenden M. Towards more human-like concept learning in machines: Compositionality, causality, and learning-to-learn. PhD thesis, Massachusetts Institute of Technology, 2014.
  22. Larsen, Jan, Svarer, Claus, Andersen, Lars Nonboe, and Hansen, Lars Kai. Adaptive regularization in neural net- work modeling. In Neural Networks: Tricks of the Trade, pp. 113-132. Springer, 1998.
  23. MacKay, David J.C. Information theory, inference, and learning algorithms. Cambridge University Press, 2003.
  24. MacKay, David J.C. and Neal, Radford M. Automatic rel- evance determination for neural networks. In Technical Report. Cambridge University, 1994.
  25. Martens, James and Sutskever, Ilya. Training deep and recurrent networks with hessian-free optimization. In Neural Networks: Tricks of the Trade, pp. 479-535. Springer, 2012.
  26. Oliphant, Travis E. Python for scientific computing. Com- puting in Science & Engineering, 9(3):10-20, 2007.
  27. Pascanu, Razvan, Mikolov, Tomas, and Bengio, Yoshua. Understanding the exploding gradient problem. arXiv preprint arXiv:1211.5063, 2012. · arXiv
  28. Pearlmutter, Barak. An investigation of the gradient de- scent process in neural networks. PhD thesis, Carnegie Mellon University, 1996.
  29. Pearlmutter, Barak A. Fast exact multiplication by the Hes- sian. Neural computation, 6(1):147-160, 1994.
  30. Pearlmutter, Barak A. and Siskind, Jeffrey Mark. Reverse- mode AD in a functional framework: Lambda the ulti- mate backpropagator. ACM Transactions on Program- ming Languages and Systems (TOPLAS), 30(2):7, 2008.
  31. Rasmussen, Carl E. and Williams, Christopher K.I. Gaus- sian Processes for Machine Learning. The MIT Press, Cambridge, MA, USA, 2006.
  32. Salimans, Tim, Kingma, Diederik P., and Welling, Max. Markov chain Monte Carlo and variational inference: Bridging the gap. arXiv preprint arXiv:1410.6460, 2014. · arXiv
  33. Solak, E., Murray Smith, R., Leithead, W.E., Leith, D., and Rasmussen, Carl E. Derivative observations in Gaussian process models of dynamic systems. Advances in Neural Information Processing Systems, pp. 1057-1064, 2003.
  34. Sutskever, Ilya, Martens, James, Dahl, George, and Hin- ton, Geoffrey. On the importance of initialization and momentum in deep learning. In Proceedings of the 30th International Conference on Machine Learning (ICML- 13), pp. 1139-1147, 2013.
  35. Sutskever, Ilya, Vinyals, Oriol, and Le, Quoc V. V. Se- quence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27, pp. 3104-3112.

Cited by

Sorted by citations

Authors
Maclaurin, D., Duvenaud, D., & Adams, R. P.
Venue
arXiv
Publisher
arXiv
Published
2015
DOI
10.48550/arxiv.1502.03492
arXiv
1502.03492
References
2
Type
Preprint
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review