CoChat
Sign Up
PreprintCited by 551Advanced Bandit Algorithms ResearchMachine Learning and Data ClassificationStochastic Gradient Optimization Techniques

BOHB: Robust and Efficient Hyperparameter Optimization at Scale

Open access
Falkner, Stefan, Klein, Aaron, Hutter, Frank·2018·arXiv
DOI 10.48550/arxiv.1807.01774arXiv 1807.01774
Open this paper in CoChat — free accountOpen PDF

Abstract

Modern deep learning methods are very sensitive to many hyperparameters, and, due to the long training times of state-of-the-art models, vanilla Bayesian hyperparameter optimization is typically computationally infeasible. On the other hand, bandit-based configuration evaluation approaches based on random search lack guidance and do not converge to the best configurations as quickly. Here, we propose to combine the benefits of both Bayesian optimization and bandit-based methods, in order to achieve the best of both worlds: strong anytime performance and fast convergence to optimal configurations. We propose a new practical state-of-the-art hyperparameter optimization method, which consistently outperforms both Bayesian optimization and Hyperband on a wide range of problem types, including high-dimensional toy functions, support vector machines, feed-forward neural networks, Bayesian neural networks, deep reinforcement learning, and convolutional neural networks. Our method is robust and versatile, while at the same time being conceptually simple and easy to implement.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1807.01774

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Hyperband allocates resources through early stopping, and BOHB combines model-guided search with bandit resource allocation.

    “Hyperband allocates resources through early stopping (Li et al., 2018), and BOHB combines model-guided search with bandit resource allocation (Falkner et al., 2018).”

    From SGM: A Statistical Godel Machine for Risk-Controlled Recursive Self-Modification (2025) · cites this paper as Falkner et al., 2018

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 3 · Cited by 551

Research works printed but not linked: 42

  1. Proceedings of the International Conference on Learning Representations (ICLR'17) , 2017. Published online: iclr.cc .
  2. Bach, F. and Blei, D. (eds.). Proceedings of the 32nd International Conference on Machine Learning (ICML'15) , volume 37, 2015. Omnipress.
  3. Bergstra, J., Bardenet, R., Bengio, Y ., and Kégl, B. Algorithms for hyper-parameter optimization. In Shawe-Taylor, J., Zemel, R., Bartlett, P., Pereira, F., and Weinberger, K. (eds.), Proceedings of the 25th International Conference on Advances in Neural Information Processing Systems (NIPS'11) , pp. 2546-2554, 2011.
  4. Bergstra, J., Yamins, D., and Cox, D. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In Dasgupta, S. and McAllester, D. (eds.), Proceedings of the 30th International Conference on Machine Learning (ICML'13) , pp. 115-123. Omnipress, 2014.
  5. Bertrand, H., Ardon, R., Perrot, M., and Bloch, I. Hyperparameter optimization of deep neural networks: Combining hyperband with Bayesian model selection. Proceedings of Conférence sur l'Apprentissage Automatique (CAP 2017) , 2017.
  6. Brochu, E., Cora, V., and de Freitas, N. A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv:1012.2599, 2010. · arXiv
  7. Cattral, R., Oppacher, F., and Deugo, D. Evolutionary data mining with automatic rule generalization. Recent Advances in Computers, Computing and Communications , 1(1):296-300, 2002.
  8. Chen, T., Fox, E., and Guestrin, C. Stochastic gradient Hamiltonian Monte Carlo. In Xing, E. and Jebara, T. (eds.), Proceedings of the 31th International Conference on Machine Learning, (ICML'14) . Omnipress, 2014.
  9. DeVries, T. and Taylor, G. W. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 , 2017. · arXiv
  10. Eggensperger, K., Feurer, M., Hutter, F., Bergstra, J., Snoek, J., Hoos, H., and Leyton-Brown, K. Towards an empirical foundation for assessing Bayesian optimization of hyperparameters. In NIPS Workshop on Bayesian Optimization in Theory and Practice (BayesOpt'13) , 2013.
  11. Eggensperger, K., Hutter, F., Hoos, H., and Leyton-Brown, K. Efficient benchmarking of hyperparameter optimizers via surrogates. In Bonet, B. and Koenig, S. (eds.), Proceedings of the Twenty-nineth National Conference on Artificial Intelligence (AAAI'15) , pp. 1114-1120. AAAI Press, 2015.
  12. Frey, P. W. and Slate, D. J. Letter recognition using holland-style adaptive classifiers. Machine Learning , 6(2):161-182, Mar 1991.
  13. Gastaldi, X. Shake-shake regularization. arXiv preprint arXiv:1705.07485 , 2017. · arXiv
  14. Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., and Meger, D. Deep reinforcement learning that matters. arXiv preprint arXiv:1709.06560 , 2017. · arXiv
  15. Hutter, F., Hoos, H., and Leyton-Brown, K. Sequential modelbased optimization for general algorithm configuration. In Coello, C. (ed.), Proceedings of the Fifth International Conference on Learning and Intelligent Optimization (LION'11) , volume 6683 of Lecture Notes in Computer Science , pp. 507523. Springer-Verlag, 2011.
  16. Jamieson, K. and Talwalkar, A. Non-stochastic best arm identification and hyperparameter optimization. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics (AISTATS) , 2016.
  17. Kandasamy, K., Dasarathy, G., Schneider, J., and Poczos, B. Multifidelity bayesian optimisation with continuous approximations. arXiv preprint arXiv:1703.06240 , 2017. · arXiv
  18. Klein, A., Falkner, S., Bartels, S., Hennig, P., and Hutter, F. Fast Bayesian hyperparameter optimization on large datasets. Electron. J. Statist. , 11(2):4945-4968, 2017a.
  19. Klein, A., Falkner, S., Mansur, N., and Hutter, F. Robo: A flexible and robust bayesian optimization framework in python. In NIPS 2017 Bayesian Optimization Workshop , December 2017b.
  20. Klein, A., Falkner, S., Springenberg, J. T., and Hutter, F. Learning curve prediction with Bayesian neural networks. In Proceedings of the International Conference on Learning Representations (ICLR'17) icl (2017). Published online: iclr.cc .
  21. Kohavi, R. Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid. In KDD , volume 96, pp. 202-207, 1996.
  22. LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. In Haykin, S. and Kosko, B. (eds.), Intelligent Signal Processing , pp. 306-351. IEEE Press, 2001. URL http://www.iro.umontreal. ca/~lisa/pointeurs/lecun-01a.pdf .
  23. Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., and Talwalkar, A. Hyperband: Bandit-based configuration evaluation for hyperparameter optimization. In Proceedings of the International Conference on Learning Representations (ICLR'17) icl (2017). Published online: iclr.cc .
  24. Li, L., Jamieson, K., Rostamizadeh, A., Gonina, K., Hardt, M., Recht, B., and Talwalkar, A. Massively parallel hyperparameter tuning, 2018. URL https://openreview.net/forum? id=S1Y7OOlRZ .
  25. Liu, C., Zoph, B., Shlens, J., Hua, W., Li, L.-J., Fei-Fei, L., Yuille, A., Huang, J., and Murphy, K. Progressive neural architecture search. arXiv preprint arXiv:1712.00559 , 2017. · arXiv
  26. Melis, G., Dyer, C., and Blunsom, P. On the state of the art of evaluation in neural language models. arXiv preprint arXiv:1707.05589 , 2017. · arXiv
  27. Mendoza, H., Klein, A., Feurer, M., Springenberg, J., and Hutter, F. Towards automatically-tuned neural networks. In ICML 2016 AutoML Workshop , 2016.
  28. Perrone, V., Jenatton, R., Seeger, M., and Archambeau, C. Multiple adaptive bayesian linear regression for scalable bayesian optimization with warm start. arXiv preprint arXiv:1712.02902 , 2017. · arXiv
  29. Poloczek, M., Wang, J., and Frazier, P. Multi-information source optimization. In Advances in Neural Information Processing Systems , pp. 4291-4301, 2017.
  30. Real, E., Aggarwal, A., Huang, Y., and Le, Q. V. Regularized Evolution for Image Classifier Architecture Search. In arXiv:1802.01548 [cs] , February 2018. · arXiv
  31. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017. · arXiv
  32. Seabold, S. and Perktold, J. Statsmodels: Econometric and statistical modeling with python. In 9th Python in Science Conference , 2010.
  33. Shahriari, B., Swersky, K., Wang, Z., Adams, R., and de Freitas, N. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE , 104(1):148-175, 2016.
  34. Snoek, J., Larochelle, H., and Adams, R. P. Practical Bayesian optimization of machine learning algorithms. In Bartlett, P., Pereira, F., Burges, C., Bottou, L., and Weinberger, K. (eds.), Proceedings of the 26th International Conference on Advances in Neural Information Processing Systems (NIPS'12) , pp. 29602968, 2012.
  35. Snoek, J., Rippel, O., Swersky, K., Kiros, R., Satish, N., Sundaram, N., Patwary, M., Prabhat, and Adams, R. Scalable Bayesian optimization using deep neural networks. In Bach & Blei (2015), pp. 2171-2180.
  36. Springenberg, J., Klein, A., Falkner, S., and Hutter, F. Bayesian optimization with robust bayesian neural networks. In Lee, D., Sugiyama, M., von Luxburg, U., Guyon, I., and Garnett, R. (eds.), Proceedings of the 30th International Conference on Advances in Neural Information Processing Systems (NIPS'16) , 2016.
  37. Swersky, K., Snoek, J., and Adams, R. Multi-task Bayesian optimization. In Burges, C., Bottou, L., Welling, M., Ghahramani, Z., and Weinberger, K. (eds.), Proceedings of the 27th International Conference on Advances in Neural Information Processing Systems (NIPS'13) , pp. 2004-2012, 2013.
  38. Swersky, K., Snoek, J., and Adams, R. Freeze-thaw bayesian optimization. arXiv:1406.3896, 2014. · arXiv
  39. Vanschoren, J., van Rijn, J., Bischl, B., and Torgo, L. OpenML: Networked science in machine learning. SIGKDD Explor. Newsl. , 15(2):49-60, June 2014.
  40. Wang, J., Xu, J., and Wang, X. Combination of hyperband and bayesian optimization for hyperparameter optimization in deep learning. arXiv preprint arxiv:1801.01596 , 01 2018. · arXiv
  41. Zoph, B. and Le, Q. V. Neural architecture search with reinforcement learning. In Proceedings of the International Conference on Learning Representations (ICLR'17) icl (2017). Published online: iclr.cc .
  42. Zoph, B., Vasudevan, V., Shlens, J., and Le, Q. V. Learning transferable architectures for scalable image recognition. In arXiv:1707.07012 [cs.CV] , 2017. · arXiv

Authors
Falkner, S., Klein, A., & Hutter, F.
Affiliations
University of Freiburg (DE)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2018
Pages
1437-1446
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1807.01774
arXiv
1807.01774
OpenAlex
W2804268694
Cited by
551
References
3
Citations by year
2026: 6 · 2025: 61 · 2024: 106 · 2023: 119 · 2022: 86 · 2021: 95 · 2020: 52 · 2019: 25 · 2018: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • International Conference on Machine Learning
Funders
  • European Commission (H2020-ICT, 645403)
  • Deutsche Forschungsgemeinschaft (BR 3815/8-1, INST 39/963-1, “INST 39/963-1 FUGG”, SPP 1527, 39/963-1)
  • Horizon 2020 Framework Programme (INST 39/963-1 FUGG)
Topics
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
Keywords
hyperparameter optimization, Bayesian optimization, Hyperband, random search, computational efficiency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review