The DOI matches its DataCite record; 23 of the 23 references it cites were found in a registry or an index.
Practical Bayesian Optimization of Machine Learning Algorithms
Open accessAbstract
Machine learning algorithms frequently require careful tuning of model hyperparameters, regularization terms, and optimization parameters. Unfortunately, this tuning is often a "black art" that requires expert experience, unwritten rules of thumb, or sometimes brute-force search. Much more appealing is the idea of developing automatic approaches which can optimize the performance of a given learning algorithm to the task at hand. In this work, we consider the automatic tuning problem within the framework of Bayesian optimization, in which a learning algorithm's generalization performance is modeled as a sample from a Gaussian process (GP). The tractable posterior distribution induced by the GP leads to efficient use of the information gathered by previous experiments, enabling optimal choices about what parameters to try next. Here we show how the effects of the Gaussian process prior and the associated inference procedure can have a large impact on the success or failure of Bayesian optimization. We show that thoughtful choices can lead to results that exceed expert-level performance in tuning machine learning algorithms. We also describe new algorithms that take into account the variable cost (duration) of learning experiments and that can leverage the presence of multiple cores for parallel experimentation. We show that these proposed algorithms improve on previous automatic procedures and can reach or surpass human expert-level optimization on a diverse set of contemporary algorithms including latent Dirichlet allocation, structured SVMs and convolutional neural networks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Validation accuracy can also guide hyperparameter and architecture search (Snoek et al., 2012;Zoph & Le, 2017); both uses reduce the validation set to a single number.”
From VERSE: Verified Self-Evolving Optimizer for Agent Harnesses (2026) · cites this paper as Snoek et al., 2012
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,889
Gaussian Processes for Machine Learning
Rasmussen, C. E., & Williams, C. K. I. (2005) · doi:10.7551/mitpress/3206.001.0001 · Cited by 11,705
(2012) · Cited by 8,009
Bayesian calibration of computer models
Kennedy, M. C., & O'Hagan, A. (2001) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/1467-9868.00294 · Cited by 4,222
(2011) · Cited by 3,939
Multi-column deep neural networks for image classification
Ciresan, D. C., Meier, U., & Schmidhuber, J. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6248110 · Cited by 3,880
Brochu, E., Cora, V. M., & de Freitas, N. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1012.2599 · Cited by 2,568
Sequential Model-Based Optimization for General Algorithm Configuration
Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-25566-3_40 · Cited by 2,469
A Taxonomy of Global Optimization Methods Based on Response Surfaces
Jones, D. R. (2001) · Journal of Global Optimization · doi:10.1023/a:1012771025575 · Cited by 2,160
Information-Theoretic Regret Bounds for Gaussian Process Optimization in the Bandit Setting
Srinivas, N., Krause, A., Kakade, S. M., & Seeger, M. (2012) · IEEE Transactions on Information Theory · doi:10.48550/arxiv.0912.3995 · Cited by 2,100
Research works printed but not linked: 2
- Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In ICML , 2010.
- Kevin Miller, M. Pawan Kumar, Benjamin Packer, Danny Goodman, and Daphne Koller. Max-margin minentropy models. In AISTATS , 2012.
Cited by
Sorted by citations
Items 1,001–1,026 of 1,026
Machine learning prediction of tau‐PET in Alzheimer's disease using plasma, MRI, and clinical data
Karlsson, L., Vogel, J., Arvidsson, I., Åström, K., Strandberg, O., Seidlitz, J., Bethlehem, R. A. I., Stomrud, E., Ossenkoppele, R., Ashton, N. J., et al. (2025) · Alzheimer's & Dementia · doi:10.1002/alz.14600 · Cited by 43
Zhang, X., Kuenzel, S., Colombo, N., & Watkins, C. (2022) · Journal of Modern Power Systems and Clean Energy · doi:10.35833/mpce.2021.000276 · Cited by 43
Borra, D., Fantozzi, S., & Magosso, E. (2021) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2021.655840 · Cited by 36
“In addition, automatic hyper-parameter search (Snoek et al., 2012) will be exploited to further improve the MS-EEGNet design and other explanation techniques, such as layer-wise relevance propagation, will be investigated, carefully analyzing the effect of different propagation rules and parameters…”
MethodTian, H., Chen, S.-C., & Shyu, M.-L. (2020) · Information Systems Frontiers · doi:10.1007/s10796-020-10023-6 · Cited by 34
Ding, H., Wang, B., Hamel, A. P., Melkonyan, M., Ang, T. F. A., Au, R., & Lin, H. (2023) · Frontiers in Dementia · doi:10.3389/frdem.2023.1271680 · Cited by 32
Chmiel, F. P., Burns, D. K., Azor, M., Borca, F., Boniface, M. J., Zlatev, Z. D., White, N. M., Daniels, T. W. V., & Kiuber, M. (2020) · doi:10.1101/2020.12.02.20239194 · Cited by 30
The Use of Machine Learning Models with Optuna in Disease Prediction
Lai, L.-H., Lin, Y.-L., Liu, Y.-H., Lai, J.-P., Yang, W.-C., Hou, H.-P., & Pai, P.-F. (2024) · Electronics · doi:10.3390/electronics13234775 · Cited by 30
Villanueva Zacarias, A. G., Reimann, P., & Mitschang, B. (2018) · Procedia CIRP · doi:10.1016/j.procir.2018.03.215 · Cited by 26
Bayesian Optimization of HPC Systems for Energy Efficiency
Miyazaki, T., Sato, I., & Shimizu, N. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-92040-5_3 · Cited by 27
GPTuner: An LLM-Based Database Tuning System
Lao, J., Wang, Y., Li, Y., Wang, J., Zhang, Y., Cheng, Z., Chen, W., Tang, M., & Wang, J. (2025) · ACM SIGMOD Record · doi:10.1145/3733620.3733641 · Cited by 17
Stacking machine learning algorithms for biomarker-based preoperative diagnosis of a pelvic mass
Shaw, R., Lokshin, A. E., Miller, M. C., Messerlian-Lambert, G., & Moore, R. G. (2022) · Cancers · doi:10.3390/cancers14051291 · Cited by 16
Kelley, J., Schneider, V., Poll, G., & Marian, M. (2024) · Tribology International · doi:10.1016/j.triboint.2024.109988 · Cited by 20
Hidden Incentives for Auto-Induced Distributional Shift
Krueger, D., Maharaj, T., & Leike, J. (2020) · arXiv · doi:10.48550/arxiv.2009.09153 · Cited by 14
“Two distinctive features of PBT are notable because they give the OL more control than most meta-learning algorithms (e.g. Bayesian optimization (Snoek et al., 2012)) over the dynamics and outcome of the learning process: (1) OL applies optimization to parameters, not just hyperparam-eters.”
BackgroundMethodAn Explainable Machine Learning Model for Early Prediction of Sepsis Using ICU Data
Nesaragi, N., & Patidar, S. (2021) · Infections and Sepsis Development · doi:10.5772/intechopen.98957 · Cited by 11
Green Containerized Service Consolidation in Cloud
Nath, S. B., Addya, S. K., Chakraborty, S., & Ghosh, S. K. (2020) · ICC 2020 - 2020 IEEE International Conference on Communications (ICC) · doi:10.1109/icc40277.2020.9149173 · Cited by 11
Nafea, M. S., & Ismail, Z. H. (2025) · AI · doi:10.3390/ai6060120 · Cited by 8
Solís-Martín, D., Galán-Páez, J., & Borrego-Díaz, J. (2025) · Mathematics · doi:10.3390/math13040555 · Cited by 7
Automatic Discovery of Privacy–Utility Pareto Fronts
Avent, B., González, J., Diethe, T., Paleyes, A., & Balle, B. (2020) · Proceedings on Privacy Enhancing Technologies · doi:10.2478/popets-2020-0060 · Cited by 3
VERSE: Verified Self-Evolving Optimizer for Agent Harnesses
Wang, Z., Ge, Y., Wang, Z., Wang, H., Xu, Y., Frandsen, J., Fu, S., Wilson, A. C., & Reddy, C. K. (2026) · arXiv · doi:10.48550/arxiv.2610.02616
“Validation accuracy can also guide hyperparameter and architecture search (Snoek et al., 2012; Zoph & Le, 2017); both uses reduce the validation set to a single number.”
(2014) · Cited by 33,419
(2013) · Cited by 2,056
Predictive Entropy Search for Efficient Global Optimization of Black-box Functions
Hernández-Lobato, J. M., Hoffman, M. W., & Ghahramani, Z. (2014) · arXiv · doi:10.48550/arxiv.1406.2541 · Cited by 407
(2017) · Networked Systems Design and Implementation · Cited by 329
On the Robustness of Interpretability Methods
Alvarez-Melis, D., & Jaakkola, T. S. (2018) · arXiv · doi:10.48550/arxiv.1806.08049 · Cited by 310
Progressive Neural Architecture Search
Liu, C., Zoph, B., Neumann, M., Shlens, J., Hua, W., Li, L.-J., Fei-Fei, L., Yuille, A., Huang, J., & Murphy, K. (2017) · arXiv · doi:10.48550/arxiv.1712.00559 · Cited by 171
Hyperparameter ensembles for robustness and uncertainty quantification
Wenzel, F., Snoek, J., Tran, D., & Jenatton, R. (2020) · arXiv · doi:10.48550/arxiv.2006.13570 · Cited by 122
Self-citation
Metadata
- Authors
- Snoek, J., Larochelle, H., & Adams, R. P.
- Affiliations
- University of Toronto (CA)
- Université de Sherbrooke (CA)
- Harvard University (US)
- ORCID
- Adams, Ryan P.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2012
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1206.2944
- OpenAlex
- W2131241448
- Cited by
- 6,691
- References
- 23
- Citations by year
- 2026: 62 · 2025: 630 · 2024: 779 · 2023: 712 · 2022: 720 · 2021: 997 · 2020: 864 · 2019: 680 · 2018: 465 · 2017: 289 · 2016: 212 · 2015: 144 · 2014: 94 · 2013: 34 · 2012: 9
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Funders
- Defense Advanced Research Projects Agency
- Natural Sciences and Engineering Research Council of Canada
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- Bayesian optimization, hyperparameter optimization, Gaussian process priors, latent Dirichlet allocation, structured support vector machines, convolutional neural networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
