CoChat
Sign Up
PreprintCited by 6,691Advanced Bandit Algorithms ResearchMachine Learning and Data ClassificationMachine Learning and Algorithms

The DOI matches its DataCite record; 23 of the 23 references it cites were found in a registry or an index.

Practical Bayesian Optimization of Machine Learning Algorithms

Open access
Snoek, Jasper, Larochelle, Hugo, Adams, Ryan P.·2012·arXiv
DOI 10.48550/arxiv.1206.2944arXiv 1206.2944
Open this paper in CoChat — free accountOpen PDF

Abstract

Machine learning algorithms frequently require careful tuning of model hyperparameters, regularization terms, and optimization parameters. Unfortunately, this tuning is often a "black art" that requires expert experience, unwritten rules of thumb, or sometimes brute-force search. Much more appealing is the idea of developing automatic approaches which can optimize the performance of a given learning algorithm to the task at hand. In this work, we consider the automatic tuning problem within the framework of Bayesian optimization, in which a learning algorithm's generalization performance is modeled as a sample from a Gaussian process (GP). The tractable posterior distribution induced by the GP leads to efficient use of the information gathered by previous experiments, enabling optimal choices about what parameters to try next. Here we show how the effects of the Gaussian process prior and the associated inference procedure can have a large impact on the success or failure of Bayesian optimization. We show that thoughtful choices can lead to results that exceed expert-level performance in tuning machine learning algorithms. We also describe new algorithms that take into account the variable cost (duration) of learning experiments and that can leverage the presence of multiple cores for parallel experimentation. We show that these proposed algorithms improve on previous automatic procedures and can reach or surpass human expert-level optimization on a diverse set of contemporary algorithms including latent Dirichlet allocation, structured SVMs and convolutional neural networks.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1206.2944

Cites

23 of 23 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 23 references

Research works printed but not linked: 2

  1. Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In ICML , 2010.
  2. Kevin Miller, M. Pawan Kumar, Benjamin Packer, Danny Goodman, and Daphne Koller. Max-margin minentropy models. In AISTATS , 2012.

Cited by

Sorted by citations

See all 6,691

Related

Metadata

Authors
Snoek, J., Larochelle, H., & Adams, R. P.
Affiliations
  • University of Toronto (CA)
  • Université de Sherbrooke (CA)
  • Harvard University (US)
ORCID
Adams, Ryan P.
Venue
arXiv
Publisher
arXiv
Published
2012
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1206.2944
arXiv
1206.2944
OpenAlex
W2131241448
Cited by
6,691
References
23
Citations by year
2026: 62 · 2025: 630 · 2024: 779 · 2023: 712 · 2022: 720 · 2021: 997 · 2020: 864 · 2019: 680 · 2018: 465 · 2017: 289 · 2016: 212 · 2015: 144 · 2014: 94 · 2013: 34 · 2012: 9
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
  • arXiv (Cornell University)
Funders
  • Defense Advanced Research Projects Agency
  • Natural Sciences and Engineering Research Council of Canada
Topics
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
Keywords
Bayesian optimization, hyperparameter optimization, Gaussian process priors, latent Dirichlet allocation, structured support vector machines, convolutional neural networks

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review