Initializing Bayesian Hyperparameter Optimization via Meta-Learning
Open accessThe DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Abstract
Model selection and hyperparameter optimization is crucial in applying machine learning to a novel dataset. Recently, a subcommunity of machine learning has focused on solving this problem with Sequential Model-based Bayesian Optimization (SMBO), demonstrating substantial successes in many applications. However, for computationally expensive algorithms the overhead of hyperparameter optimization can still be prohibitive. In this paper we mimic a strategy human domain experts use: speed up optimization by starting from promising configurations that performed well on similar datasets. The resulting initialization technique integrates naturally into the generic SMBO framework and can be trivially applied to any SMBO method. To validate our approach, we perform extensive experiments with two established SMBO frameworks (Spearmint and SMAC) with complementary strengths; optimizing two machine learning frameworks on 57 datasets. Our initialization procedure yields mild improvements for low-dimensional hyperparameter optimization and substantially improves the state of the art for the more complex combined algorithm selection and hyperparameter optimization problem.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
The WEKA data mining software: an update
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1656274.1656278 · Cited by 17,750 · cited 2×
“performs combined algorithm selection and hyperparameter optimization in the space of algorithms defined by the WEKA package (Hall et al. 2009).”
MethodGaussian Processes for Machine Learning
Rasmussen, C. E., & Williams, C. K. I. (2005) · doi:10.7551/mitpress/3206.001.0001 · Cited by 11,705
Efficient Global Optimization of Expensive Black-Box Functions
Jones, D. R., Schonlau, M., & Welch, W. J. (1998) · Journal of Global Optimization · doi:10.1023/a:1008306431147 · Cited by 8,070 · cited 2×
“A classic approach is to initialize SMBO with a space-filling design (Jones, Schonlau, and Welch 1998).”
Method(2012) · Cited by 8,009
Practical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691
(2011) · Cited by 3,939
Sequential Model-Based Optimization for General Algorithm Configuration
Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-25566-3_40 · Cited by 2,469
Self-citationMachine Learning, Neural and Statistical Classification
Fulkerson, B. (1995) · Technometrics · doi:10.1080/00401706.1995.10484383 · Cited by 2,209
Brochu, E., Cora, V. M., & de Freitas, N. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1012.2599 · Cited by 2,011
Bergstra, J., Yamins, D. L. K., & Cox, D. (2013) · Digital Access to Scholarship at Harvard (DASH) (Harvard University) · Cited by 1,852
Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms
Thornton, C., Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2013) · Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2487575.2487629 · Cited by 1,359 · cited 4×
“Here, we evaluated MI-SMBO on small and mediumsized hyperparameter optimiziation problems, and an important open research question is to extend it to even larger configuration spaces, such as those of Auto-WEKA (Thornton et al. 2013) and Hyperopt-Sklearn (Komer, Bergstra, and Eliasmith 2014).”
MethodInfluentialSelf-citationImproving deep neural networks for LVCSR using rectified linear units and dropout
Dahl, G. E., Sainath, T. N., & Hinton, G. E. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639346 · Cited by 1,310 · cited 1×
“When manually optimizing hyperparameters for D N +1 , they would begin the search with hyperparameter configurations that were optimal for the most similar previous datasets (see, e.g., Dahl, Sainath, and Hinton (2013); Goodfellow et al. (2013)).”
MethodOpenML: networked science in machine learning
Vanschoren, J., van Rijn, J. N., Bischl, B., & Torgo, L. (2014) · ACM SIGKDD Explorations Newsletter · doi:10.1145/2641190.2641198 · Cited by 1,141
Multi-Task Bayesian Optimization
Swersky, K., Snoek, J., & Adams, R. P. (2013) · Digital Access to Scholarship at Harvard (DASH) (Harvard University) · Cited by 502
Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification AlgorithmsPreprint
Thornton, C., Hutter, F., Hoos, H. H., & Leyton‐Brown, K. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1208.3719 · Cited by 374
Self-citationMeta-Learning by Landmarking Various Learning Algorithms
Pfahringer, B., Bensusan, H. N., & Giraud-Carrier, C. G. (2000) · Cited by 302
Hyperopt-Sklearn: Automatic Hyperparameter Configuration for Scikit-Learn
Komer, B., Bergstra, J., & Eliasmith, C. (2014) · Proceedings of the Python in Science Conference · doi:10.25080/majora-14bd3278-006 · Cited by 295
Collaborative hyperparameter tuning
Bardenet, R., Brendel, M., gl, B. Z. K., & le Sebag, M. (2013) · HAL · Cited by 293
Metalearning: Applications to Data Mining
Brazdil, P., Giraud-Carrier, C., Soares, C., & Vilalta, R. (2009) · Cognitive Technologies · doi:10.1007/978-3-540-73263-1 · Cited by 264
Efficient Transfer Learning Method for Automatic Hyperparameter Tuning
Yogatama, D., & Mann, G. (2014) · International Conference on Artificial Intelligence and Statistics · Cited by 254
Meta-learning for evolutionary parameter optimization of classifiers
Reif, M., Faisal Shafait, B., & Dengel, A. R. (2012) · Machine Learning · doi:10.1007/s10994-012-5286-7 · Cited by 193
Combining meta-learning and search techniques to select parameters for support vector machines
Gomes, T. A. F., Prudêncio, R. B. C., Soares, C., Rossi, A. L. D., & de Carvalho, A. C. P. L. F. (2011) · Neurocomputing · doi:10.1016/j.neucom.2011.07.005 · Cited by 178
Selecting Classification Algorithms with Active Testing
Leite, R., Brazdil, P., & Vanschoren, J. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-31537-4_10 · Cited by 125
Using meta-learning to initialize bayesian optimization of hyperparameters
Feurer, M., Springenberg, J. T., & Hutter, F. (2014) · Cited by 108
Self-citationHoffman, M. D., Shahriari, B., & de Freitas, N. (2014) · Oxford University Research Archive (ORA) (University of Oxford) · Cited by 88
Algorithm selection via meta-learning
Kalousis, A. (2002) · Archive ouverte UNIGE (University of Geneva) · doi:10.13097/archive-ouverte/unige:104435 · Cited by 88
OpenML: networked science in machine learning
Vanschoren, J., van Rijn, J. N., Bischl, B., et al. (2014) · doi:10.48550/arxiv.1407.7722 · Cited by 53
Continuous Search in Constraint Programming
Arbeláez, A., Hamadi, Y., & Sébag, M. (2010) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai.2010.17 · Cited by 38
Continuous Search in Constraint Programming
Arbeláez, A., Hamadi, Y., & Sébag, M. (2011) · Springer eBooks · doi:10.1007/978-3-642-21434-9_9 · Cited by 14
Towards Life-Long Meta Learning
Gagliolo, M., & Schmidhuber, J. (2005) · Cited by 1
Cited by
Taking the Human Out of the Loop: A Review of Bayesian Optimization
Shahriari, B., Swersky, K., Wang, Z., Adams, R. P., & de Freitas, N. (2016) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2494218 · Cited by 6,049
Automated Machine Learning: Methods, Systems, Challenges
(2019) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-030-05318-5 · Cited by 1,485
(2015) · neural information processing systems · Cited by 1,305
Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges
Bischl, B., Binder, M., Lang, M., Pielok, T., Richter, J., Coors, S., Thomas, J., Ullmann, T., Becker, M., Boulesteix, A., et al. (2023) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1484 · Cited by 930
Automated machine learning: Review of the state-of-the-art and opportunities for healthcare
Waring, J., Lindvall, C., & Umeton, R. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2020.101822 · Cited by 834
Metadata
- Authors
- Feurer, M., Springenberg, J., & Hutter, F.
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2015-02-16
- Volume
- 29
- Issue
- 1
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W2200000192
- Cited by
- 390
- References
- 32
- FWCI
- 19.13
- Type
- conference paper
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.