The DOI matches its Crossref record; 201 of the 201 references it cites were found in a registry or an index.
Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges
Open accessAbstract
Most machine learning algorithms are configured by a set of hyperparameters whose values must be carefully chosen and which often considerably impact performance. To avoid a time‐consuming and irreproducible manual process of trial‐and‐error to find well‐performing hyperparameter configurations, various automatic hyperparameter optimization (HPO) methods—for example, based on resampling error estimation for supervised machine learning—can be employed. After introducing HPO from a general perspective, this paper reviews important HPO methods, from simple techniques such as grid or random search to more advanced methods like evolution strategies, Bayesian optimization, Hyperband, and racing. This work gives practical recommendations regarding important choices to be made when conducting HPO, including the HPO algorithms themselves, performance evaluation, how to combine HPO with machine learning pipelines, runtime improvements, and parallelization. This article is categorized under: Algorithmic Development > Statistics Technologies > Machine Learning Technologies > Prediction
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482
LIBSVM: A library for support vector machines
Chang, C.-C., & Lin, C.-J. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1961189.1961199 · Cited by 40,918
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970 · cited 1×
“Hastie et al. (2009) suggest the “ one-standard-error rule ” and use the simplest model within one standard error of the performance of the best model.”
MethodRegularization and Variable Selection Via the Elastic Net
Zou, H., & Hastie, T. (2005) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00503.x · Cited by 20,957
A decision-theoretic generalization of on-line learning and an application to boosting
Freund, Y., & Schapire, R. E. (1997) · Journal of Computer and System Sciences · doi:10.1006/jcss.1997.1504 · Cited by 20,466
Regularization Paths for Generalized Linear Models via Coordinate Descent
Friedman, J., Hastie, T., & Tibshirani, R. (2010) · Journal of Statistical Software · doi:10.18637/jss.v033.i01 · Cited by 19,927
Nearest neighbor pattern classification
Cover, T., & Hart, P. (1967) · IEEE Transactions on Information Theory · doi:10.1109/tit.1967.1053964 · Cited by 16,936
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
A training algorithm for optimal margin classifiers
Boser, B. E., Guyon, I. M., & Vapnik, V. N. (1992) · Proceedings of the fifth annual workshop on Computational learning theory · doi:10.1145/130385.130401 · Cited by 11,992
Proceedings of the 24th international conference on Machine learning
(2007) · doi:10.1145/1273496 · Cited by 11,630
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
Kohavi, R. (1995) · Zenodo · doi:10.5281/zenodo.19712698 · Cited by 10,909
Efron, B., Hastie, T., Johnstone, I., & Tibshirani, R. (2004) · The Annals of Statistics · doi:10.1214/009053604000000067 · Cited by 9,357
(2012) · Cited by 9,062
Building Predictive Models in R Using the caret Package
Kuhn, M. (2008) · Journal of Statistical Software · doi:10.18637/jss.v028.i05 · Cited by 8,863
Larian Studios Dev Team (2024) · doi:10.1145/3292500.3330701 · Cited by 8,618
Efficient Global Optimization of Expensive Black-Box Functions
Jones, D. R., Schonlau, M., & Welch, W. J. (1998) · Journal of Global Optimization · doi:10.1023/a:1008306431147 · Cited by 8,070
(2012) · Cited by 8,009
Wolpert, D. H. (1992) · Neural Networks · doi:10.1016/s0893-6080(05)80023-1 · Cited by 7,951
Nelder, J. A., & Wedderburn, R. W. M. (1972) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2344614 · Cited by 7,194
Friedman, J., Hastie, T., & Tibshirani, R. (2000) · The Annals of Statistics · doi:10.1214/aos/1016218223 · Cited by 6,884
Practical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691
Proceedings of the 25th international conference on Machine learning - ICML '08
(2008) · doi:10.1145/1390156 · Cited by 6,680
Learning Transferable Architectures for Scalable Image Recognition
Zoph, B., Vasudevan, V., Shlens, J., & Le, Q. V. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00907 · Cited by 6,249
Evolutionary Algorithms for Solving Multi-Objective Problems
(2007) · Genetic and Evolutionary Computation Series · doi:10.1007/978-0-387-36797-2 · Cited by 5,686
SGDR: Stochastic Gradient Descent with Warm Restarts
Loshchilov, I., & Hutter, F. (2016) · arXiv · doi:10.48550/arxiv.1608.03983 · Cited by 5,309
Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods
Platt, J. C. (1999) · Cited by 4,827
Support Vector Regression Machines
Drucker, H., Burges, C. J. C., Kaufman, L., Smola, A. J., & Vapnik, V. N. (1996) · Cited by 4,658
The Strength of Weak Learnability
Schapire, R. E. (1990) · Machine Learning · doi:10.1007/bf00116037 · Cited by 4,600
Unbiased Recursive Partitioning: A Conditional Inference Framework
Hothorn, T., Hornik, K., & Zeileis, A. (2006) · Journal of Computational and Graphical Statistics · doi:10.1198/106186006x133933 · Cited by 4,021
McKay, M. D., Beckman, R. J., & Conover, W. J. (1979) · Technometrics · doi:10.2307/1268522 · Cited by 3,757
5th International Conference on Learning Representations (ICLR 17)
Havrylov, S., & Titov, I. (2017) · Cited by 3,503
DARTS: Differentiable architecture search
Liu, H., Simonyan, K., & Yang, Y. (2018) · arXiv · doi:10.48550/arxiv.1806.09055 · Cited by 3,248
On hyperparameter optimization of machine learning algorithms: Theory and practice
Yang, L., & Shami, A. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.07.061 · Cited by 3,232
Adaptive Control Processes: A Guided Tour
Bellman, R. E. (1961) · doi:10.1515/9781400874668 · Cited by 2,802
(1999) · doi:10.1162/089976600300015565 · Cited by 2,799
Evolution strategies – A comprehensive introduction
Beyer, H.-G., & Schwefel, H.-P. (2002) · Natural Computing · doi:10.1023/a:1015059928466 · Cited by 2,790
Sequential Model-Based Optimization for General Algorithm Configuration
Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-25566-3_40 · Cited by 2,469
Learning from imbalanced data: open challenges and future directions
Krawczyk, B. (2016) · Progress in Artificial Intelligence · doi:10.1007/s13748-016-0094-0 · Cited by 2,459
Advances in Neural Information Processing Systems 32 (NeurIPS 2019)
Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché–Buc, F., Fox, E. B., & Garnett, R. (2020) · HAL · Cited by 2,450
A Taxonomy of Global Optimization Methods Based on Response Surfaces
Jones, D. R. (2001) · Journal of Global Optimization · doi:10.1023/a:1012771025575 · Cited by 2,159
Proceedings of the 19th International Conference on Artificial Intelligence and Statistics 2016
Murray, I., & Graham, M. M. (2016) · Cited by 2,115
Automatic Differentiation in Machine Learning: a Survey
Baydin, A. G., Pearlmutter, B. A., Radul, A., & Siskind, J. M. (2015) · Maynooth University ePrints and eTheses Archive (Maynooth University) · Cited by 2,014
Estimation of Distribution Algorithms: A New Tool for Evolutionary Computation
Larraanaga, P., & Lozano, J. A. (2001) · Kluwer Academic Publishers eBooks · Cited by 1,914
Information-Theoretic Regret Bounds for Gaussian Process Optimization in the Bandit Setting
Srinivas, N., Krause, A., Kakade, S. M., & Seeger, M. W. (2012) · IEEE Transactions on Information Theory · doi:10.1109/tit.2011.2182033 · Cited by 1,850
MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
Chen, T., Li, M., Li, Y., Lin, M., Wang, N., Wang, M., Xiao, T., Xu, B., Zhang, C., & Zhang, Z. (2015) · arXiv · doi:10.48550/arxiv.1512.01274 · Cited by 1,819
kernlab- An S4 Package for Kernel Methods in R
Karatzoglou, A., Smola, A., Hornik, K., & Zeileis, A. (2004) · Journal of Statistical Software · doi:10.18637/jss.v011.i09 · Cited by 1,786
Neural Architecture Search: A Survey
Elsken, T., Metzen, J. H., & Hutter, F. (2018) · arXiv · doi:10.48550/arxiv.1808.05377 · Cited by 1,763
The irace package: Iterated racing for automatic algorithm configuration
López-Ibáñez, M., Dubois-Lacoste, J., Pérez Cáceres, L., Birattari, M., & Stützle, T. (2016) · Operations Research Perspectives · doi:10.1016/j.orp.2016.09.002 · Cited by 1,761
Graph-based approach for airborne light detection and ranging segmentation
Vilariño, D. L., Cabaleiro, J. C., Martínez, J., Rivera, F. F., & Pena, T. F. (2017) · Journal of Applied Remote Sensing · doi:10.1117/1.jrs.11.015020 · Cited by 1,721
AutoML: A survey of the state-of-the-art
He, X., Zhao, K., & Chu, X. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106622 · Cited by 1,691
Boosting a Weak Learning Algorithm by Majority
Freund, Y. (1995) · Information and Computation · doi:10.1006/inco.1995.1136 · Cited by 1,674
DEAP: evolutionary algorithms made easy
Fortin, F.-A., Rainville, F.-M. D., Gardner, M.-A., Parizeau, M., & Gagné, C. (2012) · Cited by 1,642
Automated Machine Learning: Methods, Systems, Challenges
(2019) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-030-05318-5 · Cited by 1,485
Feurer, M., & Hutter, F. (2019) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-030-05318-5_1 · Cited by 1,414
Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms
Thornton, C., Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2013) · Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2487575.2487629 · Cited by 1,359
Prediction error estimation: a comparison of resampling methods
Molinaro, A. M., Simon, R., & Pfeiffer, R. M. (2005) · Bioinformatics · doi:10.1093/bioinformatics/bti499 · Cited by 1,339
(2015) · neural information processing systems · Cited by 1,305
The Algorithm Selection Problem
Rice, J. R. (1976) · Advances in Computers · doi:10.1016/s0065-2458(08)60520-3 · Cited by 1,195
OpenML: networked science in machine learning
Vanschoren, J., van Rijn, J. N., Bischl, B., & Torgo, L. (2014) · ACM SIGKDD Explorations Newsletter · doi:10.1145/2641190.2641198 · Cited by 1,141
Self-citationHyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization
Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., & Talwalkar, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.06560 · Cited by 1,018
Auto-Keras: An Efficient Neural Architecture Search System
Jin, H., Song, Q., & Hu, X. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330648 · Cited by 891
No Unbiased Estimator of the Variance of K-Fold Cross-ValidationPreprint
Bengio, Y., & Grandvalet, Y. (2003) · RePEc: Research Papers in Economics · Cited by 883
ParamILS: An Automatic Algorithm Configuration Framework
Hutter, F., Hoos, H. H., Leyton-Brown, K., & Stuetzle, T. (2009) · Journal of Artificial Intelligence Research · doi:10.1613/jair.2861 · Cited by 882
SATzilla: Portfolio-based Algorithm Selection for SAT
Xu, L., Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2008) · Journal of Artificial Intelligence Research · doi:10.1613/jair.2490 · Cited by 857
Bishop, C. M. (2004) · Handbook of Neural Computation · doi:10.1887/0750303123/b365c43 · Cited by 833
Automatic differentiation in machine learning: a survey
Baydin, A. G., Pearlmutter, B. A., Radul, A., & Siskind, J. M. (2015) · arXiv (Cornell University) · doi:10.4230/lipics.ecoop.2019.25 · Cited by 823
Ridge Regression: Biased Estimation for Nonorthogonal Problems
Hoerl, A. E., & Kennard, R. W. (2000) · Technometrics · doi:10.1080/00401706.2000.10485983 · Cited by 808
Dask: Parallel Computation with Blocked algorithms and Task Scheduling
Rocklin, M. (2015) · Proceedings of the Python in Science Conference · doi:10.25080/majora-7b98e3ed-013 · Cited by 801
Hyperopt: A Python Library for Optimizing the Hyperparameters of Machine Learning Algorithms
Bergstra, J., Yamins, D., & Cox, D. (2013) · Proceedings of the Python in Science Conference · doi:10.25080/majora-8b375195-003 · Cited by 781
Evaluating Learning Algorithms: A Classification Perspective
Japkowicz, N., & Shah, M. (2011) · doi:10.1017/cbo9780511921803 · Cited by 767
A note on the validity of cross-validation for evaluating autoregressive time series prediction
Bergmeir, C., Hyndman, R. J., & Koo, B. (2018) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2017.11.003 · Cited by 743
Benchmark for filter methods for feature selection in high-dimensional classification data
Bommert, A., Sun, X., Bischl, B., Rahnenführer, J., & Lang, M. (2020) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2019.106839 · Cited by 725
Self-citationDEoptim: An R Package for Global Optimization by Differential Evolution
Mullen, K., Ardia, D., Gil, D., Windover, D., & Cline, J. (2011) · Journal of Statistical Software · doi:10.18637/jss.v040.i06 · Cited by 687
Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 1
Zoubin Ghahramani, A., Welling, M., Cortes, C., Lawrence, N. D., & Weinberger, K. Q. (2014) · Cited by 656
Genetic Optimization Using Derivatives: The rgenoud Package for R
Mebane, W. R., & Sekhon, J. S. (2011) · Journal of Statistical Software · doi:10.18637/jss.v042.i11 · Cited by 638
Survey on categorical data for neural networks
Hancock, J. T., & Khoshgoftaar, T. M. (2020) · Journal of Big Data · doi:10.1186/s40537-020-00305-w · Cited by 611
No Unbiased Estimator of the Variance of K-Fold Cross-Validation
BengioYoshua, & GrandvaletYves (2004) · Journal of Machine Learning Research · doi:10.5555/1005332.1044695 · Cited by 607
Evaluation of a Tree-based Pipeline Optimization Tool for Automating Data Science
Olson, R. S., Bartley, N., Urbanowicz, R. J., & Moore, J. H. (2016) · Proceedings of the Genetic and Evolutionary Computation Conference 2016 · doi:10.1145/2908812.2908918 · Cited by 595
Roustant, O., Ginsbourger, D., & Deville, Y. (2012) · Journal of Statistical Software · doi:10.18637/jss.v051.i01 · Cited by 594
Tunability: Importance of Hyperparameters of Machine Learning Algorithms
Probst, P., Bischl, B., & Boulesteix, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.09596 · Cited by 565
Self-citationBOHB: Robust and Efficient Hyperparameter Optimization at Scale
Falkner, S., Klein, A., & Hutter, F. (2018) · arXiv · doi:10.48550/arxiv.1807.01774 · Cited by 551
Feature Engineering and Selection: A Practical Approach for Predictive Models
Kuhn, M., & Johnson, K. (2019) · doi:10.1201/9781315108230 · Cited by 523
Misc Functions of the Department of Statistics, ProbabilityTheory Group (Formerly: E1071), TU Wien
MEYER, D. A., Dimitriadou, E., Hornik, K., Weingessel, A., & Leisch, F. (2015) · Cited by 523
Multi-Task Bayesian Optimization
Swersky, K., Snoek, J., & Adams, R. P. (2013) · Digital Access to Scholarship at Harvard (DASH) (Harvard University) · Cited by 502
A Racing Algorithm for Configuring Metaheuristics
Birattari, M., Stützle, T., Paquete, L., & Varrentrapp, K. (2002) · Cited by 498
GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration
Gardner, J. R., Pleiss, G., Bindel, D., Weinberger, K. Q., & Wilson, A. G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.11165 · Cited by 496
Comparison of performance of data imputation methods for numeric dataset
Jadhav, A., Pramod, D., & Ramanathan, K. (2019) · Applied Artificial Intelligence · doi:10.1080/08839514.2019.1637138 · Cited by 471
mlr3: A modern object-oriented machine learning framework in R
Lang, M., Binder, M., Richter, J., Schratz, P., Pfisterer, F., Coors, S., Au, Q., Casalicchio, G., Kotthoff, L., & Bischl, B. (2019) · Journal of Open Source Software · doi:10.21105/joss.01903 · Cited by 449
Self-citationEntropy search for information-efficient global optimization
Hennig, P., & Schuler, C. J. (2012) · Journal of Machine Learning Research · doi:10.5555/2188385.2343701 · Cited by 443
Generalized Linear Models With Examples in R
Dunn, P. K., & Smyth, G. K. (2018) · Springer Texts in Statistics · doi:10.1007/978-1-4419-0118-7 · Cited by 432
Sahidullah, M., & Saha, G. (2012) · Speech Communication · doi:10.1016/j.specom.2011.11.004 · Cited by 431
Exploration of Metamodeling Sampling Criteria for Constrained Global Optimization
Sasena, M. J., Papalambros, P. Y., & Goovaerts, P. (2002) · Engineering Optimization · doi:10.1080/03052150211751 · Cited by 428
Advances in Neural Information Processing Systems 26 (NIPS 2013)
Cseke, B., Opper, M., & Sanguinetti, G. (2013) · Cited by 424
F-Race and Iterated F-Race: An Overview
Birattari, M., Yuan, Z., Balaprakash, P., & Stützle, T. (2010) · Experimental Methods for the Analysis of Optimization Algorithms · doi:10.1007/978-3-642-02538-9_13 · Cited by 413
COCO: a platform for comparing continuous optimizers in a black-box setting
Hansen, N., Auger, A., Ros, R., Mersmann, O., Tušar, T., & Brockhoff, D. (2021) · Optimization Methods and Software · doi:10.1080/10556788.2020.1808977 · Cited by 408
Kriging Is Well-Suited to Parallelize Optimization
Ginsbourger, D., Le Riche, R., & Carraro, L. (2010) · Adaptation Learning and Optimization · doi:10.1007/978-3-642-10701-6_6 · Cited by 396
Optimally splitting cases for training and testing high dimensional classifiers
Dobbin, K. K., & Simon, R. M. (2011) · BMC Medical Genomics · doi:10.1186/1755-8794-4-31 · Cited by 392
Bayesian Optimization in a Billion Dimensions via Random Embeddings
Wang, Z., Hutter, F., Zoghi, M., Matheson, D., & De Feitas, N. (2016) · Journal of Artificial Intelligence Research · doi:10.1613/jair.4806 · Cited by 390
Initializing Bayesian Hyperparameter Optimization via Meta-Learning
Feurer, M., Springenberg, J., & Hutter, F. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9354 · Cited by 390
Regularized Evolution for Image Classifier Architecture Search
Real, E., Aggarwal, A., Huang, Y., & Le, Q. V. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33014780 · Cited by 328
A benchmark of kriging-based infill criteria for noisy optimization
Picheny, V., Wagner, T., & Ginsbourger, D. (2013) · Structural and Multidisciplinary Optimization · doi:10.1007/s00158-013-0919-4 · Cited by 325
Scalable Global Optimization via Local Bayesian OptimizationPreprint
Eriksson, D., Pearce, M., Gardner, J. R., Turner, R., & Poloczek, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.01739 · Cited by 318
Weighted k-Nearest-Neighbor Techniques and Ordinal Classification
Hechenbichler, K., & Schliep, K. P. (2004) · Econstor (Econstor) · doi:10.5282/ubm/epub.1769 · Cited by 307
Bayesian Optimization with Unknown Constraints
Gelbart, M. A., Snoek, J., & Adams, R. P. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1403.5607 · Cited by 299
Garrido-Merchán, E. C., & Hernández-Lobato, D. (2020) · Neurocomputing · doi:10.1016/j.neucom.2019.11.004 · Cited by 291
GPflow: A Gaussian process library using TensorFlow
Matthews, A., van der Wilk, M., Nickson, T., Fujii, K., Boukouvalas, A., León-Villagrá, P., Ghahramani, Z., & Hensman, J. J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.08733 · Cited by 289
Bischl, B., Lang, M., Kotthoff, L., Schratz, P., Schiffner, J., Richter, J., Jones, Z., Casalicchio, G., Gallo, M., & Binder, M. (2013) · CRAN: Contributed Packages · doi:10.32614/cran.package.mlr · Cited by 277
Self-citationMicci-Barreca, D. (2001) · ACM SIGKDD Explorations Newsletter · doi:10.1145/507533.507538 · Cited by 276
Hoeffding Races: Accelerating Model Selection Search for Classification and Function Approximation
Maron, O., & Moore, A. (2018) · doi:10.1184/r1/6555050 · Cited by 265
BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search
White, C., Neiswanger, W., & Savani, Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i12.17233 · Cited by 261
Non-stochastic Best Arm Identification and Hyperparameter Optimization
Jamieson, K., & Talwalkar, A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.07943 · Cited by 232
An economic method of computing LPτ-sequences
Antonov, I. A., & Saleev, V. M. (1979) · USSR Computational Mathematics and Mathematical Physics · doi:10.1016/0041-5553(79)90085-5 · Cited by 226
A Comparison of Optimization Algorithms for Deep Learning
Soydaner, D. (2020) · International Journal of Pattern Recognition and Artificial Intelligence · doi:10.1142/s0218001420520138 · Cited by 219
ASlib: A benchmark library for algorithm selection
Bischl, B., Kerschke, P., Kotthoff, L., Lindauer, M., Malitsky, Y., Fréchette, A., Hoos, H., Hutter, F., Leyton-Brown, K., Tierney, K., et al. (2016) · Artificial Intelligence · doi:10.1016/j.artint.2016.04.003 · Cited by 214
Self-citationHyperparameter Importance Across Datasets
van Rijn, J. N., & Hutter, F. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3220058 · Cited by 210
Scott, W., Frazier, P., & Powell, W. (2011) · SIAM Journal on Optimization · doi:10.1137/100801275 · Cited by 209
mlrMBO: A Modular Framework for Model-Based Optimization of Expensive Black-Box FunctionsPreprint
Bischl, B., Richter, J., Bossek, J., Horn, D., Thomas, J., & Lang, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.03373 · Cited by 198
Self-citationSMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization
Lindauer, M., Eggensperger, K., Feurer, M., Biedenkapp, A., Deng, D., Benjamins, C., Ruhopf, T., Sass, R., & Hutter, F. (2021) · arXiv · doi:10.48550/arxiv.2109.09831 · Cited by 195
Self-citationA Unifying Framework for Parallel and Distributed Processing in R using Futures
Bengtsson, H. (2021) · The R Journal · doi:10.32614/rj-2021-048 · Cited by 191
A survey on hyperparameters optimization algorithms of forecasting models in smart grid
Khalid, R., & Javaid, N. (2020) · Sustainable Cities and Society · doi:10.1016/j.scs.2020.102275 · Cited by 190
BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization
Balandat, M., Karrer, B., Jiang, D., Daulton, S., Letham, B., Wilson, A. G., & Bakshy, E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.06403 · Cited by 190
Auto-Pytorch: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL
Zimmer, L., Lindauer, M., & Hutter, F. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3067763 · Cited by 188
Self-citationPreventing Overfitting of Cross-Validation Data
Ng, A. Y. (1997) · International Conference on Machine Learning · Cited by 187
Resampling Methods for Meta-Model Validation with Recommendations for Evolutionary Computation
Bischl, B., Mersmann, O., Trautmann, H., & Weihs, C. (2012) · Evolutionary Computation · doi:10.1162/evco_a_00069 · Cited by 186
Self-citationTowards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter SearchPreprint
Zela, A., Klein, A., Falkner, S., & Hutter, F. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.06906 · Cited by 186
Meta-Learning of Neural Architectures for Few-Shot Learning
Elsken, T., Staffler, B., Metzen, J. H., & Hutter, F. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01238 · Cited by 180
Hyperparameter optimization in learning systems
Andonie, R. (2019) · Journal of Membrane Computing · doi:10.1007/s41965-019-00023-0 · Cited by 175
ML-Plan: Automated machine learning via hierarchical planning
Mohr, F., Wever, M., & Hüllermeier, E. (2018) · Machine Learning · doi:10.1007/s10994-018-5735-z · Cited by 174
High Dimensional Bayesian Optimisation and Bandits via Additive ModelsPreprint
Kandasamy, K., Schneider, J., & Póczos, B. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1503.01673 · Cited by 173
Particle Swarm Model Selection
Escalante, H. J., Montes, M., & Sucar, L. E. (2009) · doi:10.5555/1577069.1577084 · Cited by 170
Pargent, F., Pfisterer, F., Thomas, J., & Bischl, B. (2022) · Computational Statistics · doi:10.1007/s00180-022-01207-6 · Cited by 165
Self-citationPredictive Entropy Search for Multi-objective Bayesian OptimizationPreprint
Hernández-Lobato, D., Hernández-Lobato, J. M., Shah, A., & Adams, R. P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.05467 · Cited by 157
Thresholding for making classifiers cost-sensitive
Sheng, V. S., & Ling, C. X. (2006) · National Conference on Artificial Intelligence · Cited by 153
Model Selection: Beyond the Bayesian/Frequentist Divide
Guyon, I., Saffari, A., Dror, G., & Cawley, G. C. (2010) · Cited by 150
Fast Computation of the Multi-Points Expected Improvement with Applications in Batch Selection
Chevalier, C., & Ginsbourger, D. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-44973-4_7 · Cited by 147
Improvements in multiprocessor system design
Rodgers, D. P. (1985) · ACM SIGARCH Computer Architecture News · doi:10.1145/327070.327215 · Cited by 142
Feature Engineering and Selection: A Practical Approach for Predictive Models
Kühn, M., & Johnson, K. (2019) · Cited by 142
Optimizing Millions of Hyperparameters by Implicit DifferentiationPreprint
Lorraine, J., Vicol, P. A., & Duvenaud, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.02590 · Cited by 133
Greed Is Good: Exploration and Exploitation Trade-offs in Bayesian Optimisation
Ath, G. D., Everson, R., Rahat, A. A.-A. M., & Fieldsend, J. E. (2021) · ACM Transactions on Evolutionary Learning and Optimization · doi:10.1145/3425501 · Cited by 127
Performance Prediction and Automated Tuning of Randomized and Parametric Algorithms
Hutter, F., Hamadi, Y., Hoos, H. H., et al. (2006) · Lecture notes in computer science · doi:10.1007/11889205_17 · Cited by 125
Garciarena, U., & Santana, R. (2017) · Expert Systems with Applications · doi:10.1016/j.eswa.2017.07.026 · Cited by 123
Wah, Y. B., Ibrahim, N., Hamid, H. A., et al. (2018) · Pertanika journal of science & technology · Cited by 123
Tunability: Importance of Hyperparameters of Machine Learning AlgorithmsPreprint
Probst, P., Boulesteix, A., & Bischl, B. (2019) · arXiv (Cornell University) · Cited by 123
Self-citationGenetic Optimization Using Derivatives
Sekhon, J. S., & Mebane, W. R. (1998) · Political Analysis · doi:10.1093/pan/7.1.187 · Cited by 122
Oversearching and layered search in empirical learning
Quinlan, J. R., & Cameron-Jones, R. M. (1995) · Cited by 119
Drozdal, J., Weisz, J., Wang, D., Dass, G., Yao, B., Zhao, C., Muller, M., Ju, L., & Su, H. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377501 · Cited by 119
Evaluating Microarray-based Classifiers: An Overview
Boulesteix, A., Strobl, C., Augustin, T., & Däumer, M. (2008) · Cancer Informatics · doi:10.4137/cin.s408 · Cited by 116
Self-citationAn Investigation of Missing Data Methods for Classification Trees Applied to Binary Response Data
Ding, Y., & Simonoff, J. S. (2010) · doi:10.5555/1756006.1756012 · Cited by 114
Kandasamy, K., Vysyaraju, K. R., Neiswanger, W., Paria, B., Collins, C. R., Schneider, J., Póczos, B., & Xing, E. P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.06694 · Cited by 114
Standard errors for bagged and random forest estimators
Sexton, J., & Laake, P. (2008) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2008.08.007 · Cited by 107
Learning hyperparameter optimization initializations
Wistuba, M., Schilling, N., & Schmidt-Thieme, L. (2015) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2015.7344817 · Cited by 106
Best Practices for Scientific Research on Neural Architecture SearchPreprint
Lindauer, M., & Hutter, F. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.02453 · Cited by 106
Self-citationMixed Integer Evolution Strategies for Parameter Optimization
Li, R., Emmerich, M., Eggermont, J., Bäck, T., Schütz, M., Dijkstra, J., & Reiber, J. H. C. (2011) · Evolutionary Computation · doi:10.1162/evco_a_00059 · Cited by 104
Comparison of Kriging-based algorithms for simulation optimization with heterogeneous noise
Jalali, H., Nieuwenhuyse, I. V., & Picheny, V. (2017) · European Journal of Operational Research · doi:10.1016/j.ejor.2017.01.035 · Cited by 100
BOCK : Bayesian Optimization with Cylindrical KernelsPreprint
Oh, C., Gavves, E., & Welling, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.01619 · Cited by 93
Automated Machine Learning on Graphs: A Survey
Zhang, Z., Wang, X., & Zhu, W. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/637 · Cited by 91
Quantifying Model Complexity via Functional Decomposition for Better Post-hoc Interpretability
Molnar, C., Casalicchio, G., & Bischl, B. (2020) · Communications in computer and information science · doi:10.1007/978-3-030-43823-4_17 · Cited by 86
Self-citationPractical Design Space Exploration
Nardi, L., Koeplinger, D., & Olukotun, K. (2019) · Proceedings - International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems · doi:10.1109/mascots.2019.00045 · Cited by 85
OpenBox: A Generalized Black-box Optimization Service
Li, Y., Shen, Y., Zhang, W., Chen, Y., Jiang, H., Liu, M., Jiang, J., Gao, J., Wu, W., Yang, Z., et al. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467061 · Cited by 85
Parallel Algorithm Configuration
Hutter, F., Hoos, H. H., & Leyton‐Brown, K. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-34413-8_5 · Cited by 82
Raiders of the Lost Architecture: Kernels for Bayesian Optimization in Conditional Parameter SpacesPreprint
Swersky, K., Duvenaud, D., Snoek, J., Hutter, F., & Osborne, M. A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.4011 · Cited by 82
Provides Foreach Looping Construct for R
Analytics, R., & Weston, S. (2015) · Cited by 80
A Comprehensive Comparison on Evolutionary Feature Selection Approaches to Classification
Xue, B., Zhang, M., & Browne, W. N. (2015) · International Journal of Computational Intelligence and Applications · doi:10.1142/s146902681550008x · Cited by 79
OpenML: A Collaborative Science Platform
van Rijn, J. N., Bischl, B., Torgo, L., Gao, B., Umaashankar, V., Fischer, S., Winter, P., Wiswedel, B., Berthold, M. R., & Vanschoren, J. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-40994-3_46 · Cited by 77
Self-citationLearning Step Size Controllers for Robust Neural Network Training
Daniel, C., Taylor, J., & Nowozin, S. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10187 · Cited by 72
DEHB: Evolutionary Hyberband for Scalable, Robust and Efficient Hyperparameter Optimization
Awad, N. H., Mallik, N., & Hutter, F. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/296 · Cited by 71
Futility Analysis in the Cross-Validation of Machine Learning ModelsPreprint
Kühn, M. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1405.6974 · Cited by 66
Sweeney, R. E., & Ulveling, E. F. (1972) · The American Statistician · doi:10.1080/00031305.1972.10478949 · Cited by 64
MOI-MBO: Multiobjective Infill for Parallel Model-Based Optimization
Bischl, B., Wessing, S., Bauer, N., Friedrichs, K., & Weihs, C. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-09584-4_17 · Cited by 63
Self-citationMulti-objective hyperparameter tuning and feature selection using filter ensembles
Binder, M., Moosbauer, J., Thomas, J., & Bischl, B. (2020) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/3377930.3389815 · Cited by 63
Self-citationPitfalls and Best Practices in Algorithm Configuration
Eggensperger, K., Lindauer, M., & Hutter, F. (2019) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11420 · Cited by 60
Self-citationA Generalized Framework for Population Based Training
Li, A., Spyra, O., Perel, S., Dalibard, V., Jaderberg, M., Gu, C., Budden, D. M., Harley, T., & Gupta, P. (2019) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3292500.3330649 · Cited by 58
Wistuba, M., Schilling, N., & Schmidt-Thieme, L. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-23525-7_7 · Cited by 52
BatchJobsandBatchExperiments: Abstraction Mechanisms for UsingRin Batch Environments
Bischl, B., Lang, M., Mersmann, O., Rahnenführer, J., & Weihs, C. (2015) · Journal of Statistical Software · doi:10.18637/jss.v064.i11 · Cited by 51
Self-citationEvolution Strategies for Robust Optimization
Beyer, H.-G., & Sendhoff, B. (2006) · IEEE Congress on Evolutionary Computation (CEC) · doi:10.1109/cec.2006.1688465 · Cited by 51
Automatic model selection for high-dimensional survival analysis
Lang, M., Kotthaus, H., Marwedel, P., Weihs, C., Rahnenführer, J., & Bischl, B. (2014) · Journal of Statistical Computation and Simulation · doi:10.1080/00949655.2014.929131 · Cited by 50
Self-citationResampling Strategies for Model Assessment and Selection
Simon, R. M. (2007) · Springer eBooks · doi:10.1007/978-0-387-47509-7_8 · Cited by 48
Provably Efficient Online Hyperparameter Optimization with Population-Based BanditsPreprint
Parker-Holder, J., Nguyen, V., & Roberts, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.02518 · Cited by 48
Horn, D., & Bischl, B. (2016) · IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci.2016.7850221 · Cited by 40
Self-citationLearning multiple defaults for machine learning algorithms
Pfisterer, F., van Rijn, J. N., Probst, P., Müller, A. C., & Bischl, B. (2021) · Genetic and Evolutionary Computation Conference Companion (GECCO Companion) · doi:10.1145/3449726.3459523 · Cited by 37
Self-citationBossek, J., Doerr, C., & Kerschke, P. (2020) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/3377930.3390155 · Cited by 34
Fast, Accurate, and Simple Models for Tabular Data via Augmented DistillationPreprint
Fakoor, R., Mueller, J. W., Erickson, N., Chaudhari, P., & Smola, A. J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.14284 · Cited by 30
A parameterized lower confidence bounding scheme for adaptive metamodel-based design optimization
Zheng, J. J., Li, Z., Gao, L., & Jiang, G. (2016) · Engineering Computations · doi:10.1108/ec-04-2015-0088 · Cited by 26
Does imputation matter? Benchmark for predictive modelsPreprint
Woźnica, K., & Biecek, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.02837 · Cited by 26
General-purpose hierarchical optimisation of machine learning pipelines with grammatical evolution
Estévez-Velarde, S., Gutiérrez, Y., Almeida-Cruz, Y., & Montoyo, A. (2020) · Information Sciences · doi:10.1016/j.ins.2020.07.035 · Cited by 25
Warmstarting of Model-Based Algorithm Configuration
Lindauer, M., & Hutter, F. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11532 · Cited by 23
Self-citationMeta-Surrogate Benchmarking for Hyperparameter OptimizationPreprint
Klein, A., Dai, Z., Hutter, F., Lawrence, N. D., & González, J. (2019) · arXiv (Cornell University) · doi:10.17863/cam.54600 · Cited by 23
Hornung, R., Bernau, C., Truntzer, C., Wilson, R., Stadler, T., & Boulesteix, A. (2015) · BMC Medical Research Methodology · doi:10.1186/s12874-015-0088-9 · Cited by 21
Self-citationParallel computing with R: A brief review
Eddelbüttel, D. (2020) · Wiley Interdisciplinary Reviews Computational Statistics · doi:10.1002/wics.1515 · Cited by 16
Automated Benchmark-Driven Design and Explanation of Hyperparameter Optimizers
Moosbauer, J., Binder, M., Schneider, L., Pfisterer, F., Becker, M., Lang, M., Kotthoff, L., & Bischl, B. (2022) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2022.3211336 · Cited by 9
Self-citationMeta-learning for symbolic hyperparameter defaults
Gijsbers, P., Pfisterer, F., van Rijn, J. N., Bischl, B., & Vanschoren, J. (2021) · Genetic and Evolutionary Computation Conference Companion (GECCO Companion) · doi:10.1145/3449726.3459532 · Cited by 7
Self-citationLassoBench : A High-Dimensional Hyperparameter Optimization Benchmark Suite for LassoPreprint
Šehić, K., Gramfort, A., Salmon, J., & Nardi, L. (2021) · arXiv (Cornell University) · Cited by 7
YAHPO Gym -- An Efficient Multi-Objective Multi-Fidelity Benchmark for Hyperparameter OptimizationPreprint
Pfisterer, F., Schneider, L., Moosbauer, J., Binder, M., & Bischl, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.03670 · Cited by 7
Self-citation
Cited by
General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Models
Molnar, C., König, G., Herbinger, J., Freiesleben, T., Dandl, S., Scholbeck, C. A., Casalicchio, G., Grosse-Wentrup, M., & Bischl, B. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-04083-2_4 · Cited by 211
Self-citationMirzaei, S., Mao, H., Al-Nima, R. R. O., & Woo, W. L. (2023) · Information · doi:10.3390/info15010004 · Cited by 25
Mehraban, M. H., Mirzabeigi, S., Faraji, S., Soltanian-Zadeh, S., & Sepasgozar, S. M. E. (2025) · Buildings · doi:10.3390/buildings15213950 · Cited by 11
Metadata
- Authors
- Bischl, B., Binder, M., Lang, M., Pielok, T., Richter, J., Coors, S., Thomas, J., Ullmann, T., Becker, M., Boulesteix, A., Deng, D., & Lindauer, M.
- Venue
- WIREs Data Mining and Knowledge Discovery
- Publisher
- Wiley
- Published
- 2023-01-16
- Volume
- 13
- Issue
- 2
- ISSN
- 1942-4787 (ISSN-L)
- 1942-4795
- OpenAlex
- W3179950556
- Cited by
- 930
- References
- 201
- FWCI
- 80.56
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
