A survey on multi-objective hyperparameter optimization algorithms for machine learning
Open accessThe DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
Hyperparameter optimization (HPO) is a necessary step to ensure the best possible performance of Machine Learning (ML) algorithms. Several methods have been developed to perform HPO; most of these are focused on optimizing one performance measure (usually an error-based measure), and the literature on such single-objective HPO problems is vast. Recently, though, algorithms have appeared that focus on optimizing multiple conflicting objectives simultaneously. This article presents a systematic survey of the literature published between 2014 and 2020 on multi-objective HPO algorithms, distinguishing between metaheuristic-based algorithms, metamodel-based algorithms and approaches using a mixture of both. We also discuss the quality metrics used to compare multi-objective HPO procedures and present future research directions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A fast and elitist multiobjective genetic algorithm: NSGA-II
Deb, K., Pratap, A., Agarwal, S., & Meyarivan, T. (2002) · IEEE Transactions on Evolutionary Computation · doi:10.1109/4235.996017 · Cited by 50,053
Storn, R., & Price, K. (1997) · Journal of Global Optimization · doi:10.1023/a:1008202821328 · Cited by 29,569
A new optimizer using particle swarm theory
Eberhart, R., & Kennedy, J. (2002) · MHS'95. Proceedings of the Sixth International Symposium on Micro Machine and Human Science · doi:10.1109/mhs.1995.494215 · Cited by 16,067 · cited 7×
“Analogous to Genetic Algorithms, Particle Swarm Optimization (PSO) (Eberhart & Kennedy, 1995) works with a population of candidate solutions, known as particles.”
MethodInfluentialAn Introduction to Genetic Algorithms
Mitchell, M. (1996) · doi:10.7551/mitpress/3927.001.0001 · Cited by 11,340
MOEA/D: A Multiobjective Evolutionary Algorithm Based on Decomposition
Zhang, Q., & Li, H. (2007) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2007.892759 · Cited by 10,069 · cited 1×
“Contrary to NSGA-II, the Multi-Objective Evolutionary Algorithm based on Decomposition (MOEA/D) (Q. Zhang & Li, 2007) uses scalarization to solve the multi-objective HPO problem.”
MethodLarian Studios Dev Team (2024) · doi:10.1145/3292500.3330701 · Cited by 8,618
Efficient Global Optimization of Expensive Black-Box Functions
Jones, D. R., Schonlau, M., & Welch, W. J. (1998) · Journal of Global Optimization · doi:10.1023/a:1008306431147 · Cited by 8,070
Muiltiobjective Optimization Using Nondominated Sorting in Genetic Algorithms
Srinivas, N., & Deb, K. (1994) · Evolutionary Computation · doi:10.1162/evco.1994.2.3.221 · Cited by 6,935 · cited 1×
“…they cannot only obtain CNN configurations from scratch, but can also refine state-of-the-art CNNs. NSGA-II builds on the original NSGA algorithm (Srinivas & Deb, 1994); yet, it is computationally less expensive (a temporal complexity of O(MN2) versus O(MN3) for the original algorithm, where M…”
BackgroundMagnitude and complex based diffusion signal reconstruction
(2014) · HAL · doi:10.1007/978-3-319 · Cited by 6,731
Artificial intelligence in healthcare: past, present and future
Jiang, F., Jiang, Y., Zhi, H., Dong, Y., Li, H., Ma, S., Wang, Y., Dong, Q., Shen, H., & Wang, Y. (2017) · Stroke and Vascular Neurology · doi:10.1136/svn-2017-000101 · Cited by 4,837
Particle swarm optimization algorithm: An overview
Wang, D., Tan, D., & Liu, L. (2018) · Soft Computing · doi:10.1007/s00500-016-2474-6 · Cited by 3,343
Hansen, N., Müller, S. D., & Koumoutsakos, P. (2003) · Evolutionary Computation · doi:10.1162/106365603321828970 · Cited by 2,601
Classification assessment methods
Tharwat, A. (2021) · Applied Computing and Informatics · doi:10.1016/j.aci.2018.08.003 · Cited by 2,417
The irace package: Iterated racing for automatic algorithm configuration
López-Ibáñez, M., Dubois-Lacoste, J., Pérez Cáceres, L., Birattari, M., & Stützle, T. (2016) · Operations Research Perspectives · doi:10.1016/j.orp.2016.09.002 · Cited by 1,761
Feurer, M., & Hutter, F. (2019) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-030-05318-5_1 · Cited by 1,414
Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms
Thornton, C., Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2013) · Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2487575.2487629 · Cited by 1,359
Blumer, A., Ehrenfeucht, A., Haussler, D., & Warmuth, M. K. (1987) · Information Processing Letters · doi:10.1016/0020-0190(87)90114-1 · Cited by 1,054
BOHB: Robust and Efficient Hyperparameter Optimization at Scale
Falkner, S., Klein, A., & Hutter, F. (2018) · arXiv · doi:10.48550/arxiv.1807.01774 · Cited by 551
A review of automatic selection methods for machine learning algorithms and hyper-parameter values
Luo, G. (2016) · Network Modeling Analysis in Health Informatics and Bioinformatics · doi:10.1007/s13721-016-0125-6 · Cited by 357 · cited 4×
“, 2017; Singh, Kumar, & Kaur, 2020); yet, their optimization (hereafter referred to as hyperparameter optimization or HPO) is a challenging task, as traditional optimization methods are often not applicable (Luo, 2016).”
BackgroundInfluential50 Years of object recognition: Directions forward
Andreopoulos, A., & Tsotsos, J. K. (2013) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2013.04.005 · Cited by 347 · cited 1×
“…huge amounts of data to generate knowledge that is used in many different ways, e.g. for automatic user recommendations (Cai, Hu, Zhao, Zhang, & Chen, 2020), image recognition (Andreopoulos & Tsotsos, 2013; Phillips et al., 2005), and supporting healthcare related tasks (F. Jiang et al., 2017).”
BackgroundPopulation Based Training of Neural Networks
Jaderberg, M., Dalibard, V., Osindero, S., Czarnecki, W. M., Donahue, J., Razavi, A., Vinyals, O., Green, T., Dunning, I., Simonyan, K., et al. (2017) · arXiv · doi:10.48550/arxiv.1711.09846 · Cited by 284
HyperPower: Power- and memory-constrained hyper-parameter optimization for neural networks
Stamoulis, D., Cai, E., Juan, D.-C., & Marculescu, D. (2018) · 2018 Design, Automation & Test in Europe Conference & Exhibition (DATE) · doi:10.23919/date.2018.8341973 · Cited by 67
Designing adaptive neural networks for energy-constrained image classification
Stamoulis, D., Chin, T.-W. (., Prakash, A. K., Fang, H., Sajja, S., Bognar, M., & Marculescu, D. (2018) · Proceedings of the International Conference on Computer-Aided Design · doi:10.1145/3240765.3240796 · Cited by 63
Cited by
A systematic review of hyperparameter optimization techniques in Convolutional Neural Networks
Raiaan, M. A. K., Sakib, S., Fahad, N. M., Mamun, A. A., Rahman, M. A., Shatabda, S., & Mukta, M. S. H. (2024) · Decision Analytics Journal · doi:10.1016/j.dajour.2024.100470 · Cited by 254
Metadata
- Authors
- Morales-Hernández, A., Van Nieuwenhuyse, I., & Rojas Gonzalez, S.
- Venue
- Artificial Intelligence Review
- Published
- 2022-12-24
- Volume
- 56
- Issue
- 8
- Pages
- 8043-8093
- ISSN
- 0269-2821 (ISSN-L)
- 1573-7462
- OpenAlex
- W3216755252
- Cited by
- 204
- References
- 23
- FWCI
- 11.87
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.