The DOI matches its Crossref record; 89 of the 89 references it cites were found in a registry or an index.
A Survey of Automatic Parameter Tuning Methods for Metaheuristics
Open accessAbstract
Parameter tuning, that is, to find appropriate parameter settings (or configurations) of algorithms so that their performance is optimized, is an important task in the development and application of metaheuristics. Automating this task, i.e., developing algorithmic procedure to address parameter tuning task, is highly desired and has attracted significant attention from the researchers and practitioners. During last two decades, many automatic parameter tuning approaches have been proposed. This paper presents a comprehensive survey of automatic parameter tuning methods for metaheuristics. A new classification (or taxonomy) of automatic parameter tuning methods is introduced according to the structure of tuning methods. The existing automatic parameter tuning approaches are consequently classified into three categories: 1) simple generate-evaluate methods; 2) iterative generate-evaluate methods; and 3) high-level generate-evaluate methods. Then, these three categories of tuning methods are reviewed in sequence. In addition to the description of each tuning method, its main strengths and weaknesses are discussed, which is helpful for new researchers or practitioners to select appropriate tuning methods to use. Furthermore, some challenges and directions of this field are pointed out for further research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Optimization by Simulated Annealing
Kirkpatrick, S., Gelatt, C. D., & Vecchi, M. P. (1983) · Science · doi:10.1126/science.220.4598.671 · Cited by 45,134
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
Particle swarm optimization: An overview
Poli, R., Kennedy, J., & Blackwell, T. (2007) · Swarm Intelligence · doi:10.1007/s11721-007-0002-0 · Cited by 21,101
No free lunch theorems for optimization
Wolpert, D. H., & Macready, W. G. (1997) · IEEE Transactions on Evolutionary Computation · doi:10.1109/4235.585893 · Cited by 14,237
Efficient Global Optimization of Expensive Black-Box Functions
Jones, D. R., Schonlau, M., & Welch, W. J. (1998) · Journal of Global Optimization · doi:10.1023/a:1008306431147 · Cited by 8,070
Design and Analysis of Computer Experiments
Sacks, J., Welch, W. J., Mitchell, T. J., & Wynn, H. P. (1989) · Statistical Science · doi:10.1214/ss/1177012413 · Cited by 6,919
(2007) · Handbook of Approximation Algorithms and Metaheuristics · doi:10.1201/9781420010749-33 · Cited by 6,607
Information theory, inference, and learning algorithms
(2004) · Choice Reviews Online · doi:10.5860/choice.41-5949 · Cited by 6,362
Information Theory, Inference, and Learning Algorithms
(2004) · Kybernetes · doi:10.1108/03684920410534506 · Cited by 4,937
Future paths for integer programming and links to artificial intelligence
Glover, F. (1986) · Computers & Operations Research · doi:10.1016/0305-0548(86)90048-1 · Cited by 4,181
Evolutionary programming made faster
Yao, X., Liu, Y., & Lin, G. (1999) · IEEE Transactions on Evolutionary Computation · doi:10.1109/4235.771163 · Cited by 3,910
Self-citationDesign and Analysis of Experiments
Dean, A., Voss, D., & Draguljić, D. (2017) · Springer Texts in Statistics · doi:10.1007/978-3-319-52250-0 · Cited by 2,891
Optimization of Control Parameters for Genetic Algorithms
Grefenstette, J. (1986) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/tsmc.1986.289288 · Cited by 2,854
Hoos (2000) · doi:10.1016/s0167-739x(00)00043-1 · Cited by 2,737
(2005) · Cited by 2,565
Sequential Model-Based Optimization for General Algorithm Configuration
Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-25566-3_40 · Cited by 2,469
Parameter control in evolutionary algorithms
Eiben, A. E., Hinterding, R., & Michalewicz, Z. (1999) · IEEE Transactions on Evolutionary Computation · doi:10.1109/4235.771166 · Cited by 1,980
Estimation of Distribution Algorithms: A New Tool for Evolutionary Computation
Larraanaga, P., & Lozano, J. A. (2001) · Kluwer Academic Publishers eBooks · Cited by 1,914
Variable neighborhood search: Principles and applications
Hansen, P., & Mladenović, N. (2001) · European Journal of Operational Research · doi:10.1016/s0377-2217(00)00100-4 · Cited by 1,902
OR-Library: Distributing Test Problems by Electronic Mail
Beasley, J. E. (1990) · Journal of the Operational Research Society · doi:10.1057/jors.1990.166 · Cited by 1,899
Dorigo, M., Birattari, M., & Stutzle, T. (2006) · IEEE Computational Intelligence Magazine · doi:10.1109/ci-m.2006.248054 · Cited by 1,880
The CMA Evolution Strategy: A Comparing Review
Hansen, N. (2007) · Studies in Fuzziness and Soft Computing · doi:10.1007/3-540-32494-1_4 · Cited by 1,797
The irace package: Iterated racing for automatic algorithm configuration
López-Ibáñez, M., Dubois-Lacoste, J., Pérez Cáceres, L., Birattari, M., & Stützle, T. (2016) · Operations Research Perspectives · doi:10.1016/j.orp.2016.09.002 · Cited by 1,761 · cited 4×
“[30] provided a software package implemented in programming language R, called irace, that includes the iterated F-Race algorithm as well”
BackgroundMethodInfluentialA survey on optimization metaheuristics
Boussaïd, I., Lepagnot, J., & Siarry, P. (2013) · Information Sciences · doi:10.1016/j.ins.2013.02.041 · Cited by 1,655
The Traveling Salesman Problem
(2019) · Fundamentals of Supply Chain Theory · doi:10.1002/9781119584445.ch10 · Cited by 1,614
Estimation of Distribution Algorithms
(2002) · Genetic Algorithms and Evolutionary Computation · doi:10.1007/978-1-4615-1539-5 · Cited by 1,273
Mesh Adaptive Direct Search Algorithms for Constrained Optimization
Audet, C., & Dennis, J. E. (2006) · SIAM Journal on Optimization · doi:10.1137/040603371 · Cited by 1,243
Introduction to Evolutionary Computing
Eiben, A. E., & Smith, J. E. (2015) · Natural Computing Series · doi:10.1007/978-3-662-44874-8 · Cited by 1,237
Stochastic Local Search: Foundations & Applications
Hoos, H. H., & St�tzle, T. (2004) · Medical Entomology and Zoology · Cited by 1,227
Lourenço, H. R., Martin, O. C., & Stützle, T. (2006) · International Series in Operations Research & Management Science · doi:10.1007/0-306-48056-5_11 · Cited by 1,164
The BOBYQA algorithm for bound constrained optimization without derivatives
Powell, M. J. D. (2009) · Cited by 1,094
A Restart CMA Evolution Strategy With Increasing Population Size
Auger, A., & Hansen, N. (2005) · 2005 IEEE Congress on Evolutionary Computation · doi:10.1109/cec.2005.1554902 · Cited by 952
ParamILS: An Automatic Algorithm Configuration Framework
Hutter, F., Hoos, H. H., Leyton-Brown, K., & Stuetzle, T. (2009) · Journal of Artificial Intelligence Research · doi:10.1613/jair.2861 · Cited by 882
Information Theory, Inference, and Learning Algorithms
(2004) · IEEE Transactions on Information Theory · doi:10.1109/tit.2004.834752 · Cited by 866
Global Optimization of Stochastic Black-Box Systems via Sequential Kriging Meta-Models
Huang, D., Allen, T. T., Notz, W. I., & Zeng, N. (2006) · Journal of Global Optimization · doi:10.1007/s10898-005-2454-3 · Cited by 625
Parameter tuning for configuring and analyzing evolutionary algorithms
Eiben, A. E., & Smit, S. K. (2011) · Swarm and Evolutionary Computation · doi:10.1016/j.swevo.2011.02.001 · Cited by 622
Parameter Control in Evolutionary Algorithms: Trends and Challenges
Karafotias, G., Hoogendoorn, M., & Eiben, A. E. (2015) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2014.2308294 · Cited by 516
A Racing Algorithm for Configuring Metaheuristics
Birattari, M., Stützle, T., Paquete, L., & Varrentrapp, K. (2002) · Cited by 498
Testing heuristics: We have it all wrong
Hooker, J. N. (1995) · Journal of Heuristics · doi:10.1007/bf02430364 · Cited by 437
Fine-Tuning of Algorithms Using Fractional Experimental Designs and Local Search
Adenso-Díaz, B., & Laguna, M. (2006) · Operations Research · doi:10.1287/opre.1050.0243 · Cited by 435
F-Race and Iterated F-Race: An Overview
Birattari, M., Yuan, Z., Balaprakash, P., & Stützle, T. (2010) · Experimental Methods for the Analysis of Optimization Algorithms · doi:10.1007/978-3-642-02538-9_13 · Cited by 413
(2017) · Encyclopedia of GIS · doi:10.1007/978-3-319-17885-1_100388 · Cited by 362
A Gender-Based Genetic Algorithm for the Automatic Configuration of Algorithms
Ansótegui, C., Sellmann, M., & Tierney, K. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-04244-7_14 · Cited by 304
Sequential Parameter Optimization
Bartz-Beielstein, T., Lasarczyk, C. W. G., & Preuss, M. (2005) · 2005 IEEE Congress on Evolutionary Computation · doi:10.1109/cec.2005.1554761 · Cited by 300
Kammerdiner, A., Gevezes, T., Pasiliao, E., Pitsoulis, L., & Pardalos, P. M. (2013) · Encyclopedia of Operations Research and Management Science · doi:10.1007/978-1-4419-1153-7_1152 · Cited by 273
Hoeffding Races: Accelerating Model Selection Search for Classification and Function Approximation
Maron, O., & Moore, A. (2018) · doi:10.1184/r1/6555050 · Cited by 265
A theoretician’s guide to the experimental analysis of algorithms
Johnson, D. (2002) · DIMACS Series in Discrete Mathematics and Theoretical Computer Science · doi:10.1090/dimacs/059/11 · Cited by 261
OR-Library: Distributing Test Problems by Electronic Mail
Beasley, J. E. (1990) · Journal of the Operational Research Society · doi:10.1038/sj/jors/0411109 · Cited by 256
Automatic algorithm configuration based on local search
Hutter, F., Hoos, H. H., & Stützle, T. (2007) · Cited by 240
Introduction to the Theory of Complexity.
Rayward-Smith, V. J., Bovet, D. P., & Crescenzi, P. (1995) · The Journal of the Operational Research Society · doi:10.2307/2584070 · Cited by 239
Automated Algorithm Configuration and Parameter Tuning
Hoos, H. H. (2011) · Autonomous Search · doi:10.1007/978-3-642-21434-9_3 · Cited by 202
Lozano, M., Molina, D., & Herrera, F. (2011) · Soft Computing · doi:10.1007/s00500-010-0639-2 · Cited by 201
The Racing Algorithm: Model Selection for Lazy Learners
Maron, O., & Moore, A. W. (1997) · Artificial Intelligence Review · doi:10.1023/a:1006556606079 · Cited by 198
Improvement Strategies for the F-Race Algorithm: Sampling Design and Iterative Refinement
Balaprakash, P., Birattari, M., & Stützle, T. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-75514-2_9 · Cited by 197
Self-Adaptation in Evolutionary Algorithms
Meyer-Nieberg, S., & Beyer, H.-G. (2007) · Studies in Computational Intelligence · doi:10.1007/978-3-540-69432-8_3 · Cited by 184
Model-based methods for continuous and discrete global optimization
Bartz–Beielstein, T., & Zaefferer, M. (2017) · Applied Soft Computing · doi:10.1016/j.asoc.2017.01.039 · Cited by 184
Birattari, M. (2009) · Studies in Computational Intelligence · doi:10.1007/978-3-642-00483-4 · Cited by 176
Evolutionary Algorithm Parameters and Methods to Tune Them
Eiben, A. E., & Smit, S. K. (2011) · Springer eBooks · doi:10.1007/978-3-642-21434-9_2 · Cited by 161
Relevance estimation and value calibration of evolutionary algorithm parametersPreprint
Nannen, V., & Eiben, A. E. (2007) · Cited by 157
Improvement strategies for the F-Race algorithm: sampling design and iterative refinement
Balaprakash, P., Birattari, M., & Stützle, T. (2007) · Dépôt institutionnel de l'Université libre de Bruxelles (Université Libre de Bruxelles) · Cited by 151
Automated configuration of algorithms for solving hard computational problems
Hutter, F. (2009) · cIRcle (University of British Columbia) · doi:10.14288/1.0051652 · Cited by 141
An experimental investigation of model-based parameter optimisation: SPO and beyond
Hutter, F., Hoos, H. H., Leyton‐Brown, K., & Murphy, K. P. (2009) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/1569901.1569940 · Cited by 122
A multi-sexual genetic algorithm for multiobjective optimization
Lis, J., & Eiben, A. E. (2002) · IEEE Congress on Evolutionary Computation (CEC) · doi:10.1109/icec.1997.592269 · Cited by 114
A method for parameter calibration and relevance estimation in evolutionary algorithms
Nannen, V., & Eiben, A. E. (2006) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/1143997.1144029 · Cited by 111
Rao, R. V. (2015) · Springer eBooks · doi:10.1007/978-3-319-22732-0_1 · Cited by 101
A beginner's guide to tuning methods
Montero, E., Riff, M.-C., & Neveu, B. (2014) · Applied Soft Computing · doi:10.1016/j.asoc.2013.12.017 · Cited by 93
Parameter Tuning of Evolutionary Algorithms: Generalist vs. Specialist
Smit, S. K., & Eiben, A. E. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-12239-2_56 · Cited by 90
Instance-Specific Algorithm Configuration
Malitsky, Y. (2014) · Springer eBooks · doi:10.1007/978-3-319-11230-5 · Cited by 78
Time-Bounded Sequential Parameter Optimization
Hutter, F., Hoos, H. H., Leyton‐Brown, K., & Murphy, K. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-13800-3_30 · Cited by 76
AClib: A Benchmark Library for Algorithm Configuration
Hutter, F., López‐Ibáñez, M., Fawcett, C., Lindauer, M., Hoos, H. H., Leyton‐Brown, K., & Stützle, T. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-09584-4_4 · Cited by 73
Design and Analysis of Optimization Algorithms Using Computational Statistics
Bartz–Beielstein, T., Parsopoulos, K. E., & Vrahatis, M. N. (2004) · Applied Numerical Analysis & Computational Mathematics · doi:10.1002/anac.200410007 · Cited by 71
MO-ParamILS: A Multi-objective Automatic Algorithm Configuration Framework
Blot, A., Hoos, H. H., Jourdan, L., Kessaci, M., & Trautmann, H. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-50349-3_3 · Cited by 70
Yuan, Z. J., de Oca, M. A. M., Birattari, M., & Stützle, T. (2011) · Swarm Intelligence · doi:10.1007/s11721-011-0065-9 · Cited by 60
Space-Filling Designs for Computer Experiments
Santner, T. J., Williams, B. J., & Notz, W. I. (2003) · Springer series in statistics · doi:10.1007/978-1-4757-3799-8_5 · Cited by 51
Performance evaluation of automatically tuned continuous optimizers on different benchmark sets
Liao, T., Molina, D., & Stützle, T. (2014) · Applied Soft Computing · doi:10.1016/j.asoc.2014.11.006 · Cited by 49
Fine-Tuning Algorithm Parameters Using the Design of Experiments Approach
Gunawan, A., Lau, H. C., & Lindawati, L. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-25566-3_21 · Cited by 42
Liao, T., de, M. A. M., & Stützle, T. (2012) · Soft Computing · doi:10.1007/s00500-012-0946-x · Cited by 37
The ACO/F-Race Algorithm for Combinatorial Optimization Under Uncertainty
Birattari, M., Balaprakash, P., & Dorigo, M. (2007) · Springer eBooks · doi:10.1007/978-0-387-71921-4_10 · Cited by 32
Parameter meta-optimization of metaheuristics of solving specific NP-hard facility location problem
Skakov, E., & Malysh, V. (2018) · Journal of Physics Conference Series · doi:10.1088/1742-6596/973/1/012063 · Cited by 27
S-Race: a multi-objective racing algorithm
Zhang, T., Georgiopoulos, M., & Anagnostopoulos, G. C. (2013) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/2463372.2463561 · Cited by 24
Hutter, F., Bartz–Beielstein, T., Hoos, H. H., Leyton‐Brown, K., & Murphy, K. P. (2010) · Springer eBooks · doi:10.1007/978-3-642-02538-9_15 · Cited by 23
Multi-Objective Model Selection via Racing
Zhang, T., Georgiopoulos, M., & Anagnostopoulos, G. C. (2015) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2015.2456187 · Cited by 17
An analysis of post-selection in automatic configuration
Yuan, Z. J., Stützle, T., de, M. A. M., Lau, H. C., & Birattari, M. (2013) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/2463372.2463562 · Cited by 16
Recent Development in Automatic Parameter Tuning for Metaheuristics
Dobslaw, F. (2010) · Cited by 16
Exploring variable neighborhood search for automatic algorithm configuration
Cáceres, L. P., & Stützle, T. (2017) · Electronic Notes in Discrete Mathematics · doi:10.1016/j.endm.2017.03.022 · Cited by 11
In a Nutshell: Sequential Parameter Optimization
Bartz–Beielstein, T., Gentile, L., & Zaefferer, M. (2017) · Cologne Open Science (RFH - University of Applied Sciences Cologne) · Cited by 9
The race package for R. Racing methods for the selection of the best
Birattari, M. (2003) · Dépôt institutionnel de l'Université libre de Bruxelles (Université Libre de Bruxelles) · Cited by 4
Cited by
Karimi-Mamaghan, M., Mohammadi, M., Meyer, P., Karimi-Mamaghan, A. M., & Talbi, E.-G. (2022) · European Journal of Operational Research · doi:10.1016/j.ejor.2021.04.032 · Cited by 462
Osaba, E., Villar-Rodriguez, E., Del Ser, J., Nebro, A. J., Molina, D., LaTorre, A., Suganthan, P. N., Coello Coello, C. A., & Herrera, F. (2021) · Swarm and Evolutionary Computation · doi:10.1016/j.swevo.2021.100888 · Cited by 406
Uddin, M. G., Nash, S., Rahman, A., & Olbert, A. I. (2023) · Process Safety and Environmental Protection · doi:10.1016/j.psep.2022.11.073 · Cited by 288
Learning Performance-Improving Code Edits
Shypula, A., Madaan, A., Zeng, Y., Alon, U., Gardner, J., Hashemi, M., Neubig, G., Ranganathan, P., Bastani, O., & Yazdanbakhsh, A. (2023) · arXiv · doi:10.48550/arxiv.2302.07867 · Cited by 146
“, 1994; Talaashrafi, 2022), parameter search (Hamadi & H madi, 2013; Huang et al., 2019; Kaufman et al., 2021), auto-vectorization (Nuzman et al.”
BackgroundInfluential
Metadata
- Authors
- Huang, C., Li, Y., & Yao, X.
- Venue
- IEEE Transactions on Evolutionary Computation
- Published
- 2020-04
- Volume
- 24
- Issue
- 2
- Pages
- 201-216
- ISSN
- 1089-778X (ISSN-L)
- 1941-0026
- OpenAlex
- W2961614712
- Cited by
- 371
- References
- 89
- FWCI
- 16.58
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.