CoChat
Sign Up
PreprintCited by 214Metaheuristic Optimization Algorithms ResearchAdvanced Optimization Algorithms ResearchSimulation Techniques and Applications

The DOI matches its DataCite record; 5 of the 5 references it cites were found in a registry or an index.

Benchmarking in Optimization: Best Practice and Open Issues

Open access
Bartz-Beielstein, Thomas, Doerr, Carola, Berg, Daan van den, Bossek, Jakob, Chandrasekaran, Sowmya, Eftimov, Tome, Fischbach, Andreas, Kerschke, Pascal, La Cava, William, Lopez-Ibanez, Manuel, Malan, Katherine M., Moore, Jason H., Naujoks, Boris, Orzechowski, Patryk, Volz, Vanessa, Wagner, Markus, Weise, Thomas·2020·arXiv
DOI 10.48550/arxiv.2007.03488arXiv 2007.03488
Open this paper in CoChat — free accountOpen PDF

Abstract

This survey compiles ideas and recommendations from more than a dozen researchers with different backgrounds and from different institutes around the world. Promoting best practice in benchmarking is its main goal. The article discusses eight essential topics in benchmarking: clearly stated goals, well-specified problems, suitable algorithms, adequate performance measures, thoughtful analysis, effective and efficient designs, comprehensible presentations, and guaranteed reproducibility. The final goal is to provide well-accepted guidelines (rules) that might be useful for authors and reviewers. As benchmarking in optimization is an active and evolving field of research this manuscript is meant to co-evolve over time by means of periodic updates.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2007.03488

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Benchmarks are rarely impartial and are shaped by the environments in which they are made.

    “Research has found that benchmarks are rarely impartial and instead shaped by the environments in which the benchmarks are made”

    From The Leaderboard Illusion (2025) · cites this paper as Bartz-Beielstein et al., 2020

    Section 4 (‘Algorithms’; page unavailable) calls the choice of algorithms ‘one of the most subjective choices’ in a benchmarking study, and Section 3 (‘Problem Instances’; page unavailable) says the choice of problem instances can heavily influence results. These passages support the point that benchmark design shapes outcomes, but they do not establish that benchmarks are rarely impartial or that they are shaped specifically by the environments in which they are made. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Poor data quality and limited reproducibility can lead to unreliable evaluations and undermine scientific credibility.

    “poor data quality and limited reproducibility can lead to unreliable evaluations and undermine scientific credibility.”

    From The Leaderboard Illusion (2025) · cites this paper as Bartz-Beielstein et al., 2020

    Section 9 (Additional information; page not provided) treats reproducibility as important to obtaining measurements that lead to the same conclusions. The Results/Visualization passage warns of spurious results, supporting the concern about unreliable evaluations. The copy does not specifically link poor data quality to those evaluations or state that these problems undermine scientific credibility. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 170

  1. Stavros P Adam, Stamatios-Aggelos N Alexandropoulos, Panos M Pardalos, and Michael N Vrahatis. No Free Lunch Theorem: A Review. In Approximation and Optimization, pages 57 -82. Springer, 2019.
  2. Amritanshu Agrawal, Tim Menzies, Leandro L. Minku, Markus Wagner, and Zhe Yu. Better software analytics via "duo": Data mining algorithms using/used-by optimizers. Empirical Software Engineering, 25(3):2099-2136, 2020. doi:10.1007/s10664-020-09808-9. URL https://doi.org/10.1007/s10664-020-09808-9. · DOI
  3. Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2623-2631, 2019.
  4. Mohammad M Amini and Richard S Barr. Network reoptimization algorithms: A statistically designed comparison. ORSA Journal on Computing, 5(4):395-409, 1993.
  5. Carlos Ansótegui, Yuri Malitsky, Horst Samulowitz, Meinolf Sellmann, and Kevin Tierney. Model-based genetic algorithms for algorithm configuration. In Proc. of International Conf. on Artificial Intelligence (IJCAI'15), pages 733-739. AAAI, 2015.
  6. Dirk V. Arnold. Noisy Optimization with Evolution Strategies, volume 8. Springer, 2012.
  7. Anne Auger and Benjamin Doerr. Theory of Randomized Search Heuristics. World Scientific, 2011.
  8. Anne Auger and Nikolaus Hansen. Performance evaluation of an advanced local search evolutionary algorithm. In Proceedings of the IEEE Congress on Evolutionary Computation, pages 1777-1784. IEEE, 2005.
  9. Thomas Bäck, David B. Fogel, and Zbigniew Michalewicz. Handbook of Evolutionary Computation. IOP Publishing Ltd., GBR, 1st edition, 1997.
  10. Richard S Barr, Bruce L Golden, James P Kelly, Mauricio GC Resende, and William R Stewart. Designing and reporting on computational experiments with heuristic methods. Journal of Heuristics, 1(1):9-32, 1995.
  11. Maurice Stevenson Bartlett. Properties of sufficiency and statistical tests. Proceedings of the Royal Society of London. Series A-Mathematical and Physical Sciences, 160(901):268-282, 1937.
  12. Thomas Bartz-Beielstein. Experimental Research in Evolutionary Computation-The New Experimentalism. Natural Computing Series. Springer, 2006.
  13. Thomas Bartz-Beielstein and Mike Preuss. The Future of Experimental Research. In Thomas Bartz-Beielstein, Marco Chiarandini, Luis Paquete, and Mike Preuss, editors, Experimental Methods for the Analysis of Optimization Algorithms, pages 17-46. Springer, Berlin, Heidelberg, New York, 2010.
  14. Thomas Bartz-Beielstein, Konstantinos E Parsopoulos, and Michael N Vrahatis. Design and analysis of optimization algorithms using computational statistics. Applied Numerical Analysis & Computational Mathematics, 1(2):413- 433, 2004.
  15. Thomas Bartz-Beielstein, Christian WG Lasarczyk, and Mike Preuss. Sequential Parameter Optimization. In Pro- ceedings of the 2005 IEEE Congress on Evolutionary Computation, volume 1, pages 773 -780. IEEE, 2005.
  16. Thomas Bartz-Beielstein, Marco Chiarandini, Luís Paquete, and Mike Preuss. Experimental methods for the analysis of optimization algorithms. Springer, 2010.
  17. Thomas Bartz-Beielstein, Carola Doerr, Jakob Bossek, Sowmya Chandrasekaran, Tome Eftimov, Andreas Fischbach, Pascal Kerschke, Manuel Lopez-Ibanez, Katherine M. Malan, Jason H. Moore, Boris Naujoks, Patryk Orzechowski, Vanessa Volz, Markus Wagner, and Thomas Weise. Benchmarking in Optimization: Best Practice and Open Issues. arXiv e-prints, art. arXiv:2007.03488, July 2020.
  18. Vahid Beiranvand, Warren Hare, and Yves Lucet. Best Practices for Comparing Optimization Algorithms. Opti- mization and Engineering, 18(4):815 -848, 2017.
  19. James Bergstra, Dan Yamins, and David D Cox. Hyperopt: A python library for optimizing the hyperparameters of machine learning algorithms. In Proceedings of the 12th Python in science conference, volume 13, page 20. Citeseer, 2013.
  20. Mauro Birattari, Thomas Stützle, Luís Paquete, and Klaus Varrentrapp. A racing algorithm for configuring meta- heuristics. In W. B. Langdon et al., editors, Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2002, pages 11-18.
  21. Mauro Birattari, Luis Paquete, and Thomas Stützle. Classification of metaheuristics and design of experiments for the analysis of components. https://www.researchgate.net/publication/2557723_Classification_of_ Metaheuristics_and_Design_of_Experiments_for_the_Analysis_of_Components, 2003. technical report.
  22. Laurens Bliek, Sicco Verwer, and Mathijs de Weerdt. Black-box mixed-variable optimisation using a surrogate model that satisfies integer constraints. arXiv preprint arXiv:2006.04508, 2020. · arXiv
  23. Mahmoud A. Bokhari, Brad Alexander, and Markus Wagner. Towards rigorous validation of energy optimisa- tion experiments. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference, GECCO '20, page 1232-1240, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450371285. doi:10.1145/3377930.3390245. URL https://doi.org/10.1145/3377930.3390245. · DOI
  24. Mohammad Reza Bonyadi, Zbigniew Michalewicz, and Luigi Barone. The Travelling Thief Problem: The First Step in the Transition from Theoretical Problems to Realistic Problems. In 2013 IEEE Congress on Evolutionary Computation, pages 1037 -1044. IEEE, 2013.
  25. Mohammad Reza Bonyadi, Zbigniew Michalewicz, Markus Wagner, and Frank Neumann. Evolutionary Computa- tion for Multicomponent Problems: Opportunities and Future Directions. In Optimization in Industry: Present Practices and Future Scopes, pages 13 -30. Springer, 2019. doi:10.1007/978-3-030-01641-8 · DOI
  26. Jakob Bossek, Pascal Kerschke, Aneta Neumann, Markus Wagner, Frank Neumann, and Heike Trautmann. Evolving Diverse TSP Instances by Means of Novel and Creative Mutation Operators. In Proc. of the 15th ACM/SIGEVO Conference on Foundations of Genetic Algorithms, pages 58 -71. ACM, 2019. · DOI
  27. Jakob Bossek, Pascal Kerschke, and Heike Trautmann. A Multi-Objective Perspective on Performance Assessment and Automated Selection of Single-Objective Optimization Algorithms. Applied Soft Computing Journal (ASOC), 88:105901, March 2020a.
  28. Jakob Bossek, Pascal Kerschke, and Heike Trautmann. Anytime Behavior of Inexact TSP Solvers and Perspectives for Automated Algorithm Selection. In Proc. of the IEEE Congress on Evolutionary Computation. IEEE, 2020b. A preprint of this manuscript can be found at https://arxiv.org/abs/2005.13289. · arXiv
  29. Ilhem Boussaïd, Julien Lepagnot, and Patrick Siarry. A survey on optimization metaheuristics. Information Sciences, 237:82-117, 2013. doi:10.1016/j.ins.2013.02.041. URL https://doi.org/10.1016/j.ins.2013.02.041. · DOI
  30. Jürgen Branke. Creating Robust Solutions by Means of Evolutionary Algorithms. In International Conference on Parallel Problem Solving from Nature, pages 119 -128. Springer, 1998.
  31. Jürgen Branke, Christian Schmidt, and Hartmut Schmeck. Efficient Fitness Estimation in Noisy Environments. In Genetic and Evolutionary Computation Conference (GECCO'01), pages 243 -250.
  32. Morgan Kaufmann, 2001.
  33. Eduardo G Carrano, Elizabeth F Wanner, and Ricardo HC Takahashi. A multicriteria statistical based comparison methodology for evaluating evolutionary algorithms. IEEE Transactions on Evolutionary Computation, 15(6): 848-870, 2011.
  34. Marie-Liesse Cauwet and Olivier Teytaud. Noisy Optimization: Fast Convergence Rates with Comparison-based Algorithms. In Proceedings of the Genetic and Evolutionary Computation Conference 2016, pages 1101 -1106, 2016.
  35. Nicos Christofides. The Vehicle Routing Problem. Revue française d'automatique, d'informatique et de recherche opérationnelle (RAIRO). Recherche opérationnelle, 10(1):55 -70, 1976. URL http://www.numdam.org/item?id= RO_1976__10_1_55_0.
  36. Harlan Crowder, Ron S Dembo, and John M Mulvey. On reporting computational experiments with mathematical software. ACM Transactions on Mathematical Software (TOMS), 5(2):193-203, 1979.
  37. Joseph C. Culberson. On the futility of blind search: An algorithmic view of "no free lunch". Evolutionary Compu- tation, 6(2):109-127, 1998. doi:10.1162/evco.1998.6.2.109. · DOI
  38. Kenneth De Jong. Parameter Setting in EAs: a 30 Year Perspective. In Parameter Setting in Evolutionary Algorithms, pages 1 -18. Springer, 2007.
  39. Lucas Augusto Müller de Souza, José Eduardo Henriques da Silva, Luciano Jerez Chaves, and Heder Soares Bernardino. A benchmark suite for designing combinational logic circuits via metaheuristics. Applied Soft Comput- ing, 91:106246, June 2020. doi:10.1016/j.asoc.2020.106246. URL https://doi.org/10.1016/j.asoc.2020.106246. · DOI
  40. Joaquín Derrac, Salvador García, Daniel Molina, and Francisco Herrera. A practical tutorial on the use of nonpara- metric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms. Swarm and Evolutionary Computation, 1(1):3-18, 2011.
  41. Benjamin Doerr and Frank Neumann. Theory of Evolutionary Computation -Recent Developments in Discrete Optimization. Springer, 2020.
  42. Benjamin Doerr, Carola Doerr, and Johannes Lengler. Self-adjusting mutation rates with provably optimal success rules. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 1479 -1487. ACM, 2019.
  43. Carola Doerr, Hao Wang, Furong Ye, Sander van Rijn, and Thomas Bäck. IOHprofiler: A Benchmarking and Profiling Tool for Iterative Optimization Heuristics. arXiv e-prints, art. arXiv:1810.05281, Oct 2018. Wiki page of IOHprofiler is available at https://iohprofiler.github.io/. · arXiv
  44. Elizabeth D Dolan and Jorge J Moré. Benchmarking optimization software with performance profiles. Mathematical programming, 91(2):201-213, 2002.
  45. Marco Dorigo, Mauro Birattari, and Thomas Stützle. Ant colony optimization: Artificial ants as a computational intelligence technique. IEEE Computational Intelligence Magazine, 1(4):28-39, 2006.
  46. Tome Eftimov and Peter Korošec. The impact of statistics for benchmarking in evolutionary computation research. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, pages 1329-1336, 2018.
  47. Katharina Eggensperger, Matthias Feurer, Frank Hutter, James Bergstra, Jasper Snoek, Holger H. Hoos, and Kevin Leyton-Brown. Towards an Empirical Foundation for Assessing Bayesian Optimization of Hyperparameters. In NIPS Workshop on Bayesian Optimization in Theory and Practice, volume 10, December 2013.
  48. Ágoston Endre Eiben and Márk Jelasity. A Critical Note on Experimental Research Methodology in EC. In Proceed- ings of the 2002 IEEE Congress on Evolutionary Computation, volume 1, pages 582 -587. IEEE, 2002.
  49. Ágoston Endre Eiben and Selmar K Smit. Evolutionary Algorithm Parameters and Methods to Tune Them. In Autonomous search, pages 15 -36. Springer, 2011.
  50. Ágoston Endre Eiben and James E Smith. Introduction to Evolutionary Computing. Natural Computing. Springer, 2 edition, 2015.
  51. Álvaro Fialho, Luís Da Costa, Marc Schoenauer, and Michèle Sebag. Analyzing bandit-based adaptive operator selection mechanisms. Annals of Mathematics and Artificial Intelligence, 60:25 -64, 2010.
  52. Andreas Fischbach and Thomas Bartz-Beielstein. Improving the reliability of test functions generators. Applied Soft Computing, 92:106315, 2020. ISSN 1568-4946. doi:https://doi.org/10.1016/j.asoc.2020.106315. URL http: //www.sciencedirect.com/science/article/pii/S1568494620302556. · DOI
  53. Philip J. Fleming and John J. Wallace. How not to lie with statistics: The correct way to summarize benchmark results. Commun. ACM, 29(3):218-221, March 1986.
  54. Roger Fletcher. Conjugate gradient methods for indefinite systems. In Numerical analysis, pages 73-89. Springer, 1976.
  55. Alexandre Fréchette, Lars Kotthoff, Tomasz Michalak, Talal Rahwan, Holger H. Hoos, and Kevin Leyton-Brown. Using the shapley value to analyze algorithm portfolios. In Proc. of the Thirtieth AAAI Conference on Artificial Intelligence, pages 3397 --3403. AAAI Press, 2016.
  56. Milton Friedman. The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the american statistical association, 32(200):675-701, 1937.
  57. Marcus Gallagher. Towards improved benchmarking of black-box optimization algorithms using clustering problems. Soft Computing, 20(10):3835-3849, March 2016. doi:10.1007/s00500-016-2094-1. URL https://doi.org/10.1007/ s00500-016-2094-1. · DOI
  58. Salvador García, Daniel Molina, Manuel Lozano, and Francisco Herrera. A study on the use of non-parametric tests for analyzing the evolutionary algorithms' behaviour: a case study on the cec'2005 special session on real parameter optimization. Journal of Heuristics, 15(6):617, 2009.
  59. Carlos García-Martínez, Francisco J. Rodríguez, and Manuel Lozano. Arbitrary function optimisation with meta- heuristics: No free lunch and real-world problems. Soft Computing, 16(12):2115-2133, 2012. doi:10.1007/s00500- 012-0881-x. · DOI
  60. Robert W. Garden and Andries P. Engelbrecht. Analysis and classification of optimisation benchmark func- tions and benchmark suites. In 2014 IEEE Congress on Evolutionary Computation (CEC). IEEE, July 2014. doi:10.1109/cec.2014.6900240. URL https://doi.org/10.1109/cec.2014.6900240. · DOI
  61. Ian P. Gent and Toby Walsh. How not to do it. In AAAI Workshop on Experimental Evaluation of Reasoning and Search Methods, 1994.
  62. Fred Glover. Tabu search-part i. ORSA Journal on computing, 1(3):190-206, 1989.
  63. Sim Kuan Goh, Kay Chen Tan, Abdullah Al-Mamun, and Hussein A. Abbass. Evolutionary big optimiza- tion (BigOpt) of signals. In 2015 IEEE Congress on Evolutionary Computation (CEC). IEEE, May 2015. doi:10.1109/cec.2015.7257307. URL https://doi.org/10.1109/cec.2015.7257307. · DOI
  64. David E Goldberg. Genetic Algorithms in Search, Optimization, and Machine Learning. Addison-Wesley, Reading MA, 1989.
  65. Bruce L Golden, Arjang A Assad, Edward A Wasil, and Edward Baker. Experimentation in optimization. European Journal of Operational Research, 27(1):1-16, 1986.
  66. Raphael T. Haftka. Requirements for papers focusing on new or improved global optimization algorithms. Structural and Multidisciplinary Optimization, 54(1):1-1, 2016.
  67. Doug Hains, L. Darrell Whitley, Adele E. Howe, and Wenxiang Chen. Hyperplane Initialized Local Search for MAXSAT. In Proc. of the Genetic and Evolutionary Computation Conference, pages 805 -812. ACM, 2013.
  68. Nikolaus Hansen. Invariance, self-adaptation and correlated mutations in evolution strategies. In Proc. of Interna- tional Conference on Parallel Problem Solving from Nature, pages 355-364. Springer, 2000. ISBN 978-3-540-45356-7.
  69. Nikolaus Hansen, Sibylle D Müller, and Petros Koumoutsakos. Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (cma-es). Evolutionary computation, 11(1):1-18, 2003.
  70. Nikolaus Hansen, Anne Auger, Dimo Brockhoff, Dejan Tušar, and Tea Tušar. COCO: performance assessment. CoRR, abs/1605.03560, 2016a. URL http://arxiv.org/abs/1605.03560. · arXiv
  71. Nikolaus Hansen, Anne Auger, Olaf Mersmann, Tea Tušar, and Dimo Brockhoff. COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting. arXiv preprint, abs/1603.08785v3, August 2016b. URL http: //arxiv.org/abs/1603.08785v3. · arXiv
  72. Anna Hart. Mann-whitney test is not just a test of medians: differences in spread can be important. Bmj, 323(7309): 391-393, 2001.
  73. Michael Hellwig and Hans-Georg Beyer. Benchmarking Evolutionary Algorithms For Single-Objective Real-valued Constrained Optimization -A Critical Review. Swarm and Evolutionary Computation, 44:927-944, 2019.
  74. John N Hooker. Needed: An Empirical Science of Algorithms. Operations research, 42(2):201 -212, 1994.
  75. John N. Hooker. Testing heuristics: We have it all wrong. Journal of Heuristics, 1(1):33-42, 1996. doi:10.1007/BF02430364. · DOI
  76. Frank Hutter, Holger H. Hoos, Kevin Leyton-Brown, and Thomas Stützle. ParamILS: an automatic algorithm configuration framework. Journal of Artificial Intelligence Research, 36:267-306, October 2009.
  77. Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. Sequential Model-Based Optimization for General Algorithm Configuration. In International Conference on Learning and Intelligent Optimization, pages 507 -523. Springer, 2011.
  78. Frank Hutter, Manuel López-Ibáñez, Chris Fawcett, Marius Thomas Lindauer, Holger H. Hoos, Kevin Leyton-Brown, and Thomas Stützle. AClib: a benchmark library for algorithm configuration. In Panos M. Pardalos, Mauricio G. C. Resende, Chrysafis Vogiatzis, and Jose L. Walteros, editors, Learning and Intelligent Optimization, 8th International Conference, LION 8, volume 8426 of Lecture Notes in Computer Science, pages 36-40. Springer, Heidelberg, Germany, 2014. doi:10.1007/978-3-319-09584-4 · DOI
  79. Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren. Automated Machine Learning: Methods, Systems, Challenges. Springer, 2019.
  80. Alexandre D. Jesus, Arnaud Liefooghe, Bilel Derbel, and Luís Paquete. Algorithm Selection of Anytime Algorithms. In Proc. of the 2020 Genetic and Evolutionary Computation Conference, pages 850 -858. ACM, 2020.
  81. Yaochu Jin and Jürgen Branke. Evolutionary Optimization in Uncertain Environments -A Survey. IEEE Transac- tions on Evolutionary Computation, 9(3):303 -317, 2005.
  82. David S Johnson, Cecilia R Aragon, Lyle A McGeoch, and Catherine Schevon. Optimization by Simulated Annealing: An Experimental Evaluation. Part I, Graph Partitioning. Operations Research, 37(6):865 -892, 1989.
  83. David S Johnson, Cecilia R Aragon, Lyle A McGeoch, and Catherine Schevon. Optimization by Simulated Annealing: An Experimental Evaluation. Part II, Graph Coloring and Number Partitioning. Operations Research, 39(3):378 - 406, 1991.
  84. David Stifler Johnson. A Theoretician's Guide to the Experimental Analysis of Algorithms. In Proc. of a DIMACS Workshop on Data Structures, Near Neighbor Searches, and Methodology: Fifth and Sixth DIMACS Implementation Challenges, volume 59 of DIMACS -Series in Discrete Mathematics and Theoretical Computer Science, pages 215- 250, 2002a.
  85. David Stifler Johnson and Lyle A. McGeoch. Experimental Analysis of Heuristics for the STSP. In The Traveling Salesman Problem and its Variations, volume 12 of Combinatorial Optimization, chapter 9, pages 369 -443. Kluwer Academic Publishers, 2002.
  86. Giorgos Karafotias, Mark Hoogendoorn, and Ágoston E. Eiben. Parameter control in evolutionary algorithms: Trends and challenges. IEEE Transactions on Evolutionary Computation, 19(2):167-187, April 2015.
  87. Stuart A. Kauffman. The Origins of Order: Self-Organization and Selection in Evolution. Oxford University Press, USA, 1993.
  88. James Kennedy and Russell Eberhart. Particle swarm optimization. In Neural Networks, 1995. Proceedings., IEEE International Conference on, volume 4, pages 1942-1948. IEEE, 1995.
  89. Pascal Kerschke and Heike Trautmann. Automated Algorithm Selection on Continuous Black-Box Problems by Combining Exploratory Landscape Analysis and Machine Learning. Evolutionary Computation (ECJ), 27(1): 99 -127, 2019a.
  90. Pascal Kerschke and Heike Trautmann. Comprehensive Feature-Based Landscape Analysis of Continuous and Con- strained Optimization Problems Using the R-package flacco. In Applications in Statistical Computing, pages 93 -123. Springer, 2019b.
  91. Pascal Kerschke, Jakob Bossek, and Heike Trautmann. Parameterization of State-of-the-Art Performance Indicators: A Robustness Study based on Inexact TSP Solvers. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, pages 1737 -1744. ACM, 2018a.
  92. Pascal Kerschke, Lars Kotthoff, Jakob Bossek, Holger H. Hoos, and Heike Trautmann. Leveraging TSP Solver Complementarity through Machine Learning. Evolutionary Computation (ECJ), 26(4):597 -620, December 2018b.
  93. Jack P. C. Kleijnen. Analyzing Simulation Experiments with Common Random Numbers. Management Science, 34 (1):65 -74, 1988.
  94. Jack P. C. Kleijnen. Design and Analysis of Simulation Experiments. In International Workshop on Simulation, pages 3-22. Springer, 2015.
  95. Jack P. C. Kleijnen. Regression and Kriging Metamodels with their Experimental Designs in Simulation: A Review. European Journal of Operational Research, 256(1):1-16, 2017.
  96. Lasse Kliemann and Peter Sanders. Algorithm Engineering: Selected Results and Surveys, volume 9220. Springer, 2016.
  97. Howard Levene. Robust tests for equality of variances. Contributions to probability and statistics. Essays in honor of Harold Hotelling, pages 279-292, 1961.
  98. Lisha Li, Kevin G. Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization. Journal of Machine Learning Research, 18:185:1-185:52, 2017.
  99. Rui Li, Michael T. M. Emmerich, Jeroen Eggermont, Ernst G. P. Bovenkamp, Thomas Bäck, Jouke Dijkstra, and Johan H. C. Reiber. Mixed-Integer NK Landscapes. In Proc. of Parallel Problem Solving from Nature, pages 42-51. Springer, 2006.
  100. Tianjun Liao, Krzysztof Socha, Marco A. Montes de Oca, Thomas Stützle, and Marco Dorigo. Ant colony optimization for mixed-variable optimization problems. IEEE Transactions on Evolutionary Computation, 18(4):503-518, 2014.
  101. Jialin Liu, Antoine Moreau, Mike Preuss, Baptiste Rozière, Jérémy Rapin, Fabien Teytaud, and Olivier Teytaud. Versatile black-box optimization. CoRR, abs/2004.14014, 2020. URL https://arxiv.org/abs/2004.14014. · arXiv
  102. Qunfeng Liu, William V. Gehrlein, Ling Wang, Yuan Yan, Yingying Cao, Wei Chen, and Yun Li. Paradoxes in Numerical Comparison of Optimization Algorithms. IEEE Transactions on Evolutionary Computation, pages 1-15, 2019.
  103. Manuel López-Ibáñez, Jérémie Dubois-Lacoste, Leslie Pérez Cáceres, Thomas Stützle, and Mauro Birattari. The irace package: Iterated racing for automatic algorithm configuration. Operations Research Perspectives, 3:43-58, 2016. doi:10.1016/j.orp.2016.09.002. · DOI · Paper page
  104. Jiří Matoušek. Geometric Discrepancy. Springer, Berlin, 2 edition, 2009.
  105. Deborah G Mayo and Aris Spanos. Severe testing as a basic concept in a neyman-pearson philosophy of induction. The British Journal for the Philosophy of Science, 57(2):323-357, 2006.
  106. Kent McClymont and Ed Keedwell. Benchmark Multi-Objective Optimisation Test Problems with Mixed Encodings. In Proceedings of the 2011 IEEE Congress on Evolutionary Computation, pages 2131 -2138. IEEE, 2011.
  107. James McDermott. When and why metaheuristics researchers can ignore "no free lunch" theorems. SN Computer Science, 1(60):1-18, 2020. doi:10.1007/s42979-020-0063-3. · DOI
  108. Catherine C McGeoch. Experimental Analysis of Algorithms. PhD thesis, Carnegie Mellon University, Pittsburgh PA, 1986.
  109. Catherine C McGeoch. Toward an experimental method for algorithm simulation. INFORMS Journal on Computing, 8(1):1-15, 1996.
  110. McKay, Michael D and Beckman, Richard J and Conover, William J. A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output from a Computer Code. Technometrics, 42(1):55 -61, 2000.
  111. Olaf Mersmann, Mike Preuss, and Heike Trautmann. Benchmarking Evolutionary Algorithms: Towards Exploratory Landscape Analysis. In Proc. of International Conference on Parallel Problem Solving from Nature, pages 73 -82. Springer, 2010.
  112. Olaf Mersmann, Bernd Bischl, Heike Trautmann, Mike Preuss, Claus Weihs, and Günter Rudolph. Exploratory Landscape Analysis. In Proc. of the 13th Annual Conference on Genetic and Evolutionary Computation, pages 829 -836. ACM, 2011.
  113. Olaf Mersmann, Bernd Bischl, Heike Trautmann, Markus Wagner, Jakob Bossek, and Frank Neumann. A Novel Feature-Based Approach to Characterize Algorithm Performance for the Traveling Salesperson Problem. Annals of Mathematics and Artificial Intelligence, 69(2):151 -182, 2013.
  114. Nenad Mladenović and Pierre Hansen. Variable neighborhood search. Comput. Oper. Res., 24(11):1097-1100, 1997. ISSN 0305-0548. doi:10.1016/S0305-0548(97)00031-2. URL https://doi.org/10.1016/S0305-0548(97)00031-2. · DOI
  115. Douglas C Montgomery. Design and Analysis of Experiments. John Wiley & Sons, 9 edition, 2017.
  116. Jorge J Moré and Stefan M Wild. Benchmarking derivative-free optimization algorithms. SIAM Journal on Opti- mization, 20(1):172-191, 2009.
  117. Jorge J Moré, Burton S Garbow, and Kenneth E Hillstrom. Testing Unconstrained Optimization Software. ACM Transactions on Mathematical Software (TOMS), 7(1):17 -41, 1981.
  118. Mario Andrés Muñoz Acosta, Michael Kirley, and Saman K. Halgamuge. Exploratory Landscape Analysis of Contin- uous Space Optimization Problems Using Information Content. IEEE Transactions on Evolutionary Computation (TEVC), 19(1):74 -87, 2015a.
  119. H. Mühlenbein and G. Paaß. From recombination of genes to the estimation of distributions i. binary parameters. In Hans-Michael Voigt, Werner Ebeling, Ingo Rechenberg, and Hans-Paul Schwefel, editors, Proc. of Parallel Problem Solving from Nature (PPSN'96), pages 178-187. Springer, 1996. ISBN 978-3-540-70668-7.
  120. Matthias Müller-Hannemann and Stefan Schirra. Algorithm Engineering: Bridging the Gap Between Algorithm Theory and Practice. Springer, 2010.
  121. Mario A. Muñoz and Kate A. Smith-Miles. Performance Analysis of Continuous Black-Box Optimization Algorithms via Footprints in Instance Space. Evolutionary Computation, 25(4):529-554, December 2017.
  122. Dima Nazzal, Mansooreh Mollaghasemi, H Hedlund, and A Bozorgi. Using Genetic Algorithms and an Indifference- Zone Ranking and Selection Procedure Under Common Random Numbers for Simulation Optimisation. Journal of Simulation, 6(1):56 -66, 2012.
  123. John A Nelder and Roger Mead. A simplex method for function minimization. The computer journal, 7(4):308-313, 1965.
  124. Aneta Neumann, Wanru Gao, Markus Wagner, and Frank Neumann. Evolutionary diversity optimization using multi-objective indicators. In Proceedings of the Genetic and Evolutionary Computation Conference, GECCO '19, page 837-845, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450361118. doi:10.1145/3321707.3321796. URL https://doi.org/10.1145/3321707.3321796. · DOI
  125. Frank Neumann and Carsten Witt. Bioinspired Computation in Combinatorial Optimization -Algorithms and Their Computational Complexity. Springer, 2010.
  126. Trung Thanh Nguyen, Shengxiang Yang, and Juergen Branke. Evolutionary dynamic optimization: A survey of the state of the art. Swarm and Evolutionary Computation, 6:1-24, October 2012. doi:10.1016/j.swevo.2012.05.001. URL https://doi.org/10.1016/j.swevo.2012.05.001. · DOI
  127. Brian A. Nosek, Charles R. Ebersole, Alexander C. DeHaven, and David T. Mellor. The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11):2600-2606, March 2018. ISSN 0027-8424, 1091-6490. doi:10.1073/pnas.1708274114. · DOI · Paper page
  128. Randal S Olson and Jason H Moore. TPOT: A tree-based pipeline optimization tool for automating machine learning. In Workshop on automatic machine learning, pages 66-74, 2016.
  129. Randal S Olson, William La Cava, Patryk Orzechowski, Ryan J Urbanowicz, and Jason H Moore. Pmlb: a large benchmark suite for machine learning evaluation and comparison. BioData mining, 10(1):1-13, 2017.
  130. Patryk Orzechowski, William La Cava, and Jason H Moore. Where are we now? a large benchmark study of recent symbolic regression methods. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 1183-1190, 2018.
  131. Patryk Orzechowski, Franciszek Magiera, and Jason H Moore. Benchmarking manifold learning methods on a large collection of datasets. In European Conference on Genetic Programming (Part of EvoStar), pages 135-150. Springer, 2020.
  132. Tom Packebusch and Stephan Mertens. Low autocorrelation binary sequences. Journal of Physics A: Mathematical and Theoretical, 49(16):165001, 2016. doi:10.1088/1751-8113/49/16/165001. URL https://doi.org/10.1088% 2F1751-8113%2F49%2F16%2F165001. · DOI · Paper page
  133. Ingo Paenke, Jürgen Branke, and Yaochu Jin. Efficient search for robust solutions by means of evolutionary algorithms and fitness approximation. IEEE Trans. Evolutionary Computation, 10(4):405-420, 2006.
  134. Sinno Jialin Pan and Qiang Yang. A Survey on Transfer Learning. IEEE Transactions on Knowledge and Data Engineering, 22(10):1345-1359, Oct 2010.
  135. Fortunato Pesarin. Multivariate permutation tests: with applications in biostatistics, volume 240. Wiley Chichester, 2001.
  136. Robin L Plackett and J Peter Burman. The design of optimum multifactorial experiments. Biometrika, 33(4):305-325, 1946.
  137. Sergey Polyakovskiy, Mohammad Reza Bonyadi, Markus Wagner, Zbigniew Michalewicz, and Frank Neumann. A Comprehensive Benchmark Set and Heuristics for the Traveling Thief Problem. In Proc. of the 2014 Annual Conference on Genetic and Evolutionary Computation, pages 477 --484. ACM, 2014. ISBN 9781450326629.
  138. Karl Raimund Popper. The Logic of Scientific Discovery. Hutchinson & Co, 2 edition, 1959.
  139. Karl Raimund Popper. Objective Knowledge: An Evolutionary Approach. Oxford University Press, 1975.
  140. Mike Preuss. Experimentation in Evolutionary Computation, pages 27-54. Springer, 2015.
  141. Kenneth V. Price. Differential Evolution vs. The Functions of the 2 nd ICEO. In Proc. of the IEEE International Conference on Evolutionary Computation, pages 153 -157. IEEE, 1997.
  142. F. Pukelsheim. Optimal Design of Experiments. Wiley, New York NY, 1993.
  143. Howard Harry Rosenbrock. An Automatic Method for Finding the Greatest or Least Value of a Function. The Computer Journal, 3(3):175 -184, 1960.
  144. Jonathan Rowe and Michael Vose. Unbiased black box search algorithms. In Proc. of Genetic and Evolutionary Computation Conference, pages 2035 -2042. ACM, 2011.
  145. Ranjit K Roy. Design of experiments using the Taguchi approach: 16 steps to product and process improvement. John Wiley & Sons, 2001.
  146. Thomas J Santner, Brian J Williams, William I Notz, and Brain J Williams. The design and analysis of computer experiments, volume 1. Springer, 2003.
  147. Hans-Paul Schwefel. Evolutionsstrategie und numerische Optimierung. PhD thesis, Technische Universität Berlin, Fachbereich Verfahrenstechnik, Berlin, Germany, 1975.
  148. David F Shanno. Conditioning of quasi-newton methods for function minimization. Mathematics of computation, 24 (111):647-656, 1970.
  149. David J Sheskin. Handbook of parametric and nonparametric statistical procedures. crc Press, 2003.
  150. Yuhui Shi and Russell Eberhart. A Modified Particle Swarm Optimizer. In Proc. of the 1998 IEEE International Conference on Evolutionary Computation, within the IEEE World Congress on Computational Intelligence, pages 69-73. IEEE, 1998.
  151. Ofer M. Shir, Carola Doerr, and Thomas Bäck. Compiling a Benchmarking Test-Suite for Combinatorial Black-Box Optimization: A Position Paper. In Proc. of Genetic and Evolutionary Computation Conference, pages 1753 -1760. ACM, 2018.
  152. Kate Smith-Miles and Simon Bowly. Generating new test instances by evolving in instance space. Computers & Operations Research, 63:102-113, November 2015. doi:10.1016/j.cor.2015.04.022. URL https://doi.org/10. 1016/j.cor.2015.04.022. · DOI
  153. Kate Smith-Miles and Thomas T. Tan. Measuring Algorithm Footprints in Instance Space. In 2012 IEEE Congress on Evolutionary Computation. IEEE, 2012.
  154. Krzysztof Socha and Marco Dorigo. Ant colony optimization for continuous domains. European journal of operational research, 185(3):1155-1173, 2008.
  155. Jörg Stork, A. E. Eiben, and Thomas Bartz-Beielstein. A new taxonomy of global optimization algorithms, Nov 2020. ISSN 1572-9796. URL https://doi.org/10.1007/s11047-020-09820-4. · DOI
  156. El-Ghazali Talbi. Metaheuristics: From Design to Implementation. John Wiley & Sons Inc., July 2009. ISBN 978-0-470-27858-1.
  157. Shigeyoshi Tsutsui, Ashish Ghosh, and Yoshiji Fujimoto. A Robust Solution Searching Scheme in Genetic Search. In International Conference on Parallel Problem Solving from Nature, pages 543 -552. Springer, 1996.
  158. Tea Tušar, Dimo Brockhoff, and Nikolaus Hansen. Mixed-integer benchmark problems for single-and bi-objective optimization. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 718 --726. ACM, 2019.
  159. Vanessa Volz, Boris Naujoks, Pascal Kerschke, and Tea Tušar. Single-and Multi-Objective Game-Benchmark for Evolutionary Algorithms. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 647 - 655. ACM, 2019.
  160. Wouter Vrielink and Daan van den Berg. Fireworks algorithm versus plant propagation algorithm. In IJCCI, pages 101-112, 2019.
  161. Tobias Wagner. A subjective review of the state of the art in model-based parameter tuning. In Thomas Bartz- Beielstein, Marco Chiarandini, Luis Paquete, and Mike Preuss, editors, Workshop on Experimental Methods for the Assessment of Computational Systems (WEMACS 2010), Algorithm Engineering Report, pages 1-13. TU Dortmund, Faculty of Computer Science, Algorithm Engineering (Ls11), 2010.
  162. Hao Wang, Diederick Vermetten, Furong Ye, Carola Doerr, and Thomas Bäck. Iohanalyzer: Performance analysis for iterative optimization heuristic. CoRR, abs/2007.03953, 2020. URL https://arxiv.org/abs/2007.03953. A benchmark data repository is available through the web-based GUI at iohprofiler.liacs.nl/. · arXiv
  163. Jann Michael Weinand, Kenneth Sörensen, Pablo San Segundo, Max Kleinebrahm, and Russell McKenna. Research trends in combinatorial optimisation. arXiv e-prints, art. arXiv:2012.01294, December 2020. · arXiv
  164. Thomas Weise and Zijun Wu. Difficult features of combinatorial optimization problems and the tunable w-model benchmark problem for simulating them. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, GECCO '18, pages 1769 --1776. ACM, 2018. ISBN 9781450357647.
  165. Thomas Weise, Raymond Chiong, Ke Tang, Jörg Lässig, Shigeyoshi Tsutsui, Wenxiang Chen, Zbigniew Michalewicz, and Xin Yao. Benchmarking optimization algorithms: An open source framework for the traveling salesman problem. IEEE Computational Intelligence Magazine (CIM), 9:40-52, August 2014.
  166. L. Darrell Whitley, Soraya B. Rana, John Dzubera, and Keith E. Mathias. Evaluating Evolutionary Algorithms. Artificial Intelligence (AIJ), 85(1-2):245 -276, 1996.
  167. L. Darrell Whitley, Jean-Paul Watson, Adele Howe, and Laura Barbulescu. Testing, Evaluation and Performance of Optimization and Learning Systems. In Ian C. Parmee, editor, Adaptive Computing in Design and Manufacture V, pages 27 -39, London, 2002. Springer.
  168. Frank Wilcoxon. Individual comparisons by ranking methods. Biometrics bulletin, 1(6):80-83, 1945.
  169. David H. Wolpert and William G. Macready. No Free Lunch Theorems for Optimization. IEEE Transactions on Evolutionary Computation, 1(1):67-82, April 1997.
  170. Lin Xu, Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. SATzilla: portfolio-based algorithm selection for SAT. Journal of Artificial Intelligence Research, 32:565-606, June 2008.

Cited by

Sorted by citations

Metadata

Authors
Bartz-Beielstein, T., Doerr, C., Berg, D. V. D., Bossek, J., Chandrasekaran, S., Eftimov, T., Fischbach, A., Kerschke, P., La Cava, W., Lopez-Ibanez, M., Malan, K. M., Moore, J. H., Naujoks, B., Orzechowski, P., Volz, V., Wagner, M., & Weise, T.
Affiliations
  • Centre National de la Recherche Scientifique (FR)
  • University of Münster (DE)
  • TH Köln - University of Applied Sciences (DE)
  • Jožef Stefan Institute (SI)
  • University of South Africa (ZA)
  • University of Pennsylvania (US)
  • AGH University of Krakow (PL)
  • The University of Adelaide (AU)
  • Hefei University (CN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2007.03488
arXiv
2007.03488
OpenAlex
W3038891809
Cited by
214
References
5
Citations by year
2026: 34 · 2025: 47 · 2024: 34 · 2023: 38 · 2022: 34 · 2021: 23 · 2020: 3
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, other-oa)
  • Cologne Open Science (RFH - University of Applied Sciences Cologne) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University)
Funders
  • European Commission (722734)
  • National Research Foundation (120837)
  • Deutscher Akademischer Austauschdienst (57515062)
  • Agence Nationale de la Recherche (ANR-11-LABX-0056)
  • National Natural Science Foundation of China (61673359)
  • Bundesministerium für Bildung und Forschung (13FH007IB6)
  • Javna Agencija za Raziskovalno Dejavnost RS (Z2-1867, P2-0098)
  • Ministerio de Ciencia e Innovación
  • Ministerium für Kultur und Wissenschaft des Landes Nordrhein-Westfalen
  • National Institutes of Health
  • Horizon 2020 Framework Programme (ANR-11-LABX-0056-LMH, H2020-MSCA, MSCA-ITN-2016, 722734, H2020-MSCA-ITN, H2020‐MSCA‐ITN‐2016)
  • U.S. National Library of Medicine
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Metaheuristic Optimization Algorithms Research
  • Physical Sciences › Mathematics › Numerical Analysis › Advanced Optimization Algorithms Research
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Simulation Techniques and Applications
Keywords
benchmarking, optimization, best practice guidelines, reproducibility, performance indicators, benchmark design, problem specification, algorithm comparison

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review