Derivative-free optimization: a review of algorithms and comparison of software implementations
Open accessThe DOI matches its Crossref record; 130 of the 132 references it cites were found in a registry or an index.
Abstract
This paper addresses the solution of bound-constrained optimization problems using algorithms that require only the availability of objective function values but no derivative information. We refer to these algorithms as derivative-free algorithms. Fueled by a growing number of applications in science and engineering, the development of derivative-free optimization algorithms has long been studied, and it has found renewed interest in recent time. Along with many derivative-free algorithms, many software implementations have also appeared. The paper presents a review of derivative-free algorithms, followed by a systematic comparison of 22 related implementations using a test set of 502 problems. The test bed includes convex and nonconvex problems, smooth as well as nonsmooth problems. The algorithms were tested under the same conditions and ranked under several criteria, including their ability to find near-global solutions for nonconvex problems, improve a given starting point, and refine a near-optimal solution. A total of 112,448 problem instances were solved. We find that the ability of all these solvers to obtain good solutions diminishes with increasing problem size. For the problems used in this study, , , and are better, on average, than other derivative-free solvers in terms of solution quality within 2,500 function evaluations. These global solvers outperform local solvers even for convex problems. Finally, , , and show superior performance in terms of refining a near-optimal solution.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
































Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In the context of quantum chemistry, research has focused on the analog simulation of molecules by mapping the electronic structure problem to a Hamiltonian of spins with additional coupling terms 366 , although proposals exist for simulating chemical reactions 367-369 and protein folding simulations 370,371 .”
From Quantum Chemistry in the Age of Quantum Computing (2019) · cites this paper as [371]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Kennedy, J., & Eberhart, R. (2002) · Proceedings of ICNN'95 - International Conference on Neural Networks · doi:10.1109/icnn.1995.488968 · Cited by 58,192
Holland, J. H. (1992) · The MIT Press eBooks · doi:10.7551/mitpress/1090.001.0001 · Cited by 47,002 · cited 3×
“Examples of methods Table 1 Citations of most cited works in derivative-free algorithms Nelder and Mead [99] 1965 13,486 Brent [25] 1973 2,019 Holland [58] 1975 31,494 Kirkpatrick et al. [78] 1983 23,053 Eberhart and Kennedy [44,77] 1995 20,369 using information from multiple previously evaluated points are genetic al…”
MethodOptimization by Simulated Annealing
Kirkpatrick, S., Gelatt, C. D., & Vecchi, M. P. (1983) · Science · doi:10.1126/science.220.4598.671 · Cited by 45,134 · cited 2×
“Initially proposed to handle combinatorial optimization problems [78], the algorithm was later extended to continuous problems [17].”
Equation of State Calculations by Fast Computing Machines
Metropolis, N., Rosenbluth, A. W., Rosenbluth, M. N., Teller, A. H., & Teller, E. (1953) · The Journal of Chemical Physics · doi:10.1063/1.1699114 · Cited by 37,192
A Simplex Method for Function Minimization
Nelder, J. A., & Mead, R. (1965) · The Computer Journal · doi:10.1093/comjnl/7.4.308 · Cited by 28,905 · cited 11×
“The development of derivative-free algorithms dates back to the works of Spendley et al. [132] and Nelder and Mead [99] with their simplex-based algorithms.”
BackgroundMethodInfluentialA new optimizer using particle swarm theory
Eberhart, R., & Kennedy, J. (2002) · MHS'95. Proceedings of the Sixth International Symposium on Micro Machine and Human Science · doi:10.1109/mhs.1995.494215 · Cited by 16,067 · cited 3×
“Examples of methods Table 1 Citations of most cited works in derivative-free algorithms Nelder and Mead [99] 1965 13,486 Brent [25] 1973 2,019 Holland [58] 1975 31,494 Kirkpatrick et al. [78] 1983 23,053 Eberhart and Kennedy [44,77] 1995 20,369 using information from multiple previously evaluated points are genetic al…”
MethodEfficient Global Optimization of Expensive Black-Box Functions
Jones, D. R., Schonlau, M., & Welch, W. J. (1998) · Journal of Global Optimization · doi:10.1023/a:1008306431147 · Cited by 8,070 · cited 3×
“Although the expected improvement function can be reduced to a closed-form expression [73], it can be highly multimodal.”
BackgroundMethodConvergence Properties of the Nelder--Mead Simplex Method in Low Dimensions
Lagarias, J. C., Reeds, J. A., Wright, M. H., & Wright, P. E. (1998) · SIAM Journal on Optimization · doi:10.1137/s1052623496303470 · Cited by 7,447 · cited 1×
“[81].”
Design and Analysis of Computer Experiments
Sacks, J., Welch, W. J., Mitchell, T. J., & Wynn, H. P. (1989) · Statistical Science · doi:10.1214/ss/1177012413 · Cited by 6,919
(2019) · Handbook of Machine Learning · doi:10.1142/9789811205675_0004 · Cited by 6,270
Research works printed but not linked: 9
- Awasthi, S.: Molecular Docking by Derivative-Free Optimization Solver. Master's thesis, Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh (2008)
- Booker, A.J., Dennis, J.E. Jr., Frank, P.D., Serafini, D.B., Torczon, V.J., Trosset, M.W.: A rigorous framework for optimization of expensive functions by surrogates. Struct. Optim. 17, 1-13 (1999) · DOI · Paper page
- Chang, K.-F.: Modeling and Optimization of Polymerase Chain Reaction Using Derivative-Free Opti- mization. Master's thesis, Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh (2011)
- Desai, R.: A Comparison of Algorithms for Optimizing the Omega Function. Master's thesis, Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh (2010)
- Holland, J.H.: Adaptation in Natural and Artificial Systems. The University of Michigan Press, Ann Arbor (1975)
- Powell, M.J.D.: A direct search optimization method that models the objective by quadratic interpolation. In: Presentation at the 5th Stockholm Optimization Days (1994) 111. · DOI · Paper page
- Wang, H.: Application of Derivative-Free Algorithms in Powder Diffraction. Master's thesis, Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh (2011) 140.
- Winfield, D.: Function and Functional Optimization by Interpolation in Data Tables. PhD thesis, Harvard University, Cambridge (1969) 142.
- Zheng, Y.: Pairs Trading and Portfolio Optimization. Master's thesis, Department of Chemical Engi- neering, Carnegie Mellon University, Pittsburgh (2011)
Cited by
The theory of variational hybrid quantum-classical algorithms
McClean, J. R., Romero, J., Babbush, R., & Aspuru-Guzik, A. (2016) · New Journal of Physics · doi:10.1088/1367-2630/18/2/023023 · Cited by 2,227
Quantum Chemistry in the Age of Quantum Computing
Cao, Y., Romero, J., Olson, J. P., Degroote, M., Johnson, P. D., Kieferová, M., Kivlichan, I. D., Menke, T., Peropadre, B., Sawaya, N. P. D., et al. (2019) · Chemical Reviews · doi:10.1021/acs.chemrev.8b00803 · Cited by 1,560
Accessorize to a Crime: Real and Stealthy Attacks on State-of-the-Art Face Recognition
Sharif, M., Bhagavatula, S., Bauer, L., & Reiter, M. K. (2016) · Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2976749.2978392 · Cited by 1,530
“…over other black-box optimization meth-ods such as surrogate models [3]—which require that the adversary has the training data and enough computational resources to train an effective surrogate—and genetic algorithms [2]—which though similar to PSO are much less computationally efficient [15, 34].”
BackgroundLattimore, T., & Szepesvári, C. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108571401 · Cited by 976
A Survey of Optimization Methods From a Machine Learning Perspective
Sun, S., Cao, Z., Zhu, H., & Zhao, J. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2019.2950779 · Cited by 789
“The derivative-free optimization methods [11], [12] are mainly used in the case that the derivative of the objective function may not exist or be difficult to calculate.”
Method
Metadata
- Authors
- Rios, L. M., & Sahinidis, N. V.
- Affiliations
- Carnegie Mellon University (US)
- Corresponding author
- Sahinidis, Nikolaos V.
- Venue
- Journal of Global Optimization
- Published
- 2012-07-12
- Volume
- 56
- Issue
- 3
- Pages
- 1247-1293
- ISSN
- 0925-5001 (ISSN-L)
- 1573-2916
- OpenAlex
- W2160960847
- Cited by
- 1,276
- References
- 132
- Citations by year
- 2026: 57 · 2025: 65 · 2024: 79 · 2023: 99 · 2022: 114 · 2021: 138 · 2020: 167 · 2019: 125 · 2018: 108 · 2017: 93 · 2016: 87 · 2015: 57 · 2014: 58 · 2013: 15 · 2012: 8
- FWCI
- 52.92
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/2.0 (tdm)
- creativecommons.org/licenses/by/2.0 (vor)
- Locations
- Journal of Global Optimization (publishedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- Funders
- National Science Foundation (1033661, DE-FE-0004000)
- National Institutes of Health
- National Energy Technology Laboratory (DE-FE-0004000, 0004000)
- National Institute of General Medical Sciences (GM072023)
- Division of Chemical, Bioengineering, Environmental, and Transport Systems
- Topics
- Physical Sciences › Mathematics › Numerical Analysis › Advanced Optimization Algorithms Research
- Physical Sciences › Computer Science › Artificial Intelligence › Metaheuristic Optimization Algorithms Research
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Optimization and Variational Analysis
- Keywords
- derivative-free optimization, bound-constrained optimization, global optimization, local optimization, solver comparison, non-convex optimization, nonsmooth optimization, convex optimization, solution refinement, function evaluations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Derivative-free optimization: a review of algorithms and comparison of software implementations. https://app.cochat.ai/papers/10.1007/s10898-012-9951-y
- The paper
- Rios, L. M., & Sahinidis, N. V. (2013). Derivative-free optimization: a review of algorithms and comparison of software implementations. Journal of Global Optimization. https://doi.org/10.1007/s10898-012-9951-y
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.