The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
A review of multiobjective test problems and a scalable test problem toolkit
Abstract
When attempting to better understand the strengths and weaknesses of an algorithm, it is important to have a strong understanding of the problem at hand. This is true for the field of multiobjective evolutionary algorithms (EAs) as it is for any other field. Many of the multiobjective test problems employed in the EA literature have not been rigorously analyzed, which makes it difficult to draw accurate conclusions about the strengths and weaknesses of the algorithms tested on them. In this paper, we systematically review and analyze many problems from the EA literature, each belonging to the important class of real-valued, unconstrained, multiobjective test problems. To support this, we first introduce a set of test problem criteria, which are in turn supported by a set of definitions. Our analysis of test problems highlights a number of areas requiring attention. Not only are many test problems poorly constructed but also the important class of nonseparable problems, particularly nonseparable multimodal problems, is poorly represented. Motivated by these findings, we present a flexible toolkit for constructing well-designed test problems. We also present empirical results demonstrating how the toolkit can be used to test an optimizer in ways that existing test suites do not
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A fast and elitist multiobjective genetic algorithm: NSGA-II
Deb, K., Pratap, A., Agarwal, S., & Meyarivan, T. (2002) · IEEE Transactions on Evolutionary Computation · doi:10.1109/4235.996017 · Cited by 50,053
Multi-Objective Optimization Using Evolutionary Algorithms
Deb, K., & Kalyanmoy, D. (2001) · Andalas University Repository (Andalas University) · Cited by 17,326
No free lunch theorems for optimization
Wolpert, D. H., & Macready, W. G. (1997) · IEEE Transactions on Evolutionary Computation · doi:10.1109/4235.585893 · Cited by 14,237
Goldberg, D. E. (1989) · Medical Entomology and Zoology · Cited by 10,109
Comparison of Multiobjective Evolutionary Algorithms: Empirical Results
Zitzler, E., Deb, K., & Thiele, L. (2000) · Evolutionary Computation · doi:10.1162/106365600568202 · Cited by 5,697
Performance assessment of multiobjective optimizers: an analysis and review
Zitzler, E., Thiele, L., Laumanns, M., Fonseca, C. M., & da Fonseca, V. G. (2003) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2003.810758 · Cited by 4,330
Randomization, Bootstrap and Monte Carlo Methods in Biology
Manly, B. F. J. (2018) · Chapman & Hall/CRC eBooks · doi:10.1201/9781315273075 · Cited by 4,194
Multiple Objective Optimization with Vector Evaluated Genetic Algorithms
Schaffer, J. D. (1985) · international conference on Genetic algorithms · Cited by 3,326
Handbook of Evolutionary Computation
(1997) · Institute of Physics eBooks · doi:10.1887/0750308958 · Cited by 2,990
Genetic Algorithms in Search, Optimization & Machine Learning
Goldberg, D. E. (1989) · Medical Entomology and Zoology · Cited by 2,304
Cited by
Platemo: a matlab platform for evolutionary multi-objective optimization [educational forum]
Tian, Y., Cheng, R., Zhang, X., & Jin, Y. (2017) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2017.2742868 · Cited by 2,777
Multiobjective Optimization Problems With Complicated Pareto Sets, MOEA/D and NSGA-II
Li, H., & Zhang, Q. (2009) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2008.925798 · Cited by 2,543
Multiobjective evolutionary algorithms: A survey of the state of the art
Zhou, A., Qu, B.-Y., Li, H., Zhao, S.-Z., Suganthan, P. N., & Zhang, Q. (2011) · Swarm and Evolutionary Computation · doi:10.1016/j.swevo.2011.03.001 · Cited by 2,255
HypE: An Algorithm for Fast Hypervolume-Based Many-Objective Optimization
Bader, J., & Zitzler, E. (2011) · Evolutionary Computation · doi:10.1162/evco_a_00009 · Cited by 2,081
A Reference Vector Guided Evolutionary Algorithm for Many-Objective Optimization
Cheng, R., Jin, Y., Olhofer, M., & Sendhoff, B. (2016) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2016.2519378 · Cited by 1,760
An Evolutionary Many-Objective Optimization Algorithm Based on Dominance and Decomposition
Li, K., Deb, K., Zhang, Q., & Kwong, S. (2015) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2014.2373386 · Cited by 1,208
jMetal: A Java framework for multi-objective optimization
Durillo, J. J., & Nebro, A. J. (2011) · Advances in Engineering Software · doi:10.1016/j.advengsoft.2011.05.014 · Cited by 1,134
A Grid-Based Evolutionary Algorithm for Many-Objective Optimization
Yang, S., Li, M., Liu, X., & Zheng, J. (2013) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2012.2227145 · Cited by 893
A Knee Point-Driven Evolutionary Algorithm for Many-Objective Optimization
Zhang, X., Tian, Y., & Jin, Y. (2015) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2014.2378512 · Cited by 867
Many-Objective Evolutionary Algorithms: A Survey
Li, B., Li, J., Tang, K., & Yao, X. (2015) · ACM Computing Surveys · doi:10.1145/2792984 · Cited by 815
Liu, H.-L., Gu, F., & Zhang, Q. (2014) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2013.2281533 · Cited by 813
A New Dominance Relation-Based Evolutionary Algorithm for Many-Objective Optimization
Yuan, Y., Xu, H., Wang, B., & Yao, X. (2016) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2015.2420112 · Cited by 752
Borg: An Auto-Adaptive Many-Objective Evolutionary Computing Framework
Hadka, D., & Reed, P. (2013) · Evolutionary Computation · doi:10.1162/evco_a_00075 · Cited by 717
Tian, Y., Cheng, R., Zhang, X., Cheng, F., & Jin, Y. (2018) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2017.2749619 · Cited by 699
Review: Multi-objective optimization methods and application in energy saving
Cui, Y., Geng, Z., Zhu, Q., & Han, Y. (2017) · Energy · doi:10.1016/j.energy.2017.02.174 · Cited by 685
Zhang, X., Tian, Y., Cheng, R., & Jin, Y. (2018) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2016.2600642 · Cited by 660
Performance of Decomposition-Based Many-Objective Algorithms Strongly Depends on Pareto Front Shapes
Ishibuchi, H., Setoguchi, Y., Masuda, H., & Nojima, Y. (2017) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2016.2587749 · Cited by 592
Two_Arch2: An Improved Two-Archive Algorithm for Many-Objective Optimization
Wang, H., Jiao, L., & Yao, X. (2015) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2014.2350987 · Cited by 558
A Survey of Multiobjective Evolutionary Algorithms based on Decomposition
Trivedi, A., Srinivasan, D., Sanyal, K., & Ghosh, A. (2016) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2016.2608507 · Cited by 549
Ma, X., Liu, F., Qi, Y., Wang, X., Li, L., Jiao, L., Yin, M., & Gong, M. (2016) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2015.2455812 · Cited by 523
IGD Indicator-Based Evolutionary Algorithm for Many-Objective Optimization Problems
Sun, Y., Yen, G. G., & Yi, Z. (2019) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2018.2791283 · Cited by 519
Liu, Z.-Z., & Wang, Y. (2019) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2019.2894743 · Cited by 486
Efficient Large-Scale Multiobjective Optimization Based on a Competitive Swarm Optimizer
Tian, Y., Zheng, X., Zhang, X., & Jin, Y. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2019.2906383 · Cited by 480
Ma, Z., & Wang, Y. (2019) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2019.2896967 · Cited by 458
Pan, L., He, C., Tian, Y., Wang, H., Zhang, X., & Jin, Y. (2019) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2018.2802784 · Cited by 456
An improved epsilon constraint-handling method in MOEA/D for CMOPs with large infeasible regions
Fan, Z., Li, W., Cai, X., Huang, H., Fang, Y., You, Y., Mo, J., Wei, C., & Goodman, E. (2019) · Soft Computing · doi:10.1007/s00500-019-03794-x · Cited by 450
An Evolutionary Algorithm for Large-Scale Sparse Multiobjective Optimization Problems
Tian, Y., Zhang, X., Wang, C., & Jin, Y. (2020) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2019.2918140 · Cited by 440
Automated Algorithm Selection: Survey and Perspectives
Kerschke, P., Hoos, H. H., Neumann, F., & Trautmann, H. (2019) · Evolutionary Computation · doi:10.1162/evco_a_00242 · Cited by 435
Test Problems for Large-Scale Multiobjective and Many-Objective Optimization
Cheng, R., Jin, Y., Olhofer, M., & sendhoff, B. (2017) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2016.2600577 · Cited by 427
A competitive mechanism based multi-objective particle swarm optimizer with fast convergence
Zhang, X., Zheng, X., Cheng, R., Qiu, J., & Jin, Y. (2018) · Information Sciences · doi:10.1016/j.ins.2017.10.037 · Cited by 425
Li, K., Fialho, A., Kwong, S., & Zhang, Q. (2014) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2013.2239648 · Cited by 418
A Vector Angle-Based Evolutionary Algorithm for Unconstrained Many-Objective Optimization
Xiang, Y., Zhou, Y., Li, M., & Chen, Z. (2017) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2016.2587808 · Cited by 415
Preference-Inspired Coevolutionary Algorithms for Many-Objective Optimization
Wang, R., Purshouse, R. C., & Fleming, P. J. (2013) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2012.2204264 · Cited by 406
A Decomposition-Based Evolutionary Algorithm for Many Objective Optimization
Asafuddoula, M., Ray, T., & Sarker, R. (2015) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2014.2339823 · Cited by 403
Evolutionary Large-Scale Multi-Objective Optimization: A Survey
Tian, Y., Si, L., Zhang, X., Cheng, R., He, C., Tan, K. C., & Jin, Y. (2022) · ACM Computing Surveys · doi:10.1145/3470971 · Cited by 401
Balancing Convergence and Diversity in Decomposition-Based Many-Objective Optimizers
Yuan, Y., Xu, H., Wang, B., Zhang, B., & Yao, X. (2016) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2015.2443001 · Cited by 400
Direct Multisearch for Multiobjective Optimization
Custódio, A. L., Madeira, J. F. A., Vaz, A. I. F., & Vicente, L. N. (2011) · SIAM Journal on Optimization · doi:10.1137/10079731x · Cited by 392
Tian, Y., Cheng, R., Zhang, X., Su, Y., & Jin, Y. (2019) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2018.2866854 · Cited by 384
A Survey on the Hypervolume Indicator in Evolutionary Multiobjective Optimization
Shang, K., Ishibuchi, H., He, L., & Pang, L. M. (2021) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2020.3013290 · Cited by 380
A Multiobjective Evolutionary Algorithm Using Gaussian Process-Based Inverse Modeling
Cheng, R., Jin, Y., Narukawa, K., & Sendhoff, B. (2015) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2015.2395073 · Cited by 377
A Framework for Large-Scale Multiobjective Optimization Based on Problem Transformation
Zille, H., Ishibuchi, H., Mostaghim, S., & Nojima, Y. (2018) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2017.2704782 · Cited by 377
Jiang, S., & Yang, S. (2017) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2016.2592479 · Cited by 376
Zhou, A., Zhang, Q., & Jin, Y. (2009) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2009.2021467 · Cited by 362
Localized Weighted Sum Method for Many-Objective Optimization
Wang, R., Zhou, Z., Ishibuchi, H., Liao, T., & Zhang, T. (2018) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2016.2611642 · Cited by 362
Accelerating Large-Scale Multiobjective Optimization via Problem Reformulation
He, C., Li, L., Tian, Y., Zhang, X., Cheng, R., Jin, Y., & Yao, X. (2019) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2019.2896002 · Cited by 357
Fuzzy-Based Pareto Optimality for Many-Objective Evolutionary Algorithms
He, Z., Yen, G. G., & Zhang, J. (2014) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2013.2258025 · Cited by 350
Zhan, Z.-H., Li, J., Cao, J., Zhang, J., Chung, H. S.-H., & Shi, Y.-H. (2013) · IEEE Transactions on Cybernetics · doi:10.1109/tsmcb.2012.2209115 · Cited by 346
Hua, Y., Liu, Q., Hao, K., & Jin, Y. (2021) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2021.1003817 · Cited by 346
A Multimodal Multiobjective Evolutionary Algorithm Using Two-Archive and Recombination Strategies
Liu, Y., Yen, G. G., & Gong, D. (2019) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2018.2879406 · Cited by 325
An easy-to-use real-world multi-objective optimization problem suite
Tanabe, R., & Ishibuchi, H. (2020) · Applied Soft Computing · doi:10.1016/j.asoc.2020.106078 · Cited by 324
Ma, L., Huang, M., Yang, S., Wang, R., & Wang, X. (2022) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.3041212 · Cited by 318
Pareto or Non-Pareto: Bi-Criterion Evolution in Multiobjective Optimization
Li, M., Yang, S., & Liu, X. (2016) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2015.2504730 · Cited by 309
Difficulty Adjustable and Scalable Constrained Multiobjective Test Problem Toolkit
Fan, Z., Li, W., Cai, X., Li, H., Wei, C., Zhang, Q., Deb, K., & Goodman, E. (2020) · Evolutionary Computation · doi:10.1162/evco_a_00259 · Cited by 308
Objective Reduction in Many-Objective Optimization: Linear and Nonlinear Algorithms
Saxena, D. K., Duro, J. A., Tiwari, A., Deb, K., & Zhang, Q. (2013) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2012.2185847 · Cited by 302
A survey on multi-objective evolutionary algorithms for many-objective problems
von Lücken, C., Barán, B., & Brizuela, C. (2014) · Computational Optimization and Applications · doi:10.1007/s10589-014-9644-1 · Cited by 297
A New Decomposition-Based NSGA-II for Many-Objective Optimization
Elarbi, M., Bechikh, S., Gupta, A., Ben Said, L., & Ong, Y.-S. (2018) · IEEE Transactions on Systems, Man, and Cybernetics: Systems · doi:10.1109/tsmc.2017.2654301 · Cited by 290
Decomposition-Based Algorithms Using Pareto Adaptive Scalarizing Methods
Wang, R., Zhang, Q., & Zhang, T. (2016) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2016.2521175 · Cited by 283
The jMetal framework for multi-objective optimization: Design and architecture
Durillo, J. J., Nebro, A. J., & Alba, E. (2010) · IEEE Congress on Evolutionary Computation · doi:10.1109/cec.2010.5586354 · Cited by 283
Use of cooperative coevolution for solving large scale multiobjective optimization problems
Antonio, L. M., & Coello, C. A. C. (2013) · 2013 IEEE Congress on Evolutionary Computation · doi:10.1109/cec.2013.6557903 · Cited by 276
Methods for multi-objective optimization: An analysis
Giagkiozis, I., & Fleming, P. J. (2015) · Information Sciences · doi:10.1016/j.ins.2014.08.071 · Cited by 275
Diversity Assessment in Many-Objective Optimization
Wang, H., Jin, Y., & Yao, X. (2017) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2016.2550502 · Cited by 274
A Many-Objective Evolutionary Algorithm Using A One-by-One Selection Strategy
Liu, Y., Gong, D., Sun, J., & Jin, Y. (2017) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2016.2638902 · Cited by 271
Diversity Management in Evolutionary Many-Objective Optimization
Adra, S. F., & Fleming, P. J. (2011) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2010.2058117 · Cited by 262
A Kriging-Assisted Two-Archive Evolutionary Algorithm for Expensive Many-Objective Optimization
Song, Z., Wang, H., He, C., & Jin, Y. (2021) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2021.3073648 · Cited by 260
A Simple and Fast Hypervolume Indicator-Based Multiobjective Evolutionary Algorithm
Jiang, S., Zhang, J., Ong, Y.-S., Zhang, A. N., & Tan, P. S. (2015) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2014.2367526 · Cited by 259
Benchmarking in Optimization: Best Practice and Open Issues
Bartz-Beielstein, T., Doerr, C., Berg, D. V. D., Bossek, J., Chandrasekaran, S., Eftimov, T., Fischbach, A., Kerschke, P., La Cava, W., Lopez-Ibanez, M., et al. (2020) · arXiv · doi:10.48550/arxiv.2007.03488 · Cited by 214
Bayesian networks for interpretable machine learning and optimization
Mihaljević, B., Bielza, C., & Larrañaga, P. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.01.138 · Cited by 123
“Joint modeling of objectives and variables for the 5-objective WFG1 optimization problem [164].”
MethodMulti-objective Bayesian optimization using Pareto-frontier entropy
Suzuki, S., Takeno, S., Tamura, T., Shitara, K., & Karasuyama, M. (2019) · arXiv · doi:10.48550/arxiv.1906.00127 · Cited by 40
Metadata
- Authors
- Huband, S., Hingston, P., Barone, L., & While, L.
- Venue
- IEEE Transactions on Evolutionary Computation
- Published
- 2006-10
- Volume
- 10
- Issue
- 5
- Pages
- 477-506
- ISSN
- 1089-778X (ISSN-L)
- OpenAlex
- W2143185749
- Cited by
- 2,090
- References
- 40
- FWCI
- 20.16
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.