The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Resampling Methods for Meta-Model Validation with Recommendations for Evolutionary Computation
Abstract
Meta-modeling has become a crucial tool in solving expensive optimization problems. Much of the work in the past has focused on finding a good regression method to model the fitness function. Examples include classical linear regression, splines, neural networks, Kriging and support vector regression. This paper specifically draws attention to the fact that assessing model accuracy is a crucial aspect in the meta-modeling framework. Resampling strategies such as cross-validation, subsampling, bootstrapping, and nested resampling are prominent methods for model validation and are systematically discussed with respect to possible pitfalls, shortcomings, and specific features. A survey of meta-modeling techniques within evolutionary optimization is provided. In addition, practical examples illustrating some of the pitfalls associated with model selection and performance assessment are presented. Finally, recommendations are given for choosing a model validation technique for a particular setting.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
Kohavi, R. (1995) · Zenodo · doi:10.5281/zenodo.19712698 · Cited by 10,909
Efficient Global Optimization of Expensive Black-Box Functions
Jones, D. R., Schonlau, M., & Welch, W. J. (1998) · Journal of Global Optimization · doi:10.1023/a:1008306431147 · Cited by 8,070
Estimating the error rate of a prediction rule: improvement on cross-validation
Efron, B. (1983) · Journal of the American Statistical Association · doi:10.1080/01621459.1983.10477973 · Cited by 2,300
Linear Model Selection by Cross-validation
Shao, J. (1993) · Journal of the American Statistical Association · doi:10.1080/01621459.1993.10476299 · Cited by 1,790
An asymptotic equivalence of choice of model by cross validation and Akaike's criterion
Stone, M. (1977) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1977.tb01603.x · Cited by 1,286
Cited by
Automated Machine Learning: Methods, Systems, Challenges
(2019) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-030-05318-5 · Cited by 1,485
Feurer, M., & Hutter, F. (2019) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-030-05318-5_1 · Cited by 1,414
Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges
Bischl, B., Binder, M., Lang, M., Pielok, T., Richter, J., Coors, S., Thomas, J., Ullmann, T., Becker, M., Boulesteix, A., et al. (2023) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1484 · Cited by 930
Self-citationRandom forest versus logistic regression: a large-scale benchmark experiment
Couronné, R., Probst, P., & Boulesteix, A.-L. (2018) · BMC Bioinformatics · doi:10.1186/s12859-018-2264-5 · Cited by 864
Benchmark for filter methods for feature selection in high-dimensional classification data
Bommert, A., Sun, X., Bischl, B., Rahnenführer, J., & Lang, M. (2020) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2019.106839 · Cited by 725
Self-citationAdvances in surrogate based modeling, feasibility analysis, and optimization: A review
Bhosekar, A., & Ierapetritou, M. (2018) · Computers & Chemical Engineering · doi:10.1016/j.compchemeng.2017.09.017 · Cited by 707
Global prevalence of non-perennial rivers and streams
Messager, M. L., Lehner, B., Cockburn, C., Lamouroux, N., Pella, H., Snelder, T., Tockner, K., Trautmann, T., Watt, C., & Datry, T. (2021) · Nature · doi:10.1038/s41586-021-03565-5 · Cited by 581
Alibrahim, H., & Ludwig, S. A. (2021) · 2021 IEEE Congress on Evolutionary Computation (CEC) · doi:10.1109/cec45853.2021.9504761 · Cited by 489
Design and Analysis of Simulation Experiments
Kleijnen, J. P. C. (2018) · Springer Proceedings in Mathematics & Statistics · doi:10.1007/978-3-319-76035-3_1 · Cited by 449
TSE-IDS: A Two-Stage Classifier Ensemble for Intelligent Anomaly-Based Intrusion Detection System
Tama, B. A., Comuzzi, M., & Rhee, K.-H. (2019) · IEEE Access · doi:10.1109/access.2019.2928048 · Cited by 386
Regression and Kriging metamodels with their experimental designs in simulation: A review
Kleijnen, J. P. C. (2017) · European Journal of Operational Research · doi:10.1016/j.ejor.2016.06.041 · Cited by 324
General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Models
Molnar, C., König, G., Herbinger, J., Freiesleben, T., Dandl, S., Scholbeck, C. A., Casalicchio, G., Grosse-Wentrup, M., & Bischl, B. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-04083-2_4 · Cited by 211
“These resampling procedures are readily available in software [58,76], and well-studied in theory as well as practice [3,9,88], although rigorous analysis of cross-validation is still considered an open problem [87].”
BackgroundSelf-citationBenchmark of filter methods for feature selection in high-dimensional gene expression survival data
Bommert, A., Welchowski, T., Schmid, M., & Rahnenführer, J. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbab354 · Cited by 182
A Machine Learning Tutorial for Operational Meteorology. Part I: Traditional Machine Learning
Chase, R. J., Harrison, D. R., Burke, A., Lackmann, G. M., & McGovern, A. (2022) · Weather and Forecasting · doi:10.1175/waf-d-22-0070.1 · Cited by 25
“In the event that the total dataset is small and splitting the data into smaller subsets creates less robust statistics, a resampling method known as k-fold cross validation (e.g., Bischl et al. 2012; Goodfellow et al. 2016) can be used.”
Method
Metadata
- Authors
- Bischl, B., Mersmann, O., Trautmann, H., & Weihs, C.
- Venue
- Evolutionary Computation
- Publisher
- MIT Press
- Published
- 2012-06
- Volume
- 20
- Issue
- 2
- Pages
- 249-275
- ISSN
- 1063-6560 (ISSN-L)
- 1530-9304
- OpenAlex
- W2083542829
- Cited by
- 186
- References
- 5
- FWCI
- 8.15
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.