The DOI matches its Crossref record; 72 of the 72 references it cites were found in a registry or an index.
Cross-disciplinary perspectives on meta-learning for algorithm selection
Abstract
The algorithm selection problem [Rice 1976] seeks to answer the question:Which algorithm is likely to perform best for my problem?Recognizing the problem as a learning task in the early 1990's, the machine learning community has developed the field of meta-learning, focused on learning about learning algorithm performance on classification problems. But there has been only limited generalization of these ideas beyond classification, and many related attempts have been made in other disciplines (such as AI and operations research) to tackle the algorithm selection problem in different ways, introducing different terminology, and overlooking the similarities of approaches. In this sense, there is much to be gained from a greater awareness of developments in meta-learning, and how these ideas can be generalized to learn about the behaviors of other (nonlearning) algorithms. In this article we present a unified framework for considering the algorithm selection problem as a learning problem, and use this framework to tie together the crossdisciplinary developments in tackling the algorithm selection problem. We discuss the generalization of meta-learning concepts to algorithms focused on tasks including sorting, forecasting, constraint satisfaction, and optimization, and the extension of these ideas to bioinformatics, cryptography, and other fields.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb01338.x · Cited by 88,486
Lecture Notes in Computer Science 1205
(1999) · Industrial Robot: An International Journal · doi:10.1108/ir.1999.04926fae.001 · Cited by 38,662
Data Mining: Practical Machine Learning Tools and Techniques
(2011) · Elsevier eBooks · doi:10.1016/c2009-0-19715-5 · Cited by 25,437
UCI Machine Learning Repository
Asuncion, A. (2007) · Medical Entomology and Zoology · Cited by 25,410
No free lunch theorems for optimization
Wolpert, D. H., & Macready, W. G. (1997) · IEEE Transactions on Evolutionary Computation · doi:10.1109/4235.585893 · Cited by 14,237
Proceedings of the 24th international conference on Machine learning
(2007) · doi:10.1145/1273496 · Cited by 11,630
Witten, I. H. (2008) · Cited by 9,143
Self-Organization and Associative Memory
Kohonen, T. (1989) · Springer Series in Information Sciences · doi:10.1007/978-3-642-88163-3 · Cited by 8,740
Wolpert, D. H. (1992) · Neural Networks · doi:10.1016/s0893-6080(05)80023-1 · Cited by 7,951 · cited 2×
“…adjusting the inherent bias in a learning model [DesJardins and Gordon 1995; Castiello et al. 2005]; combining models via stacked generalization [Wolpert 1992; Chan and Stolfo 1997], ensembles [Opitz and Maclin 1999] and other approaches [Gama and Brazdil 2000; Todorovski and Dzeroski 2003;…”
MethodSelf-Organization and Associative Memory
Kohonen, T. (1988) · Springer Series in Information Sciences · doi:10.1007/978-3-662-00784-6 · Cited by 3,812
(2003) · International Series in Operations Research & Management Science · doi:10.1007/b101874 · Cited by 3,430
Popular Ensemble Methods: An Empirical Study
Opitz, D., & Maclin, R. (1999) · Journal of Artificial Intelligence Research · doi:10.1613/jair.614 · Cited by 2,953 · cited 1×
“…and Gordon 1995; Castiello et al. 2005]; combining models via stacked generalization [Wolpert 1992; Chan and Stolfo 1997], ensembles [Opitz and Maclin 1999] and other approaches [Gama and Brazdil 2000; Todorovski and Dzeroski 2003; Peterson and Martinez 2005]; meta-learning of broader…”
MethodProceedings of the International Joint Conference on Artificial Intelligence 2007
Bundy, A. (2007) · International Joint Conference on Artificial Intelligence · Cited by 2,761
The M3-Competition: results, conclusions and implications
Makridakis, S., & Hibon, M. (2000) · International Journal of Forecasting · doi:10.1016/s0169-2070(00)00057-1 · Cited by 1,695
Self-organization and associative memory: 3rd edition
Kohonen, T. (1989) · Springer eBooks · Cited by 1,223
The Algorithm Selection Problem
Rice, J. R. (1976) · Advances in Computers · doi:10.1016/s0065-2458(08)60520-3 · Cited by 1,195
Lim, T.-S., Loh, W.-Y., & Shih, Y.-S. (2000) · Machine Learning · doi:10.1023/a:1007608224229 · Cited by 1,133
A perspective view and survey of meta-learning
Vilalta, R., & Drissi, Y. (2002) · Artificial Intelligence Review · doi:10.1023/a:1019956318069 · Cited by 1,108
An Information Measure for Classification
Wallace, C. S., & Boulton, D. M. (1968) · The Computer Journal · doi:10.1093/comjnl/11.2.185 · Cited by 933
Methods for Statistical Data Analysis of Multivariate Observations.
Sen, P. K., & Gnanadesikan, R. (1978) · Journal of the American Statistical Association · doi:10.2307/2286626 · Cited by 897
Classes of recursively enumerable sets and their decision problems
Rice, H. G. (1953) · Transactions of the American Mathematical Society · doi:10.1090/s0002-9947-1953-0053041-6 · Cited by 869
Characteristic-Based Clustering for Time Series Data
Wang, X., Smith, K., & Hyndman, R. (2006) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-005-0039-x · Cited by 679
Hyper-Heuristics: An Emerging Direction in Modern Search Technology
Burke, E., Kendall, G., Newall, J., Hart, E., Ross, P., & Schulenburg, S. (2006) · International Series in Operations Research & Management Science · doi:10.1007/0-306-48056-5_16 · Cited by 665
Proceedings of the European Conference on Machine Learning
Brazdil, P. (1993) · Cited by 549
An Introduction to Bioinformatics Algorithms
Jones, N., & Pevzner, P. A. (2004) · Cited by 500
Fitness Distance Correlation as a Measure of Problem Difficulty for Genetic Algorithms
Jones, T., & Forrest, S. (1995) · Cited by 469
Letter recognition using Holland-style adaptive classifiers
Frey, P. W., & Slate, D. J. (1991) · Machine Learning · doi:10.1007/bf00114162 · Cited by 409
Ranking Learning Algorithms: Using IBL and Meta-Learning on Accuracy and Time Results
Brazdil, P. B., Soares, C., & da Costa, J. P. (2003) · Machine Learning · doi:10.1023/a:1021713901879 · Cited by 406
On learning algorithm selection for classification
Ali, S., & Smith, K. A. (2006) · Applied Soft Computing · doi:10.1016/j.asoc.2004.12.002 · Cited by 374
Meta-Learning by Landmarking Various Learning Algorithms
Pfahringer, B., Bensusan, H. N., & Giraud-Carrier, C. G. (2000) · Cited by 302
Generating hard satisfiability problems
Selman, B., Mitchell, D. G., & Levesque, H. J. (1996) · Artificial Intelligence · doi:10.1016/0004-3702(95)00045-3 · Cited by 296
Methods for Statistical Data Analysis of Multivariate Observations
Gnanadesikan, R. (1997) · Wiley Series in Probability and Statistics · doi:10.1002/9781118032671 · Cited by 295
OpitzDavid, & MaclinRichard (1999) · doi:10.5555/3013545.3013549 · Cited by 293
Gama, J., & Brazdil, P. (2000) · Machine Learning · doi:10.1023/a:1007652114878 · Cited by 281
Rigorous location of phase transitions in hard optimization problems
Achlioptas, D., Naor, A., & Peres, Y. (2005) · Nature · doi:10.1038/nature03602 · Cited by 279
Generalizing from Case Studies: A Case Study
Aha, D. W. (1992) · Elsevier eBooks · doi:10.1016/b978-1-55860-247-2.50006-1 · Cited by 216
Combining Classifiers with Meta Decision Trees
Todorovski, L., & Džeroski, S. (2003) · Machine Learning · doi:10.1023/a:1021709817809 · Cited by 204
The Racing Algorithm: Model Selection for Lazy Learners
Maron, O., & Moore, A. W. (1997) · Artificial Intelligence Review · doi:10.1023/a:1006556606079 · Cited by 198
Bernstein, A., Provost, F., & Hill, S. (2005) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2005.67 · Cited by 194
A Meta-Learning Method to Select the Kernel Width in Support Vector Regression
Soares, C., Brazdil, P. B., & Kuba, P. (2004) · Machine Learning · doi:10.1023/b:mach.0000015879.28004.9b · Cited by 187
Understanding Random SAT: Beyond the Clauses-to-Variables Ratio
Nudelman, E., Leyton-Brown, K., Hoos, H. H., Devkar, A., & Shoham, Y. (2004) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-30201-8_33 · Cited by 183
On the Accuracy of Meta-learning for Scalable Data Mining
Chan, P. K., & Stolfo, S. J. (1997) · Journal of Intelligent Information Systems · doi:10.1023/a:1008640732416 · Cited by 178
Advanced Fitness Landscape Analysis and the Performance of Memetic Algorithms
Merz, P. (2004) · Evolutionary Computation · doi:10.1162/1063656041774956 · Cited by 166
Meta-Learning in Distributed Data Mining Systems: Issues and Approaches
Prodromidis, A. L., Chan, P. K., & Stolfo, S. J. (2000) · Cited by 156
Meta-learning approaches to selecting time series models
Prudêncio, R. B. C., & Ludermir, T. B. (2004) · Neurocomputing · doi:10.1016/j.neucom.2004.03.008 · Cited by 154
Addressing the Selective Superiority Problem: Automatic Algorithm/Model Class Selection
Brodley, C. E. (1993) · Elsevier eBooks · doi:10.1016/b978-1-55860-307-3.50009-5 · Cited by 151
A meta-learning approach to automatic kernel selection for support vector machines
Ali, S., & Smith‐Miles, K. (2006) · Neurocomputing · doi:10.1016/j.neucom.2006.03.004 · Cited by 148
Self-citationA portfolio approach to algorithm select
Leyton‐Brown, K., Nudelman, E., Andrew, G., McFadden, J., & Shoham, Y. (2003) · International Joint Conference on Artificial Intelligence · Cited by 128
The Facilities Layout Problem in Perspective
Vollmann, T. E., & Buffa, E. S. (1966) · Management Science · doi:10.1287/mnsc.12.10.b450 · Cited by 125
Characterization of classification algorithms
da Gama, J. M. P., & Brazdil, P. (1995) · Lecture notes in computer science · doi:10.1007/3-540-60428-6_16 · Cited by 123
A Bayesian Approach to Tackling Hard Computational ProblemsPreprint
Horvitz, E., Ruan, Y., Gomes, C. P., Kautz, H., Selman, B., & Chickering, D. M. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.2279 · Cited by 114
SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT
Xu, L. H., Hutter, F., Hoos, H. H., & Leyton‐Brown, K. (2007) · Springer eBooks · doi:10.1007/978-3-540-74970-7_50 · Cited by 109
Learning dynamic algorithm portfolios
Gagliolo, M., & Schmidhuber, J. (2007) · Annals of Mathematics and Artificial Intelligence · doi:10.1007/s10472-006-9036-z · Cited by 107
Estimating the Predictive Accuracy of a Classifier
Bensusan, H. N., & Kalousis, A. (2001) · Lecture notes in computer science · doi:10.1007/3-540-44795-4_3 · Cited by 100
Meta-data: Characterization of Input Features for Meta-learning
Castiello, C., Castellano, G., & Fanelli, A. M. (2005) · Lecture notes in computer science · doi:10.1007/11526018_45 · Cited by 95
Kalousis, A. (1999) · Intelligent Data Analysis · doi:10.1016/s1088-467x(99)00026-8 · Cited by 88
Empirical Learning as a Function of Concept Character
Rendell, L. A., & Cho, H. (1990) · Machine Learning · doi:10.1023/a:1022651406695 · Cited by 88
An Evaluation of Landmarking Variants
Fürnkranz, J., & Petrak, J. (2001) · Cited by 85
Brazdil, P. B., & Henery, R. J. (1995) · Ellis Horwood eBooks · Cited by 80
A higher-order approach to meta-learning
Bensusan, H. N., Giraud-Carrier, C. G., & Kennedy, C. J. (2000) · Bristol Research (University of Bristol) · Cited by 80
Samulowitz, H., & Memisevic, R. (2007) · Cited by 79
Combining multiple heuristics online
Streeter, M. J. V., Golovin, D., & Smith, S. F. (2007) · Cited by 72
Boosting as a Metaphor for Algorithm Design
Leyton‐Brown, K., Nudelman, E., Andrew, G., McFadden, J., & Shoham, Y. (2003) · Lecture notes in computer science · doi:10.1007/978-3-540-45193-8_75 · Cited by 57
Evolving Combinatorial Problem Instances That Are Difficult to Solve
Hemert, J. I. V. (2006) · Evolutionary Computation · doi:10.1162/evco.2006.14.4.433 · Cited by 54
Sampling-Based Relative Landmarks: Systematically Test-Driving Algorithms before Choosing
Soares, C., Petrak, J., & Brazdil, P. (2001) · Lecture notes in computer science · doi:10.1007/3-540-45329-6_12 · Cited by 52
An intelligent model selection and forecasting system
Venkatachalam, A. R., & Sohl, J. E. (1999) · Journal of Forecasting · doi:10.1002/(sici)1099-131x(199905)18:3<167::aid-for715>3.0.co;2-f · Cited by 50
Ranking with Predictive Clustering Trees
Todorovski, L., Blockeel, H., & Džeroski, S. (2002) · Lecture notes in computer science · doi:10.1007/3-540-36755-1_37 · Cited by 49
Neural network system for forecasting method selection
Chu, C., & Widjaja, D. (1994) · Decision Support Systems · doi:10.1016/0167-9236(94)90071-x · Cited by 47
Proceedings of 8th International Conference on Tools with Artificial Intelligence
(1996) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/tai.1996.560387 · Cited by 38
Algorithm selection for sorting and probabilistic inference: a machine learning-based approach
Guo, H., & Hsu, W. H. (2003) · Cited by 29
Proceedings of the 14th international conference on Discovery science
Elomaa, T., Hollmén, J., & Mannila, H. (2011) · Cited by 23
Kernal Width Selection for SVM Classification
Ali, S., & Smith‐Miles, K. (2005) · International Journal of Data Warehousing and Mining · doi:10.4018/jdwm.2005100104 · Cited by 18
Self-citation
Cited by
AutoML: A survey of the state-of-the-art
He, X., Zhao, K., & Chu, X. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106622 · Cited by 1,691
Automated Machine Learning: Methods, Systems, Challenges
(2019) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-030-05318-5 · Cited by 1,485
Hyper-heuristics: a survey of the state of the art
Burke, E. K., Gendreau, M., Hyde, M., Kendall, G., Ochoa, G., Özcan, E., & Qu, R. (2013) · Journal of the Operational Research Society · doi:10.1057/jors.2013.71 · Cited by 1,263
Evaluation of clustering algorithms for financial risk analysis using MCDM methods
Kou, G., Peng, Y., & Wang, G. (2014) · Information Sciences · doi:10.1016/j.ins.2014.02.137 · Cited by 764
(2011) · Cited by 675
Metadata
- Authors
- Smith-Miles, K. A.
- Venue
- ACM Computing Surveys
- Published
- 2009-01-15
- Volume
- 41
- Issue
- 1
- Pages
- 1-25
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W2089213632
- Cited by
- 552
- References
- 72
- FWCI
- 30.00
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.