Metalearning: A survey of trends and technologies
Open accessAbstract
Metalearning attracted considerable interest in the machine learning community in the last years. Yet, some disagreement remains on what does or what does not constitute a metalearning problem and in which contexts the term is used in. This survey aims at giving an all-encompassing overview of the research directions pursued under the umbrella of metalearning, reconciling different definitions given in scientific literature, listing the choices involved when designing a metalearning system and identifying some of the future research challenges in this domain.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Design issues on data collection, pre-processing, cleaning, labeling and augmentation, including big data challenges, exist for ML applications.
“-Data Design: Design issues on data collection, pre-processing, cleaning, labeling and augmentation, including big data challenges [17,23,25-27, 31,33,41,44,74-76,105,114-116].”
From Software engineering challenges for machine learning applications: A literature review (2020) · cites this paper as [31]
Training data, ML models and ML platforms are key components of ML applications, and challenges on the quality of each component have been discussed, including data anomaly, imbalanced or biased data, encrypted data, data evaluation/cleaning, and novelty detection.
“-Component Quality: Training data, ML models and ML platforms (e.g.scikit-learn [125], Tensor flow [126], Weka [127]) is the key components of ML applications.The challenges on the quality of each components have been discussed.The challenge issues on data quality for ML applications includes data anomaly [23], imbalanced or biased data, encrypted data [60], data evaluation/cleaning [39][40][41]74”
From Software engineering challenges for machine learning applications: A literature review (2020) · cites this paper as [31]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Pan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387 · see citing works
A decision-theoretic generalization of on-line learning and an application to boosting
Freund, Y., & Schapire, R. E. (1997) · Journal of Computer and System Sciences · doi:10.1006/jcss.1997.1504 · Cited by 20,466 · see citing works
Neural networks for pattern recognition
Bishop, C. (1994) · Choice Reviews Online · doi:10.5860/choice.31-5500 · Cited by 18,537 · see citing works
Breiman, L. (1996) · Machine Learning · doi:10.1007/bf00058655 · Cited by 16,325 · see citing works
Neural Networks for Pattern Recognition
Bishop, C. M. (1995) · doi:10.1093/oso/9780198538493.001.0001 · Cited by 12,287 · see citing works
Ensemble methods in machine learning
Dietterich, T. G. (2000) · Lecture Notes in Computer Science · doi:10.1007/3-540-45014-9_1 · Cited by 8,408 · see citing works
Wolpert, D. H. (1992) · Neural Networks · doi:10.1016/s0893-6080(05)80023-1 · Cited by 7,951 · see citing works
Mining high-speed data streams
Domingos, P., & Hulten, G. (2000) · ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/347090.347107 · Cited by 2,213 · see citing works
Bleiholder, J., & Naumann, F. (2009) · ACM Computing Surveys · doi:10.1145/1456650.1456651 · Cited by 1,847 · see citing works
The Algorithm Selection Problem
Rice, J. R. (1976) · Advances in computers · doi:10.1016/s0065-2458(08)60520-3 · Cited by 1,210 · see citing works
A perspective view and survey of meta-learning
Vilalta, R., & Drissi, Y. (2002) · Artificial Intelligence Review · doi:10.1023/a:1019956318069 · Cited by 1,108 · see citing works
Learning Multiple Tasks with Kernel Methods
Evgeniou, T., Micchelli, C. A., & Pontil, M. (2005) · doi:10.5555/1046920.1088693 · Cited by 820 · see citing works
THE ROLE OF METALEARNING IN STUDY PROCESSES
BIGGS, J. B. (1985) · British Journal of Educational Psychology · doi:10.1111/j.2044-8279.1985.tb02625.x · Cited by 550 · see citing works
Cross-disciplinary perspectives on meta-learning for algorithm selection
Smith‐Miles, K. (2009) · ACM Computing Surveys · doi:10.1145/1456650.1456656 · Cited by 548 · see citing works
Proceedings of the Seventeenth International Conference on Machine Learning
Langley, P. (2000) · International Conference on Machine Learning · Cited by 545 · see citing works
Ranking Learning Algorithms: Using IBL and Meta-Learning on Accuracy and Time Results
Brazdil, P., Soares, C., & da Costa, J. P. (2003) · Machine Learning · doi:10.1023/a:1021713901879 · Cited by 409 · see citing works
Cortés, R., Bonnaire, X., Marin, O., & Sens, P. (2015) · Procedia Computer Science · doi:10.1016/j.procs.2015.05.093 · Cited by 356 · see citing works
Meta-Learning by Landmarking Various Learning Algorithms
Pfahringer, B., Bensusan, H. N., & Giraud-Carrier, C. G. (2000) · Cited by 299 · see citing works
MetaFraud: A Meta-Learning Framework for Detecting Financial Fraud1
Abbasi, A., Albrecht, C., Vance, A., & Hansen, J. (2012) · MIS Quarterly · doi:10.2307/41703508 · Cited by 298 · see citing works
da Gama, J. M. P., & Brazdil, P. (2000) · Machine Learning · doi:10.1023/a:1007652114878 · Cited by 281 · see citing works
Brazdil, P., Giraud-Carrier, C. G., Soares, C., & Vilalta, R. (2008) · Cognitive technologies · doi:10.1007/978-3-540-73263-1 · Cited by 265 · see citing works
Meta-learning for time series forecasting and forecast combination
Lemke, C., & Gabryś, B. (2010) · Neurocomputing · doi:10.1016/j.neucom.2009.09.020 · Cited by 244 · see citing works
Self-citationDATA STREAM MINING A Practical Approach
Bifet, A., & Kirkby, R. (2009) · Cited by 221 · see citing works
Combining Classifiers with Meta Decision Trees
Todorovski, L., & Džeroski, S. (2003) · Machine Learning · doi:10.1023/a:1021709817809 · Cited by 204 · see citing works
Bernstein, A., Provost, F. J., & Hill, S. (2005) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2005.67 · Cited by 195 · see citing works
Wang, X., Smith‐Miles, K., & Hyndman, R. J. (2008) · Neurocomputing · doi:10.1016/j.neucom.2008.10.017 · Cited by 176 · see citing works
A Comparison of Ranking Methods for Classification Algorithm Selection
Brazdil, P., & Soares, C. (2000) · Lecture notes in computer science · doi:10.1007/3-540-45164-1_8 · Cited by 174 · see citing works
PMML: An Open Standard for Sharing Models
Guazzelli, A., Zeller, M., Lin, W.-C., & Williams, G. (2009) · The R Journal · doi:10.32614/rj-2009-010 · Cited by 162 · see citing works
Domain-Driven Data Mining: Challenges and Prospects
Cao, L. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2010.32 · Cited by 154 · see citing works
Meta-learning for evolutionary parameter optimization of classifiers
Reif, M., Faisal Shafait, B., & Dengel, A. R. (2012) · Machine Learning · doi:10.1007/s10994-012-5286-7 · Cited by 153 · see citing works
Meta-learning approaches to selecting time series models
Prudêncio, R. B. C., & Ludermir, T. B. (2004) · Neurocomputing · doi:10.1016/j.neucom.2004.03.008 · Cited by 148 · see citing works
Combining meta-learning and search techniques to select parameters for support vector machines
Gomes, T. A. F., Prudêncio, R. B. C., Soares, C., et al. (2011) · Neurocomputing · doi:10.1016/j.neucom.2011.07.005 · Cited by 144 · see citing works
Automatic classifier selection for non-experts
Reif, M., Shafait, F., Goldstein, M., et al. (2012) · Pattern Analysis and Applications · doi:10.1007/s10044-012-0280-z · Cited by 140 · see citing works
Evolving artificial neural network ensembles
Yao, X., & Islam, M. M. (2008) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2007.913386 · Cited by 129 · see citing works
Ramakrishnan, R. (2000) · Medical Entomology and Zoology · Cited by 107 · see citing works
Flexible latent variable models for multi-task learning
Zhang, J., Zoubin Ghahramani, A., & Yang, Y. (2008) · Machine Learning · doi:10.1007/s10994-008-5050-1 · Cited by 97 · see citing works
Meta-Learning in Computational Intelligence
Jankowski, N., Gra̧bczewski, K., & Duch, W. (2011) · Studies in computational intelligence · doi:10.1007/978-3-642-20980-2 · Cited by 95 · see citing works
Kalousis, A. (1999) · Intelligent Data Analysis · doi:10.1016/s1088-467x(99)00026-8 · Cited by 88 · see citing works
The MiningMart Approach to Knowledge Discovery in Databases
Morik, K. J., & Scholz, M. (2004) · Springer eBooks · doi:10.1007/978-3-662-07952-2_3 · Cited by 81 · see citing works
A higher-order approach to meta-learning
Bensusan, H. N., Giraud-Carrier, C. G., & Kennedy, C. J. (2000) · Bristol Research (University of Bristol) · Cited by 75 · see citing works
Load forecasting using a multivariate meta-learning system
Matijaš, M., Suykens, J. A. K., & Krajcar, S. (2013) · Expert Systems with Applications · doi:10.1016/j.eswa.2013.01.047 · Cited by 64 · see citing works
Feature Selection for Meta-learning
Kalousis, A., & Hilario, M. (2001) · Lecture notes in computer science · doi:10.1007/3-540-45357-1_26 · Cited by 59 · see citing works
Postprocessing in machine learning and data mining
Brůha, I., & Famili, A. F. (2000) · ACM SIGKDD Explorations Newsletter · doi:10.1145/380995.381059 · Cited by 52 · see citing works
Ranking with Predictive Clustering Trees
Todorovski, L., Blockeel, H., & Džeroski, S. (2002) · Lecture notes in computer science · doi:10.1007/3-540-36755-1_37 · Cited by 49 · see citing works
Inductive transfer with context-sensitive neural networks
Silver, D. L., Poirier, R., & Currie, D. (2008) · Machine Learning · doi:10.1007/s10994-008-5088-0 · Cited by 49 · see citing works
Architecture for development of adaptive on-line prediction models
Kadlec, P., & Gabryś, B. (2009) · Memetic Computing · doi:10.1007/s12293-009-0017-8 · Cited by 49 · see citing works
Experiments in Meta-level Learning with ILP
Todorovski, L., & Džeroski, S. (1999) · Lecture notes in computer science · doi:10.1007/978-3-540-48247-5_11 · Cited by 42 · see citing works
A Meta‐learning Framework for Bankruptcy Prediction
Tsai, C., & Hsu, Y. (2011) · Journal of Forecasting · doi:10.1002/for.1264 · Cited by 38 · see citing works
Guest editor’s introduction: special issue on inductive transfer learning
Silver, D. L., & Bennett, K. P. (2008) · Machine Learning · doi:10.1007/s10994-008-5087-1 · Cited by 37 · see citing works
Using Machine Learning Techniques to Combine Forecasting Methods
Prudêncio, R. B. C., & Ludermir, T. B. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-30549-1_111 · Cited by 34 · see citing works
Towards process-oriented tool support for knowledge discovery in databases
Wirth, R., Shearer, C., Grimmer, U., et al. (1997) · Lecture notes in computer science · doi:10.1007/3-540-63223-9_123 · Cited by 33 · see citing works
The Data Mining Advisor: Meta-learning at the Service of Practitioners
Giraud-Carrier, C. G. (2006) · International Conference on Machine Learning and Applications · doi:10.1109/icmla.2005.65 · Cited by 32 · see citing works
UCI++: Improved Support for Algorithm Selection Using Datasetoids
Soares, C. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-01307-2_46 · Cited by 30 · see citing works
Meta-learning Approach for Automatic Parameter Tuning: A case of study with educational datasets.
Molina, M. D. M., Romero, C., Ventura, S., & Luna, J. M. (2012) · Educational Data Mining · Cited by 25 · see citing works
Robust predictive modelling of water pollution using biomarker data
Budka, M., Gabryś, B., & Ravagnan, E. (2010) · Water Research · doi:10.1016/j.watres.2010.03.006 · Cited by 24 · see citing works
Self-citationUniversal Meta-Learning Architecture and Algorithms
Jankowski, N., & Grąbczewski, K. (2011) · Studies in computational intelligence · doi:10.1007/978-3-642-20980-2_1 · Cited by 21 · see citing works
Ridge regression ensemble for toxicity prediction
Budka, M., & Gabryś, B. (2010) · Procedia Computer Science · doi:10.1016/j.procs.2010.04.022 · Cited by 21 · see citing works
Self-citationLemke, C., Riedel, S., & Gabrys, B. (2009) · 2009 IEEE Symposium on Computational Intelligence for Financial Engineering · doi:10.1109/cifer.2009.4937507 · Cited by 20 · see citing works
Self-citationde Miranda, P. B. C., Prudêncio, R. B. C., de Carvalho, A. C. P. L. F., et al. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-31137-6_43 · Cited by 15 · see citing works
Lazy Meta-Learning: Creating Customized Model Ensembles on Demand
Bonissone, P. P. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-30687-7_1 · Cited by 14 · see citing works
A Characterization of Difficult Problems in Classification.
Vilalta, R., & Drissi, Y. (2002) · Cited by 11 · see citing works
Learnt Topology Gating Artificial Neural Networks
Kadlec, P., & Gabryś, B. (2008) · IEEE International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2008.4634162 · Cited by 10 · see citing works
Towards understanding learning behavior
Vanschoren, J., & Blockeel, H. (2006) · Lirias · Cited by 9 · see citing works
On the Use of Fast Subsampling Estimates for Algorithm Recommendation
Fürnkranz, J., Petrak, J., Brazdil, P., et al. (2002) · Cited by 8 · see citing works
Transfer Learning by Kernel Meta-Learning.
Aiolli, F. (2011) · Padua Research Archive (University of Padova) · Cited by 8 · see citing works
Complexity Measures for Meta-learning and Their Optimality
Jankowski, N. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-44958-1_15 · Cited by 6 · see citing works
Cleland, K. D., & Dobrea-Grindahl, M. (2014) · doi:10.4324/9780203738474-129 · Cited by 5 · see citing works
METALA: a J2EE technology based framework for web mining
Hernansaez, J. M., Botía, J. A., & Skármeta, A. (2004) · Revista Colombiana de Computación · doi:10.29375/25392115.1080 · Cited by 4 · see citing works
Meta-learning from an experiment database
Driessens, K., Vanwinckelen, G., & Blockeel, H. (2012) · Lirias · Cited by 2 · see citing works
Flagged first, then by citations
Research works printed but not linked: 5
- Brazdil P Giraud-Carrier C Soares C Vilalta R Metalearning: applications to data mining 2009 Berlin Springer
- Köpf C, Iglezakis I (2002) Combination of task description strategies and case base properties for meta-learning. In: Proceedings of the 2nd international workshop on integration and collaboration aspects of data mining, decision support and meta-learning, pp 65–76
- Nguyen P, Kalousis A, Hilario M (2011) A meta-mining infrastructure to support kd workflow optimization. In: eCML PKDD 2011, p 1
- Nguyen P, Kalousis A, Hilario M (2012) Experimental evaluation of the e-lico meta-miner. In: 5th planning to learn workshop WS28 at ECAI 2012, p 18
- Vanschoren J (2010) Understanding machine learning performance with experiment databases. PhD thesis, Arenberg Doctoral School of Science, Engineering & Technology, Katholieke Universiteit Leuven
Cited by
Sorted by citations
More Than Privacy: Applying Differential Privacy in Key Areas of Artificial Intelligence
Zhu, T., Ye, D., Wang, W., Zhou, W., & Yu, P. S. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2020.3014246 · Cited by 219 · see citing works
A review on the self and dual interactions between machine learning and optimisation
Song, H., Triguero, I., & Özcan, E. (2019) · Progress in Artificial Intelligence · doi:10.1007/s13748-019-00185-z · Cited by 101 · see citing works
Triguero, I., Molina, D., Poyatos, J., Del Ser, J., & Herrera, F. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102135 · Cited by 90 · see citing works
Hemani, G., Bowden, J., Haycock, P., Zheng, J., Davis, O., Flach, P., Gaunt, T., & Smith, G. D. (2017) · doi:10.1101/173682 · Cited by 85 · see citing works
Software engineering challenges for machine learning applications: A literature review
Kumeno, F. (2020) · Intelligent Decision Technologies · doi:10.3233/idt-190160 · Cited by 81 · see citing works
Optimizing quantum heuristics with meta-learning
Wilson, M., Stromswold, R., Wudarski, F., Hadfield, S., Tubman, N. M., & Rieffel, E. G. (2021) · Quantum Machine Intelligence · doi:10.1007/s42484-020-00022-w · Cited by 79 · see citing works
- Authors
- Lemke, C., Budka, M., & Gabrys, B.
- Affiliations
- Bournemouth University (GB)
- Corresponding author
- Lemke, Christiane
- Budka, Marcin
- ORCID
- Budka, Marcin
- Venue
- Artificial Intelligence Review
- Published
- 2013-07-20
- Volume
- 44
- Issue
- 1
- Pages
- 117-130
- ISSN
- 0269-2821 (ISSN-L)
- 1573-7462
- PMC
- PMC4459543
- OpenAlex
- W2091118421
- Cited by
- 365
- References
- 69
- Citations by year
- 2026: 11 · 2025: 18 · 2024: 30 · 2023: 39 · 2022: 39 · 2021: 64 · 2020: 55 · 2019: 44 · 2018: 30 · 2017: 21 · 2016: 7 · 2015: 5 · 2014: 2
- FWCI
- 5.15
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Artificial Intelligence Review (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Bournemouth University Research Online (Bournemouth University) (submittedVersion)
- Funders
- European Commission (2007–2013, FP7 2007-2013, FP7/2007, 251617)
- Seventh Framework Programme (FP7/2007, FP7/2007 - 2013, 2007‐2013)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- meta-learning, research directions, future research challenges
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
