A review of automatic selection methods for machine learning algorithms and hyper-parameter values
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The challenges and solutions on ML techniques such as data processing, ML algorithms/models, and specific ML functions can influence the structure and implementation of ML applications.
“The contrary can also be true; that is, the challenges and solutions on ML techniques such as data processing (e.g., feature extraction) [62,70,124], ML algorithms/ models (e.g., transfer learning) [43,68,69,79,80,86,97,101,[110][111][112]116], and specific ML functions (e.g., interpretability) [32,104,105,121] can influence the structure and implementation of ML applications.”
From Software engineering challenges for machine learning applications: A literature review (2020) · cites this paper as [32]
Developing the skill set for ML applications is the major challenge related to the software engineering professional practice knowledge area.
“Developing the skill set for ML applications is the major challenge related to this KA [15,16,27,32,39,40,59].”
From Software engineering challenges for machine learning applications: A literature review (2020) · cites this paper as [32]
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · see citing works
Boyd, S., & Vandenberghe, L. (2004) · doi:10.1017/cbo9780511804441 · Cited by 56,601 · see citing works
Model selection and multimodel inference: a practical information-theoretic approach.
Guthery, F. S., Burnham, K. P., & Anderson, D. R. (2003) · The Journal of Wildlife Management · doi:10.2307/3802723 · Cited by 41,495 · see citing works
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500 · see citing works
Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2013) · doi:10.1201/b16018 · Cited by 27,276 · see citing works
Data Mining: Practical Machine Learning Tools and Techniques
Witten, I. H., Frank, E., & Hall, M. A. (2011) · Elsevier eBooks · doi:10.1016/c2009-0-19715-5 · Cited by 25,493 · see citing works
Braun, W. J., & Murdoch, D. J. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993456.011 · Cited by 14,099 · see citing works
Bertsekas, D. P. (1997) · Journal of the Operational Research Society · doi:10.1038/sj.jors.2600425 · Cited by 11,969 · see citing works
Witten, I. H. (2008) · Cited by 9,115 · see citing works
(2012) · Cited by 8,009 · see citing works
Practical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691 · see citing works
Taking the Human Out of the Loop: A Review of Bayesian Optimization
Shahriari, B., Swersky, K., Wang, Z., Adams, R. P., & de Freitas, N. (2015) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2494218 · Cited by 6,195 · see citing works
(2010) · Cited by 4,411 · see citing works
(2011) · Cited by 3,939 · see citing works
Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281 · see citing works
MapReduce: Simplified Data Processing on Large Cluster
Dayalan, M. (2018) · INTERNATIONAL JOURNAL OF RESEARCH AND ENGINEERING · doi:10.21276/ijre.2018.5.5.4 · Cited by 2,959 · see citing works
Ensemble Methods: Foundations and Algorithms
Zhou, Z.-H. (2012) · doi:10.1201/b12207 · Cited by 2,534 · see citing works
Sequential Model-Based Optimization for General Algorithm Configuration
Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-25566-3_40 · Cited by 2,469 · see citing works
Feature Selection for Knowledge Discovery and Data Mining
Liu, H., & Motoda, H. (1998) · Springer eBooks · doi:10.1007/978-1-4615-5689-3 · Cited by 2,216 · see citing works
Ensemble Methods: Foundations and Algorithms
Zhou, Z. (2012) · Cited by 2,051 · see citing works
Bigtable: a distributed storage system for structured data
Chang, F. W., Dean, J. B., Ghemawat, S., Hsieh, W. C.-Y., Wallach, D. A., Burrows, M., Chandra, T., Fikes, A., & Gruber, R. E. (2006) · doi:10.5555/1298455.1298475 · Cited by 1,953 · see citing works
Steyerberg, E. W. (2009) · Statistics for Biology and Health · doi:10.1007/978-0-387-77244-8 · Cited by 1,491 · see citing works
(2006) · WORLD SCIENTIFIC eBooks · doi:10.1142/9789812774248_0002 · Cited by 1,361 · see citing works
Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms
Thornton, C., Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2013) · Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2487575.2487629 · Cited by 1,359 · see citing works
(2015) · neural information processing systems · Cited by 1,305 · see citing works
Ensemble selection from libraries of models
Caruana, R., Niculescu-Mizil, A., Crew, G., & Ksikes, A. (2004) · International Conference on Machine Learning · doi:10.1145/1015330.1015432 · Cited by 1,089 · see citing works
ParamILS: An Automatic Algorithm Configuration Framework
Hutter, F., Hoos, H. H., Leyton‐Brown, K., & Stützle, T. (2009) · Journal of Artificial Intelligence Research · doi:10.1613/jair.2861 · Cited by 883 · see citing works
Model selection and model averaging
Claeskens, G., & Hjort, N. L. (2008) · Cited by 872 · see citing works
Hyperopt: A Python Library for Optimizing the Hyperparameters of Machine Learning Algorithms
Bergstra, J., Yamins, D., & Cox, D. (2013) · Proceedings of the Python in Science Conferences · doi:10.25080/majora-8b375195-003 · Cited by 784 · see citing works
Introduction to Machine Learning
Suykens, J. A. K. (2014) · Academic press library in signal processing · doi:10.1016/b978-0-12-396502-8.00013-9 · Cited by 649 · see citing works
Model Selection and Model Averaging
Claeskens, G., & Hjort, N. L. (2001) · Cambridge University Press eBooks · doi:10.1017/cbo9780511790485 · Cited by 624 · see citing works
Multi-Task Bayesian Optimization
Swersky, K., Snoek, J., & Adams, R. P. (2013) · Digital Access to Scholarship at Harvard (DASH) (Harvard University) · Cited by 485 · see citing works
Smith, D. K., & Bertsekas, D. P. (1997) · Journal of the Operational Research Society · doi:10.2307/3010437 · Cited by 480 · see citing works
Gradient-Based Optimization of Hyperparameters
Bengio, Y. (2000) · Neural Computation · doi:10.1162/089976600300015187 · Cited by 467 · see citing works
Domhan, T., Springenberg, J. T., & Hutter, F. (2015) · FreiDok plus (Universitätsbibliothek Freiburg) · Cited by 431 · see citing works
Methods and Criteria for Model Selection
Kadane, J. B., & Lazar, N. A. (2004) · Journal of the American Statistical Association · doi:10.1198/016214504000000269 · Cited by 429 · see citing works
Ranking Learning Algorithms: Using IBL and Meta-Learning on Accuracy and Time Results
Brazdil, P., Soares, C., & da Costa, J. P. (2003) · Machine Learning · doi:10.1023/a:1021713901879 · Cited by 408 · see citing works
Initializing Bayesian Hyperparameter Optimization via Meta-Learning
Feurer, M., Springenberg, J. T., & Hutter, F. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9354 · Cited by 402 · see citing works
Efficient progressive sampling
Provost, F. J., Jensen, D. D., & Oates, T. (1999) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/312129.312188 · Cited by 361 · see citing works
An Efficient Approach for Assessing Hyperparameter Importance
Hutter, F., Hoos, H. H., & Leyton‐Brown, K. (2014) · Cited by 348 · see citing works
Creating Value In Health Care Through Big Data: Opportunities And Policy Implications
Roski, J., Bo‐Linn, G. W., & Andrews, T. A. (2014) · Health Affairs · doi:10.1377/hlthaff.2014.0147 · Cited by 323 · see citing works
Hyperopt-Sklearn: Automatic Hyperparameter Configuration for Scikit-Learn
Komer, B., Bergstra, J., & Eliasmith, C. (2014) · Proceedings of the Python in Science Conferences · doi:10.25080/majora-14bd3278-006 · Cited by 310 · see citing works
MLbase: A Distributed Machine-learning System
Kraska, T., Talwalkar, A., & Duchi, J. C. (2013) · Cited by 310 · see citing works
Meta-Learning by Landmarking Various Learning Algorithms
Pfahringer, B., Bensusan, H. N., & Giraud-Carrier, C. G. (2000) · Cited by 302 · see citing works
Collaborative hyperparameter tuning
Bardenet, R., Brendel, M., gl, B. Z. K., & le Sebag, M. (2013) · HAL · Cited by 293 · see citing works
Hoeffding Races: Accelerating Model Selection Search for Classification and Function Approximation
Maron, O., & Moore, A. (2018) · doi:10.1184/r1/6555050 · Cited by 271 · see citing works
A novel LS-SVMs hyper-parameter selection based on particle swarm optimization
Guo, X., Yang, J., Wu, C., et al. (2008) · Neurocomputing · doi:10.1016/j.neucom.2008.04.027 · Cited by 210 · see citing works
Model selection for the LS-SVM. Application to handwriting recognition
Adankon, M. M., & Cheriet, M. (2008) · Pattern Recognition · doi:10.1016/j.patcog.2008.10.023 · Cited by 197 · see citing works
Empirical Model Discovery and Theory Evaluation
Hendry, D. F., & Doornik, J. A. (2014) · The MIT Press eBooks · doi:10.7551/mitpress/9780262028356.001.0001 · Cited by 175 · see citing works
Static versus dynamic sampling for data mining
John, G. H., & Langley, P. (1996) · Cited by 171 · see citing works
Automating model search for large scale machine learning
Sparks, E., Talwalkar, A., Haas, D., Franklin, M., Jordan, M. I., & Kraska, T. (2015) · ACM Symposium on Cloud Computing · doi:10.1145/2806777.2806945 · Cited by 165 · see citing works
Efficient Transfer Learning Method for Automatic Hyperparameter Tuning
Yogatama, D., & Mann, G. (2014) · International Conference on Artificial Intelligence and Statistics · Cited by 154 · see citing works
Freeze-Thaw Bayesian OptimizationPreprint
Swersky, K., Snoek, J., & Adams, R. P. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.3896 · Cited by 154 · see citing works
MLI: An API for Distributed Machine Learning
Sparks, E., Talwalkar, A., Smith, V., Kottalam, J., Pan, X., Gonzalez, J. E., Franklin, M. J. M., Jordan, M. I., & Kraska, T. (2013) · Proceedings · doi:10.1109/icdm.2013.158 · Cited by 151 · see citing works
An overview of free software tools for general data mining
Jović, A., Brkić, K., & Bogunovic, N. (2014) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.1109/mipro.2014.6859735 · Cited by 133 · see citing works
Design of the 2015 ChaLearn AutoML challenge
Guyon, I., Bennett, K. P., Cawley, G. C., Escalante, H. J., Escalera, S., Ho, T. K., Macià, N., Ray, B., Saeed, M., Statnikov, A., et al. (2015) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2015.7280767 · Cited by 129 · see citing works
Learning hyperparameter optimization initializations
Wistuba, M., Schilling, N., & Schmidt-Thieme, L. (2015) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2015.7344817 · Cited by 95 · see citing works
Hoffman, M. D., Shahriari, B., & de Freitas, N. (2014) · International Conference on Artificial Intelligence and Statistics · Cited by 85 · see citing works
Luo, G. (2016) · Health Information Science and Systems · doi:10.1186/s13755-016-0015-4 · Cited by 85 · see citing works
Self-citationSelecting Classification Algorithms with Active Testing
Leite, R., Brazdil, P., & Vanschoren, J. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-31537-4_10 · Cited by 83 · see citing works
A systematic review of predictive models for asthma development in children
Luo, G., Nkoy, F. L., Stone, B. L., Schmick, D., & Johnson, M. D. (2015) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-015-0224-9 · Cited by 77 · see citing works
Self-citationFast Algorithm Selection Using Learning Curves
van Rijn, J. N., Abdulrahman, S. M., Brazdil, P., & Vanschoren, J. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24465-5_26 · Cited by 71 · see citing works
An Evaluation of Landmarking Variants
Fürnkranz, J., & Petrak, J. (2001) · Cited by 62 · see citing works
Predicting relative performance of classifiers from samples
Leite, R., & Brazdil, P. (2005) · International Conference on Machine Learning · doi:10.1145/1102351.1102414 · Cited by 59 · see citing works
Case study: a data warehouse for an academic medical center.
Einbinder, J. S., Scully, K. W., Pates, R. D., et al. (2001) · PubMed · Cited by 54 · see citing works
Empirical model discovery and theory evaluation: automatic selection methods in econometrics
(2015) · Choice Reviews Online · doi:10.5860/choice.187457 · Cited by 50 · see citing works
Wistuba, M., Schilling, N., & Schmidt-Thieme, L. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-23525-7_7 · Cited by 50 · see citing works
Bandit Algorithms for Website Optimization
White, J. M. (2012) · CERN Document Server (European Organization for Nuclear Research) · Cited by 48 · see citing works
Cleophas, T. J., & Zwinderman, A. H. (2013) · Springer eBooks · doi:10.1007/978-94-007-5824-7 · Cited by 45 · see citing works
Cleophas, T. J., & Zwinderman, A. H. (2013) · Springer eBooks · doi:10.1007/978-94-007-7869-6 · Cited by 44 · see citing works
A systematic review of predictive modeling for bronchiolitis
Luo, G., Nkoy, F. L., Gesteland, P. H., Glasgow, T. S., & Stone, B. L. (2014) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2014.07.005 · Cited by 43 · see citing works
Self-citationSampling-Based Relative Landmarks: Systematically Test-Driving Algorithms before Choosing
Soares, C., Petrak, J., & Brazdil, P. (2001) · Lecture notes in computer science · doi:10.1007/3-540-45329-6_12 · Cited by 41 · see citing works
MLBCD: a machine learning tool for big clinical data
Luo, G. (2015) · Health Information Science and Systems · doi:10.1186/s13755-015-0011-0 · Cited by 39 · see citing works
Self-citationSelecting Near-Optimal Learners via Incremental Data Allocation
Sabharwal, A., Samulowitz, H., & Tesauro, G. J. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10316 · Cited by 34 · see citing works
Efficient Hyper-parameter Optimization for NLP Applications
Wang, L., Feng, M., Zhou, B., Xiang, B., & Mahadevan, S. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1253 · Cited by 33 · see citing works
Active Testing Strategy to Predict the Best Classification Algorithm via Sampling and Metalearning
Leite, R., & Brazdil, P. (2010) · Frontiers in artificial intelligence and applications · doi:10.3233/978-1-60750-606-5-309 · Cited by 31 · see citing works
Using Computational Approaches to Improve Risk-Stratified Patient Management: Rationale and Methods
Luo, G., Stone, B. L., Sakaguchi, F. H., Sheng, X., & Murtaugh, M. A. (2015) · JMIR Research Protocols · doi:10.2196/resprot.5039 · Cited by 26 · see citing works
Self-citationActive Learning with Model Selection
Ali, A., Caruana, R., & Kapoor, A. (2014) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v28i1.9014 · Cited by 26 · see citing works
Agnostic Bayesian Learning of Ensembles
Lacoste, A., Marchand, M., Laviolette, F. O., et al. (2014) · Cited by 26 · see citing works
Efficiently Determining the Starting Sample Size for Progressive Sampling
Gu, B., Liu, B., Hu, F., et al. (2001) · Lecture notes in computer science · doi:10.1007/3-540-44795-4_17 · Cited by 23 · see citing works
Metadata-driven Software Systems in Biomedicine
Nadkarni, P. M. (2011) · Computers in health care · doi:10.1007/978-0-85729-510-1 · Cited by 23 · see citing works
Sequential Model-Based Ensemble OptimizationPreprint
Lacoste, A., Larochelle, H., Laviolette, F., et al. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1402.0796 · Cited by 22 · see citing works
Efficient Execution Methods of Pivoting for Bulk Extraction of Entity-Attribute-Value-Modeled Data
Luo, G., & Frey, L. J. (2015) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2015.2392553 · Cited by 13 · see citing works
Self-citationLuo, G., Stone, B. L., Johnson, M. D., & Nkoy, F. L. (2016) · JMIR Research Protocols · doi:10.2196/resprot.5155 · Cited by 13 · see citing works
Self-citationCleophas, T. J., & Zwinderman, A. H. (2013) · Springer eBooks · doi:10.1007/978-94-007-6886-4 · Cited by 10 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Feurer, M., & Hutter, F. (2019) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-030-05318-5_1 · Cited by 1,414 · see citing works
Bayesian reaction optimization as a tool for chemical synthesis
Shields, B. J., Stevens, J., Li, J., Parasram, M., Damani, F., Alvarado, J. I. M., Janey, J. M., Adams, R. P., & Doyle, A. G. (2021) · Nature · doi:10.1038/s41586-021-03213-y · Cited by 1,063 · see citing works
Automated machine learning: Review of the state-of-the-art and opportunities for healthcare
Waring, J., Lindvall, C., & Umeton, R. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2020.101822 · Cited by 834 · see citing works
Alibrahim, H., & Ludwig, S. A. (2021) · 2021 IEEE Congress on Evolutionary Computation (CEC) · doi:10.1109/cec45853.2021.9504761 · Cited by 489 · see citing works
Software engineering challenges for machine learning applications: A literature review
Kumeno, F. (2020) · Intelligent Decision Technologies · doi:10.3233/idt-190160 · Cited by 81 · see citing works
Lessons and tips for designing a machine learning study using EHR data
Arbet, J., Brokamp, C., Meinzen-Derr, J., Trinkley, K. E., & Spratt, H. M. (2021) · Journal of Clinical and Translational Science · doi:10.1017/cts.2020.513 · Cited by 46 · see citing works
Towards Trustworthy Machine Learning in Production: An Overview of the Robustness in MLOps Approach
Bayram, F., & Ahmed, B. S. (2025) · ACM Computing Surveys · doi:10.1145/3708497 · Cited by 48 · see citing works
van Mierlo, T., Li, X., Hyatt, D., & Ching, A. T. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.6330 · Cited by 14 · see citing works
The Impact of Data Quality on Software Testing Effort Prediction
Radliński, Ł. (2023) · Electronics · doi:10.3390/electronics12071656 · Cited by 8 · see citing works
- Authors
- Luo, G.
- Affiliations
- University of Utah (US)
- Corresponding author
- Luo, Gang
- ORCID
- Luo, Gang
- Venue
- Network Modeling Analysis in Health Informatics and Bioinformatics
- Published
- 2016-05-23
- Volume
- 5
- Issue
- 1
- ISSN
- 2192-6670 (ISSN-L)
- 2192-6662
- OpenAlex
- W2407212869
- Cited by
- 357
- References
- 85
- Citations by year
- 2026: 17 · 2025: 39 · 2024: 53 · 2023: 50 · 2022: 53 · 2021: 54 · 2020: 35 · 2019: 30 · 2018: 12 · 2017: 9 · 2016: 4 · 2015: 1
- FWCI
- 15.11
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Network Modeling Analysis in Health Informatics and Bioinformatics (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- automatic algorithm selection, hyperparameter optimization, supervised learning, model selection, biomedical data analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
