The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
On hyperparameter optimization of machine learning algorithms: Theory and practice
Open accessAbstract
Abstract Machine learning algorithms have been used widely in various applications and areas. To fit a machine learning model into different problems, its hyper-parameters must be tuned. Selecting the best hyper-parameter configuration for machine learning models has a direct impact on the model’s performance. It often requires deep knowledge of machine learning algorithms and appropriate hyper-parameter optimization techniques. Although several automatic optimization techniques exist, they have different strengths and drawbacks when applied to different types of problems. In this paper, optimizing the hyper-parameters of common machine learning models is studied. We introduce several state-of-the-art optimization techniques and discuss how to apply them to machine learning algorithms. Many available libraries and frameworks developed for hyper-parameter optimization problems are provided, and some open challenges of hyper-parameter optimization research are also discussed in this paper. Moreover, experiments are conducted on benchmark datasets to compare the performance of different optimization methods and provide practical examples of hyper-parameter optimization. This survey paper will help industrial users, data analysts, and researchers to better develop machine learning models by identifying the proper hyper-parameter configurations effectively.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482
(1990) · Choice Reviews Online · doi:10.5860/choice.27-3338 · Cited by 35,305
Hosmer, D. W., Lemeshow, S., & Sturdivant, R. X. (2013) · Wiley Series in Probability and Statistics · doi:10.1002/9781118548387 · Cited by 13,861
Machine learning: Trends, perspectives, and prospects
Jordan, M. I., & Mitchell, T. M. (2015) · Science · doi:10.1126/science.aaa8415 · Cited by 9,955
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408
Ensemble methods in machine learning
Dietterich, T. G. (2000) · Lecture Notes in Computer Science · doi:10.1007/3-540-45014-9_1 · Cited by 8,408
XGBoost: A Scalable Tree Boosting System
Chen, T., & Guestrin, C. (2016) · arXiv · doi:10.48550/arxiv.1603.02754 · Cited by 7,104
Kuhn, M., & Johnson, K. (2013) · doi:10.1007/978-1-4614-6849-3 · Cited by 6,788 · cited 1×
“model parameters; while the other, named hyper-parameters, cannot be directly estimated from data learning and must be set before training a ML model because they dene the architecture of a ML model [4]. Hyper-parameters are the parameters that are used to either congure a ML model (e.g., the penalty parameter Cin a s…”
BackgroundPractical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691
Cited by
Challenges in deploying machine learning: A survey of case studies
Paleyes, A., Urma, R.-G., & Lawrence, N. D. (2023) · ACM Computing Surveys · doi:10.1145/3533378 · Cited by 638
Application of machine learning for advanced material prediction and design
Chan, C. H., Sun, M., & Huang, B. (2022) · EcoMat · doi:10.1002/eom2.12194 · Cited by 123
ANN-based automated scaffold builder activity recognition through wearable EMG and IMU sensors
Bangaru, S. S., Wang, C., Busam, S. A., & Aghazadeh, F. (2021) · Automation in Construction · doi:10.1016/j.autcon.2021.103653 · Cited by 114
Transforming Digital Marketing with Generative AI
Islam, T., Miron, A., Nandy, M., Choudrie, J., Liu, X., & Li, Y. (2024) · Computers · doi:10.3390/computers13070168 · Cited by 68
Machine Learning Approach to Simulate Soil CO2 Fluxes under Cropping Systems
Adjuik, T. A., & Davis, S. C. (2022) · Agronomy · doi:10.3390/agronomy12010197 · Cited by 56
“The hyperparameters of an algorithm are the parameters that can be configured to minimize the loss function of the model which increases accuracy in the models being developed [41].”
Yeslam, H. E., Freifrau von Maltzahn, N., & Nassar, H. M. (2024) · PeerJ · doi:10.7717/peerj.17793 · Cited by 55
The Use of Machine Learning Models with Optuna in Disease Prediction
Lai, L.-H., Lin, Y.-L., Liu, Y.-H., Lai, J.-P., Yang, W.-C., Hou, H.-P., & Pai, P.-F. (2024) · Electronics · doi:10.3390/electronics13234775 · Cited by 30
Enhancing Debris Flow Warning via Machine Learning Feature Reduction and Model Selection
Zhou, Q., Tang, H., Hibert, C., Chmiel, M., Walter, F., Dietze, M., & Turowski, J. M. (2025) · Journal of Geophysical Research: Earth Surface · doi:10.1029/2024jf008094 · Cited by 20
Entezami, A., De Michele, C., Arslan, A. N., & Behkamal, B. (2022) · Sensors · doi:10.3390/s22134964 · Cited by 19
Jarmakovica, A. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1621514 · Cited by 10
Metadata
- Authors
- Yang, L., & Shami, A.
- Venue
- Neurocomputing
- Publisher
- Elsevier BV
- Published
- 2020-11
- Volume
- 415
- Pages
- 295-316
- ISSN
- 0925-2312 (ISSN-L)
- OpenAlex
- W3045004532
- Cited by
- 3,232
- References
- 14
- FWCI
- 99.92
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
