Tuning hyperparameters of machine learning algorithms and deep neural networks using metaheuristics: A bioinformatics study on biomedical and biological cases
The DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index.
Abstract
The performance of a model in machine learning problems highly depends on the dataset and training algorithms. Choosing the right training algorithm can change the tale of a model. While some algorithms have a great performance in some datasets, they may fall into trouble in other datasets. Moreover, by adjusting hyperparameters of an algorithm, which controls the training processes, the performance can be improved. This study contributes a method to tune hyperparameters of machine learning algorithms using Grey Wolf Optimization (GWO) and Genetic algorithm (GA) metaheuristics. Also, 11 different algorithms including Averaged Perceptron, FastTree, FastForest, Light Gradient Boost Machine (LGBM), Limited memory Broyden Fletcher Goldfarb Shanno algorithm Maximum Entropy (LbfgsMxEnt), Linear Support Vector Machine (LinearSVM), and a Deep Neural Network (DNN) including four architectures are employed on 11 datasets in different biological, biomedical, and nature categories such as molecular interactions, cancer, clinical diagnosis, behavior related predictions, RGB images of human skin, and X-rays images of Covid19 and cardiomegaly patients. Our results show that in all trials, the performance of the training phases is improved. Also, GWO demonstrates a better performance with a p-value of 2.6E-5. Moreover, in most experiment cases of this study, the metaheuristic methods demonstrate better performance and faster convergence than Exhaustive Grid Search (EGS). The proposed method just receives a dataset as an input and suggests the best-explored algorithm with related arguments. So, it is appropriate for datasets with unknown distribution, machine learning algorithms with complex behavior, or users who are not experts in analytical statistics and data science algorithms.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Pan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387
Mirjalili, S., Mirjalili, S. M., & Lewis, A. (2014) · Advances in Engineering Software · doi:10.1016/j.advengsoft.2013.12.007 · Cited by 19,343
On the limited memory BFGS method for large scale optimization
Liu, D. C., & Nocedal, J. (1989) · Mathematical Programming · doi:10.1007/bf01589116 · Cited by 8,560
FastTree: Computing Large Minimum Evolution Trees with Profiles instead of a Distance Matrix
Price, M. N., Dehal, P. S., & Arkin, A. P. (2009) · Molecular Biology and Evolution · doi:10.1093/molbev/msp077 · Cited by 5,602
Whitley, D. (1994) · Statistics and Computing · doi:10.1007/bf00175354 · Cited by 4,522
On hyperparameter optimization of machine learning algorithms: Theory and practice
Yang, L., & Shami, A. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.07.061 · Cited by 3,232
Hyperparameters and tuning strategies for random forest
Probst, P., Wright, M. N., & Boulesteix, A. (2019) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1301 · Cited by 1,640
Hyperparameter Optimization for Machine Learning Models Based on Bayesian Optimization
Wu, J., Chen, X. Y., Zhang, H., XIONG, L., Lei, H., & Deng, S. (2019) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.11989/jest.1674-862x.80904120 · Cited by 1,410
Deep-COVID: Predicting COVID-19 from chest X-ray images using deep transfer learning
Minaee, S., Kafieh, R., Sonka, M., Yazdani, S., & Jamalipour Soufi, G. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101794 · Cited by 1,024
An Analysis of Deep Neural Network Models for Practical Applications
Canziani, A., Paszke, A., & Culurciello, E. (2016) · arXiv · doi:10.48550/arxiv.1605.07678 · Cited by 969
A survey on new generation metaheuristic algorithms
Dokeroglu, T., Sevinc, E., Kucukyilmaz, T., & Cosar, A. (2019) · Computers & Industrial Engineering · doi:10.1016/j.cie.2019.106040 · Cited by 923
Metaheuristic research: a comprehensive survey
Hussain, K., Mohd Salleh, M. N., Cheng, S., & Shi, Y. (2019) · Artificial Intelligence Review · doi:10.1007/s10462-017-9605-z · Cited by 864
An Efficient Approach for Assessing Hyperparameter Importance
Hutter, F., Hoos, H. H., & Leyton‐Brown, K. (2014) · International Conference on Machine Learning · Cited by 736
Detrano, R., Janosi, A., Steinbrunn, W., Pfisterer, M., Schmid, J.-J., Sandhu, S., Guppy, K. H., Lee, S., & Froelicher, V. (1989) · The American Journal of Cardiology · doi:10.1016/0002-9149(89)90524-9 · Cited by 695
Kabir Anaraki, A., Ayati, M., & Kazemi, F. (2019) · Biocybernetics and Biomedical Engineering · doi:10.1016/j.bbe.2018.10.004 · Cited by 611
Chicco, D., & Jurman, G. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-1023-5 · Cited by 596
Support Vector Machines for Pattern Classification
Abe, S. (2010) · Advances in Pattern Recognition · doi:10.1007/978-1-84996-098-4 · Cited by 581
Tunability: Importance of Hyperparameters of Machine Learning Algorithms
Probst, P., Bischl, B., & Boulesteix, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.09596 · Cited by 565
Multiclass Support Vector Machines
Abe, S. (2010) · Advances in Pattern Recognition · doi:10.1007/978-1-84996-098-4_3 · Cited by 512
Zhou, J., Qiu, Y., Zhu, S., Armaghani, D. J., Li, C., Nguyen, H., & Yağız, S. (2020) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2020.104015 · Cited by 333
Zhang, J., Huang, Y., Wang, Y., & Ma, G. (2020) · Construction and Building Materials · doi:10.1016/j.conbuildmat.2020.119208 · Cited by 294
Performance assessment of the metaheuristic optimization algorithms: an exhaustive review
Halim, A. H., Ismail, I., & Das, S. (2021) · Artificial Intelligence Review · doi:10.1007/s10462-020-09906-6 · Cited by 293
Darwish, A., Ezzat, D., & Hassanien, A. E. (2020) · Swarm and Evolutionary Computation · doi:10.1016/j.swevo.2019.100616 · Cited by 279
Transfer Learning with Partial Observability Applied to Cervical Cancer Screening
Fernandes, K., Cardoso, J. S., & Fernandes, J. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-58838-4_27 · Cited by 230
Lujan-Moreno, G. A., Howard, P. R., Rojas, O. G., & Montgomery, D. C. (2018) · Expert Systems with Applications · doi:10.1016/j.eswa.2018.05.024 · Cited by 176
Chamasemani, F. F., & Singh, Y. P. (2011) · International Conference on Bio-Inspired Computing: Theories and Applications (BIC-TA) · doi:10.1109/bic-ta.2011.51 · Cited by 172
Bačanin, N., Bezdan, T., Tuba, E., Strumberger, I., & Tuba, M. (2020) · Algorithms · doi:10.3390/a13030067 · Cited by 172
Addressing voice recording replications for Parkinson’s disease detection
Naranjo, L., Pérez, C. J., Campos-Roca, Y., & Martín, J. (2016) · Expert Systems with Applications · doi:10.1016/j.eswa.2015.10.034 · Cited by 162
Palechor, F. M., & Manotas, A. D. L. H. (2019) · Data in Brief · doi:10.1016/j.dib.2019.104344 · Cited by 159
Machine Learning at Microsoft with ML.NET
Ahmed, Z., Amizadeh, S., Bilenko, M., Carr, R., Chin, W.-S., Dekel, Y., Dupre, X., Eksarevskiy, V., Filipi, S., Finley, T., et al. (2019) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3292500.3330667 · Cited by 109
Lee, W.-Y., Park, S.-M., & Sim, K. (2018) · Optik (Stuttgart) · doi:10.1016/j.ijleo.2018.07.044 · Cited by 104
Empirical comparison of cross-validation and internal metrics for tuning SVM hyperparameters
Duarte, E. C., & Wainer, J. (2017) · Pattern Recognition Letters · doi:10.1016/j.patrec.2017.01.007 · Cited by 91
Predicting Positive p53 Cancer Rescue Regions Using Most Informative Positive (MIP) Active Learning
Danziger, S. A., Baronio, R., Ho, L. S., Hall, L., Salmon, K. A., Hatfield, G. W., Kaiser, P., & Lathrop, R. H. (2009) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000498 · Cited by 90
Exploring Tunable Hyperparameters for Deep Neural Networks with Industrial ADME Data Sets
Zhou, Y., Cahya, S., Combs, S. A., Nicolaou, C. A., Wang, J., Desai, P. V., & Shen, J. (2019) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.8b00671 · Cited by 78
Neary, P. L. (2018) · IEEE International Conference on Cognitive Computing (ICCC) · doi:10.1109/iccc.2018.00017 · Cited by 66
Recent Advances and Application of Metaheuristic Algorithms: A Survey (2014–2020)
Khanduja, N., & Bhushan, B. (2020) · Studies in computational intelligence · doi:10.1007/978-981-15-7571-6_10 · Cited by 62
FastForest: Increasing random forest processing speed while maintaining accuracy
Yates, D., & Islam, M. Z. (2020) · Information Sciences · doi:10.1016/j.ins.2020.12.067 · Cited by 58
Grisoni, F., Consonni, V., & Ballabio, D. (2019) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.8b00794 · Cited by 57
Integrated QSAR Models to Predict Acute Oral Systemic Toxicity
Ballabio, D., Grisoni, F., Consonni, V., & Todeschini, R. (2018) · Molecular Informatics · doi:10.1002/minf.201800124 · Cited by 50
Torkamanian-Afshar, M., Nematzadeh, S., Tabarzad, M., Najafi, A., Lanjanian, H., & Masoudi‐Nejad, A. (2021) · Molecular Diversity · doi:10.1007/s11030-021-10192-9 · Cited by 37
Self-citationA Swarm Intelligence Approach to Avoid Local Optima in Fuzzy C-Means Clustering
Fuchs, C., Spolaor, S., Nobile, M. S., & Kaymak, U. (2019) · IEEE International Conference on Fuzzy Systems (FUZZ-IEEE) · doi:10.1109/fuzz-ieee.2019.8858940 · Cited by 29
An Effective Hyperparameter Optimization Algorithm for DNN to Predict Passengers at a Metro Station
Tsai, C., & Fang, Z.-Y. (2021) · ACM Transactions on Internet Technology · doi:10.1145/3410156 · Cited by 25
High-throughput analysis of the interactions between viral proteins and host cell RNAs
Lanjanian, H., Nematzadeh, S., Hosseini, S., Torkamanian-Afshar, M., Kiani, F., Moazzam‐Jazi, M., Aydın, N., & Masoudi‐Nejad, A. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104611 · Cited by 23
Self-citationRPINBASE: An online toolbox to extract features for predicting RNA-protein interactions
Torkamanian-Afshar, M., Lanjanian, H., Nematzadeh, S., Tabarzad, M., Najafi, A., Kiani, F., & Masoudi‐Nejad, A. (2020) · Genomics · doi:10.1016/j.ygeno.2020.02.013 · Cited by 23
Self-citationFine Tuning Deep Boltzmann Machines Through Meta-Heuristic Approaches
Júnior, L. A. P., Rodrigues, D., & Papa, J. P. (2018) · IEEE International Symposium on Applied Computational Intelligence and Informatics (SACI) · doi:10.1109/saci.2018.8440959 · Cited by 20
Hyperparameter Tuning using Quantum Genetic Algorithms
Lentzas, A., Nalmpantis, C., & Vrakas, D. (2019) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai.2019.00199 · Cited by 14
Two-Class Support Vector Machines
Abe, S. (2010) · Advances in computer vision and pattern recognition · doi:10.1007/978-1-84996-098-4_2 · Cited by 12
Cited by
A systematic review of hyperparameter optimization techniques in Convolutional Neural Networks
Raiaan, M. A. K., Sakib, S., Fahad, N. M., Mamun, A. A., Rahman, M. A., Shatabda, S., & Mukta, M. S. H. (2024) · Decision Analytics Journal · doi:10.1016/j.dajour.2024.100470 · Cited by 254
Machine learning and deep learning algorithms for skin cancer classification from dermoscopic images
Bechelli, S., & Delhommelle, J. (2022) · Bioengineering · doi:10.3390/bioengineering9030097 · Cited by 181
“Furthermore, optimization algorithms for the fine-tuning of hyperparameters are currently under development to improve accuracy [28] and methods to detect shortcut learning in the training of CNN models on the ISIC Archive dataset have been recently proposed [29].”
Method
Metadata
- Authors
- Nematzadeh, S., Kiani, F., Torkamanian-Afshar, M., & Aydin, N.
- Venue
- Computational Biology and Chemistry
- Publisher
- Elsevier BV
- Published
- 2022-04
- Volume
- 97
- Pages
- 107619
- ISSN
- 1476-9271 (ISSN-L)
- OpenAlex
- W4200409353
- Cited by
- 229
- References
- 47
- FWCI
- 11.20
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.