The DOI matches its Crossref record; 85 of the 89 references it cites were found in a registry or an index.
The importance of being external. methodological insights for the external validation of machine learning models in medicine
Open accessAbstract
Background and Objective Medical machine learning (ML) models tend to perform better on data from the same cohort than on new data, often due to overfitting, or co-variate shifts. For these reasons, external validation (EV) is a necessary practice in the evaluation of medical ML. However, there is still a gap in the literature on how to interpret EV results and hence assess the robustness of ML models.Methods We fill this gap by proposing a meta-validation method, to assess the soundness of EV procedures. In doing so, we complement the usual way to assess EV by considering both dataset cardinality, and the similarity of the EV dataset with respect to the training set. We then investigate how the notions of cardinality and similarity can be used to inform on the reliability of a validation procedure, by integrating them into two summative data visualizations.Results We illustrate our methodology by applying it to the validation of a state-of-the-art COVID-19 diagnostic model on 8 EV sets, collected across 3 different continents. The model performance was moderately impacted by data similarity (Pearson ρ = 0.38, p< 0.001). In the EV, the validated model reported good AUC (average: 0.84), acceptable calibration (average: 0.17) and utility (average: 0.50). The validation datasets were adequate in terms of dataset cardinality and similarity, thus suggesting the soundness of the results. We also provide a qualitative guideline to evaluate the reliability of validation procedures, and we discuss the importance of proper external validation in light of the obtained results.Conclusions In this paper, we propose a novel, lean methodology to: 1) study how the similarity between training and validation sets impacts the generalizability of a ML model; 2) assess the soundness of EV evaluations along three complementary performance dimensions: discrimination, utility and calibration; 3) draw conclusions on the robustness of the model under validation. We applied this methodology to a state-of-the-art model for the diagnosis of COVID-19 from routine blood tests, and showed how to interpret the results in light of the presented framework.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,417
Statistical Power Analysis for the Behavioral Sciences
Cohen, J. (2013) · doi:10.4324/9780203771587 · Cited by 25,373
Receiver Operating Characteristic Curve in Diagnostic Test Assessment
Mandrekar, J. N. (2010) · Journal of Thoracic Oncology · doi:10.1097/jto.0b013e3181ec173d · Cited by 4,506
Wynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070
Internal validation of predictive models
Steyerberg, E. W., Harrell, F. E., Borsboom, G. J. J. M., Eijkemans, M. J. C., Vergouwe, Y., & Habbema, J. D. F. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00341-9 · Cited by 2,746
Automated detection of COVID-19 cases using deep neural networks with X-ray images
Öztürk, T., Talo, M., Yildirim, E. A., Baloğlu, U. B., Yıldırım, Ö., & Acharya, U. R. (2020) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.103792 · Cited by 2,652
A Kernel Method for the Two-Sample-Problem
Gretton, A., Borgwardt, K. M., Rasch, M., Schölkopf, B., & Smola, A. J. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0069 · Cited by 2,167
(2010) · doi:10.5555/1756006.1859921 · Cited by 1,924
Machine learning algorithm validation with a limited sample size
Vabalas, A., Gowen, E., Poliakoff, E., & Casson, A. J. (2019) · PLOS ONE · doi:10.1371/journal.pone.0224365 · Cited by 1,901
Cited by
Sorted by citations
Evaluation of clinical prediction models (part 1): from development to external validation
Collins, G. S., Dhiman, P., Ma, J., Schlussel, M. M., Archer, L., Van Calster, B., Harrell, F. E., Martin, G. P., Moons, K. G. M., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-074819 · Cited by 798
Lekadir, K., Frangi, A. F., Porras, A. R., Glocker, B., Cintas, C., Langlotz, C. P., Weicken, E., Asselbergs, F. W., Prior, F., Collins, G. S., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081554 · Cited by 957
“45 These might include reference or benchmarking datasets that are representative for the task in question (ie, approximating the expected real world variations).”
Feng, J., Phillips, R. V., Malenica, I., Bishara, A., Hubbard, A. E., Celi, L. A., & Pirracchio, R. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00611-y · Cited by 423
Salinas, M. P., Sepúlveda, J., Hidalgo, L., Peirano, D., Morel-Marambio, M., Uribe, P., Rotemberg, V. M., Briones, J., Mery, D., & Navarrete‐Dechent, C. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01103-x · Cited by 191
Targeted validation: validating clinical prediction models in their intended population and setting
Sperrin, M., Riley, R. D., Collins, P. G. S., & Martin, G. P. (2022) · Diagnostic and Prognostic Research · doi:10.1186/s41512-022-00136-8 · Cited by 121
Rahmani, K., Thapa, R., Tsou, P., Chetty, S. C., Barnes, G., Lam, C., & Tso, C. F. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2022.104930 · Cited by 115
Cabitza, F., Campagner, A., Malgieri, G., Natali, C., Schneeberger, D., Stoeger, K., & Holzinger, A. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118888 · Cited by 89
Self-citationScientific evidence for clinical text summarization using large language models: scoping review
Bednarczyk, L., Reichenpfader, D., Gaudet-Blavignac, C., Ette, A. K., Zaghir, J., Zheng, Y., Bensahla, A., Bjelogrlic, M., & Lovis, C. (2025) · Journal of Medical Internet Research · doi:10.2196/68998 · Cited by 83
Homeyer, A., Geißler, C., Schwen, L. O., Zakrzewski, F., Evans, T., Strohmenger, K., Westphal, M., Bülow, R. D., Kargl, M., Karjauv, A., et al. (2022) · Modern Pathology · doi:10.1038/s41379-022-01147-y · Cited by 88
Artificial Intelligence in Head and Neck Cancer: A Systematic Review of Systematic Reviews
Mäkitie, A. A., Alabi, R. O., Ng, S. P., Takes, R. P., Robbins, K. T., Ronen, O., Shaha, A. R., Bradley, P. J., Saba, N. F., Nuyts, S., et al. (2023) · Advances in Therapy · doi:10.1007/s12325-023-02527-9 · Cited by 85
Clinlabomics: leveraging clinical laboratory data by data mining strategies
Wen, X., Leng, P., Wang, J., Yang, G., Zu, R., Jia, X., Zhang, K., Mengesha, B. A., Huang, J. H., Wang, D., et al. (2022) · BMC Bioinformatics · doi:10.1186/s12859-022-04926-1 · Cited by 63
Innovative applications of artificial intelligence during the COVID-19 pandemic
Lv, C., Guo, W., Yin, X., Liu, L., Huang, X., Li, S., & Zhang, L. (2024) · Infectious Medicine · doi:10.1016/j.imj.2024.100095 · Cited by 59
Recent advancements and applications of deep learning in heart failure: Α systematic review
Πετμεζάς, Γ., Papageorgiou, V. Ε., Vassilikos, V. P., Pagourelias, E. D., Tsaklidis, G. M., Katsaggelos, A. K., & Maglaveras, N. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108557 · Cited by 53
Show and tell: A critical review on robustness and uncertainty for a more responsible medical AI
Marconi, L., & Cabitza, F. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.105970 · Cited by 58
Self-citationBarandas, M., Famiglini, L., Campagner, A., Folgado, D., Simão, R., Cabitza, F., & Gamboa, H. F. S. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101978 · Cited by 44
Self-citationHussein, A. M., Sharifai, A. G., Alia, O. M., Abualigah, L., Almotairi, K. H., Abujayyab, S. K. M., & Gandomi, A. H. (2024) · Scientific Reports · doi:10.1038/s41598-023-47038-3 · Cited by 47
Machine learning in physical activity, sedentary, and sleep behavior research
Farrahi, V., & Rostami, M. (2024) · Journal of Activity Sedentary and Sleep Behaviors · doi:10.1186/s44167-024-00045-9 · Cited by 57
Carobene, A., Cabitza, F., Bernardini, S., Gopalan, R., Lennerz, J. K., Weir, C., & Cadamuro, J. (2023) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2022-1030 · Cited by 42
Self-citationRahman, T., Khandakar, A. A., Abir, F. F., Faisal, M. A. A., Hossain, M. S., Podder, K. K., Abbas, T. O., Alam, M. F., Kashem, S. B. A., Islam, M. T., et al. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105284 · Cited by 41
Yin, M., Lin, J., Wang, Y., Liu, Y., Zhang, R., Duan, W., Zhou, Z., Zhu, S., Gao, J., Liu, L., et al. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105341 · Cited by 40
Carobene, A., Milella, F., Famiglini, L., & Cabitza, F. (2022) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2022-0182 · Cited by 40
Self-citationThe challenges of using machine learning models in psychiatric research and clinical practice
Ostojic, D., Lalousis, P. A., Donohoe, G. J., & Morris, D. W. (2024) · European Neuropsychopharmacology · doi:10.1016/j.euroneuro.2024.08.005 · Cited by 43
Pasquini, L., Napolitano, A., Pignatelli, M., Tagliente, E., Parrillo, C., Nasta, F., Romano, A., Bozzao, A., & Napoli, A. D. (2022) · Pharmaceutics · doi:10.3390/pharmaceutics14112378 · Cited by 38
Bento, N., Rebelo, J., Barandas, M., Carreiro, A. V., Campagner, A., Cabitza, F., & Gamboa, H. F. S. (2022) · Sensors · doi:10.3390/s22197324 · Cited by 45
Self-citationFunctional MRI in Neuro-Oncology: State of the Art and Future Directions
Pasquini, L., Peck, K. K., Jenabi, M., & Holodny, A. I. (2023) · Radiology · doi:10.1148/radiol.222028 · Cited by 33
Machine learning algorithms in sepsis
Agnello, L., Vidali, M., Padoan, A., Lucis, R., Mancini, A., Guerranti, R., Plebani, M., Ciaccio, M., & Carobene, A. (2023) · Clinica Chimica Acta · doi:10.1016/j.cca.2023.117738 · Cited by 35
Self-citationMachine learning applications in precision medicine: Overcoming challenges and unlocking potential
Nilius, H., Tsouka, S., Nagler, M., & Masoodi, M. (2024) · TrAC Trends in Analytical Chemistry · doi:10.1016/j.trac.2024.117872 · Cited by 40
Elguoshy, A., Zedan, H., & Saito, S. (2025) · Metabolites · doi:10.3390/metabo15080514 · Cited by 36
Yuan, H. (2024) · Health Care Science · doi:10.1002/hcs2.114 · Cited by 37
“Different from internal validation, which evaluates the performance of ML using a subset of the original datasets, external validation assesses ML models in contexts that may vary subtly or considerably from the one in which they were developed [3].”
MethodAshraf, A. R., Somogyi‐Végh, A., Merczel, S., Gyimesi, N., & Fittler, A. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102844 · Cited by 37
McSweeney, T. P., Tiulpin, A., Saarakkala, S. S., Niinimäki, J. L., Windsor, R., Jamaludin, A., Kadir, T., Karppinen, J. I., & Määttä, J. H. (2022) · Spine · doi:10.1097/brs.0000000000004572 · Cited by 30
Budiarto, A., Tsang, K. C. H., Wilson, A. M., Sheikh, A., & Shah, S. A. (2023) · JMIR AI · doi:10.2196/46717 · Cited by 34
Lennerz, J. K., Salgado, R., Kim, G. E., Sirintrapun, S. J., Thierauf, J. C., Singh, A., Indave, I., Bard, A., Weissinger, S. E., Heher, Y. K., et al. (2023) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2022-1151 · Cited by 29
Self-citationHealth Risk Assessment Using Machine Learning: Systematic Review
Abhadiomhen, S. E., Nzeakor, E. O., & Oyibo, K. (2024) · Electronics · doi:10.3390/electronics13224405 · Cited by 32
Huang, Y., Bazzazzadehgan, S., Li, J., Arabshomali, A., Li, M., Bhattacharya, K., & Bentley, J. P. (2025) · BMC Medical Research Methodology · doi:10.1186/s12874-025-02694-z · Cited by 31
Mohammadzadeh, I., Hajikarimloo, B., Niroomand, B., Eini, P., Ghanbarnia, R., Habibi, M. A., Albakr, A. A., & Borghei-Razavi, H. (2025) · Neurosurgical Review · doi:10.1007/s10143-025-03419-y · Cited by 29
A robust and parsimonious machine learning method to predict ICU admission of COVID-19 patients
Famiglini, L., Campagner, A., Carobene, A., & Cabitza, F. (2022) · Medical & Biological Engineering & Computing · doi:10.1007/s11517-022-02543-x · Cited by 28
Self-citationMachine Learning Approaches to Predict Symptoms in People With Cancer: Systematic Review
Zeinali, N., Youn, N., Albashayreh, A., Fan, W., & Gilbertson White, S. (2024) · JMIR Cancer · doi:10.2196/52322 · Cited by 26
Padoan, A., Cadamuro, J., Frans, G., Cabitza, F., Tolios, A., Bruyne, S. D., van Doorn, W. P. T. M., Elias, J., Debeljak, Ž., Pérez, S. M., et al. (2024) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2024-0971 · Cited by 25
Self-citationCarobene, A., Cadamuro, J., Frans, G., Goldshmidt, H., Debeljak, Z., Bruyne, S. D., van Doorn, W. P. T. M., Elias, J., Özdemir, H., Pérez, S. M., et al. (2025) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2025-0841 · Cited by 25
Self-citationChen, T. L., Buddhiraju, A., Seo, H. H., Subih, M. A., Tuchinda, P., & Kwon, Y. (2023) · The Journal of Arthroplasty · doi:10.1016/j.arth.2023.01.065 · Cited by 25
Machine learning classification meets migraine: recommendations for study evaluation
Petrušić, I., Savić, A. M., Mitrović, K., Bačanin, N., Sebastianelli, G., Secci, D., & Coppola, G. (2024) · The Journal of Headache and Pain · doi:10.1186/s10194-024-01924-x · Cited by 23
Distante, A., Marandino, L., Bertolo, R. G., Ingels, A., Pavan, N., Pecoraro, A., Marchioni, M., Carbonara, U., Erdem, S., Amparore, D., et al. (2023) · Diagnostics · doi:10.3390/diagnostics13132294 · Cited by 28
Grignaffini, F., Simeoni, P., Alisi, A., & Frezza, F. (2024) · Electronics · doi:10.3390/electronics13163174 · Cited by 26
Multicenter Prospective Validation of an Updated Proprietary Sepsis Prediction Model
Wong, A., Currey, D., Schwinne, M., Park-Egan, B., Meyer, S., Gutting, A., Cao, J., Khan, S., Dantes, R., Pan, T., et al. (2026) · JAMA Network Open · doi:10.1001/jamanetworkopen.2026.0181 · Cited by 21
Lehmann, D. H., Gomes, B., Vetter, N., Braun, O., Amr, A., Hilbel, T., Müller, J., Köthe, U., Reich, C., Kayvanpour, E., et al. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00063-3 · Cited by 21
Three-way decision in machine learning tasks: a systematic review
Campagner, A., Milella, F., Ciucci, D., & Cabitza, F. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10845-9 · Cited by 23
Self-citationMagliacano, A., Liuzzi, P., Formisano, R., Grippo, A., Angelakis, E., Thibaut, A., Gosseries, O., Lamberti, G., Noé, E., Bagnato, S., et al. (2022) · Brain Sciences · doi:10.3390/brainsci13010051 · Cited by 21
Liu, Q., Yang, L., Shi, Z., Yu, J., Si, H., Jin, Y., Bian, Y., Li, Y., Ji, L., Qiao, X., et al. (2023) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2023.105138 · Cited by 21
Artificial intelligence in routine blood tests
Santos-Silva, M. A., Sousa, N., & Sousa, J. C. (2024) · Frontiers in Medical Engineering · doi:10.3389/fmede.2024.1369265 · Cited by 24
Cohen, J., Wright-Berryman, J. L., Rohlfs, L., Trocinski, D. R., Daniel, L., & Klatt, T. W. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.818705 · Cited by 25
Troiano, G., Fanelli, F., Rapani, A., Zotti, M., Lombardi, T., Zhurakivska, K., & Stacchi, C. (2023) · Journal Of Clinical Periodontology · doi:10.1111/jcpe.13797 · Cited by 22
Napoli, A. D., Tagliente, E., Pasquini, L., Cipriano, E., Pietrantonio, F., Ortis, P., Curti, S., Boellis, A., Stefanini, T., Bernardini, A., et al. (2022) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-022-00734-4 · Cited by 19
Integrating AI and ML in Myelodysplastic Syndrome Diagnosis: State-of-the-Art and Future Prospects
Elshoeibi, A. M., Badr, A. A., Elsayed, B., Metwally, O. H., Elshoeibi, R. M., Elhadary, M. R., Elshoeibi, A. M., Attya, M. A., Khadadah, F. M., Alshurafa, A., et al. (2023) · Cancers · doi:10.3390/cancers16010065 · Cited by 19
Berman, A., Anteby, R., Efros, O., Klang, E., & Soffer, S. (2023) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2023.04.027 · Cited by 19
Pennestrì, F., Cabitza, F., Picerno, N., & Banfi, G. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-02883-2 · Cited by 21
Self-citationAdvances and opportunities in process analytical technologies for viral vector manufacturing
Sripada, S. A., Hosseini, M., Ramesh, S., Wang, J., Ritola, K., Menegatti, S., & Daniele, M. A.-A. (2024) · Biotechnology Advances · doi:10.1016/j.biotechadv.2024.108391 · Cited by 18
Cai, C., Shi, Q., Li, J., Jiao, Y., Xu, A., Zhou, Y., Wang, X., Peng, C., Zhang, X., Cui, X., et al. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105648 · Cited by 18
Yang, H. S., Çubukçu, H. C., Ben, F. D., Duan, X., Feng, J., Frans, G., Gruson, D., King, M. C., Nieto, J., Wang, B., et al. (2026) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2025-1322 · Cited by 17
Araújo, D. C., Veloso, A., Borges, K. B. G., & Ferriani, M. D. G. C. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2022.104835 · Cited by 19
Çorumlu, V., Altıntaş, V., & Abuşka, M. (2024) · International Communications in Heat and Mass Transfer · doi:10.1016/j.icheatmasstransfer.2023.107228 · Cited by 16
External Validation of a Commercial Acute Kidney Injury Predictive Model
Dutta, S., McEvoy, D. S., Dunham, L. N., Stevens, R., Rubins, D. M., McMahon, G. M., & Samal, L. (2024) · NEJM AI · doi:10.1056/aioa2300099 · Cited by 16
Code-Free Machine Learning Approach for EVO-ICL Vault Prediction: A Retrospective Two-Center Study
Shin, D., Choi, H., Kim, D., Park, J., Yoo, T. K., & Koh, K. (2024) · Translational Vision Science & Technology · doi:10.1167/tvst.13.4.4 · Cited by 17
Li, Y., Zhang, H., Sun, Y., Fan, Q., Wang, L., Ji, C., HuiGu, Chen, B., Zhao, S., Wang, D., et al. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105487 · Cited by 17
Seok, H. S., Yu, S., Shin, K., Lee, W., Chun, S., Kim, S., & Shin, H. (2024) · Clinical Chemistry · doi:10.1093/clinchem/hvae114 · Cited by 17
Mari, T., Asgard, O., Henderson, J. R., Hewitt, D., Brown, C. A., Stančák, A., & Fallon, N. B. (2023) · Scientific Reports · doi:10.1038/s41598-022-27298-1 · Cited by 19
Everything is varied: The surprising impact of instantial variation on ML reliability
Campagner, A., Famiglini, L., Carobene, A., & Cabitza, F. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110644 · Cited by 17
Self-citationFace-based machine learning diagnostics: applications, challenges and opportunities
Song, J., He, M., Zheng, X., Zhang, Y., Bi, C., Feng, J., Du, J., Li, H., & Shen, B. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11246-2 · Cited by 15
Jadhav, S. P., Selvam, M., Zunain, S. M., & Bugalia, N. (2024) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2024.143166 · Cited by 15
Discovery the intersection of performing arts in cultural tourism: a scoping review
Mazlan, C. A. N., Abdullah, M. H., Hashim, N. S. N., Wahid, N. B. A., Pisali, A., Uyub, A. I., Nor, N. M., & Hidayatullah, R. (2025) · Discover Sustainability · doi:10.1007/s43621-025-00805-w · Cited by 15
Lampros, M. G., Symeou, S., Vlachos, N., Gkampenis, A., Zigouris, A., Voulgaris, S., & Alexiou, G. Α. (2024) · Neurosurgical Review · doi:10.1007/s10143-024-02955-3 · Cited by 14
Hassoun, S., Bruckmann, C., Ciardullo, S., Perseghin, G., Gaudio, F. D., & Broccolo, F. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2022.104932 · Cited by 15
Li, Q., Lv, H., Chen, Y., Shen, J., Shi, J., Zhou, C., & Yan, F. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105343 · Cited by 15
Xu, J., Luo, Z., Zhang, X., Li, Z., Li, D., Chen, Y., Lin, X., Guillén, F., & Xu, Y. (2025) · Comprehensive Reviews in Food Science and Food Safety · doi:10.1111/1541-4337.70360 · Cited by 14
Analysis: Flawed Datasets of Monkeypox Skin Images
Vega, C., Schneider, R., & Satagopam, V. (2023) · Journal of Medical Systems · doi:10.1007/s10916-023-01928-1 · Cited by 15
Campagner, A., Agnello, L., Carobene, A., Padoan, A., Ben, F. D., Locatelli, M., Plebani, M., Ognibene, A., Lorubbio, M., Vecchi, E. D., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/55492 · Cited by 17
Self-citationDeep learning for cardiac imaging: focus on myocardial diseases, a narrative review
Tsampras, T., Karamanidou, T., Papanastasiou, G., & Stavropoulos, T. G. (2024) · Hellenic Journal of Cardiology · doi:10.1016/j.hjc.2024.12.002 · Cited by 15
Artificial intelligence and laboratory data in rheumatic diseases
Galozzi, P., Basso, D., Plebani, M., & Padoan, A. (2023) · Clinica Chimica Acta · doi:10.1016/j.cca.2023.117388 · Cited by 12
Machine-learning-assisted Preoperative Prediction of Pediatric Appendicitis Severity
Erman, A., Ferreira, J., Abu-Ashour, W., Guadagno, E., St‐Louis, E., Emil, S. G. S., Cheung, J. S., & Poenaru, D. (2025) · Journal of Pediatric Surgery · doi:10.1016/j.jpedsurg.2024.162151 · Cited by 12
Chiang, D.-H., Jiang, Z., Tian, C., & Wang, C. (2025) · Critical Care · doi:10.1186/s13054-025-05553-x · Cited by 12
Alabi, R. O., Sjöblom, A., Carpén, T. P., Elmusrati, M. S., Leivo, I. V. V., Almangush, A., & Mäkitie, A. A. (2023) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2023.105064 · Cited by 12
External Validation of AI Models for Skin Diseases: A Systematic Review
Fernandes, T. R. S., Teles, A. S., Fernandes, J. R. N., Lima, L. D. B., Sousa, D. L., Soares, R. D. C., & Teixeira, S. S. (2025) · IEEE Access · doi:10.1109/access.2025.3584904 · Cited by 12
The role of machine learning in autism spectrum disorder assessment and management
Reilly, A., Walsh, N., O’Reilly, D. P., Smyth, M., Gorman, K. M., Ostadabbas, S., & Power, C. (2025) · Pediatric Research · doi:10.1038/s41390-025-04566-0 · Cited by 15
Good practices and common pitfalls of machine learning in nutrition research
Kirk, D. (2024) · Proceedings of The Nutrition Society · doi:10.1017/s0029665124007638 · Cited by 11
Bhattacharjee, A., Kumar, A., Ojha, P. K., & Kar, S. (2025) · Expert Opinion on Drug Discovery · doi:10.1080/17460441.2025.2491669 · Cited by 11
Ndjonko, L. C. M., Chakraborty, A., Petri, F., Alavi, S. M. A., Matsuo, T., Borgonovo, F., Comba, I. Y., Murad, M. H., Nassr, A. N., Zein, S. E., et al. (2025) · The Spine Journal · doi:10.1016/j.spinee.2025.07.032 · Cited by 12
Mesquita, F., Bernardino, J., Henriques, J., Raposo, J., Ribeiro, R., & Paredes, S. (2023) · Journal of Diabetes and Metabolic Disorders · doi:10.1007/s40200-023-01357-4 · Cited by 11
Automated determination of transport and depositional environments in sand and sandstones
Hasson, M., Marvin, M. C., & Lapôtre, M. G. A. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2407655121 · Cited by 11
Study and prediction of photocurrent density with external validation using machine learning models
Sahu, N., Azad, C., & Kumar, U. (2024) · International Journal of Hydrogen Energy · doi:10.1016/j.ijhydene.2024.10.339 · Cited by 10
Mehri-Kakavand, G., Mdletshe, S., & Wang, A. (2025) · Journal of Medical Radiation Sciences · doi:10.1002/jmrs.860 · Cited by 10
Yismaw, M. B., Tafere, C., Tefera, B. B., Demsie, D. G., Feyisa, K., Addisu, Z. D., Zeleke, T. K., Siraj, E. A., Worku, M. C., & Berihun, F. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.105858 · Cited by 10
Zhang, S., Ding, H., Shen, Y., Shao, B., Gu, Y.-Y., Chen, Q., Zhang, H., Pei, Y., & Jiang, H. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-02976-y · Cited by 14
Cazzaniga, G., L’Imperio, V., Bonoldi, E., Londoño, M., Madaleno, J., Cipriano, A., Gevers, T. J. G., Samarska, I. V., Koc, Ö. M., Villamil, A., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01870-1 · Cited by 10
Methaneethorn, J., Duangchaemkarn, K., Reisfeld, B., & Habiballah, S. (2025) · Clinical Pharmacokinetics · doi:10.1007/s40262-025-01547-8 · Cited by 10
Moustakli, E., Grigoriadis, T., Stavros, S., Potiris, A., Zikopoulos, A., Gerede, A., Tsimpoukis, I., Papageorgiou, C., Louis, K., & Domali, E. (2025) · Diagnostics · doi:10.3390/diagnostics15162075 · Cited by 11
Dadabhoy, F. Z., Driver, L., McEvoy, D. S., Stevens, R., Rubins, D., & Dutta, S. (2023) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2022.11.012 · Cited by 10
Comparing code-free and bespoke deep learning approaches in ophthalmology
Wong, C. Y. T., O’Byrne, C., Taribagil, P., Liu, T., Antaki, F., & Keane, P. A. (2024) · Graefe s Archive for Clinical and Experimental Ophthalmology · doi:10.1007/s00417-024-06432-x · Cited by 10
Applying Artificial Intelligence for Phenotyping of Inherited Arrhythmia Syndromes
Sigfstead, S., Jiang, R., Avram, R., Davies, B., Krahn, A. D., & Cheung, C. C. (2024) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2024.04.014 · Cited by 9
Rigorous validation of machine learning in laboratory medicine: guidance toward quality improvement
Miller, H. A., & Valdes, R. (2025) · Critical Reviews in Clinical Laboratory Sciences · doi:10.1080/10408363.2025.2488842 · Cited by 13
Vagliano, I., Schut, M. C., Abu–Hanna, A., Dongelmans, D. A., de Lange, D. W., Gommers, D. A. M. P. J., Cremer, O. L., Bosman, R. J., Rigter, S., Wils, E., et al. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2022.104863 · Cited by 9
Metadata
- Authors
- Cabitza, F., Campagner, A., Soares, F., García de Guadiana-Romualdo, L., Challa, F., Sulejmani, A., Seghezzi, M., & Carobene, A.
- Affiliations
- University of Milano-Bicocca (IT)
- Universidade Federal do Rio Grande do Sul (BR)
- Ethiopian Public Health Institute (ET)
- Ospedale Papa Giovanni XXIII (IT)
- IRCCS Ospedale San Raffaele (IT)
- Corresponding author
- Cabitza, Federico
- Venue
- Computer Methods and Programs in Biomedicine
- Publisher
- Elsevier BV
- Published
- 2021-09
- Volume
- 208
- Pages
- 106288
- ISSN
- 0169-2607 (ISSN-L)
- OpenAlex
- W3183421885
- Cited by
- 284
- References
- 89
- Citations by year
- 2026: 71 · 2025: 83 · 2024: 70 · 2023: 33 · 2022: 26 · 2021: 1
- FWCI
- 28.33
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- Locations
- Computer Methods and Programs in Biomedicine (publishedVersion)
- PubMed (publishedVersion)
- BOA (University of Milano-Bicocca) (submittedVersion, other-oa)
- Topics
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- external validation, machine learning in medicine, covariate shift, overfitting, generalizability, dataset similarity, model robustness, COVID-19 diagnosis, routine blood tests, discrimination (statistics), calibration, clinical utility
- MeSH terms
- Machine Learning, COVID-19, SARS-CoV-2, Humans, Reproducibility of Results, Cohort Studies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
