The importance of being external. methodological insights for the external validation of machine learning models in medicine
Open accessAbstract
Background and Objective Medical machine learning (ML) models tend to perform better on data from the same cohort than on new data, often due to overfitting, or co-variate shifts. For these reasons, external validation (EV) is a necessary practice in the evaluation of medical ML. However, there is still a gap in the literature on how to interpret EV results and hence assess the robustness of ML models.Methods We fill this gap by proposing a meta-validation method, to assess the soundness of EV procedures. In doing so, we complement the usual way to assess EV by considering both dataset cardinality, and the similarity of the EV dataset with respect to the training set. We then investigate how the notions of cardinality and similarity can be used to inform on the reliability of a validation procedure, by integrating them into two summative data visualizations.Results We illustrate our methodology by applying it to the validation of a state-of-the-art COVID-19 diagnostic model on 8 EV sets, collected across 3 different continents. The model performance was moderately impacted by data similarity (Pearson ρ = 0.38, p< 0.001). In the EV, the validated model reported good AUC (average: 0.84), acceptable calibration (average: 0.17) and utility (average: 0.50). The validation datasets were adequate in terms of dataset cardinality and similarity, thus suggesting the soundness of the results. We also provide a qualitative guideline to evaluate the reliability of validation procedures, and we discuss the importance of proper external validation in light of the obtained results.Conclusions In this paper, we propose a novel, lean methodology to: 1) study how the similarity between training and validation sets impacts the generalizability of a ML model; 2) assess the soundness of EV evaluations along three complementary performance dimensions: discrimination, utility and calibration; 3) draw conclusions on the robustness of the model under validation. We applied this methodology to a state-of-the-art model for the diagnosis of COVID-19 from routine blood tests, and showed how to interpret the results in light of the presented framework.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · see citing works
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,406 · see citing works
Statistical Power Analysis for the Behavioral Sciences
Cohen, J. (2013) · doi:10.4324/9780203771587 · Cited by 25,373 · see citing works
Receiver Operating Characteristic Curve in Diagnostic Test Assessment
Mandrekar, J. N. (2010) · Journal of Thoracic Oncology · doi:10.1097/jto.0b013e3181ec173d · Cited by 4,506 · see citing works
Wynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070 · see citing works
Internal validation of predictive models
Steyerberg, E. W., Harrell, F. E., Borsboom, G. J. J. M., Eijkemans, M. J. C., Vergouwe, Y., & Habbema, J. D. F. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00341-9 · Cited by 2,746 · see citing works
Automated detection of COVID-19 cases using deep neural networks with X-ray images
Öztürk, T., Talo, M., Yildirim, E. A., Baloğlu, U. B., Yıldırım, Ö., & Acharya, U. R. (2020) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.103792 · Cited by 2,607 · see citing works
A Kernel Method for the Two-Sample-Problem
Gretton, A., Borgwardt, K. M., Rasch, M., Schölkopf, B., & Smola, A. J. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0069 · Cited by 2,167 · see citing works
(2010) · doi:10.5555/1756006.1859921 · Cited by 1,924 · see citing works
Machine learning algorithm validation with a limited sample size
Vabalas, A., Gowen, E., Poliakoff, E., & Casson, A. J. (2019) · PLOS ONE · doi:10.1371/journal.pone.0224365 · Cited by 1,901 · see citing works
The Balanced Accuracy and Its Posterior Distribution
Brodersen, K. H., Ong, C. S., Stephan, K. E., & Buhmann, J. M. (2010) · 2010 20th International Conference on Pattern Recognition · doi:10.1109/icpr.2010.764 · Cited by 1,816 · cited 1× · see citing works
“In particular, we suggest to used balanced performance metrics to better account for potential imbalances in the target distribution [9].”
MethodReal-time tracking of self-reported symptoms to predict potential COVID-19
Menni, C., Valdes, A. M., Freidin, M. B., Sudre, C. H., Nguyen, L. H., Drew, D. A., Ganesh, S., Varsavsky, T., Cardoso, M. J., El-Sayed Moustafa, J. S., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0916-2 · Cited by 1,517 · see citing works
CoroNet: A deep neural network for detection and diagnosis of COVID-19 from chest x-ray images
Khan, A. I., Shah, J. L., & Bhat, M. M. (2020) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2020.105581 · Cited by 1,312 · see citing works
Prediction models need appropriate internal, internal-external, and external validation
Steyerberg, E. W., & Harrell, F. E. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.04.005 · Cited by 1,304 · see citing works
2010 20th International Conference on Pattern Recognition
(2010) · doi:10.1109/icpr16097.2010 · Cited by 1,183 · see citing works
Roberts, M., Driggs, D., Thorpe, M., Gilbey, J., Yeung, M., Ursprung, S., Aviles-Rivero, A. I., Etmann, C., McCague, C., Beer, L., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00307-0 · Cited by 1,044 · cited 5× · see citing works
“The same reasoning obviously applies, and if possible even exacerbated, in the case of models that rely on less stable tests, such as imaging [50].”
BackgroundMethodInfluentialA calibration hierarchy for risk models was defined: from utopia to empirical data
Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina, M. J., & Steyerberg, E. W. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.12.005 · Cited by 910 · see citing works
Roberts, M. T., Driggs, D., Thorpe, M., Gilbey, J., Yeung, M., Ursprung, S., Avilés-Rivero, A. I., Etmann, C., McCague, C., Beer, L., et al. (2020) · Apollo (University of Cambridge) · doi:10.17863/cam.64000 · Cited by 902 · see citing works
Machine Learning Testing: Survey, Landscapes and Horizons
Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2962027 · Cited by 897 · see citing works
Chicco, D., Tötsch, N., & Jurman, G. (2021) · BioData Mining · doi:10.1186/s13040-021-00244-z · Cited by 893 · see citing works
Ardakani, A. A., Kanafi, A. R., Acharya, U. R., Khadem, N., & Mohammadi, A. (2020) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.103795 · Cited by 848 · see citing works
Estimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap
Kim, J. (2009) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2009.04.009 · Cited by 792 · see citing works
Collins, G. S., de Groot, J. A., Dutton, S., Omar, O., Shanyinde, M., Tajar, A., Voysey, M., Wharton, R., Yu, L.-M., Moons, K. G., et al. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-40 · Cited by 729 · see citing works
Model selection and overfitting
Lever, J., Krzywinski, M. I., & Altman, N. (2016) · Nature Methods · doi:10.1038/nmeth.3968 · Cited by 727 · see citing works
External validation is necessary in prediction research:
Bleeker, S. E., Moll, H. A., Steyerberg, E. W., Donders, A. R. T., Derksen-Lubsen, G., Grobbee, D. E., & Moons, K. G. M. (2003) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(03)00207-5 · Cited by 710 · see citing works
Collins, G. S., Ogundimu, E. O., & Altman, D. G. (2016) · Statistics in Medicine · doi:10.1002/sim.6787 · Cited by 665 · see citing works
Vergouwe, Y., Steyerberg, E. W., Eijkemans, M. J. C., & Habbema, J. D. F. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.06.017 · Cited by 636 · see citing works
Explainable Deep Learning for Pulmonary Disease and Coronavirus COVID-19 Detection from X-rays
Brunese, L., Mercaldo, F., Reginelli, A., & Santone, A. (2020) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2020.105608 · Cited by 623 · see citing works
Steyerberg, E. W., Bleeker, S. E., Moll, H. A., Grobbee, D. E., & Moons, K. G. M. (2003) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(03)00047-7 · Cited by 611 · see citing works
Onukwugha, E., Yong, C., Hussain, A., Seal, B., & Mullins, C. D. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-1 · Cited by 589 · see citing works
COVID-19 identification in chest X-ray images on flat and hierarchical classification scenarios
Pereira, R. M., Bertolini, D. S., Teixeira, L., Junior, C. N. S., & Costa, Y. M. G. (2020) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2020.105532 · Cited by 511 · see citing works
2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI)
Supatto, W. (2020) · doi:10.1109/isbi45749.2020 · Cited by 472 · see citing works
The myth of generalisability in clinical research and machine learning in health care
Futoma, J., Simons, M., Panch, T., Doshi-Velez, F., & Celi, L. A. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30186-2 · Cited by 467 · see citing works
Li, X., Zhang, S., Zhang, Q., Wei, X., Pan, Y., Zhao, J., Xin, X., Qin, C., Wang, X., Li, J., et al. (2018) · The Lancet Oncology · doi:10.1016/s1470-2045(18)30762-9 · Cited by 433 · see citing works
Hu, F., Jiang, X. W., Guo, C., Li, Y., Chen, S., Zhang, W., Du, Y., Wang, P., Zheng, X., Fang, X., et al. (2020) · Nature Communications · doi:10.1038/s41467-020-15700-3 · Cited by 419 · see citing works
Hernandez-Boussard, T., Bozkurt, S., Ioannidis, J. P. A., & Shah, N. H. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa088 · Cited by 410 · see citing works
Minimum sample size for external validation of a clinical prediction model with a binary outcome
Riley, R. D., Debray, T. P. A., Collins, G. S., Archer, L., Ensor, J., van Smeden, M., & Snell, K. I. E. (2021) · Statistics in Medicine · doi:10.1002/sim.9025 · Cited by 400 · see citing works
Underspecification Presents Challenges for Credibility in Modern Machine Learning
D'Amour, A., Heller, K., Moldovan, D., Adlam, B., Alipanahi, B., Beutel, A., Chen, C., Deaton, J., Eisenstein, J., Hoffman, M. D., et al. (2020) · arXiv · doi:10.48550/arxiv.2011.03395 · Cited by 398 · see citing works
Challenges to the Reproducibility of Machine Learning Models in Health Care
Beam, A. L., Manrai, A. K., & Ghassemi, M. (2020) · JAMA · doi:10.1001/jama.2019.20866 · Cited by 366 · see citing works
On the effectiveness of preprocessing methods when dealing with different levels of class imbalance
García, V., Sánchez, J. S., & Mollineda, R. A. (2011) · Knowledge-Based Systems · doi:10.1016/j.knosys.2011.06.013 · Cited by 358 · see citing works
Kim, S., Lee, K., Hwang, H. Y., & Yoo, S. (2015) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-016-0249-8 · Cited by 290 · see citing works
Explaining physicians’ acceptance of EHCR systems: An extension of TAM with trust and risk factors
Ortega Egea, J. M., & Román González, M. V. (2011) · Computers in Human Behavior · doi:10.1016/j.chb.2010.08.010 · Cited by 287 · see citing works
The need to separate the wheat from the chaff in medical informatics
Cabitza, F., & Campagner, A. (2021) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2021.104510 · Cited by 283 · see citing works
Self-citationRoutine blood tests as a potential diagnostic tool for COVID-19
Ferrari, D., Motta, A., Strollo, M., Banfi, G., & Locatelli, M. (2020) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2020-0398 · Cited by 273 · see citing works
Menni, C., Valdes, A. M., Freidin, M. B., et al. (2020) · medRxiv · doi:10.1101/2020.04.05.20048421 · Cited by 232 · see citing works
Minimum sample size for external validation of a clinical prediction model with a continuous outcome
Archer, L., Snell, K. I. E., Ensor, J., Hudda, M. T., Collins, G. S., & Riley, R. D. (2021) · Statistics in Medicine · doi:10.1002/sim.8766 · Cited by 198 · see citing works
Wang, G., Liu, X., Shen, J., Wang, C., Li, Z., Ye, L., Wu, X., Chen, T., Wang, K., Zhang, X., et al. (2021) · Nature Biomedical Engineering · doi:10.1038/s41551-021-00704-1 · Cited by 192 · see citing works
Ateya, M., Delaney, B. C., & Speedie, S. M. (2015) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-016-0239-x · Cited by 172 · see citing works
Cabitza, F., Campagner, A., Ferrari, D., Resta, C. D., Ceriotti, D., Sabetta, E., Colombini, A., Vecchi, E. D., Banfi, G., Locatelli, M., et al. (2020) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2020-1294 · Cited by 168 · see citing works
Self-citationRousson, V., & Zumbrunn, T. (2011) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-11-45 · Cited by 165 · see citing works
Soltan, A. A. S., Kouchaki, S., Zhu, T., Kiyasseh, D., Taylor, T., Hussain, Z. B., Peto, T., Brent, A. J., Eyre, D. W., & Clifton, D. A. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30274-0 · Cited by 150 · see citing works
Clinician checklist for assessing suitability of machine learning applications in healthcare
Scott, I. A., Carter, S. M., & Coiera, E. W. (2021) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2020-100251 · Cited by 140 · see citing works
Practical experiences on the necessity of external validation
König, I. R., Malley, J. D., Weimar, C., Diener, H., & Ziegler, A. (2007) · Statistics in Medicine · doi:10.1002/sim.3069 · Cited by 136 · see citing works
Routine Laboratory Blood Tests Predict SARS-CoV-2 Infection Using Machine Learning
Yang, H. S., Hou, Y., Vasović, L. V., Steel, P. A. D., Chadburn, A., Racine‐Brzostek, S., Velu, P., Cushing, M. M., Loda, M. F., Kaushal, R., et al. (2020) · Clinical Chemistry · doi:10.1093/clinchem/hvaa200 · Cited by 125 · see citing works
Snell, K. I. E., Archer, L., Ensor, J., Bonnett, L. J., Debray, T. P. A., Phillips, B., Collins, P. G. S., & Riley, R. D. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.02.011 · Cited by 125 · see citing works
Song, X., Yu, A. S. L., Kellum, J. A., Waitman, L. R., Matheny, M. E., Simpson, S. Q., Hu, Y., & Liu, M. (2020) · Nature Communications · doi:10.1038/s41467-020-19551-w · Cited by 120 · see citing works
Sampling Uncertainty and Confidence Intervals for the Brier Score and Brier Skill Score
Bradley, A., Schwartz, S., & Hashino, T. (2008) · Weather and Forecasting · doi:10.1175/2007waf2007049.1 · Cited by 101 · see citing works
Machine Learning for Prediction in Electronic Health Data
Rose, S. (2018) · JAMA Network Open · doi:10.1001/jamanetworkopen.2018.1404 · Cited by 95 · see citing works
Race‐specific WBC and neutrophil count reference intervals
Lim, E., Cembrowski, G. S., Cembrowski, M. S., & Clarke, G. (2010) · International Journal of Laboratory Hematology · doi:10.1111/j.1751-553x.2010.01223.x · Cited by 93 · see citing works
Bierbooms, J. J. P. A., Bongers, I. M. B., & OERS, H. A. M. V. (2011) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-11-1 · Cited by 88 · see citing works
Theoretical Analysis of a Performance Measure for Imbalanced Data
García, V., Mollineda, R. A., & Sánchez, J. S. (2010) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2010.156 · Cited by 73 · see citing works
Cabitza, F., & Zeitoun, J.-D. (2019) · Annals of Translational Medicine · doi:10.21037/atm.2019.04.07 · Cited by 72 · see citing works
Self-citationAdvances in Domain Adaptation Theory
Redko, I., Morvant, E., Habrard, A., Sebban, M., & Bennani, Y. (2019) · HAL · Cited by 65 · see citing works
Diagnostics of prior-data agreement in applied Bayesian analysis
Bousquet, N. (2008) · Journal of Applied Statistics · doi:10.1080/02664760802192981 · Cited by 61 · see citing works
Estimation of required sample size for external validation of risk models for binary outcomes
Pavlou, M., Chen, Q., Omar, R., Seaman, S. R., Steyerberg, E. W., White, I. R., & Ambler, G. (2021) · Statistical Methods in Medical Research · doi:10.1177/09622802211007522 · Cited by 58 · see citing works
Coşkun, A., Braga, F., Carobene, A., Tejedor-Ganduxé, X., Aarsand, A. K., Fernández–Calle, P., Díaz–Garzón, J., Bartlett, W. A., Jonker, N., Aslan, B., et al. (2019) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2019-0658 · Cited by 55 · see citing works
Self-citationAs if sand were stone. New concepts and metrics to probe the ground on which to build trustable AI
Cabitza, F., Campagner, A., & Sconfienza, L. M. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-01224-9 · Cited by 46 · see citing works
Self-citationPlante, T. B., Blau, A. M., Berg, A. N., Weinberg, A. S., Jun, I. C., Tapson, V. F., Kanigan, T. S., & Adib, A. A. (2020) · Journal of Medical Internet Research · doi:10.2196/24048 · Cited by 46 · see citing works
The optimal diagnostic methods for COVID-19
Harahwa, T., Yau, T., Lim‐Cooke, M., Al-Haddi, S., Zeinah, M., & Harky, A. (2020) · Diagnosis · doi:10.1515/dx-2020-0058 · Cited by 45 · see citing works
Identification of risk factors for patients with diabetes: diabetic polyneuropathy case study
Metsker, O., Magoev, K., Yakovlev, A., Yanishevskiy, S., Kopanitsa, G., Kovalchuk, S., & Krzhizhanovskaya, V. V. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-01215-w · Cited by 35 · see citing works
Schat, E., van de Schoot, R., Kouw, W. M., Veen, D., & Mendrik, A. M. (2020) · PLoS ONE · doi:10.1371/journal.pone.0237009 · Cited by 28 · see citing works
kNN-based high-dimensional Kullback-Leibler distance for tracking
Boltz, S., Debreuve, É., & Barlaud, M. (2007) · International Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS) · doi:10.1109/wiamis.2007.53 · Cited by 22 · see citing works
Harmonization: Its Time Has Come
Miller, W. G. (2017) · Clinical Chemistry · doi:10.1373/clinchem.2017.274860 · Cited by 20 · see citing works
Vidali, M., Carobene, A., Esposito, S. A., Napolitano, G., Caracciolo, A., Seghezzi, M., Previtali, G., Lippi, G., & Buoro, S. (2020) · International Journal of Laboratory Hematology · doi:10.1111/ijlh.13379 · Cited by 19 · see citing works
Self-citationUsing the Data Agreement Criterion to Rank Experts' Beliefs
Veen, D., Stoel, D., Schalken, N., Mulder, K., van de Schoot, R., Sciences, M. A. S. F. T. B. A. S., & Hoijtink, L. (2016) · Utrecht University Repository (Utrecht University) · Cited by 17 · see citing works
Abegaz, F., Lishout, F. V., John, J. M., Chiachoompu, K., Bhardwaj, A., Duroux, D., Gusareva, E. S., Wei, Z., Hakonarson, H. H., & Steen, K. V. (2021) · BioData Mining · doi:10.1186/s13040-021-00247-w · Cited by 17 · see citing works
Using the Data Agreement Criterion to Rank Experts’ Beliefs
Veen, D., Stoel, D., Schalken, N., Mulder, K., & van de Schoot, R. (2018) · Entropy · doi:10.3390/e20080592 · Cited by 15 · see citing works
Estimating Kullback-Leibler Divergence Using Kernel Machines
Ahuja, K. (2019) · 2018 52nd Asilomar Conference on Signals, Systems, and Computers · doi:10.1109/ieeeconf44664.2019.9049082 · Cited by 12 · see citing works
Detecting Concept Drift In Medical Triage
Huggard, H., Koh, Y. S., Dobbie, G., & Zhang, E. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401228 · Cited by 11 · see citing works
Learning An Mr Acquisition-Invariant Representation Using Siamese Neural Networks
Kouw, W. M., Loog, M., Bartels, L. W., & Mendrik, A. M. (2019) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2019.8759281 · Cited by 9 · see citing works
Franzini, C. (1995) · PubMed · Cited by 8 · see citing works
Carobene, A., Kiarie, A. G., Tsegaye, M., Seghezzi, M., Buoro, S., & Mburugu, R. W. (2020) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2020-0364 · Cited by 2 · see citing works
Self-citation
Flagged first, then by citations
4 of the 89 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- Jiangpeng Wu, Pengyi Zhang, Liting Zhang, Wenbo Meng, Junfeng Li, Chongxiang Tong, Yonghong Li, Jing Cai, Zengwei Yang, Jinhong Zhu, et al. Rapid and accurate identification of covid-19 infection through ma- chine learning based on clinical available blood test results. medRxiv, 2020.
Cited by
Sorted by citations
Evaluation of clinical prediction models (part 1): from development to external validation
Collins, G. S., Dhiman, P., Ma, J., Schlussel, M. M., Archer, L., Van Calster, B., Harrell, F. E., Martin, G. P., Moons, K. G. M., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-074819 · Cited by 798 · see citing works
Lekadir, K., Frangi, A. F., Porras, A. R., Glocker, B., Cintas, C., Langlotz, C. P., Weicken, E., Asselbergs, F. W., Prior, F., Collins, G. S., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081554 · Cited by 957 · see citing works
“45 These might include reference or benchmarking datasets that are representative for the task in question (ie, approximating the expected real world variations).”
Feng, J., Phillips, R. V., Malenica, I., Bishara, A., Hubbard, A. E., Celi, L. A., & Pirracchio, R. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00611-y · Cited by 423 · see citing works
Salinas, M. P., Sepúlveda, J., Hidalgo, L., Peirano, D., Morel-Marambio, M., Uribe, P., Rotemberg, V. M., Briones, J., Mery, D., & Navarrete‐Dechent, C. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01103-x · Cited by 165 · see citing works
Targeted validation: validating clinical prediction models in their intended population and setting
Sperrin, M., Riley, R. D., Collins, P. G. S., & Martin, G. P. (2022) · Diagnostic and Prognostic Research · doi:10.1186/s41512-022-00136-8 · Cited by 115 · see citing works
Rahmani, K., Thapa, R., Tsou, P., Chetty, S. C., Barnes, G., Lam, C., & Tso, C. F. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2022.104930 · Cited by 115 · see citing works
Cabitza, F., Campagner, A., Malgieri, G., Natali, C., Schneeberger, D., Stoeger, K., & Holzinger, A. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118888 · Cited by 89 · see citing works
Self-citationScientific evidence for clinical text summarization using large language models: scoping review
Bednarczyk, L., Reichenpfader, D., Gaudet-Blavignac, C., Ette, A. K., Zaghir, J., Zheng, Y., Bensahla, A., Bjelogrlic, M., & Lovis, C. (2025) · Journal of Medical Internet Research · doi:10.2196/68998 · Cited by 83 · see citing works
Homeyer, A., Geißler, C., Schwen, L. O., Zakrzewski, F., Evans, T., Strohmenger, K., Westphal, M., Bülow, R. D., Kargl, M., Karjauv, A., et al. (2022) · Modern Pathology · doi:10.1038/s41379-022-01147-y · Cited by 83 · see citing works
Artificial Intelligence in Head and Neck Cancer: A Systematic Review of Systematic Reviews
Mäkitie, A. A., Alabi, R. O., Ng, S. P., Takes, R. P., Robbins, K. T., Ronen, O., Shaha, A. R., Bradley, P. J., Saba, N. F., Nuyts, S., et al. (2023) · Advances in Therapy · doi:10.1007/s12325-023-02527-9 · Cited by 81 · see citing works
Clinlabomics: leveraging clinical laboratory data by data mining strategies
Wen, X., Leng, P., Wang, J., Yang, G., Zu, R., Jia, X., Zhang, K., Mengesha, B. A., Huang, J. H., Wang, D., et al. (2022) · BMC Bioinformatics · doi:10.1186/s12859-022-04926-1 · Cited by 63 · see citing works
Innovative applications of artificial intelligence during the COVID-19 pandemic
Lv, C., Guo, W., Yin, X., Liu, L., Huang, X., Li, S., & Zhang, L. (2024) · Infectious Medicine · doi:10.1016/j.imj.2024.100095 · Cited by 59 · see citing works
Recent advancements and applications of deep learning in heart failure: Α systematic review
Πετμεζάς, Γ., Papageorgiou, V. Ε., Vassilikos, V. P., Pagourelias, E. D., Tsaklidis, G. M., Katsaggelos, A. K., & Maglaveras, N. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108557 · Cited by 50 · see citing works
Show and tell: A critical review on robustness and uncertainty for a more responsible medical AI
Marconi, L., & Cabitza, F. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.105970 · Cited by 49 · see citing works
Self-citationBarandas, M., Famiglini, L., Campagner, A., Folgado, D., Simão, R., Cabitza, F., & Gamboa, H. F. S. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101978 · Cited by 44 · see citing works
Self-citationHussein, A. M., Sharifai, A. G., Alia, O. M., Abualigah, L., Almotairi, K. H., Abujayyab, S. K. M., & Gandomi, A. H. (2024) · Scientific Reports · doi:10.1038/s41598-023-47038-3 · Cited by 43 · see citing works
Machine learning in physical activity, sedentary, and sleep behavior research
Farrahi, V., & Rostami, M. (2024) · Journal of Activity Sedentary and Sleep Behaviors · doi:10.1186/s44167-024-00045-9 · Cited by 53 · see citing works
Carobene, A., Cabitza, F., Bernardini, S., Gopalan, R., Lennerz, J. K., Weir, C., & Cadamuro, J. (2023) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2022-1030 · Cited by 42 · see citing works
Self-citationRahman, T., Khandakar, A. A., Abir, F. F., Faisal, M. A. A., Hossain, M. S., Podder, K. K., Abbas, T. O., Alam, M. F., Kashem, S. B. A., Islam, M. T., et al. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105284 · Cited by 41 · see citing works
Yin, M., Lin, J., Wang, Y., Liu, Y., Zhang, R., Duan, W., Zhou, Z., Zhu, S., Gao, J., Liu, L., et al. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105341 · Cited by 40 · see citing works
Carobene, A., Milella, F., Famiglini, L., & Cabitza, F. (2022) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2022-0182 · Cited by 37 · see citing works
Self-citationThe challenges of using machine learning models in psychiatric research and clinical practice
Ostojic, D., Lalousis, P. A., Donohoe, G. J., & Morris, D. W. (2024) · European Neuropsychopharmacology · doi:10.1016/j.euroneuro.2024.08.005 · Cited by 40 · see citing works
Pasquini, L., Napolitano, A., Pignatelli, M., Tagliente, E., Parrillo, C., Nasta, F., Romano, A., Bozzao, A., & Napoli, A. D. (2022) · Pharmaceutics · doi:10.3390/pharmaceutics14112378 · Cited by 34 · see citing works
Bento, N., Rebelo, J., Barandas, M., Carreiro, A. V., Campagner, A., Cabitza, F., & Gamboa, H. F. S. (2022) · Sensors · doi:10.3390/s22197324 · Cited by 36 · see citing works
Self-citationFunctional MRI in Neuro-Oncology: State of the Art and Future Directions
Pasquini, L., Peck, K. K., Jenabi, M., & Holodny, A. I. (2023) · Radiology · doi:10.1148/radiol.222028 · Cited by 33 · see citing works
Machine learning algorithms in sepsis
Agnello, L., Vidali, M., Padoan, A., Lucis, R., Mancini, A., Guerranti, R., Plebani, M., Ciaccio, M., & Carobene, A. (2023) · Clinica Chimica Acta · doi:10.1016/j.cca.2023.117738 · Cited by 35 · see citing works
Self-citationMachine learning applications in precision medicine: Overcoming challenges and unlocking potential
Nilius, H., Tsouka, S., Nagler, M., & Masoodi, M. (2024) · TrAC Trends in Analytical Chemistry · doi:10.1016/j.trac.2024.117872 · Cited by 36 · see citing works
Elguoshy, A., Zedan, H., & Saito, S. (2025) · Metabolites · doi:10.3390/metabo15080514 · Cited by 34 · see citing works
Yuan, H. (2024) · Health Care Science · doi:10.1002/hcs2.114 · Cited by 31 · see citing works
“Different from internal validation, which evaluates the performance of ML using a subset of the original datasets, external validation assesses ML models in contexts that may vary subtly or considerably from the one in which they were developed [3].”
MethodAshraf, A. R., Somogyi‐Végh, A., Merczel, S., Gyimesi, N., & Fittler, A. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102844 · Cited by 31 · see citing works
McSweeney, T. P., Tiulpin, A., Saarakkala, S. S., Niinimäki, J. L., Windsor, R., Jamaludin, A., Kadir, T., Karppinen, J. I., & Määttä, J. H. (2022) · Spine · doi:10.1097/brs.0000000000004572 · Cited by 30 · see citing works
Budiarto, A., Tsang, K. C. H., Wilson, A. M., Sheikh, A., & Shah, S. A. (2023) · JMIR AI · doi:10.2196/46717 · Cited by 32 · see citing works
Lennerz, J. K., Salgado, R., Kim, G. E., Sirintrapun, S. J., Thierauf, J. C., Singh, A., Indave, I., Bard, A., Weissinger, S. E., Heher, Y. K., et al. (2023) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2022-1151 · Cited by 29 · see citing works
Self-citationHealth Risk Assessment Using Machine Learning: Systematic Review
Abhadiomhen, S. E., Nzeakor, E. O., & Oyibo, K. (2024) · Electronics · doi:10.3390/electronics13224405 · Cited by 29 · see citing works
Huang, Y., Bazzazzadehgan, S., Li, J., Arabshomali, A., Li, M., Bhattacharya, K., & Bentley, J. P. (2025) · BMC Medical Research Methodology · doi:10.1186/s12874-025-02694-z · Cited by 30 · see citing works
Mohammadzadeh, I., Hajikarimloo, B., Niroomand, B., Eini, P., Ghanbarnia, R., Habibi, M. A., Albakr, A. A., & Borghei-Razavi, H. (2025) · Neurosurgical Review · doi:10.1007/s10143-025-03419-y · Cited by 29 · see citing works
A robust and parsimonious machine learning method to predict ICU admission of COVID-19 patients
Famiglini, L., Campagner, A., Carobene, A., & Cabitza, F. (2022) · Medical & Biological Engineering & Computing · doi:10.1007/s11517-022-02543-x · Cited by 28 · see citing works
Self-citationMachine Learning Approaches to Predict Symptoms in People With Cancer: Systematic Review
Zeinali, N., Youn, N., Albashayreh, A., Fan, W., & Gilbertson White, S. (2024) · JMIR Cancer · doi:10.2196/52322 · Cited by 26 · see citing works
Padoan, A., Cadamuro, J., Frans, G., Cabitza, F., Tolios, A., Bruyne, S. D., van Doorn, W. P. T. M., Elias, J., Debeljak, Ž., Pérez, S. M., et al. (2024) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2024-0971 · Cited by 25 · see citing works
Self-citationCarobene, A., Cadamuro, J., Frans, G., Goldshmidt, H., Debeljak, Z., Bruyne, S. D., van Doorn, W. P. T. M., Elias, J., Özdemir, H., Pérez, S. M., et al. (2025) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2025-0841 · Cited by 25 · see citing works
Self-citationChen, T. L., Buddhiraju, A., Seo, H. H., Subih, M. A., Tuchinda, P., & Kwon, Y. (2023) · The Journal of Arthroplasty · doi:10.1016/j.arth.2023.01.065 · Cited by 25 · see citing works
Machine learning classification meets migraine: recommendations for study evaluation
Petrušić, I., Savić, A. M., Mitrović, K., Bačanin, N., Sebastianelli, G., Secci, D., & Coppola, G. (2024) · The Journal of Headache and Pain · doi:10.1186/s10194-024-01924-x · Cited by 23 · see citing works
Distante, A., Marandino, L., Bertolo, R. G., Ingels, A., Pavan, N., Pecoraro, A., Marchioni, M., Carbonara, U., Erdem, S., Amparore, D., et al. (2023) · Diagnostics · doi:10.3390/diagnostics13132294 · Cited by 25 · see citing works
Grignaffini, F., Simeoni, P., Alisi, A., & Frezza, F. (2024) · Electronics · doi:10.3390/electronics13163174 · Cited by 22 · see citing works
Multicenter Prospective Validation of an Updated Proprietary Sepsis Prediction Model
Wong, A., Currey, D., Schwinne, M., Park-Egan, B., Meyer, S., Gutting, A., Cao, J., Khan, S., Dantes, R., Pan, T., et al. (2026) · JAMA Network Open · doi:10.1001/jamanetworkopen.2026.0181 · Cited by 21 · see citing works
Lehmann, D. H., Gomes, B., Vetter, N., Braun, O., Amr, A., Hilbel, T., Müller, J., Köthe, U., Reich, C., Kayvanpour, E., et al. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00063-3 · Cited by 21 · see citing works
Three-way decision in machine learning tasks: a systematic review
Campagner, A., Milella, F., Ciucci, D., & Cabitza, F. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10845-9 · Cited by 21 · see citing works
Self-citationMagliacano, A., Liuzzi, P., Formisano, R., Grippo, A., Angelakis, E., Thibaut, A., Gosseries, O., Lamberti, G., Noé, E., Bagnato, S., et al. (2022) · Brain Sciences · doi:10.3390/brainsci13010051 · Cited by 21 · see citing works
Liu, Q., Yang, L., Shi, Z., Yu, J., Si, H., Jin, Y., Bian, Y., Li, Y., Ji, L., Qiao, X., et al. (2023) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2023.105138 · Cited by 21 · see citing works
Artificial intelligence in routine blood tests
Santos-Silva, M. A., Sousa, N., & Sousa, J. C. (2024) · Frontiers in Medical Engineering · doi:10.3389/fmede.2024.1369265 · Cited by 24 · see citing works
Cohen, J. T., Wright-Berryman, J., Rohlfs, L., Trocinski, D., Daniel, L., & Klatt, T. W. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.818705 · Cited by 19 · see citing works
Troiano, G., Fanelli, F., Rapani, A., Zotti, M., Lombardi, T., Zhurakivska, K., & Stacchi, C. (2023) · Journal Of Clinical Periodontology · doi:10.1111/jcpe.13797 · Cited by 19 · see citing works
Napoli, A. D., Tagliente, E., Pasquini, L., Cipriano, E., Pietrantonio, F., Ortis, P., Curti, S., Boellis, A., Stefanini, T., Bernardini, A., et al. (2022) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-022-00734-4 · Cited by 19 · see citing works
Integrating AI and ML in Myelodysplastic Syndrome Diagnosis: State-of-the-Art and Future Prospects
Elshoeibi, A. M., Badr, A. A., Elsayed, B., Metwally, O. H., Elshoeibi, R. M., Elhadary, M. R., Elshoeibi, A. M., Attya, M. A., Khadadah, F. M., Alshurafa, A., et al. (2023) · Cancers · doi:10.3390/cancers16010065 · Cited by 19 · see citing works
Berman, A., Anteby, R., Efros, O., Klang, E., & Soffer, S. (2023) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2023.04.027 · Cited by 18 · see citing works
Pennestrì, F., Cabitza, F., Picerno, N., & Banfi, G. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-02883-2 · Cited by 21 · see citing works
Self-citationAdvances and opportunities in process analytical technologies for viral vector manufacturing
Sripada, S. A., Hosseini, M., Ramesh, S., Wang, J., Ritola, K., Menegatti, S., & Daniele, M. A.-A. (2024) · Biotechnology Advances · doi:10.1016/j.biotechadv.2024.108391 · Cited by 18 · see citing works
Cai, C., Shi, Q., Li, J., Jiao, Y., Xu, A., Zhou, Y., Wang, X., Peng, C., Zhang, X., Cui, X., et al. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105648 · Cited by 18 · see citing works
Yang, H. S., Çubukçu, H. C., Ben, F. D., Duan, X., Feng, J., Frans, G., Gruson, D., King, M. C., Nieto, J., Wang, B., et al. (2026) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2025-1322 · Cited by 17 · see citing works
Araújo, D. C., Veloso, A., Borges, K. B. G., & Ferriani, M. D. G. C. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2022.104835 · Cited by 19 · see citing works
Çorumlu, V., Altıntaş, V., & Abuşka, M. (2024) · International Communications in Heat and Mass Transfer · doi:10.1016/j.icheatmasstransfer.2023.107228 · Cited by 17 · see citing works
External Validation of a Commercial Acute Kidney Injury Predictive Model
Dutta, S., McEvoy, D. S., Dunham, L. N., Stevens, R., Rubins, D. M., McMahon, G. M., & Samal, L. (2024) · NEJM AI · doi:10.1056/aioa2300099 · Cited by 17 · see citing works
Code-Free Machine Learning Approach for EVO-ICL Vault Prediction: A Retrospective Two-Center Study
Shin, D., Choi, H., Kim, D., Park, J., Yoo, T. K., & Koh, K. (2024) · Translational Vision Science & Technology · doi:10.1167/tvst.13.4.4 · Cited by 17 · see citing works
Li, Y., Zhang, H., Sun, Y., Fan, Q., Wang, L., Ji, C., HuiGu, Chen, B., Zhao, S., Wang, D., et al. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105487 · Cited by 17 · see citing works
Seok, H. S., Yu, S., Shin, K., Lee, W., Chun, S., Kim, S., & Shin, H. (2024) · Clinical Chemistry · doi:10.1093/clinchem/hvae114 · Cited by 16 · see citing works
Mari, T., Asgard, O., Henderson, J. R., Hewitt, D., Brown, C. A., Stančák, A., & Fallon, N. B. (2023) · Scientific Reports · doi:10.1038/s41598-022-27298-1 · Cited by 16 · see citing works
Everything is varied: The surprising impact of instantial variation on ML reliability
Campagner, A., Famiglini, L., Carobene, A., & Cabitza, F. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110644 · Cited by 17 · see citing works
Self-citationFace-based machine learning diagnostics: applications, challenges and opportunities
Song, J., He, M., Zheng, X., Zhang, Y., Bi, C., Feng, J., Du, J., Li, H., & Shen, B. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11246-2 · Cited by 15 · see citing works
Jadhav, S. P., Selvam, M., Zunain, S. M., & Bugalia, N. (2024) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2024.143166 · Cited by 15 · see citing works
Discovery the intersection of performing arts in cultural tourism: a scoping review
Mazlan, C. A. N., Abdullah, M. H., Hashim, N. S. N., Wahid, N. B. A., Pisali, A., Uyub, A. I., Nor, N. M., & Hidayatullah, R. (2025) · Discover Sustainability · doi:10.1007/s43621-025-00805-w · Cited by 15 · see citing works
Lampros, M. G., Symeou, S., Vlachos, N., Gkampenis, A., Zigouris, A., Voulgaris, S., & Alexiou, G. Α. (2024) · Neurosurgical Review · doi:10.1007/s10143-024-02955-3 · Cited by 14 · see citing works
Hassoun, S., Bruckmann, C., Ciardullo, S., Perseghin, G., Gaudio, F. D., & Broccolo, F. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2022.104932 · Cited by 14 · see citing works
Li, Q., Lv, H., Chen, Y., Shen, J., Shi, J., Zhou, C., & Yan, F. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105343 · Cited by 14 · see citing works
Xu, J., Luo, Z., Zhang, X., Li, Z., Li, D., Chen, Y., Lin, X., Guillén, F., & Xu, Y. (2025) · Comprehensive Reviews in Food Science and Food Safety · doi:10.1111/1541-4337.70360 · Cited by 13 · see citing works
Analysis: Flawed Datasets of Monkeypox Skin Images
Vega, C., Schneider, R., & Satagopam, V. (2023) · Journal of Medical Systems · doi:10.1007/s10916-023-01928-1 · Cited by 14 · see citing works
Campagner, A., Agnello, L., Carobene, A., Padoan, A., Ben, F. D., Locatelli, M., Plebani, M., Ognibene, A., Lorubbio, M., Vecchi, E. D., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/55492 · Cited by 16 · see citing works
Self-citationDeep learning for cardiac imaging: focus on myocardial diseases, a narrative review
Tsampras, T., Karamanidou, T., Papanastasiou, G., & Stavropoulos, T. G. (2024) · Hellenic Journal of Cardiology · doi:10.1016/j.hjc.2024.12.002 · Cited by 13 · see citing works
Artificial intelligence and laboratory data in rheumatic diseases
Galozzi, P., Basso, D., Plebani, M., & Padoan, A. (2023) · Clinica Chimica Acta · doi:10.1016/j.cca.2023.117388 · Cited by 12 · see citing works
Machine-learning-assisted Preoperative Prediction of Pediatric Appendicitis Severity
Erman, A., Ferreira, J., Abu-Ashour, W., Guadagno, E., St‐Louis, E., Emil, S. G. S., Cheung, J. S., & Poenaru, D. (2025) · Journal of Pediatric Surgery · doi:10.1016/j.jpedsurg.2024.162151 · Cited by 12 · see citing works
Chiang, D.-H., Jiang, Z., Tian, C., & Wang, C. (2025) · Critical Care · doi:10.1186/s13054-025-05553-x · Cited by 12 · see citing works
Alabi, R. O., Sjöblom, A., Carpén, T. P., Elmusrati, M. S., Leivo, I. V. V., Almangush, A., & Mäkitie, A. A. (2023) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2023.105064 · Cited by 12 · see citing works
External Validation of AI Models for Skin Diseases: A Systematic Review
Fernandes, T. R. S., Teles, A. S., Fernandes, J. R. N., Lima, L. D. B., Sousa, D. L., Soares, R. D. C., & Teixeira, S. S. (2025) · IEEE Access · doi:10.1109/access.2025.3584904 · Cited by 12 · see citing works
The role of machine learning in autism spectrum disorder assessment and management
Reilly, A., Walsh, N., O’Reilly, D. P., Smyth, M., Gorman, K. M., Ostadabbas, S., & Power, C. (2025) · Pediatric Research · doi:10.1038/s41390-025-04566-0 · Cited by 11 · see citing works
Good practices and common pitfalls of machine learning in nutrition research
Kirk, D. (2024) · Proceedings of The Nutrition Society · doi:10.1017/s0029665124007638 · Cited by 11 · see citing works
Bhattacharjee, A., Kumar, A., Ojha, P. K., & Kar, S. (2025) · Expert Opinion on Drug Discovery · doi:10.1080/17460441.2025.2491669 · Cited by 11 · see citing works
Ndjonko, L. C. M., Chakraborty, A., Petri, F., Alavi, S. M. A., Matsuo, T., Borgonovo, F., Comba, I. Y., Murad, M. H., Nassr, A. N., Zein, S. E., et al. (2025) · The Spine Journal · doi:10.1016/j.spinee.2025.07.032 · Cited by 12 · see citing works
Mesquita, F., Bernardino, J., Henriques, J., Raposo, J., Ribeiro, R., & Paredes, S. (2023) · Journal of Diabetes and Metabolic Disorders · doi:10.1007/s40200-023-01357-4 · Cited by 11 · see citing works
Automated determination of transport and depositional environments in sand and sandstones
Hasson, M., Marvin, M. C., & Lapôtre, M. G. A. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2407655121 · Cited by 10 · see citing works
Study and prediction of photocurrent density with external validation using machine learning models
Sahu, N., Azad, C., & Kumar, U. (2024) · International Journal of Hydrogen Energy · doi:10.1016/j.ijhydene.2024.10.339 · Cited by 10 · see citing works
Mehri-Kakavand, G., Mdletshe, S., & Wang, A. (2025) · Journal of Medical Radiation Sciences · doi:10.1002/jmrs.860 · Cited by 10 · see citing works
Yismaw, M. B., Tafere, C., Tefera, B. B., Demsie, D. G., Feyisa, K., Addisu, Z. D., Zeleke, T. K., Siraj, E. A., Worku, M. C., & Berihun, F. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.105858 · Cited by 10 · see citing works
Zhang, S., Ding, H., Shen, Y., Shao, B., Gu, Y.-Y., Chen, Q., Zhang, H., Pei, Y., & Jiang, H. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-02976-y · Cited by 14 · see citing works
Cazzaniga, G., L’Imperio, V., Bonoldi, E., Londoño, M., Madaleno, J., Cipriano, A., Gevers, T. J. G., Samarska, I. V., Koc, Ö. M., Villamil, A., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01870-1 · Cited by 10 · see citing works
Methaneethorn, J., Duangchaemkarn, K., Reisfeld, B., & Habiballah, S. (2025) · Clinical Pharmacokinetics · doi:10.1007/s40262-025-01547-8 · Cited by 10 · see citing works
Moustakli, E., Grigoriadis, T., Stavros, S., Potiris, A., Zikopoulos, A., Gerede, A., Tsimpoukis, I., Papageorgiou, C., Louis, K., & Domali, E. (2025) · Diagnostics · doi:10.3390/diagnostics15162075 · Cited by 10 · see citing works
Dadabhoy, F. Z., Driver, L., McEvoy, D. S., Stevens, R., Rubins, D., & Dutta, S. (2023) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2022.11.012 · Cited by 10 · see citing works
Comparing code-free and bespoke deep learning approaches in ophthalmology
Wong, C. Y. T., O’Byrne, C., Taribagil, P., Liu, T., Antaki, F., & Keane, P. A. (2024) · Graefe s Archive for Clinical and Experimental Ophthalmology · doi:10.1007/s00417-024-06432-x · Cited by 10 · see citing works
Applying Artificial Intelligence for Phenotyping of Inherited Arrhythmia Syndromes
Sigfstead, S., Jiang, R., Avram, R., Davies, B., Krahn, A. D., & Cheung, C. C. (2024) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2024.04.014 · Cited by 8 · see citing works
Rigorous validation of machine learning in laboratory medicine: guidance toward quality improvement
Miller, H. A., & Valdes, R. (2025) · Critical Reviews in Clinical Laboratory Sciences · doi:10.1080/10408363.2025.2488842 · Cited by 9 · see citing works
Vagliano, I., Schut, M. C., Abu–Hanna, A., Dongelmans, D. A., de Lange, D. W., Gommers, D. A. M. P. J., Cremer, O. L., Bosman, R. J., Rigter, S., Wils, E., et al. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2022.104863 · Cited by 9 · see citing works
- Authors
- Cabitza, F., Campagner, A., Soares, F., García de Guadiana-Romualdo, L., Challa, F., Sulejmani, A., Seghezzi, M., & Carobene, A.
- Affiliations
- University of Milano-Bicocca (IT)
- Universidade Federal do Rio Grande do Sul (BR)
- Ethiopian Public Health Institute (ET)
- Ospedale Papa Giovanni XXIII (IT)
- IRCCS Ospedale San Raffaele (IT)
- Corresponding author
- Cabitza, Federico
- Venue
- Computer Methods and Programs in Biomedicine
- Publisher
- Elsevier BV
- Published
- 2021-09
- Volume
- 208
- Pages
- 106288
- ISSN
- 0169-2607 (ISSN-L)
- OpenAlex
- W3183421885
- Cited by
- 284
- References
- 89
- Citations by year
- 2026: 71 · 2025: 83 · 2024: 70 · 2023: 33 · 2022: 26 · 2021: 1
- FWCI
- 28.33
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- Locations
- Computer Methods and Programs in Biomedicine (publishedVersion)
- PubMed (publishedVersion)
- BOA (University of Milano-Bicocca) (submittedVersion, other-oa)
- Topics
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- external validation, machine learning in medicine, covariate shift, overfitting, generalizability, dataset similarity, model robustness, COVID-19 diagnosis, routine blood tests, discrimination (statistics), calibration, clinical utility
- MeSH terms
- Machine Learning, COVID-19, SARS-CoV-2, Humans, Reproducibility of Results, Cohort Studies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
