The DOI matches its Crossref record; 85 of the 89 references it cites were found in a registry or an index.
The importance of being external. methodological insights for the external validation of machine learning models in medicine
Open accessAbstract
Background and Objective Medical machine learning (ML) models tend to perform better on data from the same cohort than on new data, often due to overfitting, or co-variate shifts. For these reasons, external validation (EV) is a necessary practice in the evaluation of medical ML. However, there is still a gap in the literature on how to interpret EV results and hence assess the robustness of ML models.Methods We fill this gap by proposing a meta-validation method, to assess the soundness of EV procedures. In doing so, we complement the usual way to assess EV by considering both dataset cardinality, and the similarity of the EV dataset with respect to the training set. We then investigate how the notions of cardinality and similarity can be used to inform on the reliability of a validation procedure, by integrating them into two summative data visualizations.Results We illustrate our methodology by applying it to the validation of a state-of-the-art COVID-19 diagnostic model on 8 EV sets, collected across 3 different continents. The model performance was moderately impacted by data similarity (Pearson ρ = 0.38, p< 0.001). In the EV, the validated model reported good AUC (average: 0.84), acceptable calibration (average: 0.17) and utility (average: 0.50). The validation datasets were adequate in terms of dataset cardinality and similarity, thus suggesting the soundness of the results. We also provide a qualitative guideline to evaluate the reliability of validation procedures, and we discuss the importance of proper external validation in light of the obtained results.Conclusions In this paper, we propose a novel, lean methodology to: 1) study how the similarity between training and validation sets impacts the generalizability of a ML model; 2) assess the soundness of EV evaluations along three complementary performance dimensions: discrimination, utility and calibration; 3) draw conclusions on the robustness of the model under validation. We applied this methodology to a state-of-the-art model for the diagnosis of COVID-19 from routine blood tests, and showed how to interpret the results in light of the presented framework.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,417
Statistical Power Analysis for the Behavioral Sciences
Cohen, J. (2013) · doi:10.4324/9780203771587 · Cited by 25,373
Receiver Operating Characteristic Curve in Diagnostic Test Assessment
Mandrekar, J. N. (2010) · Journal of Thoracic Oncology · doi:10.1097/jto.0b013e3181ec173d · Cited by 4,506
Wynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070
Internal validation of predictive models
Steyerberg, E. W., Harrell, F. E., Borsboom, G. J. J. M., Eijkemans, M. J. C., Vergouwe, Y., & Habbema, J. D. F. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00341-9 · Cited by 2,746
Automated detection of COVID-19 cases using deep neural networks with X-ray images
Öztürk, T., Talo, M., Yildirim, E. A., Baloğlu, U. B., Yıldırım, Ö., & Acharya, U. R. (2020) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.103792 · Cited by 2,652
A Kernel Method for the Two-Sample-Problem
Gretton, A., Borgwardt, K. M., Rasch, M., Schölkopf, B., & Smola, A. J. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0069 · Cited by 2,167
(2010) · doi:10.5555/1756006.1859921 · Cited by 1,924
Machine learning algorithm validation with a limited sample size
Vabalas, A., Gowen, E., Poliakoff, E., & Casson, A. J. (2019) · PLOS ONE · doi:10.1371/journal.pone.0224365 · Cited by 1,901
The Balanced Accuracy and Its Posterior Distribution
Brodersen, K. H., Ong, C. S., Stephan, K. E., & Buhmann, J. M. (2010) · 2010 20th International Conference on Pattern Recognition · doi:10.1109/icpr.2010.764 · Cited by 1,816 · cited 1×
“In particular, we suggest to used balanced performance metrics to better account for potential imbalances in the target distribution [9].”
MethodReal-time tracking of self-reported symptoms to predict potential COVID-19
Menni, C., Valdes, A. M., Freidin, M. B., Sudre, C. H., Nguyen, L. H., Drew, D. A., Ganesh, S., Varsavsky, T., Cardoso, M. J., El-Sayed Moustafa, J. S., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0916-2 · Cited by 1,517
CoroNet: A deep neural network for detection and diagnosis of COVID-19 from chest x-ray images
Khan, A. I., Shah, J. L., & Bhat, M. M. (2020) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2020.105581 · Cited by 1,351
Prediction models need appropriate internal, internal-external, and external validation
Steyerberg, E. W., & Harrell, F. E. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.04.005 · Cited by 1,304
2010 20th International Conference on Pattern Recognition
(2010) · doi:10.1109/icpr16097.2010 · Cited by 1,183
Roberts, M., Driggs, D., Thorpe, M., Gilbey, J., Yeung, M., Ursprung, S., Aviles-Rivero, A. I., Etmann, C., McCague, C., Beer, L., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00307-0 · Cited by 1,044 · cited 5×
“The same reasoning obviously applies, and if possible even exacerbated, in the case of models that rely on less stable tests, such as imaging [50].”
BackgroundMethodInfluentialA calibration hierarchy for risk models was defined: from utopia to empirical data
Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina, M. J., & Steyerberg, E. W. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.12.005 · Cited by 910
Roberts, M. T., Driggs, D., Thorpe, M., Gilbey, J., Yeung, M., Ursprung, S., Avilés-Rivero, A. I., Etmann, C., McCague, C., Beer, L., et al. (2020) · Apollo (University of Cambridge) · doi:10.17863/cam.64000 · Cited by 902
Machine Learning Testing: Survey, Landscapes and Horizons
Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2962027 · Cited by 897
Chicco, D., Tötsch, N., & Jurman, G. (2021) · BioData Mining · doi:10.1186/s13040-021-00244-z · Cited by 893
Ardakani, A. A., Kanafi, A. R., Acharya, U. R., Khadem, N., & Mohammadi, A. (2020) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.103795 · Cited by 864
Estimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap
Kim, J. (2009) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2009.04.009 · Cited by 792
Collins, G. S., de Groot, J. A., Dutton, S., Omar, O., Shanyinde, M., Tajar, A., Voysey, M., Wharton, R., Yu, L.-M., Moons, K. G., et al. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-40 · Cited by 729
Model selection and overfitting
Lever, J., Krzywinski, M. I., & Altman, N. (2016) · Nature Methods · doi:10.1038/nmeth.3968 · Cited by 727
External validation is necessary in prediction research:
Bleeker, S. E., Moll, H. A., Steyerberg, E. W., Donders, A. R. T., Derksen-Lubsen, G., Grobbee, D. E., & Moons, K. G. M. (2003) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(03)00207-5 · Cited by 710
Collins, G. S., Ogundimu, E. O., & Altman, D. G. (2016) · Statistics in Medicine · doi:10.1002/sim.6787 · Cited by 665
Vergouwe, Y., Steyerberg, E. W., Eijkemans, M. J. C., & Habbema, J. D. F. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.06.017 · Cited by 636
Explainable Deep Learning for Pulmonary Disease and Coronavirus COVID-19 Detection from X-rays
Brunese, L., Mercaldo, F., Reginelli, A., & Santone, A. (2020) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2020.105608 · Cited by 623
Steyerberg, E. W., Bleeker, S. E., Moll, H. A., Grobbee, D. E., & Moons, K. G. M. (2003) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(03)00047-7 · Cited by 611
Onukwugha, E., Yong, C., Hussain, A., Seal, B., & Mullins, C. D. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-1 · Cited by 589
COVID-19 identification in chest X-ray images on flat and hierarchical classification scenarios
Pereira, R. M., Bertolini, D. S., Teixeira, L., Junior, C. N. S., & Costa, Y. M. G. (2020) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2020.105532 · Cited by 527
2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI)
Supatto, W. (2020) · doi:10.1109/isbi45749.2020 · Cited by 472
The myth of generalisability in clinical research and machine learning in health care
Futoma, J., Simons, M., Panch, T., Doshi-Velez, F., & Celi, L. A. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30186-2 · Cited by 467
Li, X., Zhang, S., Zhang, Q., Wei, X., Pan, Y., Zhao, J., Xin, X., Qin, C., Wang, X., Li, J., et al. (2018) · The Lancet Oncology · doi:10.1016/s1470-2045(18)30762-9 · Cited by 439
Hu, F., Jiang, X. W., Guo, C., Li, Y., Chen, S., Zhang, W., Du, Y., Wang, P., Zheng, X., Fang, X., et al. (2020) · Nature Communications · doi:10.1038/s41467-020-15700-3 · Cited by 419
Hernandez-Boussard, T., Bozkurt, S., Ioannidis, J. P. A., & Shah, N. H. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa088 · Cited by 410
Minimum sample size for external validation of a clinical prediction model with a binary outcome
Riley, R. D., Debray, T. P. A., Collins, G. S., Archer, L., Ensor, J., van Smeden, M., & Snell, K. I. E. (2021) · Statistics in Medicine · doi:10.1002/sim.9025 · Cited by 400
Underspecification Presents Challenges for Credibility in Modern Machine Learning
D'Amour, A., Heller, K., Moldovan, D., Adlam, B., Alipanahi, B., Beutel, A., Chen, C., Deaton, J., Eisenstein, J., Hoffman, M. D., et al. (2020) · arXiv · doi:10.48550/arxiv.2011.03395 · Cited by 398
On the effectiveness of preprocessing methods when dealing with different levels of class imbalance
García, V., Sánchez, J. S., & Mollineda, R. A. (2011) · Knowledge-Based Systems · doi:10.1016/j.knosys.2011.06.013 · Cited by 368
Challenges to the Reproducibility of Machine Learning Models in Health Care
Beam, A. L., Manrai, A. K., & Ghassemi, M. (2020) · JAMA · doi:10.1001/jama.2019.20866 · Cited by 366
Kim, S., Lee, K., Hwang, H. Y., & Yoo, S. (2015) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-016-0249-8 · Cited by 321
The need to separate the wheat from the chaff in medical informatics
Cabitza, F., & Campagner, A. (2021) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2021.104510 · Cited by 288
Self-citationExplaining physicians’ acceptance of EHCR systems: An extension of TAM with trust and risk factors
Ortega Egea, J. M., & Román González, M. V. (2011) · Computers in Human Behavior · doi:10.1016/j.chb.2010.08.010 · Cited by 287
Routine blood tests as a potential diagnostic tool for COVID-19
Ferrari, D., Motta, A., Strollo, M., Banfi, G., & Locatelli, M. (2020) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2020-0398 · Cited by 273
Menni, C., Valdes, A. M., Freidin, M. B., et al. (2020) · medRxiv · doi:10.1101/2020.04.05.20048421 · Cited by 232
Wang, G., Liu, X., Shen, J., Wang, C., Li, Z., Ye, L., Wu, X., Chen, T., Wang, K., Zhang, X., et al. (2021) · Nature Biomedical Engineering · doi:10.1038/s41551-021-00704-1 · Cited by 202
Minimum sample size for external validation of a clinical prediction model with a continuous outcome
Archer, L., Snell, K. I. E., Ensor, J., Hudda, M. T., Collins, G. S., & Riley, R. D. (2021) · Statistics in Medicine · doi:10.1002/sim.8766 · Cited by 198
Cabitza, F., Campagner, A., Ferrari, D., Resta, C. D., Ceriotti, D., Sabetta, E., Colombini, A., Vecchi, E. D., Banfi, G., Locatelli, M., et al. (2020) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2020-1294 · Cited by 174
Self-citationAteya, M., Delaney, B. C., & Speedie, S. M. (2015) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-016-0239-x · Cited by 172
Rousson, V., & Zumbrunn, T. (2011) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-11-45 · Cited by 165
Soltan, A. A. S., Kouchaki, S., Zhu, T., Kiyasseh, D., Taylor, T., Hussain, Z. B., Peto, T., Brent, A. J., Eyre, D. W., & Clifton, D. A. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30274-0 · Cited by 150
Clinician checklist for assessing suitability of machine learning applications in healthcare
Scott, I. A., Carter, S. M., & Coiera, E. W. (2021) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2020-100251 · Cited by 143
Practical experiences on the necessity of external validation
König, I. R., Malley, J. D., Weimar, C., Diener, H., & Ziegler, A. (2007) · Statistics in Medicine · doi:10.1002/sim.3069 · Cited by 139
2019 53rd Asilomar Conference on Signals, Systems, and Computers
(2019) · doi:10.1109/ieeeconf44664.2019 · Cited by 130
Routine Laboratory Blood Tests Predict SARS-CoV-2 Infection Using Machine Learning
Yang, H. S., Hou, Y., Vasović, L. V., Steel, P. A. D., Chadburn, A., Racine‐Brzostek, S., Velu, P., Cushing, M. M., Loda, M. F., Kaushal, R., et al. (2020) · Clinical Chemistry · doi:10.1093/clinchem/hvaa200 · Cited by 125
Snell, K. I. E., Archer, L., Ensor, J., Bonnett, L. J., Debray, T. P. A., Phillips, B., Collins, P. G. S., & Riley, R. D. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.02.011 · Cited by 125
Song, X., Yu, A. S. L., Kellum, J. A., Waitman, L. R., Matheny, M. E., Simpson, S. Q., Hu, Y., & Liu, M. (2020) · Nature Communications · doi:10.1038/s41467-020-19551-w · Cited by 120
Sampling Uncertainty and Confidence Intervals for the Brier Score and Brier Skill Score
Bradley, A., Schwartz, S., & Hashino, T. (2008) · Weather and Forecasting · doi:10.1175/2007waf2007049.1 · Cited by 101
Machine Learning for Prediction in Electronic Health Data
Rose, S. (2018) · JAMA Network Open · doi:10.1001/jamanetworkopen.2018.1404 · Cited by 98
Race‐specific WBC and neutrophil count reference intervals
Lim, E., Cembrowski, G. S., Cembrowski, M. S., & Clarke, G. (2010) · International Journal of Laboratory Hematology · doi:10.1111/j.1751-553x.2010.01223.x · Cited by 93
Bierbooms, J. J. P. A., Bongers, I. M. B., & OERS, H. A. M. V. (2011) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-11-1 · Cited by 88
Theoretical Analysis of a Performance Measure for Imbalanced Data
García, V., Mollineda, R. A., & Sánchez, J. S. (2010) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2010.156 · Cited by 73
Cabitza, F., & Zeitoun, J.-D. (2019) · Annals of Translational Medicine · doi:10.21037/atm.2019.04.07 · Cited by 72
Self-citationAdvances in Domain Adaptation Theory
Redko, I., Morvant, E., Habrard, A., Sebban, M., & Bennani, Y. (2019) · HAL · Cited by 65
Diagnostics of prior-data agreement in applied Bayesian analysis
Bousquet, N. (2008) · Journal of Applied Statistics · doi:10.1080/02664760802192981 · Cited by 61
Estimation of required sample size for external validation of risk models for binary outcomes
Pavlou, M., Chen, Q., Omar, R., Seaman, S. R., Steyerberg, E. W., White, I. R., & Ambler, G. (2021) · Statistical Methods in Medical Research · doi:10.1177/09622802211007522 · Cited by 58
Coşkun, A., Braga, F., Carobene, A., Tejedor-Ganduxé, X., Aarsand, A. K., Fernández–Calle, P., Díaz–Garzón, J., Bartlett, W. A., Jonker, N., Aslan, B., et al. (2019) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2019-0658 · Cited by 56
Self-citationAs if sand were stone. New concepts and metrics to probe the ground on which to build trustable AI
Cabitza, F., Campagner, A., & Sconfienza, L. M. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-01224-9 · Cited by 46
Self-citationThe optimal diagnostic methods for COVID-19
Harahwa, T., Yau, T., Lim‐Cooke, M., Al-Haddi, S., Zeinah, M., & Harky, A. (2020) · Diagnosis · doi:10.1515/dx-2020-0058 · Cited by 45
Plante, T. B., Blau, A. M., Berg, A. N., Weinberg, A. S., Jun, I. C., Tapson, V. F., Kanigan, T. S., & Adib, A. A. (2020) · Journal of Medical Internet Research · doi:10.2196/24048 · Cited by 45
Identification of risk factors for patients with diabetes: diabetic polyneuropathy case study
Metsker, O., Magoev, K., Yakovlev, A., Yanishevskiy, S., Kopanitsa, G., Kovalchuk, S., & Krzhizhanovskaya, V. V. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-01215-w · Cited by 35
Schat, E., van de Schoot, R., Kouw, W. M., Veen, D., & Mendrik, A. M. (2020) · PLoS ONE · doi:10.1371/journal.pone.0237009 · Cited by 28
kNN-based high-dimensional Kullback-Leibler distance for tracking
Boltz, S., Debreuve, É., & Barlaud, M. (2007) · International Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS) · doi:10.1109/wiamis.2007.53 · Cited by 22
Harmonization: Its Time Has Come
Miller, W. G. (2017) · Clinical Chemistry · doi:10.1373/clinchem.2017.274860 · Cited by 20
Vidali, M., Carobene, A., Esposito, S. A., Napolitano, G., Caracciolo, A., Seghezzi, M., Previtali, G., Lippi, G., & Buoro, S. (2020) · International Journal of Laboratory Hematology · doi:10.1111/ijlh.13379 · Cited by 19
Self-citationUsing the Data Agreement Criterion to Rank Experts' Beliefs
Veen, D., Stoel, D., Schalken, N., Mulder, K., van de Schoot, R., Sciences, M. A. S. F. T. B. A. S., & Hoijtink, L. (2016) · Utrecht University Repository (Utrecht University) · Cited by 17
Abegaz, F., Lishout, F. V., John, J. M., Chiachoompu, K., Bhardwaj, A., Duroux, D., Gusareva, E. S., Wei, Z., Hakonarson, H. H., & Steen, K. V. (2021) · BioData Mining · doi:10.1186/s13040-021-00247-w · Cited by 17
Using the Data Agreement Criterion to Rank Experts’ Beliefs
Veen, D., Stoel, D., Schalken, N., Mulder, K., & van de Schoot, R. (2018) · Entropy · doi:10.3390/e20080592 · Cited by 15
Estimating Kullback-Leibler Divergence Using Kernel Machines
Ahuja, K. (2019) · 2018 52nd Asilomar Conference on Signals, Systems, and Computers · doi:10.1109/ieeeconf44664.2019.9049082 · Cited by 12
Detecting Concept Drift In Medical Triage
Huggard, H., Koh, Y. S., Dobbie, G., & Zhang, E. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401228 · Cited by 11
Learning An Mr Acquisition-Invariant Representation Using Siamese Neural Networks
Kouw, W. M., Loog, M., Bartels, L. W., & Mendrik, A. M. (2019) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2019.8759281 · Cited by 9
Franzini, C. (1995) · PubMed · Cited by 8
Carobene, A., Kiarie, A. G., Tsegaye, M., Seghezzi, M., Buoro, S., & Mburugu, R. W. (2020) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2020-0364 · Cited by 2
Self-citation
Flagged first, then by citations
4 of the 89 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Evaluation of clinical prediction models (part 1): from development to external validation
Collins, G. S., Dhiman, P., Ma, J., Schlussel, M. M., Archer, L., Van Calster, B., Harrell, F. E., Martin, G. P., Moons, K. G. M., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-074819 · Cited by 798
Lekadir, K., Frangi, A. F., Porras, A. R., Glocker, B., Cintas, C., Langlotz, C. P., Weicken, E., Asselbergs, F. W., Prior, F., Collins, G. S., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081554 · Cited by 957
“45 These might include reference or benchmarking datasets that are representative for the task in question (ie, approximating the expected real world variations).”
Feng, J., Phillips, R. V., Malenica, I., Bishara, A., Hubbard, A. E., Celi, L. A., & Pirracchio, R. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00611-y · Cited by 423
Salinas, M. P., Sepúlveda, J., Hidalgo, L., Peirano, D., Morel-Marambio, M., Uribe, P., Rotemberg, V. M., Briones, J., Mery, D., & Navarrete‐Dechent, C. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01103-x · Cited by 191
Targeted validation: validating clinical prediction models in their intended population and setting
Sperrin, M., Riley, R. D., Collins, P. G. S., & Martin, G. P. (2022) · Diagnostic and Prognostic Research · doi:10.1186/s41512-022-00136-8 · Cited by 121
Metadata
- Authors
- Cabitza, F., Campagner, A., Soares, F., García de Guadiana-Romualdo, L., Challa, F., Sulejmani, A., Seghezzi, M., & Carobene, A.
- Affiliations
- University of Milano-Bicocca (IT)
- Universidade Federal do Rio Grande do Sul (BR)
- Ethiopian Public Health Institute (ET)
- Ospedale Papa Giovanni XXIII (IT)
- IRCCS Ospedale San Raffaele (IT)
- Corresponding author
- Cabitza, Federico
- Venue
- Computer Methods and Programs in Biomedicine
- Publisher
- Elsevier BV
- Published
- 2021-09
- Volume
- 208
- Pages
- 106288
- ISSN
- 0169-2607 (ISSN-L)
- OpenAlex
- W3183421885
- Cited by
- 284
- References
- 89
- Citations by year
- 2026: 71 · 2025: 83 · 2024: 70 · 2023: 33 · 2022: 26 · 2021: 1
- FWCI
- 28.33
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- Locations
- Computer Methods and Programs in Biomedicine (publishedVersion)
- PubMed (publishedVersion)
- BOA (University of Milano-Bicocca) (submittedVersion, other-oa)
- Topics
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- external validation, machine learning in medicine, covariate shift, overfitting, generalizability, dataset similarity, model robustness, COVID-19 diagnosis, routine blood tests, discrimination (statistics), calibration, clinical utility
- MeSH terms
- Machine Learning, COVID-19, SARS-CoV-2, Humans, Reproducibility of Results, Cohort Studies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
