Toward Generalizability in the Deployment of Artificial Intelligence in Radiology: Role of Computation Stress Testing to Overcome Underspecification
Open accessThe DOI matches its Crossref record; 76 of the 77 references it cites were found in a registry or an index.
Abstract
The clinical deployment of artificial intelligence (AI) applications in medical imaging is perhaps the greatest challenge facing radiology in the next decade. One of the main obstacles to the incorporation of automated AI-based decision-making tools in medicine is the failure of models to generalize when deployed across institutions with heterogeneous populations and imaging protocols. The most well-understood pitfall in developing these AI models is overfitting, which has, in part, been overcome by optimizing training protocols. However, overfitting is not the only obstacle to the success and generalizability of AI. Underspecification is also a serious impediment that requires conceptual understanding and correction. It is well known that a single AI pipeline, with prescribed training and testing sets, can produce several models with various levels of generalizability. Underspecification defines the inability of the pipeline to identify whether these models have embedded the structure of the underlying system by using a test set independent of, but distributed identically, to the training set. An underspecified pipeline is unable to assess the degree to which the models will be generalizable. Stress testing is a known tool in AI that can limit underspecification and, importantly, assure broad generalizability of AI models. However, the application of stress tests is new in radiologic applications. This report describes the concept of underspecification from a radiologist perspective, discusses stress testing as a specific strategy to overcome underspecification, and explains how stress tests could be designed in radiology—by modifying medical images or stratifying testing datasets. In the upcoming years, stress tests should become in radiology the standard that crash tests have become in the automotive industry. Keywords: Computer Applications-General, Informatics, Computer-aided Diagnosis © RSNA, 2021
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“A critical barrier to the clinical implementation of AI decision tools is their frequent failure to generalize across heterogeneous populations and imaging protocols (Eche et al., 2021).”
From Multimodal neuroimaging and AI integration in cognitive disorders: advances, challenges, and future directions for precision medicine (2026) · cites this paper as Eche et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Lecture Notes in Computer Science 1205
(1999) · Industrial Robot: An International Journal · doi:10.1108/ir.1999.04926fae.001 · Cited by 38,662
Federated Machine Learning: Concept and Applications
Yang, Q., Liu, Y., Chen, T., & Tong, Y. (2019) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3298981 · Cited by 6,322
Radiomics: the bridge between medical imaging and personalized medicine
Lambin, P., Leijenaar, R. T. H., Deist, T. M., Peerlings, J., de Jong, E. E. C., van Timmeren, J., Sanduleanu, S., Larue, R. T. H. M., Even, A. J. G., Jochems, A., et al. (2017) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2017.141 · Cited by 5,688
Zwanenburg, A., Vallières, M., Abdalah, M. A., Aerts, H. J. W. L., Andrearczyk, V., Apte, A., Ashrafinia, S., Bakas, S., Beukinga, R. J., Boellaard, R., et al. (2020) · Radiology · doi:10.1148/radiol.2020191145 · Cited by 4,177
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2016) · arXiv · doi:10.48550/arxiv.1612.01474 · Cited by 3,666
Neural Networks and the Bias/Variance Dilemma
Geman, S., Bienenstock, E., & Doursat, R. (1992) · Neural Computation · doi:10.1162/neco.1992.4.1.1 · Cited by 3,578
Shortcut learning in deep neural networks
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00257-z · Cited by 2,224
Li, L., Qin, L., Xu, Z., Yin, Y., Wang, X., Kong, B., Bai, J., Lu, Y., Fang, Z., Song, Q., et al. (2020) · Radiology · doi:10.1148/radiol.2020200905 · Cited by 1,927
Neural Architecture Search: A Survey
Elsken, T., Metzen, J. H., & Hutter, F. (2018) · arXiv · doi:10.48550/arxiv.1808.05377 · Cited by 1,763
Preparing Medical Imaging Data for Machine Learning
Willemink, M. J., Koszek, W. A., Hardell, C., Wu, J., Fleischmann, D., Harvey, H., Folio, L. R., Summers, R. M., Rubin, D. L., & Lungren, M. P. (2020) · Radiology · doi:10.1148/radiol.2020192224 · Cited by 1,020
Research works printed but not linked: 2
- 73. Belkin M, Hsu D, Ma S, Mandal S. Reconciling modern machine learning practice and the bias-variance trade-off. ArXiv 1812.11118 [preprint] https:// arxiv.org/abs/1812.11118. Posted December 28, 2018. Accessed March 2021. · arXiv
- 77. Nakkiran P, Kaplun G, Bansal Y, Yang T, Barak B, Sutskever I. Deep double descent: where bigger models and more data hurt. ArXiv 1912.02292 [preprint] https://arxiv.org/abs/1912.02292. Posted December 4, 2019. Accessed March 2021. · arXiv
Cited by
Methods for clinical evaluation of artificial intelligence algorithms for medical diagnosis
Park, S. H., Han, K., Jang, H. Y., Park, J. E., Lee, J.-G., Kim, D. W., & Choi, J. (2023) · Radiology · doi:10.1148/radiol.220182 · Cited by 163
Rahimi, A. K., Pienaar, O., Ghadimi, M., Canfell, O. J., Pole, J. D., Shrapnel, S., van der Vegt, A. H., & Sullivan, C. M. (2024) · Journal of Medical Internet Research · doi:10.2196/49655 · Cited by 164
“%) originated from the United States [20,32,33,36,37,39,43-45, Radiology was the clinical setting in 46% (12/26) of the studies [33,34,36,37,39-41, 43-45”
Deep learning in cancer genomics and histopathology
Unger, M., & Kather, J. N. (2024) · Genome Medicine · doi:10.1186/s13073-024-01315-6 · Cited by 152
“Therefore, studies which only contain a dozen participants, will usually not have sufficiently diverse data to generalize well to external datasets, particularly in clinical routine [111].”
BackgroundGeneralizability of FDA-Approved AI-Enabled Medical Devices for Clinical Use
Windecker, D., Baj, G., Shiri, I., Kazaj, P. M., Kaesmacher, J., Gräni, C., & Siontis, G. C. M. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.8052 · Cited by 142
Mitigating Bias in Radiology Machine Learning: 3. Performance Metrics
Faghani, S., Khosravi, B., Zhang, K., Moassefi, M., Jagtap, J. M., Nugen, F., Vahdati, S., Kuanar, S. P., Rassoulinejad-Mousavi, S. M., Singh, Y., et al. (2022) · Radiology: Artificial Intelligence · doi:10.1148/ryai.220061 · Cited by 94
“This approach consists of obtaining a controlled shifted dataset from the original (internal) population and testing the trained model on this shifted dataset to identify its underspecifi-cations (21,24).”
BackgroundArtificial intelligence and radiomics: fundamentals, applications, and challenges in immunotherapy
Dercle, L., McGale, J., Sun, S. H., Marabelle, A., Yeh, R., Deutsch, É., Mokrane, F., Farwell, M. D., Ammari, S., Schöder, H. M., et al. (2022) · Journal for ImmunoTherapy of Cancer · doi:10.1136/jitc-2022-005292 · Cited by 109
“33 Another radiomics signature applied to metastatic melanoma also outperformed RECIST V.1.1 in estimating OS and was better able to distinguish between pseudo-progression and true progression at 3 months.”
InfluentialSelf-citationRevolutionizing Radiology With Artificial Intelligence
Bhandari, A. (2024) · Cureus · doi:10.7759/cureus.72646 · Cited by 126
“This challenge underscores the importance of building robust, scalable AI tools that can deliver consistent performance across various clinical scenarios [69].”
BackgroundBakrania, A. K., Joshi, N., Zhao, X., Zheng, G., & Bhat, M. (2023) · Pharmacological Research · doi:10.1016/j.phrs.2023.106706 · Cited by 108
Hendrickx, J., Gracea, R. S., Vanheers, M., Winderickx, N., Preda, F., Shujaat, S., & Jacobs, R. (2024) · European Journal of Orthodontics · doi:10.1093/ejo/cjae029 · Cited by 101
“While few studies provided detailed information about patient selection, the majority relied on IEEE dataset without explicitly outlining their sampling procedures [16, 38].”
BackgroundBarragán-Montero, A., Bibal, A., Dastarac, M. H., Draguet, C., Valdés, G., Nguyen, D., Willems, S., Vandewinckele, L., Holmström, M., Löfman, F., et al. (2022) · Physics in Medicine & Biology · doi:10.1088/1361-6560/ac678a · Cited by 84
“Although the concept of stress testing is rather new, the medical community is being encouraged to apply before clinically implementing ML models ( Eche et al 2021 ) .”
BackgroundHsu, W. W. Y., Hippe, D. S., Nakhaei, N., Wang, P., Zhu, B., Siu, N., Ahsen, M. E., Lotter, W. E., Sorensen, A. G., Naeim, A., et al. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.42343 · Cited by 66
Emerging and Evolving Concepts in Cancer Immunotherapy Imaging
Dercle, L., Sun, S. H., Seban, R., Mekki, A., Sun, R., Tselikas, L., Hans, S., Bernard‐Tessier, A., Bouvier, F. M., Aide, N., et al. (2022) · Radiology · doi:10.1148/radiol.210518 · Cited by 66
Self-citationChest X-ray in Emergency Radiology: What Artificial Intelligence Applications Are Available?
Irmici, G., Cè, M., Caloro, E., Khenkina, N., Pepa, G. D., Ascenti, V., Martinenghi, C. M. A., Papa, S., Oliva, G., & Cellina, M. I. (2023) · Diagnostics · doi:10.3390/diagnostics13020216 · Cited by 76
“However, few studies have been conducted to assess the applicability of these models and the tangible advantages they offer to clinical practice in the real-world setting [92].”
BackgroundArtificial Intelligence in Emergency Radiology: Where Are We Going?
Cellina, M., Cè, M., Irmici, G., Ascenti, V., Caloro, E., Bianchi, L., Pellegrino, G., D’Amico, N., Papa, S., & Carrafiello, G. (2022) · Diagnostics · doi:10.3390/diagnostics12123223 · Cited by 62
Esmaeili, M., Toosi, A., Roshanpoor, A., Changizi, V., Ghazisaeedi, M., Rahmim, A., & Sabokrou, M. (2023) · IEEE Access · doi:10.1109/access.2023.3244741 · Cited by 64
“Currently, the main obstacle to incorporating AI-based solutions into healthcare systems is their lack of generalization power [80].”
BackgroundArtificial Intelligence Applications for Osteoporosis Classification Using Computed Tomography
Ong, W., Liu, R. W., Makmur, A., Low, X. Z., Sng, W. J., Tan, J. H., Kumar, N., & Hallinan, J. T. P. D. (2023) · Bioengineering · doi:10.3390/bioengineering10121364 · Cited by 64
“Relying on smaller datasets from a single institution can yield unreliable results when applying the AI model to a different population, posing challenges related to generalizability and re-producibility [189,190].”
The diagnosis of ASD with MRI: a systematic review and meta-analysis
Schielen, S. J. C., Pilmeyer, J., Aldenkamp, A. P., & Zinger, S. (2024) · Translational Psychiatry · doi:10.1038/s41398-024-03024-5 · Cited by 59
Mirkin, S., & Albensi, B. C. (2023) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2023.1094233 · Cited by 46
“A possible strategy to overcome overfitting is to collect more training data (Eche et al., 2021).”
BackgroundRobustness evaluation of deep neural networks for endoscopic image analysis: Insights and strategies
Jaspers, T. J. M., Boers, T. G. W., Kusters, C. H. J., Jong, M. R., Jukema, J. B., de Groof, A. J., Bergman, J. J. G. H. M., de With, P. H. N., & van der Sommen, F. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103157 · Cited by 52
Valizadeh, P., Jannatdoust, P., Pahlevan‐Fallahy, M., Hassankhani, A., Amoukhteh, M., Bagherieh, S., Ghadimi, D. J., & Gholamrezanezhad, A. (2024) · Neuroradiology · doi:10.1007/s00234-024-03485-x · Cited by 44
“Overfitting occurs when the AI model is no longer generalizable to the whole population and only works on the training data [ 37– 39].”
Boone, L., Biparva, M., Forooshani, P. M., Ramirez, J., Masellis, M., Bartha, R., Symons, S. P., Strother, S. C., Black, S. E., Heyn, C., et al. (2023) · NeuroImage · doi:10.1016/j.neuroimage.2023.120289 · Cited by 50
Ethics of artificial intelligence in prenatal and pediatric genomic medicine
Coghlan, S., Gyngell, C., & Vears, D. F. (2023) · Journal of Community Genetics · doi:10.1007/s12687-023-00678-4 · Cited by 46
“For example, an overfitted ML model may internalize “noisy” parts of the dataset (Eche et al. 2021) and thus fail to generalize well to other datasets from current children or fetuses.”
MethodA Deep Learning-Based Radiomic Classifier for Usual Interstitial Pneumonia
Chung, J. H., Chelala, L., Pugashetti, J. V., Wang, J. M., Adegunsoye, A., Matyga, A. W., Keith, L. A., Ludwig, K. D., Zafari, S., Ghodrati, S., et al. (2023) · CHEST Journal · doi:10.1016/j.chest.2023.10.012 · Cited by 41
Nair, A., Ong, W., Lee, A., Leow, N. W., Makmur, A., Ting, Y. H., Lee, Y. J., Ong, S. J., Tan, J. J. H., Kumar, N., et al. (2025) · Diagnostics · doi:10.3390/diagnostics15091146 · Cited by 45
“Although many AI models achieve high accuracy within their home institutions, differences in imaging protocols, sequences, and MRI hardware often remain underexplored, leading to model underspecification and limited generalizability [128,129].”
Pitfalls in Interpretive Applications of Artificial Intelligence in Radiology
Behzad, S., Tabatabaei, S. M. H., Lu, M. Y., Eibschutz, L. S., & Gholamrezanezhad, A. (2024) · American Journal of Roentgenology · doi:10.2214/ajr.24.31493 · Cited by 37
Pooler, B. D., Garrett, J. W., Southard, A. M., Summers, R. M., & Pickhardt, P. J. (2023) · American Journal of Roentgenology · doi:10.2214/ajr.22.28745 · Cited by 37
Alsadoun, L., Ali, H., Mushtaq, M. M., Mushtaq, M., Burhanuddin, M., Anwar, R., Liaqat, M., Bokhari, S. F. H., Hasan, A. H., & Ahmed, F. (2024) · Cureus · doi:10.7759/cureus.67844 · Cited by 38
“Efforts to establish standardized imaging protocols and quality control measures are essential to ensure the reliability and generalizability of AI models across different clinical environments [44].”
BackgroundWang, H., Lin, W.-Y., Zhou, C., Yang, Z., Kalpana, S., & Lebowitz, M. (2024) · Cancers · doi:10.3390/cancers16050862 · Cited by 42
“This characteristic is referred to as generalizability and is typically regarded as one of key characteristics for AI models [90].”
BackgroundOettl, F. C., Pareek, A., Winkler, P. W., Zsidai, B., Pruneski, J. A., Senorski, E. H., Kopf, S., Ley, C., Herbst, E., Oeding, J. F., et al. (2024) · Journal of Experimental Orthopaedics · doi:10.1002/jeo2.12039 · Cited by 32
Overcoming barriers in the use of artificial intelligence in point of care ultrasound
Vega, R., Dehghan, M., Nagdev, A. D., Buchanan, B. M., Kapur, J., Jaremko, J. L., & Zonoobi, D. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01633-y · Cited by 30
“Training a model with data acquired from only big urban areas might underperform in low-resource settings 112,119 ; however, note that a strategy that brings patients from remote locations to a well-equipped hospital in an effort to increase diversity for purposes of data collection may be…”
Duan, J., Vargas, C., Yu, N. Y., Laughlin, B. S., Toesca, D. A. S., Keole, S. R., Rwigema, J. M., Wong, W. W., Schild, S. E., Feng, X., et al. (2023) · Medical Physics · doi:10.1002/mp.16537 · Cited by 30
Multi-Center Fetal Brain Tissue Annotation (FeTA) Challenge 2022 Results
Payette, K., Steger, C., Licandro, R., de Dumast, P., Li, H., Barkovich, M. J., Li, L., Dannecker, M., Chen, C., Ouyang, C., et al. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3485554 · Cited by 38
“Domain generalizability of automatic segmentation algorithms is an urgent need and is attracting increasing attention in the medical imaging field [6], [7], [8], [9], [10].”
Esteghamati, M. Z., Bean, B., Burton, H. V., & Naser, M. Z. (2025) · Journal of Structural Engineering · doi:10.1061/jsendh.steng-13301 · Cited by 37
“Several authors define underspecification as the inability of the ML model to capture the causal relationship between response and features, and subsequently, a lack of invariability to confounding factors (Eche et al. 2021).”
Rashidisabet, H., Sethi, A. R., Jindarak, P., Edmonds, J., Chan, R. V. P., Leiderman, Y. I., Vajaranant, T. S., & Yi, D. (2023) · Translational Vision Science & Technology · doi:10.1167/tvst.12.11.8 · Cited by 29
Rethinking model prototyping through the MedMNIST+ dataset collection
Doerrich, S., Salvo, F. D., Brockmann, J., & Ledig, C. (2025) · Scientific Reports · doi:10.1038/s41598-025-92156-9 · Cited by 36
“However, the generalizability of these methods across diverse domains remains a significant challenge, constraining their real-world applicability 16 .”
Contemporary applications of artificial intelligence and machine learning in echocardiography
Raissi-Dehkordi, N., Raissi-Dehkordi, N., & Xu, B. (2025) · npj Cardiovascular Health · doi:10.1038/s44325-025-00064-8 · Cited by 28
Eriksson, M., Román, M., Gräwingholt, A., Castells, X., Nitrosi, A., Pattacini, P., Heywang-Köbrunner, S. H., & Rossi, P. G. (2023) · The Lancet Regional Health - Europe · doi:10.1016/j.lanepe.2023.100798 · Cited by 28
Brugnara, G., Preetha, C. J., Deike‐Hofmann, K., Haase, R., Pinetz, T., Foltyn‐Dumitru, M., Mahmutoglu, M. A., Wildemann, B., Diem, R., Wick, W., et al. (2024) · Radiology Artificial Intelligence · doi:10.1148/ryai.230514 · Cited by 31
Artificial intelligence and omics in malignant gliomas
Tambi, R., Zehra, B., Vijayakumar, A., Satsangi, D., Uddin, M. J., & Berdiev, B. K. (2024) · Physiological Genomics · doi:10.1152/physiolgenomics.00011.2024 · Cited by 25
Fernández‐Miranda, P. M., Fraguela, E. M., de Linera-Alperi, M. Á., Cobo, M., del Barrio, A. P., Rodriguez, D., Vega, J. A., & Iglesias, L. L. (2024) · Scientific Reports · doi:10.1038/s41598-024-64941-5 · Cited by 23
Yang, Y., Jin, Q., Zhu, Q., Wang, Z., Álvarez, F. E., Wan, N., Hou, B., & Lu, Z. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-094851 · Cited by 25
Santomartino, S. M., Putman, K., Beheshtian, E., Parekh, V. S., & Yi, P. H. (2024) · Radiology Artificial Intelligence · doi:10.1148/ryai.230240 · Cited by 23
A Review on Medical Image Segmentation: Datasets, Technical Models, Challenges and Solutions
Gan, H., Ramlee, M. H., Wang, Z., & Shimizu, A. (2025) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1574 · Cited by 29
“A worse case in underspecification is that even if the pipeline could be in principle return a model that meets all of these requirements, there is no guarantee that in practice the model will satisfy any requirement beyond accurate prediction on held-out data (Eche et al. 2021).”
Artificial intelligence in neuroradiology: a scoping review of some ethical challenges
Khosravi, P., & Schweitzer, M. (2023) · Frontiers in Radiology · doi:10.3389/fradi.2023.1149461 · Cited by 23
Investigating fairness in machine learning-based audio sentiment analysis
Luitel, S., Liu, Y., & Anwar, M. M. (2024) · AI and Ethics · doi:10.1007/s43681-024-00453-2 · Cited by 25
“Increasingly, researchers are starting to realize that a high accuracy in standard benchmark datasets does not guarantee successful deployment in real-world scenarios [2–4].”
Applications of Artificial Intelligence and Machine Learning in Spine MRI
Lee, A., Ong, W., Makmur, A., Ting, Y. H., Tan, W. C., Lim, S. W. D., Low, X. Z., Tan, J. J. H., Kumar, N., & Hallinan, J. T. P. D. (2024) · Bioengineering · doi:10.3390/bioengineering11090894 · Cited by 22
“Thirdly, “stress testing” involves evaluating an AI model under varied or extreme conditions to identify potential weak points [81].”
A critical assessment of artificial intelligence in magnetic resonance imaging of cancer
Wu, C., Andaloussi, M. A., Hormuth, D. A., Lima, E. A. B. F., Lorenzo, G., Stowers, C. E., Ravula, S., Levac, B., Dimakis, A. G., Tamir, J. I., et al. (2025) · npj Imaging · doi:10.1038/s44303-025-00076-0 · Cited by 22
“ThelimitedprogressinbeingabletotransportanAImethodfrom one institute to another, and from one disease setting to another (i.e., generalizability), isheavily in fl uencedbyboththedata and thedevicesemployed to capture the data 133 .”
Kryszan, K., Wylęgała, A., Kijonka, M., Potrawa, P., Walasz, M., Wylęgała, E., & Orzechowska-Wylęgała, B. E. (2024) · Diagnostics · doi:10.3390/diagnostics14070694 · Cited by 21
“Another strategy to address overfitting is using a hold out or a cross-validation dataset [25,26].”
MethodOng, W., Lee, A., Tan, W. C., Fong, K. T. D., Lai, D. D., Tan, Y. L., Low, X. Z., Ge, S., Makmur, A., Ong, S. J., et al. (2024) · Cancers · doi:10.3390/cancers16172988 · Cited by 21
“However, we anticipate that more AI tools will achieve regulatory clearance in the future as the technology matures and more evidence of its clinical utility emerges.”
BackgroundBartusik‐Aebisher, D., Raj, D. R. J., & Aebisher, D. (2026) · Applied Sciences · doi:10.3390/app16020728 · Cited by 24
Nanotechnology and artificial intelligence in cancer treatment
Mukheja, Y., Pal, K., Ahuja, A., Sarkar, A., Kumar, B., Kuhad, A., Chopra, K., & Jain, M. K. (2025) · Next research. · doi:10.1016/j.nexres.2025.100179 · Cited by 19
Ethics of Artificial Intelligence in Breast Imaging
Morgan, M. B., & Mates, J. (2023) · Journal of Breast Imaging · doi:10.1093/jbi/wbac076 · Cited by 21
Methods of preoperative prediction of pituitary adenoma consistency: a systematic review
Černý, M., Sedlák, V., Lesáková, V., Francůz, P., & Netuka, D. (2022) · Neurosurgical Review · doi:10.1007/s10143-022-01909-x · Cited by 19
Karnuta, J. M., Shaikh, H. J. F., Murphy, M. P., Brown, N. M., Pearle, A. D., Nawabi, D. H., Chen, A. F., & Ramkumar, P. N. (2023) · The Journal of Arthroplasty · doi:10.1016/j.arth.2023.03.039 · Cited by 17
Estimating Model Performance Under Domain Shifts with Class-Specific Confidence Scores
Li, Z., Kamnitsas, K., Islam, M., Chen, C., & Glocker, B. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16449-1_66 · Cited by 24
The AI Generalization Gap: One Size Does Not Fit All
Huisman, M., & Hannink, G. (2023) · Radiology Artificial Intelligence · doi:10.1148/ryai.230246 · Cited by 18
Naderalvojoud, B., Curtin, C. M., Yanover, C., El‐Hay, T., Choi, B., Park, R. W., Tabuenca, J. G., Reeve, M. P., Falconer, T., Humphreys, K. N., et al. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae028 · Cited by 16
Differences between human and artificial/augmented intelligence in medicine
Monteith, S., Glenn, T., Geddes, J. R., Achtyes, E. D., Whybrow, P. C., & Bauer, M. (2024) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2024.100084 · Cited by 21
Shahbazi, S., Esmaeili, S., Kavousinejad, S., Younessian, F., & Behnaz, M. (2025) · International Orthodontics · doi:10.1016/j.ortho.2025.101010 · Cited by 15
Precision Medicine in Orthobiologics: A Paradigm Shift in Regenerative Therapies
Navani, A. H., Jeyaraman, M., Jeyaraman, N., Ramasubramanian, S., Nallakumarasamy, A., Azzini, G. O. M., & Lana, J. F. (2025) · Bioengineering · doi:10.3390/bioengineering12090908 · Cited by 14
“Models trained on datasets from limited patient populations may not perform well when applied to broader, more diverse groups [100,101].”
Mehrtabar, S., Marey, A., Desai, A., Saad, A. M., Desai, V. S., Goñi, J., Pal, B., & Umair, M. (2025) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-025-01656-7 · Cited by 19
Langius-Wiffen, E., Slotman, D. J., Groeneveld, J., van Osch, J. A. C., Nijholt, I. M., de Boer, E., Nijboer-Oosterveld, J., Veldhuis, W. B., de Jong, P. A., & Boomsma, M. F. (2024) · European Journal of Radiology · doi:10.1016/j.ejrad.2024.111361 · Cited by 14
Liu, Z., Xu, J., Yin, C., Han, G., Che, Y., Fan, G., Li, X. H., Xie, L., Bao, L., Peng, Z., et al. (2024) · Research · doi:10.34133/research.0426 · Cited by 14
“…research has experienced a proliferation of literature on novel diagnostic, classification, and prediction tools developed using ML techniques in recent years, there remains a paucity of studies evaluating the practicality and real-world applicability of these models in clinical practice [37].”
CheXstray: A Real-Time Multi-Modal Monitoring Workflow for Medical Imaging AI
Merkow, J., Soin, A., Long, J., Cohen, J. P., Saligrama, S., Bridge, C., Yang, X., Kaiser, S., Borg, S., Tarapov, I., et al. (2023) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-43898-1_32 · Cited by 13
Tang, F., Zha, X.-K., Ye, W., Wang, Y., Wu, Y., Wang, L., Lyu, L.-P., & Lyu, X.-M. (2025) · BMC Pulmonary Medicine · doi:10.1186/s12890-025-03760-4 · Cited by 13
“Standardization of AI models, harmonization of imaging protocols, and validation in prospective studies across diverse populations are essential for ensuring generalizability [44].”
Artificial intelligence in immunotherapy PET/SPECT imaging
McGale, J., Chen, D. L., Trebeschi, S., Farwell, M. D., Wu, A. M., Cutler, C. S., Schwartz, L. H., & Dercle, L. (2024) · European Radiology · doi:10.1007/s00330-024-10637-3 · Cited by 13
Self-citationHoebers, F. J. P., Wee, L., Likitlersuang, J., Mak, R. H., Bitterman, D. S., Huang, Y., Dekker, A. L. A. J., Aerts, H. J. W. L., & Kann, B. H. (2023) · BJR|Open · doi:10.1093/bjro/tzae039 · Cited by 14
Revolutionizing total hip arthroplasty: The role of artificial intelligence and machine learning
Longo, U. G., Salvatore, S. D., Piccolomini, A., Ullman, N. S., Salvatore, G., D’Hooghe, M., Saccomanno, M. F., Samuelsson, K., Papalia, R., & Pareek, A. (2025) · Journal of Experimental Orthopaedics · doi:10.1002/jeo2.70195 · Cited by 12
“Access to larger data sets across distinct populations and regions is paramount to preventing over fi tting or drawing conclusions that are only applicable to identically distributed data sets [9, 41].”
Extreme cardiac MRI analysis under respiratory motion: Results of the CMRxMotion challenge
Wang, K., Qin, C., Shi, Z., Wang, H., Zhang, X., Chen, C., Ouyang, C., Dai, C., Mo, Y., Dai, C., et al. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103883 · Cited by 14
“…reliable segmentation model, it is useful to investigate when and how the model fails (Wang et al., 2020) by exposing a trained segmentation model to extreme cases in a stress test (Eche et al., 2021), such as altered image quality caused by motion artifacts that may occur during clinical practice.”
Ma, L., Zhao, X., Jiang, R., Chengke, W., Yang, Z., Tan, J., & Lei, X. (2025) · Buildings · doi:10.3390/buildings15234296 · Cited by 17
Lee, S. B., Hong, Y., Cho, Y. J., Jeong, D., Lee, J., Yoon, S. H., Lee, S., Choi, Y. H., & Cheon, J. (2023) · Korean Journal of Radiology · doi:10.3348/kjr.2022.0588 · Cited by 12
“[11] attempted to secure generalizability using computational stress testing.”
BackgroundMethodInfluentialAn Automatic DWI/FLAIR Mismatch Assessment of Stroke Patients
Johansen, J., Offersen, C. M., Carlsen, J. F., Ingala, S., Hansen, A. E., Nielsen, M. B., Darkner, S., & Pai, A. (2023) · Diagnostics · doi:10.3390/diagnostics14010069 · Cited by 11
“…choices, suggesting that the DWI/FLAIR mismatch ratio will generalize well across different scanners and acquisition protocols, as small changes in intensity did not effect the outcome significantly, something which is a large concern when deploying automatic methods in hospital settings [29,30].”
BackgroundMoon, J., Jadhav, P., & Choi, S. T. (2025) · Journal of Rheumatic Diseases · doi:10.4078/jrd.2024.0128 · Cited by 15
“One of the primary barriers to applying DL-based decision-making in medicine is the difficulty models face in generalizing across different institutions, which may have varying patient populations, imaging machines, or protocols [146].”
Pacholec, C., Flatland, B., Xie, H., & Zimmerman, K. (2025) · Veterinary Clinical Pathology · doi:10.1111/vcp.13407 · Cited by 16
Wolak, W., Plichta, A., & Orlicki, H. (2025) · Scientific Reports · doi:10.1038/s41598-025-31079-x · Cited by 14
“…in Toward Generalisability in the Deployment of Artificial Intelligence in Radiology, “underspecification is a major obstacle to the generalisability of machine learning models and defines the inability of the pipeline to ensure that the model has encoded the inner logic of the system” 33 .”
Oloruntoba, A. I., Ingvar, Å., Sashindranath, M., Anthony, O., Abbott, L. M., Guitera, P., Caccetta, T. P., Janda, M., Soyer, H. P., & Mar, V. (2024) · Australasian Journal of Dermatology · doi:10.1111/ajd.14269 · Cited by 11
Dang, M., Liu, B., Chen, Y., & Zhang, Z. (2026) · Psychoradiology · doi:10.1093/psyrad/kkag007 · Cited by 10
Chen, D., Cao, C., Kloosterman, R., Parsa, R., & Raman, S. (2024) · Journal of Medical Internet Research · doi:10.2196/58578 · Cited by 10
“There remains a need for sound methodological design in AI model training to improve generalizability in validation cohorts [12].”
Zhu, Y., Shu, X., Liu, X., Li, Y., Yi, B., & Ma, D. (2025) · Anesthesiology and Perioperative Science · doi:10.1007/s44254-025-00156-9 · Cited by 9
“Common problems include over-fitting, insufficient data diversity, and difficulties in domain transfer, which can limit model application and lead to resource wastage [69, 70].”
Chatzara, A., Maliagkani, E., Mitsopoulou, D., Katsimpris, A., Apostolopoulos, I. D., Papageorgiou, E. I., & Georgalas, I. (2025) · Bioengineering · doi:10.3390/bioengineering12050475 · Cited by 12
“Addressing the current limitations will be essential for unlocking the full potential of AI in ophthalmology and facilitating its meaningful adoption in clinical care.”
Higgins, H., Nakhla, A., Lotfalla, A., Khalil, D., Doshi, P., Thakkar, V., Shirini, D., Bebawy, M., Ammari, S., Lopci, E., et al. (2023) · Diagnostics · doi:10.3390/diagnostics13223483 · Cited by 9
Self-citationProspects and challenges of deep learning in gynecologic malignancies
Zhang, Y., & Zeng, L. (2025) · Frontiers in Oncology · doi:10.3389/fonc.2025.1592078 · Cited by 10
Johnson, L. A., Harmon, S. A., Yılmaz, E. C., Lin, Y., Belue, M. J., Merriman, K. M., Lay, N. S., Sanford, T., Sarma, K. V., Arnold, C. W., et al. (2024) · Abdominal Radiology · doi:10.1007/s00261-024-04242-7 · Cited by 8
Robustness Stress Testing in Medical Image Classification
Islam, M., Li, Z., & Glocker, B. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44336-7_17 · Cited by 14
Machine Learning Model for Chest Radiographs: Using Local Data to Enhance Performance
Mohn, S. F., Law, M., Koleva, M., Lee, B., Berg, A., Murray, N., Nicolaou, S., & Parker, W. A. (2022) · Canadian Association of Radiologists Journal · doi:10.1177/08465371221145023 · Cited by 8
“5,7,8 We demonstrated that fi ne-tuning using a small local dataset may be a solution for adapting models across different institutions.”
BackgroundArtificial Intelligence for Drug Discovery: An Update and Future Prospects
Howell, H. J., McGale, J., Choucair, A., Shirini, D., Aide, N., Postow, M. A., Wang, L., Tordjman, M., Lopci, E., Lecler, A., et al. (2025) · Seminars in Nuclear Medicine · doi:10.1053/j.semnuclmed.2025.01.004 · Cited by 9
Self-citationAlhhazmi, A., Alferidi, A., Almutawif, Y. A., Makhdoom, H. M., Albasri, H. M., & Sami, B. S. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1327355 · Cited by 8
Hamon, G., Legrand, L., Hmeydia, G., Turc, G., Hassen, W. B., Charron, S., Debacker, C., Naggara, O. N., Thirion, B., Chen, B., et al. (2024) · European Stroke Journal · doi:10.1177/23969873241263418 · Cited by 7
“Data partition based on sequence parameters would have been useful to increase data homogeneity in each subset but using smaller groups would have increased the risk of over-fitting, 29 especially using the fine-tuning strategy.”
Reproducibility of AI in Cephalometric Landmark Detection: A Preliminary Study
Fracchia, D. E., Bignotti, D., Lai, S., Cubeddu, S., Curreli, F., Lombardo, M., Verdecchia, A., & Spinas, E. (2025) · Diagnostics · doi:10.3390/diagnostics15192521 · Cited by 8
“Thus, validating AI outputs on independent datasets remains critical [27], especially given the potential for overfitting when evaluation is conducted on data similar to the training set [26].”
Smart Drilling: Integrating AI for Process Optimisation and Quality Enhancement in Manufacturing
Panico, M., & Boccarusso, L. (2025) · Journal of Manufacturing and Materials Processing · doi:10.3390/jmmp9120386 · Cited by 6
Defining operational safety in clinical artificial intelligence systems
Kim, Y.-T., Kim, H., Bahl, M., Lev, M. H., González, R. G., Gee, M. S., & Do, S. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02450-7 · Cited by 8
“This policy-driven capability is crucial because it resists the one-size-fi ts-all fallacy that often hinders AI adoption, where a single model con fi guration is expected to perform optimally across diverse clinical settings 37,38 .”
Generalizable self-supervised learning for brain CTA in acute stroke
Dong, Y., Pachade, S., Roberts, K. E., Jiang, X., Sheth, S. A., & Giancardo, L. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109337 · Cited by 6
Jiwani, R. A., Pal, K., Paolucci, I., Odisio, B. C., Brock, K. K., Tannir, N. M., Shapiro, D. D., Msaouel, P., & Sheth, R. A. (2025) · The Oncologist · doi:10.1093/oncolo/oyae337 · Cited by 6
The Ongoing Evolution of AI in Craniofacial Surgery: From Theory to Reality and Beyond
Pourriyahi, H., & Alkureishi, L. W. T. (2025) · Journal of Craniofacial Surgery · doi:10.1097/scs.0000000000012123 · Cited by 6
Adams, J. (2024) · AI and Ethics · doi:10.1007/s43681-024-00597-1 · Cited by 7
Vasilev, Y. A., Vladzymyrskyy, A. V., Arzamasov, K. M., Omelyanskaya, O. V., Михайлович, Ш. И., Kozhikhina, D. D., Goncharova, I. V., Reshetnikov, R. V., Четвериков, С. Ф., Blokhin, I. A., et al. (2023) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2023.105190 · Cited by 6
Meneses, J. P., Qadir, A., Surendran, N., Arrieta, C., Cristián Tejos, N., Andía, M. E., Chen, Z., & Uribe, S. A. (2024) · European Radiology · doi:10.1007/s00330-024-11164-x · Cited by 6
Thrall, J. H. (2024) · Radiology Artificial Intelligence · doi:10.1148/ryai.240411 · Cited by 5
Qayyum, S. N., Iftikhar, M., Rehan, M., Khan, G. A., Khan, M., Naeem, R., Ansari, R. S., Ullah, I., & Noori, S. (2025) · Annals of Medicine and Surgery · doi:10.1097/ms9.0000000000002778 · Cited by 9
“Therefore, there is the need for preparation of diversified databases available for everyone to ensure generalizability in the AI algorithm being developed in every corner of the world [ 50 ] .”
von Reppert, M., Chadha, S., Willms, K., Avesta, A., Maleki, N., Zeevi, T., Lost, J., Tillmanns, N., Jekel, L., Merkaj, S., et al. (2025) · American Journal of Neuroradiology · doi:10.3174/ajnr.a8805 · Cited by 6
Metadata
- Authors
- Eche, T., Schwartz, L. H., Mokrane, F.-Z., & Dercle, L.
- Affiliations
- NewYork–Presbyterian Hospital (US)
- Venue
- Radiology: Artificial Intelligence
- Published
- 2021-11-01
- Volume
- 3
- Issue
- 6
- Pages
- e210097-e210097
- ISSN
- 2638-6100 (ISSN-L)
- PMC
- PMC8637230
- OpenAlex
- W3210784269
- Cited by
- 163
- References
- 71
- Citations by year
- 2026: 33 · 2025: 40 · 2024: 47 · 2023: 30 · 2022: 13
- FWCI
- 6.56
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Radiology Artificial Intelligence (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Alain Rahmouni
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiology practices and education
- Keywords
- underspecification, stress testing, generalizability, AI deployment, diagnostic imaging, overfitting, model evaluation, test set design, artificial intelligence in radiology, computer-aided diagnosis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.