Toward Generalizability in the Deployment of Artificial Intelligence in Radiology: Role of Computation Stress Testing to Overcome Underspecification
Open accessThe DOI matches its Crossref record; 75 of the 76 references it cites were found in a registry or an index.
Abstract
The clinical deployment of artificial intelligence (AI) applications in medical imaging is perhaps the greatest challenge facing radiology in the next decade. One of the main obstacles to the incorporation of automated AI-based decision-making tools in medicine is the failure of models to generalize when deployed across institutions with heterogeneous populations and imaging protocols. The most well-understood pitfall in developing these AI models is overfitting, which has, in part, been overcome by optimizing training protocols. However, overfitting is not the only obstacle to the success and generalizability of AI. Underspecification is also a serious impediment that requires conceptual understanding and correction. It is well known that a single AI pipeline, with prescribed training and testing sets, can produce several models with various levels of generalizability. Underspecification defines the inability of the pipeline to identify whether these models have embedded the structure of the underlying system by using a test set independent of, but distributed identically, to the training set. An underspecified pipeline is unable to assess the degree to which the models will be generalizable. Stress testing is a known tool in AI that can limit underspecification and, importantly, assure broad generalizability of AI models. However, the application of stress tests is new in radiologic applications. This report describes the concept of underspecification from a radiologist perspective, discusses stress testing as a specific strategy to overcome underspecification, and explains how stress tests could be designed in radiology—by modifying medical images or stratifying testing datasets. In the upcoming years, stress tests should become in radiology the standard that crash tests have become in the automotive industry. Keywords: Computer Applications-General, Informatics, Computer-aided Diagnosis © RSNA, 2021
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“A critical barrier to the clinical implementation of AI decision tools is their frequent failure to generalize across heterogeneous populations and imaging protocols (Eche et al., 2021).”
From Multimodal neuroimaging and AI integration in cognitive disorders: advances, challenges, and future directions for precision medicine (2026) · cites this paper as Eche et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Lecture Notes in Computer Science 1205
(1999) · Industrial Robot: An International Journal · doi:10.1108/ir.1999.04926fae.001 · Cited by 38,662
Federated Machine Learning: Concept and Applications
Yang, Q., Liu, Y., Chen, T., & Tong, Y. (2019) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3298981 · Cited by 6,322
Radiomics: the bridge between medical imaging and personalized medicine
Lambin, P., Leijenaar, R. T. H., Deist, T. M., Peerlings, J., de Jong, E. E. C., van Timmeren, J., Sanduleanu, S., Larue, R. T. H. M., Even, A. J. G., Jochems, A., et al. (2017) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2017.141 · Cited by 5,688
Zwanenburg, A., Vallières, M., Abdalah, M. A., Aerts, H. J. W. L., Andrearczyk, V., Apte, A., Ashrafinia, S., Bakas, S., Beukinga, R. J., Boellaard, R., et al. (2020) · Radiology · doi:10.1148/radiol.2020191145 · Cited by 4,177
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2016) · arXiv · doi:10.48550/arxiv.1612.01474 · Cited by 3,666
Neural Networks and the Bias/Variance Dilemma
Geman, S., Bienenstock, E., & Doursat, R. (1992) · Neural Computation · doi:10.1162/neco.1992.4.1.1 · Cited by 3,578
Shortcut learning in deep neural networks
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00257-z · Cited by 2,224
Li, L., Qin, L., Xu, Z., Yin, Y., Wang, X., Kong, B., Bai, J., Lu, Y., Fang, Z., Song, Q., et al. (2020) · Radiology · doi:10.1148/radiol.2020200905 · Cited by 1,927
Neural Architecture Search: A Survey
Elsken, T., Metzen, J. H., & Hutter, F. (2018) · arXiv · doi:10.48550/arxiv.1808.05377 · Cited by 1,763
Preparing Medical Imaging Data for Machine Learning
Willemink, M. J., Koszek, W. A., Hardell, C., Wu, J., Fleischmann, D., Harvey, H., Folio, L. R., Summers, R. M., Rubin, D. L., & Lungren, M. P. (2020) · Radiology · doi:10.1148/radiol.2020192224 · Cited by 1,020
Park, S. H., & Han, K. (2018) · Radiology · doi:10.1148/radiol.2017171920 · Cited by 894
Convolutional Neural Networks for Radiologic Images: A Radiologist’s Guide
Soffer, S., Ben-Cohen, A., Shimon, O., Amitai, M. M., Greenspan, H., & Klang, E. (2019) · Radiology · doi:10.1148/radiol.2018180547 · Cited by 577
The myth of generalisability in clinical research and machine learning in health care
Futoma, J., Simons, M., Panch, T., Doshi-Velez, F., & Celi, L. A. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30186-2 · Cited by 467
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · European Urology · doi:10.1016/j.eururo.2014.11.025 · Cited by 439
Underspecification Presents Challenges for Credibility in Modern Machine Learning
D'Amour, A., Heller, K., Moldovan, D., Adlam, B., Alipanahi, B., Beutel, A., Chen, C., Deaton, J., Eisenstein, J., Hoffman, M. D., et al. (2020) · arXiv · doi:10.48550/arxiv.2011.03395 · Cited by 398
Lymph Node Metastasis Prediction from Primary Breast Cancer US Images Using Deep Learning
Zhou, L.-Q., Wu, X.-L., Huang, S.-Y., Wu, G.-G., Ye, H.-R., Wei, Q., Bao, L.-Y., Deng, Y.-B., Li, X.-R., Cui, X.-W., et al. (2020) · Radiology · doi:10.1148/radiol.2019190372 · Cited by 384
Bai, H. X., Wang, R., Xiong, Z., Hsieh, B., Chang, K., Halsey, K., Tran, T. M. L., Choi, J. W., Wang, D.-C., Shi, L.-B., et al. (2020) · Radiology · doi:10.1148/radiol.2020201491 · Cited by 381
Bluemke, D. A., Moy, L., Bredella, M. A., Ertl-Wagner, B. B., Fowler, K. J., Goh, V. J., Halpern, E. F., Hess, C. P., Schiebler, M. L., & Weiss, C. R. (2020) · Radiology · doi:10.1148/radiol.2019192515 · Cited by 343
Stress Test Evaluation for Natural Language InferencePreprint
Naik, A., Ravichander, A., Sadeh, N., Rosé, C. P., & Neubig, G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.00692 · Cited by 319
Deep Ensembles: A Loss Landscape Perspective
Fort, S., Hu, H., & Lakshminarayanan, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.02757 · Cited by 311
Ji, G.-W., Zhu, F.-P., Xu, Q., Wang, K., Wu, M.-Y., Tang, W.-W., Li, X.-C., & Wang, X.-H. (2020) · Radiology · doi:10.1148/radiol.2020191470 · Cited by 273
Sim, Y., Chung, M. J., Kotter, E., Yune, S., Kim, M., Do, S., Han, K., Kim, H., Yang, S., Lee, D.-J., et al. (2020) · Radiology · doi:10.1148/radiol.2019182465 · Cited by 248
van Velzen, S. G. M., Lessmann, N., Velthuis, B. K., Bank, I. E. M., van den Bongard, D. H. J. G., Leiner, T., de Jong, P. A., Veldhuis, W. B., Correa, A., Terry, J. G., et al. (2020) · Radiology · doi:10.1148/radiol.2020191621 · Cited by 244
Continuous Learning AI in Radiology: Implementation Principles and Early Applications
Pianykh, O. S., Langs, G., Dewey, M., Enzmann, D. R., Herold, C. J., Schoenberg, S. O., & Brink, J. A. (2020) · Radiology · doi:10.1148/radiol.2020200038 · Cited by 237
Artificial Intelligence in Low- and Middle-Income Countries: Innovating Global Health Radiology
Mollura, D. J., Culp, M. P., Pollack, E., Battino, G., Scheel, J. R., Mango, V. L., Elahi, A., Schweitzer, A., & Dako, F. (2020) · Radiology · doi:10.1148/radiol.2020201434 · Cited by 228
Sibille, L., Seifert, R., Avramovic, N., Vehren, T., Spottiswoode, B., Zuehlsdorff, S., & Schäfers, M. (2020) · Radiology · doi:10.1148/radiol.2019191114 · Cited by 219
COVID-19 on Chest Radiographs: A Multireader Evaluation of an Artificial Intelligence System
Murphy, K., Smits, H., Knoops, A. J. G., Korst, M. B. J. M., Samson, T., Scholten, E. T., Schalekamp, S., Schaefer-Prokop, C. M., Philipsen, R. H. H. M., Meijers, A., et al. (2020) · Radiology · doi:10.1148/radiol.2020201874 · Cited by 214
Ethics of using and sharing clinical imaging data for artificial intelligence: a proposed framework
Larson, D. B., Magnus, D. C., Lungren, M. P., Shah, N. H., & Langlotz, C. P. (2020) · Radiology · doi:10.1148/radiol.2020192536 · Cited by 199
Machine Learning for Detecting Early Infarction in Acute Stroke with Non–Contrast-enhanced CT
Qiu, W., Kuang, H., Teleg, E., Ospel, J. M., Sohn, S. I., Almekhlafi, M., Goyal, M., Hill, M. D., Demchuk, A. M., & Menon, B. K. (2020) · Radiology · doi:10.1148/radiol.2020191193 · Cited by 195
Dembrower, K., Liu, Y., Azizpour, H., Eklund, M., Smith, K., Lindholm, P., & Strand, F. (2020) · Radiology · doi:10.1148/radiol.2019190872 · Cited by 163
Deep Learning Enables Automatic Classification of Emphysema Pattern at CT
Humphries, S. M., Notary, A. M., Centeno, J. P., Strand, M. J., Crapo, J. D., Silverman, E. K., Lynch, D. A., & For the Genetic Epidemiology of COPD (COPDGene) Investigators (2020) · Radiology · doi:10.1148/radiol.2019191022 · Cited by 160
Wu, G., Woodruff, H. C., Shen, J., Refaee, T., Sanduleanu, S., Ibrahim, A., Leijenaar, R. T. H., Wang, R., Xiong, J., Bian, J., et al. (2020) · Radiology · doi:10.1148/radiol.2020192431 · Cited by 156
Han, A., Byra, M., Heba, E., Andre, M. P., Erdman, J. W., Loomba, R., Sirlin, C. B., & O’Brien, W. D. (2020) · Radiology · doi:10.1148/radiol.2020191160 · Cited by 148
Rauschecker, A. M., Rudie, J. D., Xie, L., Wang, J., Duong, M. T., Botzolakis, E. J., Kovalovich, A. M., Egan, J., Cook, T. C., Bryan, R. N., et al. (2020) · Radiology · doi:10.1148/radiol.2020190283 · Cited by 148
Deep Learning for Predicting Enhancing Lesions in Multiple Sclerosis from Noncontrast MRI
Narayana, P. A., Coronado, I., Sujit, S. J., Wolinsky, J. S., LUBLIN, F. D., & Gabr, R. E. (2019) · Radiology · doi:10.1148/radiol.2019191061 · Cited by 132
Pickhardt, P. J., Graffy, P. M., Zea, R., Lee, S. J., Liu, J., Sandfort, V., & Summers, R. M. (2020) · Radiology · doi:10.1148/radiol.2020200466 · Cited by 132
Deep Learning Single-Frame and Multiframe Super-Resolution for Cardiac MRI
Masutani, E. M., Bahrami, N., & Hsiao, A. (2020) · Radiology · doi:10.1148/radiol.2020192173 · Cited by 130
Kim, H., Goo, J. M., Lee, K. H., Kim, Y. T., & Park, C. M. (2020) · Radiology · doi:10.1148/radiol.2020192764 · Cited by 130
Eun, N. L., Kang, D., Son, E. J., Park, J. S., Youk, J. H., Kim, J., & Gweon, H. M. (2019) · Radiology · doi:10.1148/radiol.2019182718 · Cited by 126
Zhang, X.-Y., Wang, L., Zhu, H., Li, Z., Ye, M., Li, X.-T., Shi, Y., Zhu, H., & Sun, Y. (2020) · Radiology · doi:10.1148/radiol.2020190936 · Cited by 124
Zhou, Z., Sanders, J. W., Johnson, J. M., Gule-Monroe, M. K., Chen, M. M., Briere, T. M., Wang, Y., Son, J. B., Pagel, M. D., Li, J., et al. (2020) · Radiology · doi:10.1148/radiol.2020191479 · Cited by 110
Fahmy, A. S., Neisius, U., Chan, R. H. M., Rowin, E. J., Manning, W. J., Maron, M. S., & Nezafat, R. (2019) · Radiology · doi:10.1148/radiol.2019190737 · Cited by 98
von Schacky, C. E., Sohn, J. H., Liu, F., Ozhinsky, E., Jungmann, P. M., Nardo, L., Posadzy, M., Foreman, S. C., Nevitt, M. C., Link, T. M., et al. (2020) · Radiology · doi:10.1148/radiol.2020190925 · Cited by 98
Stib, M. T., Vasquez, J., Dong, M. P., Kim, Y. H., Subzwari, S. S., Triedman, H. J., Wang, A., Wang, H.-L. C., Yao, A. D., Jayaraman, M., et al. (2020) · Radiology · doi:10.1148/radiol.2020200334 · Cited by 84
Lee, J. H., Sun, H. Y., Park, S., Kim, H., Hwang, E. J., Goo, J. M., & Park, C. M. (2020) · Radiology · doi:10.1148/radiol.2020201240 · Cited by 81
Deep Learning Measurement of Leg Length Discrepancy in Children Based on Radiographs
Zheng, Q., Shellikeri, S., Huang, H., Hwang, M., & Sze, R. W. (2020) · Radiology · doi:10.1148/radiol.2020192003 · Cited by 74
Jang, S., Song, H., Shin, Y. J., Kim, J., Kim, J., Lee, K. W., Lee, S. S., Lee, W., Lee, S., & Lee, K. H. (2020) · Radiology · doi:10.1148/radiol.2020200165 · Cited by 72
Incremental Unsupervised Domain-Adversarial Training of Neural Networks
Gallego, A., Calvo-Zaragoza, J., & Fisher, R. B. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3025954 · Cited by 69
Learning Domain-Invariant Representations of Histological Images
Lafarge, M. W., Pluim, J. P. W., Eppenhof, K. A. J., & Veta, M. (2019) · Frontiers in Medicine · doi:10.3389/fmed.2019.00162 · Cited by 66
Bariatric CT Imaging: Challenges and Solutions
Fursevich, D., LiMarzi, G. M., O’Dell, M. C., Hernandez, M. A., & Sensakovic, W. F. (2016) · Radiographics · doi:10.1148/rg.2016150198 · Cited by 65
Young, A. T., Fernandez, K., Pfau, J., Reddy, R., Cao, N. A., von Franque, M. Y., Johal, A., Wu, B. V., Wu, R. R., Chen, J. Y., et al. (2021) · npj Digital Medicine · doi:10.1038/s41746-020-00380-6 · Cited by 51
Coronavirus Disease 2019 Deep Learning Models: Methodologic Considerations
D'addario, A. M. V., de Paiva, J. P. Q., Chate, R. C., Machado, B. S., & Szarf, G. (2020) · Radiology · doi:10.1148/radiol.2020201178 · Cited by 35
Choi, K. S., You, S., Han, Y., Ye, J. C., Jeong, B., & Choi, S. H. (2020) · Radiology · doi:10.1148/radiol.2020192763 · Cited by 29
FROELICH, J. W., & Salavati, A. (2019) · Radiology · doi:10.1148/radiol.2019192425 · Cited by 23
The Long Route to Standardized Radiomics: Unraveling the Knot from the End
Kuhl, C. K., & Truhn, D. (2020) · Radiology · doi:10.1148/radiol.2020200059 · Cited by 21
Using Deep Learning for MRI to Identify Responders to Chemoradiotherapy in Rectal Cancer
Koh, D. (2020) · Radiology · doi:10.1148/radiol.2020200417 · Cited by 17
Deep Learning Demonstrates Potential for Lung Cancer Detection in Chest Radiography
Armato, S. G. (2020) · Radiology · doi:10.1148/radiol.2020203538 · Cited by 16
Using Deep Learning to Predict Axillary Lymph Node Metastasis from US Images of Breast Cancer
Bae, M. S. (2019) · Radiology · doi:10.1148/radiol.2019192339 · Cited by 15
Moving Artificial Intelligence from Feasible to Real: Time to Drill for Gas and Build Roads
Chang, P. J. (2019) · Radiology · doi:10.1148/radiol.2019192527 · Cited by 15
Making Your AI Smarter: Continuous Learning Artificial Intelligence for Radiology
McMillan, A. B. (2020) · Radiology · doi:10.1148/radiol.2020202664 · Cited by 13
Three Reasons Why Artificial Intelligence Might Be the Radiologist’s Best Friend
van Rijn, R. R., & Luca, A. D. (2020) · Radiology · doi:10.1148/radiol.2020200855 · Cited by 12
Harnessing the Power of Deep Learning to Assess Breast Cancer Risk
Bahl, M. (2019) · Radiology · doi:10.1148/radiol.2019192471 · Cited by 10
Deep Learning and Lung Cancer: AI to Extract Information Hidden in Routine CT Scans
Shaffer, K. (2020) · Radiology · doi:10.1148/radiol.2020201366 · Cited by 9
Fatty Liver Disease: Artificial Intelligence Takes on the Challenge
Lockhart, M. E., & Smith, A. D. (2020) · Radiology · doi:10.1148/radiol.2020200058 · Cited by 8
Zaharchuk, G. (2020) · Radiology · doi:10.1148/radiol.2020200819 · Cited by 8
Medical Image Perception Research in the Emerging Age of Artificial Intelligence
Jacobson, F. L. (2019) · Radiology · doi:10.1148/radiol.2019192252 · Cited by 7
Nael, K. (2020) · Radiology · doi:10.1148/radiol.2020192703 · Cited by 6
Detection of Brain Metastases with Deep Learning Single-Shot Detector Algorithms
Kikinis, R., & Wells, W. M. (2020) · Radiology · doi:10.1148/radiol.2020200261 · Cited by 6
Automated Coronary Artery Calcium Scoring for Chest CT Scans
Vannier, M. W. (2020) · Radiology · doi:10.1148/radiol.2020192718 · Cited by 5
Deep Learning Improves Predictions of the Need for Total Knee Replacement
Richardson, M. L. (2020) · Radiology · doi:10.1148/radiol.2020202332 · Cited by 5
Radiomics to Predict Invasiveness of Part-Solid Adenocarcinoma of the Lung
Nishino, M. (2020) · Radiology · doi:10.1148/radiol.2020202273 · Cited by 5
Smith, A. D. (2020) · Radiology · doi:10.1148/radiol.2020202900 · Cited by 3
Diagnostic Performance of Deep Learning–augmented Radiology Residents
Deng, F. M. (2020) · Radiology · doi:10.1148/radiol.2020200227 · Cited by 2
Explainable AI for medical imaging: Explaining pneumothorax diagnoses with Bayesian TeachingPreprint
Folke, T., Yang, S. C., Anderson, S. P., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.04684 · Cited by 2
Closing the Gap between Prostate Cancer and Deep Learning Detection Tools
Meyrignac, O., Aziza, R., Roumiguié, M., & Malavaud, B. A. (2020) · Radiology · doi:10.1148/radiol.2020200187 · Cited by 1
Ospel, J. M., & Goyal, M. (2020) · Radiology · doi:10.1148/radiol.2020203527 · Cited by 1
Research works printed but not linked: 2
- 73. Belkin M, Hsu D, Ma S, Mandal S. Reconciling modern machine learning practice and the bias-variance trade-off. ArXiv 1812.11118 [preprint] https:// arxiv.org/abs/1812.11118. Posted December 28, 2018. Accessed March 2021. · arXiv
- 77. Nakkiran P, Kaplun G, Bansal Y, Yang T, Barak B, Sutskever I. Deep double descent: where bigger models and more data hurt. ArXiv 1912.02292 [preprint] https://arxiv.org/abs/1912.02292. Posted December 4, 2019. Accessed March 2021. · arXiv
Cited by
Methods for clinical evaluation of artificial intelligence algorithms for medical diagnosis
Park, S. H., Han, K., Jang, H. Y., Park, J. E., Lee, J.-G., Kim, D. W., & Choi, J. (2023) · Radiology · doi:10.1148/radiol.220182 · Cited by 163
Rahimi, A. K., Pienaar, O., Ghadimi, M., Canfell, O. J., Pole, J. D., Shrapnel, S., van der Vegt, A. H., & Sullivan, C. M. (2024) · Journal of Medical Internet Research · doi:10.2196/49655 · Cited by 164
“%) originated from the United States [20,32,33,36,37,39,43-45, Radiology was the clinical setting in 46% (12/26) of the studies [33,34,36,37,39-41, 43-45”
Deep learning in cancer genomics and histopathology
Unger, M., & Kather, J. N. (2024) · Genome Medicine · doi:10.1186/s13073-024-01315-6 · Cited by 152
“Therefore, studies which only contain a dozen participants, will usually not have sufficiently diverse data to generalize well to external datasets, particularly in clinical routine [111].”
BackgroundGeneralizability of FDA-Approved AI-Enabled Medical Devices for Clinical Use
Windecker, D., Baj, G., Shiri, I., Kazaj, P. M., Kaesmacher, J., Gräni, C., & Siontis, G. C. M. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.8052 · Cited by 142
Mitigating Bias in Radiology Machine Learning: 3. Performance Metrics
Faghani, S., Khosravi, B., Zhang, K., Moassefi, M., Jagtap, J. M., Nugen, F., Vahdati, S., Kuanar, S. P., Rassoulinejad-Mousavi, S. M., Singh, Y., et al. (2022) · Radiology: Artificial Intelligence · doi:10.1148/ryai.220061 · Cited by 94
“This approach consists of obtaining a controlled shifted dataset from the original (internal) population and testing the trained model on this shifted dataset to identify its underspecifi-cations (21,24).”
Background
Metadata
- Authors
- Eche, T., Schwartz, L. H., Mokrane, F.-Z., & Dercle, L.
- Affiliations
- NewYork–Presbyterian Hospital (US)
- Venue
- Radiology: Artificial Intelligence
- Published
- 2021-11-01
- Volume
- 3
- Issue
- 6
- Pages
- e210097-e210097
- ISSN
- 2638-6100 (ISSN-L)
- PMC
- PMC8637230
- OpenAlex
- W3210784269
- Cited by
- 163
- References
- 71
- Citations by year
- 2026: 33 · 2025: 40 · 2024: 47 · 2023: 30 · 2022: 13
- FWCI
- 6.56
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Radiology Artificial Intelligence (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Alain Rahmouni
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiology practices and education
- Keywords
- underspecification, stress testing, generalizability, AI deployment, diagnostic imaging, overfitting, model evaluation, test set design, artificial intelligence in radiology, computer-aided diagnosis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.