The DOI matches its Crossref record; 116 of the 116 references it cites were found in a registry or an index.
External Validation of Deep Learning Algorithms for Radiologic Diagnosis: A Systematic Review
Open accessAbstract
Purpose To assess generalizability of published deep learning (DL) algorithms for radiologic diagnosis. Materials and Methods In this systematic review, the PubMed database was searched for peer-reviewed studies of DL algorithms for image-based radiologic diagnosis that included external validation, published from January 1, 2015, through April 1, 2021. Studies using nonimaging features or incorporating non-DL methods for feature extraction or classification were excluded. Two reviewers independently evaluated studies for inclusion, and any discrepancies were resolved by consensus. Internal and external performance measures and pertinent study characteristics were extracted, and relationships among these data were examined using nonparametric statistics. Results Eighty-three studies reporting 86 algorithms were included. The vast majority (70 of 86, 81%) reported at least some decrease in external performance compared with internal performance, with nearly half (42 of 86, 49%) reporting at least a modest decrease (≥0.05 on the unit scale) and nearly a quarter (21 of 86, 24%) reporting a substantial decrease (≥0.10 on the unit scale). No study characteristics were found to be associated with the difference between internal and external performance. Conclusion Among published external validation studies of DL algorithms for image-based radiologic diagnosis, the vast majority demonstrated diminished algorithm performance on the external dataset, with some reporting a substantial performance decrease. Keywords: Meta-Analysis, Computer Applications–Detection/Diagnosis, Neural Networks, Computer Applications–General (Informatics), Epidemiology, Technology Assessment, Diagnosis, Informatics Supplemental material is available for this article. © RSNA, 2022
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0697 · Cited by 3,267
Zech, J. R., Badgeley, M. A., Liu, M., Costa, A. B., Titano, J. J., & Oermann, E. K. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002683 · Cited by 1,719
Rajpurkar, P., Irvin, J., Ball, R. L., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D., Bagul, A., Langlotz, C. P., et al. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002686 · Cited by 1,449
Checklist for Artificial Intelligence in Medical Imaging (CLAIM): a guide for authors and reviewers
Mongan, J., Moy, L., & Kahn, C. E. (2020) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2020200029 · Cited by 1,443
STARD 2015: An Updated List of Essential Items for Reporting Diagnostic Accuracy Studies
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2015) · Radiology · doi:10.1148/radiol.2015151516 · Cited by 1,337
Automated breast ultrasound lesions detection using convolutional neural networks
Yap, M. H., Pons, G., Martí, J., Ganau, S., Sentís, M., Zwiggelaar, R., Davison, A. K., & Martí, R. (2018) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2017.2731873 · Cited by 1,086
A unifying view on dataset shift in classification
Moreno-Torres, J. G., Raeder, T., Alaiz-Rodríguez, R., Chawla, N. V., & Herrera, F. (2012) · Pattern Recognition · doi:10.1016/j.patcog.2011.06.019 · Cited by 1,029
Deep learning algorithms for detection of critical findings in head CT scans: a retrospective study
Chilamkurthy, S., Ghosh, R., Tanamala, S., Biviji, M., Campeau, N. G., Venugopal, V. K., Mahajan, V., Rao, P., & Warier, P. (2018) · The Lancet · doi:10.1016/s0140-6736(18)31645-3 · Cited by 1,020
Park, S. H., & Han, K. (2018) · Radiology · doi:10.1148/radiol.2017171920 · Cited by 894
Kennedy-Martin, T., Curtis, S., Faries, D., Robinson, S., & Johnston, J. (2015) · Trials · doi:10.1186/s13063-015-1023-4 · Cited by 878
Leakage in data mining: Formulation, detection, and avoidance
Kaufman, S., Rosset, S., Perlich, C., & Stitelman, O. (2012) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/2382577.2382579 · Cited by 791
Bien, N., Rajpurkar, P., Ball, R. L., Irvin, J., Park, A., Jones, E., Bereket, M., Patel, B. N., Yeom, K. W., Shpanskaya, K., et al. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002699 · Cited by 734
Deep Neural Networks Improve Radiologists’ Performance in Breast Cancer Screening
Wu, N., Phang, J., Park, J., Shen, Y., Huang, Z., Zorin, M., Jastrzebski, S., Fevry, T., Katsnelson, J., Kim, E., et al. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2945514 · Cited by 689
Chen, J., Wu, L., Zhang, J., Zhang, L., Gong, D., Zhao, Y., Chen, Q., Huang, S., Yang, M., Yang, X., et al. (2020) · Scientific Reports · doi:10.1038/s41598-020-76282-0 · Cited by 675
Development and evaluation of an artificial intelligence system for COVID-19 diagnosis
Jin, C., Chen, W., Cao, Y., Xu, Z., Tan, Z., Zhang, X., Deng, L., Zheng, C., Zhou, J., Shi, H., et al. (2020) · Nature Communications · doi:10.1038/s41467-020-18685-1 · Cited by 592
Harmon, S. A., Sanford, T. H., Xu, S., Turkbey, E. B., Roth, H., Xu, Z., Yang, D., Myronenko, A., Anderson, V., Amalou, A., et al. (2020) · Nature Communications · doi:10.1038/s41467-020-17971-2 · Cited by 587
A Deep Learning Model to Predict a Diagnosis of Alzheimer Disease by Using 18 F-FDG PET of the Brain
Ding, Y., Sohn, J. H., Kawczynski, M. G., Trivedi, H., Harnish, R., Jenkins, N. W., Lituiev, D., Copeland, T. P., Aboian, M. S., Mari Aparici, C., et al. (2019) · Radiology · doi:10.1148/radiol.2018180958 · Cited by 577
Nam, J. G., Park, S., Hwang, E. J., Lee, J. H., Jin, K.-N., Lim, K. Y., Vu, T. H., Sohn, J. H., Hwang, S., Goo, J. M., et al. (2019) · Radiology · doi:10.1148/radiol.2018180237 · Cited by 539
A fully automatic deep learning system for COVID-19 diagnostic and prognostic analysis
Wang, S., Zha, Y., Li, W., Wu, Q., Li, X., Niu, M., Wang, M., Qiu, X., Li, H., Yu, H., et al. (2020) · European Respiratory Journal · doi:10.1183/13993003.00775-2020 · Cited by 490
Generalizing Evidence From Randomized Clinical Trials to Target Populations: The ACTG 320 Trial
Cole, S. R., & Stuart, E. A. (2010) · American Journal of Epidemiology · doi:10.1093/aje/kwq084 · Cited by 486
The myth of generalisability in clinical research and machine learning in health care
Futoma, J., Simons, M., Panch, T., Doshi-Velez, F., & Celi, L. A. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30186-2 · Cited by 467
Kim, D. W., Jang, H. Y., Kim, K. W., Shin, Y., & Park, S. H. (2019) · Korean Journal of Radiology · doi:10.3348/kjr.2019.0025 · Cited by 459
Hwang, E. J., Park, S., Jin, K.-N., Kim, J. I., Choi, S. Y., Lee, J. H., Goo, J. M., Aum, J., Yim, J.-J., Cohen, J. G., et al. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.1095 · Cited by 447
Wang, S., Shi, J., Ye, Z., Dong, D., Yu, D., Zhou, M., Liu, Y., Gevaert, O., Wang, K., Zhu, Y., et al. (2019) · European Respiratory Journal · doi:10.1183/13993003.00986-2018 · Cited by 435
Li, X., Zhang, S., Zhang, Q., Wei, X., Pan, Y., Zhao, J., Xin, X., Qin, C., Wang, X., Li, J., et al. (2019) · The Lancet Oncology · doi:10.1016/s1470-2045(18)30762-9 · Cited by 433
Schaffter, T., Buist, D. S. M., Lee, C. I., Nikulin, Y., Ribli, D., Guan, Y., Lotter, W., Jie, Z., Du, H., Wang, S., et al. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.0265 · Cited by 409
Chang, K., Bai, H. X., Zhou, H., Su, C., Bi, W. L., Agbodza, E., Kavouridis, V. K., Senders, J. T., Boaro, A., Beers, A., et al. (2018) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-17-2236 · Cited by 391
Walsh, S. L. F., Calandriello, L., Silva, M., & Sverzellati, N. (2018) · The Lancet Respiratory Medicine · doi:10.1016/s2213-2600(18)30286-8 · Cited by 390
Lymph Node Metastasis Prediction from Primary Breast Cancer US Images Using Deep Learning
Zhou, L.-Q., Wu, X.-L., Huang, S.-Y., Wu, G.-G., Ye, H.-R., Wei, Q., Bao, L.-Y., Deng, Y.-B., Li, X.-R., Cui, X.-W., et al. (2020) · Radiology · doi:10.1148/radiol.2019190372 · Cited by 384
Factors That Can Affect the External Validity of Randomised Controlled Trials
Rothwell, P. M. (2006) · PLoS Clinical Trials · doi:10.1371/journal.pctr.0010009 · Cited by 376
Becker, A. S., Marcon, M., Ghafoor, S., Wurnig, M. C., Frauenfelder, T., & Boss, A. (2017) · Investigative Radiology · doi:10.1097/rli.0000000000000358 · Cited by 352
Bluemke, D. A., Moy, L., Bredella, M. A., Ertl-Wagner, B. B., Fowler, K. J., Goh, V. J., Halpern, E. F., Hess, C. P., Schiebler, M. L., & Weiss, C. R. (2020) · Radiology · doi:10.1148/radiol.2019192515 · Cited by 343
Balki, I., Amirabadi, A., Levman, J., Martel, A. L., Emersic, Z., Meden, B., Garcia-Pedrero, A., Ramirez, S. C., Kong, D., Moody, A. R., et al. (2019) · Canadian Association of Radiologists Journal · doi:10.1016/j.carj.2019.06.002 · Cited by 299
Disease Staging and Prognosis in Smokers Using Deep Learning in Chest Computed Tomography
González, G., Ash, S. Y., Vegas-Sánchez-Ferrero, G., Onieva Onieva, J., Rahaghi, F. N., Ross, J. C., Díaz, A., San José Estépar, R., & Washko, G. R. (2018) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.201705-0860oc · Cited by 281
Dou, Q., So, T. Y., Jiang, M., Liu, Q., Vardhanabhuti, V., Kaissis, G., Li, Z., Si, W., Lee, H. H. C., Yu, K., et al. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00431-6 · Cited by 278
Kuo, W., Hӓne, C., Mukherjee, P., Malik, J., & Yuh, E. L. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1908021116 · Cited by 264
Wang, H., Zhou, Z., Li, Y., Chen, Z., Lu, P., Wang, W., Liu, W., & Yu, L. (2017) · EJNMMI Research · doi:10.1186/s13550-017-0260-9 · Cited by 249
Automated abnormality classification of chest radiographs using deep convolutional neural networks
Tang, Y.-X., Tang, Y.-B., Peng, Y., Yan, K., Bagheri, M., Redd, B. A., Brandon, C. J., Lu, Z., Han, M., Xiao, J., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0273-z · Cited by 248
Hwang, E. J., Park, S., Jin, K.-N., Kim, J. I., Choi, S. Y., Lee, J. H., Goo, J. M., Aum, J., Yim, J.-J., Park, C. M., et al. (2019) · Clinical Infectious Diseases · doi:10.1093/cid/ciy967 · Cited by 241
Peng, J., Kang, S., Ning, Z., Deng, H., Shen, J., Xu, Y., Zhang, J., Zhao, W., Li, X., Gong, W., et al. (2020) · European Radiology · doi:10.1007/s00330-019-06318-1 · Cited by 218
Choi, K. J., Jang, J. K., Lee, S. S., Sung, Y. S., Shim, W. H., Kim, H. S., Yun, J., Choi, J.-Y., Lee, Y., Kang, B.-K., et al. (2018) · Radiology · doi:10.1148/radiol.2018180763 · Cited by 213
Taylor, A. G., Mielke, C., & Mongan, J. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002697 · Cited by 212
Shi, Z., Miao, C., Schoepf, U. J., Savage, R. H., Dargis, D. M., Pan, C., Chai, X., Li, X. L., Xia, S., Zhang, X., et al. (2020) · Nature Communications · doi:10.1038/s41467-020-19527-w · Cited by 207
Deep learning in chest radiography: Detection of findings and presence of change
Singh, R., Kalra, M. K., Nitiwarangkul, C., Patti, J. A., Homayounieh, F., Padole, A., Rao, P., Putha, P., Muse, V. V., Sharma, A., et al. (2018) · PLOS ONE · doi:10.1371/journal.pone.0204155 · Cited by 195
Assessing the accuracy of a deep learning method to risk stratify indeterminate pulmonary nodules
Massion, P. P., Antic, S., Ather, S., Arteta, C., Brabec, J., Chen, H., Declerck, J., Dufek, D., Hickes, W., Kadir, T., et al. (2020) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.201903-0505oc · Cited by 190
Huang, S.-C., Kothari, T., Banerjee, I., Chute, C., Ball, R. L., Borus, N., Huang, A., Patel, B. N., Rajpurkar, P., Irvin, J., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0266-y · Cited by 189
Scientific Challenges in the Application of Randomized Trials
Kramer, M. S. (1984) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1984.03350190041017 · Cited by 181
Artificial Intelligence for Medical Image Analysis: A Guide for Authors and Reviewers
England, J. R., & Cheng, P. M. (2019) · American Journal of Roentgenology · doi:10.2214/ajr.18.20490 · Cited by 180
Gao, K., Su, J., Jiang, Z., Zeng, L.-L., Feng, Z., Shen, H., Rong, P., Xu, X., Qin, J., Yang, Y., et al. (2021) · Medical Image Analysis · doi:10.1016/j.media.2020.101836 · Cited by 179
Inconsistent Performance of Deep Learning Models on Mammogram Classification
Wang, X., Liang, G., Zhang, Y., Blanton, H., Bessinger, Z., & Jacobs, N. (2020) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2020.01.006 · Cited by 174
Zhang, C., Sun, X., Dang, K., Li, K., Guo, X.-W., Chang, J., Yu, Z.-Q., Huang, F.-Y., Wu, Y.-S., Liang, Z., et al. (2019) · The Oncologist · doi:10.1634/theoncologist.2018-0908 · Cited by 168
Bae, J. B., Lee, S., Jung, W., Park, S., Kim, W., Oh, H., Han, J. W., Kim, G. E., Kim, J. S., Kim, J. H., et al. (2020) · Scientific Reports · doi:10.1038/s41598-020-79243-9 · Cited by 168
Kann, B. H., Hicks, D. F., Payabvash, S., Mahajan, A. K., Du, J., Gupta, V., Park, H. S.-M., Yu, J. B., Yarbrough, W. G., Burtness, B. A., et al. (2019) · Journal of Clinical Oncology · doi:10.1200/jco.19.02031 · Cited by 160
Leßmann, N., Sánchez, C. I., Beenen, L. F. M., Boulogne, L., Brink, M., Çallı, E., Charbonnier, J. P., Dofferhoff, T., van Everdingen, W. M., Gerke, P. K., et al. (2020) · Radiology · doi:10.1148/radiol.2020202439 · Cited by 159
Multi-resolution convolutional networks for chest X-ray radiograph based lung nodule detection
Li, X., Shen, L., Xie, X., Huang, S., Xie, Z., Hong, X., & Yu, J. (2019) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2019.101744 · Cited by 150
Lee, J. H., Joo, I., Kang, T. W., Paik, Y. H., Sinn, D. H., Ha, S. Y., Kim, K., Choi, C., Lee, G., Yi, J., et al. (2020) · European Radiology · doi:10.1007/s00330-019-06407-1 · Cited by 144
Zhao, W., Yang, J., Ni, B., Bi, D., Sun, Y., Xu, M., Zhu, X., Li, C., Jin, L., Gao, P., et al. (2019) · Cancer Medicine · doi:10.1002/cam4.2233 · Cited by 133
Deep learning-based classification of primary bone tumors on radiographs: A preliminary study
He, Y., Pan, I., Bao, B., Halsey, K., Chang, M., Liu, H., Peng, S., Sebro, R., Guan, J., Yi, T. Y., et al. (2020) · EBioMedicine · doi:10.1016/j.ebiom.2020.103121 · Cited by 128
Yun, J., Park, J. E., Lee, H., Ham, S., Kim, N., & Kim, H. S. (2019) · Scientific Reports · doi:10.1038/s41598-019-42276-w · Cited by 126
Deep Learning in Diagnosis of Maxillary Sinusitis Using Conventional Radiography
Kim, Y., Lee, K. J., Sunwoo, L., Choi, D., Nam, C.-M., Cho, J., Kim, J., Bae, Y. J., Yoo, R., Choi, B. S., et al. (2018) · Investigative Radiology · doi:10.1097/rli.0000000000000503 · Cited by 122
Choi, J. W., Cho, Y. J., Lee, S., Lee, J. S., Lee, S., Choi, Y. H., Cheon, J., & Ha, J. Y. (2019) · Investigative Radiology · doi:10.1097/rli.0000000000000615 · Cited by 119
Ciritsis, A., Rossi, C., Eberhard, M., Marcon, M., Becker, A. S., & Boss, A. (2019) · European Radiology · doi:10.1007/s00330-019-06118-7 · Cited by 117
MSCS-DeepLN: Evaluating lung nodule malignancy using multi-scale cost-sensitive neural networks
Xu, X., Wang, C., Guo, J., Gan, Y., Wang, J., Bai, H., Zhang, L., Li, W., & Zhang, Y. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101772 · Cited by 117
Development and clinical application of deep learning model for lung nodules screening on CT images
Cui, S., Ming, S., Lin, Y., Chen, F., Shen, Q., Li, H., Chen, G., Gong, X., & Wang, H. (2020) · Scientific Reports · doi:10.1038/s41598-020-70629-3 · Cited by 116
Decuyper, M., Bonte, S., Deblaere, K., & Holen, R. V. (2020) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2020.101831 · Cited by 115
Tang, L. Y. W., Coxson, H. O., Lam, S. C., Leipsic, J. A., Tam, R. C., & Sin, D. D. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30064-9 · Cited by 113
Yao, A. D., Cheng, D. L., Pan, I., & Kitamura, F. (2020) · Radiology Artificial Intelligence · doi:10.1148/ryai.2020190026 · Cited by 108
Ultrasound image analysis using deep learning algorithm for the diagnosis of thyroid nodules
Song, J., Chai, Y. J., Masuoka, H., Park, S., Kim, S., Choi, J. Y., Kong, H., Lee, K. E., Lee, J., Kwak, N., et al. (2019) · Medicine · doi:10.1097/md.0000000000015133 · Cited by 106
von Schacky, C. E., Sohn, J. H., Liu, F., Ozhinsky, E., Jungmann, P. M., Nardo, L., Posadzy, M., Foreman, S. C., Nevitt, M. C., Link, T. M., et al. (2020) · Radiology · doi:10.1148/radiol.2020190925 · Cited by 105
Detection and localization of distal radius fractures: Deep learning system versus radiologists
Blüthgen, C., Becker, A. S., Martini, I. V. D., Meier, A., Martini, K., & Frauenfelder, T. (2020) · European Journal of Radiology · doi:10.1016/j.ejrad.2020.108925 · Cited by 103
Nam, J. G., Kim, M., Park, J., Hwang, E. J., Lee, J. H., Hong, J. H., Goo, J. M., & Park, C. M. (2021) · European Respiratory Journal · doi:10.1183/13993003.03061-2020 · Cited by 103
Lee, J. H., Ha, E. J., Kim, D., Jung, Y. J., Heo, S., Jang, Y.-H., An, S. H., & Lee, K. (2020) · European Radiology · doi:10.1007/s00330-019-06652-4 · Cited by 101
Zhao, X., Xie, P., Wang, M., Li, W., Pickhardt, P. J., Xia, W., Xiong, F., Zhang, R., Xie, Y. Q., Jian, J., et al. (2020) · EBioMedicine · doi:10.1016/j.ebiom.2020.102780 · Cited by 98
Wu, Q., Wang, S., Zhang, S., Wang, M., Ding, Y., Jin, F., Wu, Q., Qian, W., Liu, Z., Sun, K., et al. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.11625 · Cited by 89
Weikert, T., Noordtzij, L. A., Bremerich, J., Stieltjes, B., Parmar, V., Cyriac, J., Sommer, G., & Sauter, A. W. (2020) · Korean Journal of Radiology · doi:10.3348/kjr.2019.0653 · Cited by 88
Wang, M., Chen, X., Huang, L., Xu, S., Qin, C., Liu, J., Cao, Y., Yu, P., Zhu, T., Zhu, H., et al. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30199-0 · Cited by 88
Tackling the Radiological Society of North America Pneumonia Detection Challenge
Pan, I., Cadrin-Chênevert, A., & Cheng, P. M. (2019) · American Journal of Roentgenology · doi:10.2214/ajr.19.21512 · Cited by 87
Xiao, L.-S., Li, P., Sun, F., Zhang, Y.-P., Cheng-hai, X., Zhu, H., Cai, F.-Q., He, Y.-L., Zhang, W., Ma, S.-C., et al. (2020) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2020.00898 · Cited by 86
Joo, B., Ahn, S. S., Yoon, P. H., et al. (2020) · European Radiology · doi:10.1007/s00330-020-06966-8 · Cited by 83
Toward human intervention-free clinical diagnosis of intracranial aneurysm via deep neural network
Bo, Z.-H., Qiao, H., Tian, C., Guo, Y., Li, W., Liang, T., Li, D., Liao, D., Zeng, X., Mei, L., et al. (2021) · Patterns · doi:10.1016/j.patter.2020.100197 · Cited by 76
Park, J. J., Kim, K. A., Nam, Y., Choi, M. H., Choi, S. Y., & Rhie, J. (2020) · Scientific Reports · doi:10.1038/s41598-020-66674-7 · Cited by 71
Pan, I., Agarwal, S., & Merck, D. L. (2019) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-019-00180-9 · Cited by 70
One Algorithm May Not Fit All: How Selection Bias Affects Machine Learning Performance
Yu, A. C., & Eng, J. (2020) · Radiographics · doi:10.1148/rg.2020200040 · Cited by 70
Self-citationShin, I., Kim, H., Ahn, S. S., Sohn, B., Bae, S., Park, J. E., Kim, H. S., & Lee, S. (2021) · American Journal of Neuroradiology · doi:10.3174/ajnr.a7003 · Cited by 68
Diagnosis of thyroid nodules on ultrasonography by a deep convolutional neural network
Koh, J., Lee, E., Han, K., Kim, E., Son, E. J., Sohn, Y., Seo, M., Kwon, M.-R., Yoon, J. H., Lee, J. H., et al. (2020) · Scientific Reports · doi:10.1038/s41598-020-72270-6 · Cited by 64
Nguyen, A. V. P., Blears, E. E., Ross, E., Lall, R. R., & Ortega-Barnett, J. R. (2018) · Neurosurgical FOCUS · doi:10.3171/2018.8.focus18325 · Cited by 59
Wu, J., Xin, J., Yang, X., Sun, J., Xu, D., Zheng, N., & Yuan, C. (2019) · Medical Physics · doi:10.1002/mp.13739 · Cited by 59
Song, Z., Liu, T., Shi, L., Yu, Z., Shen, Q., Xu, M., Huang, Z., Cai, Z., Wang, W., Xu, C., et al. (2020) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-020-04986-6 · Cited by 59
Park, S. H. (2018) · Radiology · doi:10.1148/radiol.2018182294 · Cited by 58
Chee, C. G., Kim, Y., Kang, Y., Lee, K. J., Chae, H.-D., Cho, J., Nam, C.-M., Choi, D., Lee, E., Lee, J. W., et al. (2019) · American Journal of Roentgenology · doi:10.2214/ajr.18.20817 · Cited by 56
Ashraf, S. F., Yin, K., Meng, C. X., Wang, Q., Wang, Q., Pu, J., & Dhupar, R. (2021) · Journal of Thoracic and Cardiovascular Surgery · doi:10.1016/j.jtcvs.2021.02.010 · Cited by 56
Zhou, Q.-Q., Tang, W., Wang, J., Hu, Z.-C., Xia, Z., Zhang, R., Fan, X., Yong, W., Yin, X., Zhang, B., et al. (2020) · European Radiology · doi:10.1007/s00330-020-07418-z · Cited by 54
Kim, T., Heo, J., Jang, D., Sunwoo, L., Kim, J., Lee, K. J., Kang, S., Park, S. J., Kwon, O.-K., & Oh, C. W. (2018) · EBioMedicine · doi:10.1016/j.ebiom.2018.12.043 · Cited by 53
Suzuki, K., Otsuka, Y., Nomura, Y., Kumamaru, K. K., Kuwatsuru, R., & Aoki, S. (2020) · Academic Radiology · doi:10.1016/j.acra.2020.07.030 · Cited by 51
Gao, X., & Wang, X. (2019) · Diagnostic and Interventional Imaging · doi:10.1016/j.diii.2019.07.002 · Cited by 49
A CT-based deep learning model for predicting the nuclear grade of clear cell renal cell carcinoma
Lin, F., Ma, C., Xu, J., Yi, L., Li, Q., Lan, Y., Sun, M. H., Long, W., & Cui, E. (2020) · European Journal of Radiology · doi:10.1016/j.ejrad.2020.109079 · Cited by 47
Deep learning for differentiating novel coronavirus pneumonia and influenza pneumonia
Zhou, M., Yang, D., Chen, Y., Xu, Y., Xu, J., Jie, Z., Yao, W., Jin, X., Pan, Z., Tan, J., et al. (2021) · Annals of Translational Medicine · doi:10.21037/atm-20-5328 · Cited by 47
Kitamura, G., & Deible, C. R. (2020) · Clinical Imaging · doi:10.1016/j.clinimag.2020.01.008 · Cited by 44
(2018) · Nature · doi:10.1038/d41586-018-03067-x · Cited by 44
Comparison of medical image classification accuracy among three machine learning methods
Maruyama, T., Hayashi, N., Sato, Y., Hyuga, S., Wakayama, Y., Watanabe, H., Ogura, A., & Ogura, T. (2018) · Journal of X-Ray Science and Technology · doi:10.3233/xst-18386 · Cited by 43
Zou, X., Ren, Y., Feng, D.-Y., He, X., Guo, Y.-F., Yang, H., Li, X., Fang, J., Li, Q., Ye, J., et al. (2020) · PLoS ONE · doi:10.1371/journal.pone.0236378 · Cited by 42
Distributed deep learning across multisite datasets for generalized CT hemorrhage segmentation
Remedios, S. W., Roy, S., Bermudez, C., Patel, M. B., Butman, J. A., Landman, B. A., & Pham, D. L. (2020) · Medical Physics · doi:10.1002/mp.13880 · Cited by 37
Rueckel, J., Kunz, W. G., Hoppe, B. F., Patzig, M., Notohamiprodjo, M., Meinel, F. G., Cyran, C. C., Ingrisch, M., Ricke, J., & Sabel, B. O. (2020) · Critical Care Medicine · doi:10.1097/ccm.0000000000004397 · Cited by 37
Kwon, G., Ryu, J., Oh, J., Lim, J., Kang, B.-K., Ahn, C., Bae, J., & Lee, D. K. (2020) · Scientific Reports · doi:10.1038/s41598-020-74653-1 · Cited by 35
Nael, K., Gibson, E., Chen, Y., Ceccaldi, P., Yoo, Y., Das, J., Doshi, A. H., Georgescu, B., Janardhanan, N., Odry, B. L., et al. (2021) · Scientific Reports · doi:10.1038/s41598-021-86022-7 · Cited by 34
Shin, D. H., Heo, H., Song, S., Shin, N., Nam, Y., Yoo, S., Kim, J., Yoon, J. H., Lee, S. H., Sung, Y. H., et al. (2021) · Parkinsonism & Related Disorders · doi:10.1016/j.parkreldis.2021.03.004 · Cited by 34
The Challenge of Clinical Radiology Research
Blackmore, C. C. (2001) · American Journal of Roentgenology · doi:10.2214/ajr.176.2.1760327 · Cited by 29
Anderson, T. S., Odden, M. C., Penko, J. M., Kazi, D. S., Bellows, B. K., & Bibbins-Domingo, K. B. (2020) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2020.0051 · Cited by 26
Artificial Intelligence Applied to Chest X-ray for Differential Diagnosis of COVID-19 Pneumonia
Salvatore, C., Interlenghi, M., Monti, C. B., Ippolito, D., Capra, D., Cozzi, A., Schiaffino, S., Polidori, A., Gandola, D. G., Alì, M., et al. (2021) · Diagnostics · doi:10.3390/diagnostics11030530 · Cited by 26
Chen, X., Fan, Z., Li, K. K., Wu, G., Yang, Z. H., Gao, X., Liu, Y., Wu, H., Chen, H., Tang, Q., et al. (2020) · Neuro-Oncology Advances · doi:10.1093/noajnl/vdaa079 · Cited by 19
Xie, Q., Lu, Y., Xie, X., Mei, N., Xiong, Y., Li, X., Zhu, Y., Xiao, A., & Yin, B. (2020) · European Radiology · doi:10.1007/s00330-020-07553-7 · Cited by 18
Lee, K. J., Ryoo, I., Choi, D., Sunwoo, L., You, S., & Jung, H. N. (2020) · PLoS ONE · doi:10.1371/journal.pone.0241796 · Cited by 16
Cited by
AI pitfalls and what not to do: mitigating bias in AI
Gichoya, J. W., Thomas, K., Celi, L. A., Safdar, N., Banerjee, I., Banja, J. D., Seyyed-Kalantari, L., Trivedi, H., & Purkayastha, S. (2023) · The British Journal of Radiology · doi:10.1259/bjr.20230023 · Cited by 277
“57 Even on what would be considered a small sample size for external validation, nearly half of the 86 studies reported a modest increase in external performance with nearly a quarter reporting a substantial decrease, reiterating on the challenge of validation.”
External validation of AI models in health should be replaced with recurring local validation
Youssef, A., Pencina, M., Thakur, A., Zhu, T., Clifton, D., & Shah, N. H. (2023) · Nature Medicine · doi:10.1038/s41591-023-02540-z · Cited by 170
Understanding and Mitigating Bias in Imaging Artificial Intelligence
Tejani, A. S., Ng, Y. S., Xi, Y., & Rayan, J. C. (2024) · RadioGraphics · doi:10.1148/rg.230067 · Cited by 138
Mastrodicasa, D., van Assen, M., Huisman, M., Leiner, T., Williamson, E. E., Nicol, E. D., Allen, B. D., Saba, L., Vliegenthart, R., & Hanneman, K. (2025) · Radiology · doi:10.1148/radiol.240516 · Cited by 69
FDA Review of Radiologic AI Algorithms: Process and Challenges
Zhang, K., Khosravi, B., Vahdati, S., & Erickson, B. J. (2024) · Radiology · doi:10.1148/radiol.230242 · Cited by 46
Metadata
- Authors
- Yu, A. C., Mohajer, B., & Eng, J.
- Venue
- Radiology: Artificial Intelligence
- Published
- 2022-05-01
- Volume
- 4
- Issue
- 3
- Pages
- e210064-e210064
- ISSN
- 2638-6100 (ISSN-L)
- OpenAlex
- W4229036772
- Cited by
- 408
- References
- 116
- FWCI
- 5.62
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.