Assessment of Convolutional Neural Networks for Automated Classification of Chest Radiographs
Open accessThe DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
Abstract
Purpose To assess the ability of convolutional neural networks (CNNs) to enable high-performance automated binary classification of chest radiographs. Materials and Methods In a retrospective study, 216 431 frontal chest radiographs obtained between 1998 and 2012 were procured, along with associated text reports and a prospective label from the attending radiologist. This data set was used to train CNNs to classify chest radiographs as normal or abnormal before evaluation on a held-out set of 533 images hand-labeled by expert radiologists. The effects of development set size, training set size, initialization strategy, and network architecture on end performance were assessed by using standard binary classification metrics; detailed error analysis, including visualization of CNN activations, was also performed. Results Average area under the receiver operating characteristic curve (AUC) was 0.96 for a CNN trained with 200 000 images. This AUC value was greater than that observed when the same model was trained with 2000 images (AUC = 0.84, P < .005) but was not significantly different from that observed when the model was trained with 20 000 images (AUC = 0.95, P > .05). Averaging the CNN output score with the binary prospective label yielded the best-performing classifier, with an AUC of 0.98 (P < .005). Analysis of specific radiographs revealed that the model was heavily influenced by clinically relevant spatial regions but did not reliably generalize beyond thoracic disease. Conclusion CNNs trained with a modestly sized collection of prospectively labeled chest radiographs achieved high diagnostic performance in the classification of chest radiographs as normal or abnormal; this function may be useful for automated prioritization of abnormal chest radiographs. © RSNA, 2018 Online supplemental material is available for this article. See also the editorial by van Ginneken in this issue.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The clinical acceptance of AI techniques would be aided by multi-modality and information fusion of both imaging and non-imaging data [34,170].”
From Survey of explainable artificial intelligence techniques for biomedical imaging with deep neural networks (2023) · cites this paper as [170]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“deep learning models have demonstrated diagnostic accuracy comparable to that of human experts in narrow clinical tasks for several medical domains and imaging modalities, including chest and extremity X-rays8–10, computed tomography (CT)11, magnetic resonance imaging (MRI)12, whole slide images (WSI)13,14, and dermatology images15.”
From Self-supervised learning for medical image classification: a systematic review and implementation guidelines (2023) · cites this paper as [8]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The current state-of-the-art deep learning models for radiology applications consider only pixel-value information without data informing clinical context.”
From Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines (2020) · cites this paper as [13]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Early applications of CNNs for image analysis in medicine include diabetic retinopathy, skin cancer, and chest X-rays11–18.”
From Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines (2020) · cites this paper as [13]
In the citing paper
“Only eight studies in ophthalmology imaging14,21,32,33,43,55,108,109, ten studies in respiratory imaging64,66,70,72,75,79,82,87,89,110 and six studies in breast imaging62,91,97,104,106,111 mentioned adherence to the STARD-2015 guidelines or had a STARD flow diagram in the manuscript.”
From Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis (2021) · cites this paper as [79]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 3×
“When we moved from AlexNet (18) to ResNet-18 (19), we observed an 11-point improvement in negative class F1, mostly driven by higher recall.”
MethodLeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988) · Biometrics · doi:10.2307/2531595 · Cited by 22,649 · cited 1×
“The DeLong nonparametric statistical test (24) implemented in the Daim (version 1.1.0) R library was used to assess statistical differences among AUC values, wherein P values less than .05 were considered to indicate a significant difference.”
MethodInfluentialGulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697 · cited 1×
“These scalability results are in line with those of Gulshan et al (27), who found that CNN performance for classification of retinal fundus photographs reached a plateau after approximately 60 000 images.”
ResultHanley, J. A., & McNeil, B. J. (1983) · Radiology · doi:10.1148/radiology.148.3.6878708 · Cited by 7,016 · cited 1×
“The method described by Hanley and McNeil (25) was used to compute 95% confidence intervals for AUC values.”
MethodLakhani, P., & Sundaram, B. (2017) · Radiology · doi:10.1148/radiol.2017162326 · Cited by 1,775
Park, S. H., & Han, K. (2018) · Radiology · doi:10.1148/radiol.2017171920 · Cited by 894
Snorkel: rapid training data creation with weak supervision
Ratner, A., Bach, S. H., Ehrenberg, H., Fries, J., Wu, S., & Ré, C. (2017) · Proceedings of the VLDB Endowment · doi:10.14778/3157794.3157797 · Cited by 777
Self-citationEfficient Additive Kernels via Explicit Feature Maps
Vedaldi, A., & Zisserman, A. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.153 · Cited by 688
Diagnostic Accuracy of Digital Screening Mammography With and Without Computer-Aided Detection
Lehman, C. D., Wellman, R. D., Buist, D. S. M., Kerlikowske, K., Tosteson, A. N. A., & Miglioretti, D. L. (2015) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2015.5231 · Cited by 678 · cited 1×
“Further, this method of disease-specific computer-assisted diagnosis may not ultimately be beneficial to the interpreting clinician (16).”
Method
Research works printed but not linked: 10
- Rosman DA, Nshizirungu JJ, Rudakemwa E, et al. Imaging in the land of 1000 hills: Rwanda radiology country report. J Glob Radiol 2015;1(1):6. [Google Scholar]
- Ali FS, Harrington SG, Kennedy SB, Hussain S. Diagnostic radiology in Liberia: a country report. J Glob Radiol 2015;1(1):6. [Google Scholar]
- Deng J, Dong W, Socher R, Li LJ, Li K, Li F. ImageNet: a large-scale hierarchical image database. In: IEEE CVPR 2009; 248–255. [Google Scholar]
- Wang X, Peng Y, Lu L, Lu Z, Bagheri M, Summers RM. ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases. In: IEEE CVPR 2017; 3462–3471. [Google Scholar]
- Davies ER. Computer and machine vision: theory, algorithms, practicalities. Waltham, Mass: Academic Press/Elsevier, 2012. [Google Scholar]
- He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: IEEE CVPR 2016; 770–778. [Google Scholar]
- Huang G, Liu Z, Van Der Maaten L, Weinberger KQ. Densely connected convolutional networks. In: IEEE CVPR 2017;3:4700–4708. [Google Scholar]
- Paszke A, Gross S, Chintala S, et al. Automatic differentiation in PyTorch. Neural Inf Process Syst Autodiff Work, 2017. https://openreview.net/forum?id=BJJsrmfCZ. Accessed May 8, 2018.
- Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A. Learning deep features for discriminative localization. In: IEEE CVPR 2016; 2921–2929. [Google Scholar]
- Kingma DP, Ba JL. Adam: a method for stochastic optimization. In: International Conference on Learning Representations 2015. https://pdfs.semanticscholar.org/aeea/02a93d674e0a044a8715e767f3a372582604.pdf. Published 2015. Accessed March 31, 2018. [Google Scholar]
Cited by
Liu, X., Faes, L., Kale, A. U., Wagner, S. K., Fu, D. J., Bruynseels, A., Mahendiran, T., Moraes, G., Shamdas, M., Kern, C., et al. (2019) · The Lancet Digital Health · doi:10.1016/s2589-7500(19)30123-2 · Cited by 1,883
Explainable artificial intelligence (XAI) in deep learning-based medical image analysis
van der Velden, B. H. M., Kuijf, H. J., Gilhuijs, K. G. A., & Viergever, M. A. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102470 · Cited by 1,328
Preparing Medical Imaging Data for Machine Learning
Willemink, M. J., Koszek, W. A., Hardell, C., Wu, J., Fleischmann, D., Harvey, H., Folio, L. R., Summers, R. M., Rubin, D. L., & Lungren, M. P. (2020) · Radiology · doi:10.1148/radiol.2020192224 · Cited by 1,020
“As a result, strategies for extracting structured labels from unstructured text have emerged that have shown a great deal of promise for limited applications to apply structured labels in large populations and generate large labeled data sets of imaging studies (2,56).”
BackgroundSelf-citationDiagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis
Aggarwal, R., Sounderajah, V., Martin, G., Ting, D. S. W., Karthikesalingam, A., King, D., Ashrafian, H., & Darzi, A. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00438-z · Cited by 946
Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I., & Lungren, M. P. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00341-z · Cited by 934
Self-citationDeep learning for chest X-ray analysis: A survey
Çallı, E., Sogancioglu, E., van Ginneken, B., van Leeuwen, K. G., & Murphy, K. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102125 · Cited by 506
A scoping review of transfer learning research on medical image analysis using ImageNet
Morid, M. A., Borjali, A., & Del Fiol, G. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.104115 · Cited by 508
Huang, S.-C., Pareek, A., Jensen, M., Lungren, M. P., Yeung, S., & Chaudhari, A. S. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00811-0 · Cited by 479
Self-citationHidden stratification causes clinically meaningful failures in machine learning for medical imaging
Oakden-Rayner, L., Dunnmon, J., Carneiro, G., & Re, C. (2020) · Proceedings of the ACM Conference on Health, Inference, and Learning · doi:10.1145/3368555.3384468 · Cited by 411
“Similarly, Dunnmon et al. [11] report the performance of their chest radiograph triage system by pathology subtype, finding that models trained on binary triage labels achieved substantially lower performance on fracture than on other diseases.”
BackgroundSelf-citationLanglotz, C. P., Allen, B., Erickson, B. J., Kalpathy-Cramer, J., Bigelow, K., Cook, T. S., Flanders, A. E., Lungren, M. P., Mendelson, D. S., Rudie, J. D., et al. (2019) · Radiology · doi:10.1148/radiol.2019190613 · Cited by 368
Self-citationComputer‐aided diagnosis in the era of deep learning
Chan, H., Hadjiiski, L. M., & Samala, R. K. (2020) · Medical Physics · doi:10.1002/mp.13764 · Cited by 358
“Initial efforts have been made to use these tools to visualize the detected location of abnormalities 67,71 or to visualize the characteristics of the deep features 72 on medical images.”
MethodBluemke, D. A., Moy, L., Bredella, M. A., Ertl-Wagner, B. B., Fowler, K. J., Goh, V. J., Halpern, E. F., Hess, C. P., Schiebler, M. L., & Weiss, C. R. (2020) · Radiology · doi:10.1148/radiol.2019192515 · Cited by 343
AI-based computer-aided diagnosis (AI-CAD): the latest review to read first
Fujita, H. (2020) · Radiological Physics and Technology · doi:10.1007/s12194-019-00552-4 · Cited by 300
Balki, I., Amirabadi, A., Levman, J., Martel, A. L., Emersic, Z., Meden, B., Garcia-Pedrero, A., Ramirez, S. C., Kong, D., Moody, A. R., et al. (2019) · Canadian Association of Radiologists Journal · doi:10.1016/j.carj.2019.06.002 · Cited by 299
Nazir, S., Dickson, D. M., & Akram, M. U. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.106668 · Cited by 283
“The clinical acceptance of AI techniques would be aided by multi-modality and information fusion of both imaging and non-imaging data [34,170].”
BackgroundMuhammad, D., & Bendechache, M. (2024) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.08.005 · Cited by 258
Automated abnormality classification of chest radiographs using deep convolutional neural networks
Tang, Y.-X., Tang, Y.-B., Peng, Y., Yan, K., Bagheri, M., Redd, B. A., Brandon, C. J., Lu, Z., Han, M., Xiao, J., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0273-z · Cited by 248
“Dunnmon et al.(4) presented a similar system trained and tested on the radiographs from their institution, wherein they achieved an AUC of 0.”
BackgroundMethodInfluentialPrivacy preservation for federated learning in health care
Pati, S., Kumar, S., Varma, A., Edwards, B., Lu, C., Qu, L., Wang, J. J., Lakshminarayanan, A., Wang, S.-H., Sheller, M. J., et al. (2024) · Patterns · doi:10.1016/j.patter.2024.100974 · Cited by 227
Self-citationFederated Learning for Breast Density Classification: A Real-World Implementation
Roth, H. R., Chang, K., Singh, P., Neumark, N., Li, W., Gupta, V., Gupta, S., Qu, L., Ihsani, A., Bizzo, B. C., et al. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-60548-3_18 · Cited by 202
Self-citationHuang, S.-C., Kothari, T., Banerjee, I., Chute, C., Ball, R. L., Borus, N., Huang, A., Patel, B. N., Rajpurkar, P., Irvin, J., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0266-y · Cited by 189
Self-citationEl‐kenawy, E. M., Mirjalili, S., Ibrahim, A. F., Alrahmawy, M. F., El-Said, M., Zaki, R. M., & Eid, M. M. (2021) · IEEE Access · doi:10.1109/access.2021.3061058 · Cited by 150
“However, a need for sufficient amounts of the training dataset is a critical problem [13]–[15].”
BackgroundBhati, D., Neha, F., & Amiruzzaman, M. (2024) · Journal of Imaging · doi:10.3390/jimaging10100239 · Cited by 216
Deep Learning to Assess Long-term Mortality From Chest Radiographs
Lu, M. T., Ivanov, A., Mayrhofer, T., Hosny, A., Aerts, H. J. W. L., & Hoffmann, U. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.7416 · Cited by 140
Triple attention learning for classification of 14 thoracic diseases using chest radiography
Wang, H., Wang, S., Qin, Z., Zhang, Y., Li, R., & Xia, Y. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101846 · Cited by 157
Malik, H., Anees, T., din, M.-Z., & Naeem, A. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-13843-7 · Cited by 172
“Additionally, most of the researchers [3, 9, 13, 20, 23, 24, 34, 47, 65, 86, 89, 102] believe that when the number of CNN layers rises for a given binary classification task, these classifiers did not work appropriately.”
BackgroundInteractive explainable deep learning model informs prostate cancer diagnosis at MRI
Hamm, C. A., Baumgärtner, G. L., Biessmann, F., Beetz, N. L., Hartenstein, A., Savic, L. J., Froböse, K., Dräger, F., Schallenberg, S., Rudolph, M., et al. (2023) · Radiology · doi:10.1148/radiol.222276 · Cited by 133
Houssein, E. H., Gamal, A. M., Younis, E. M. G., & Mohamed, E. (2025) · Cluster Computing · doi:10.1007/s10586-025-05281-5 · Cited by 175
Explainable AI for Medical Data: Current Methods, Limitations, and Future Directions
Hossain, M. I., Zamzmi, G., Mouton, P. R., Salekin, M. S., Sun, Y., & Goldgof, D. B. (2025) · ACM Computing Surveys · doi:10.1145/3637487 · Cited by 185
Secure Evaluation of Quantized Neural Networks
Dalskov, A., Escudero, D., & Keller, M. (2020) · Proceedings on Privacy Enhancing Technologies · doi:10.2478/popets-2020-0077 · Cited by 144
Malik, H., Farooq, M. S., Khelifi, A., Abid, A., Nasir Qureshi, J., & Hussain, M. (2020) · IEEE Access · doi:10.1109/access.2020.3004766 · Cited by 104
Weak supervision as an efficient approach for automated seizure detection in electroencephalography
Saab, K., Dunnmon, J., Ré, C., Rubin, D., & Lee-Messer, C. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0264-0 · Cited by 103
“…approaches for this task, as they have recently seen widespread success in medical domains including skin lesion classi fi cation from dermatoscopy 12 , automated interpretation of chest radiographs 13 , in-hospital mortality prediction from electronic health records 14 , and many others 15 – 21 .”
MethodSelf-citationAutonomous Chest Radiograph Reporting Using AI: Estimation of Clinical Impact
Plesner, L. L., Müller, F. C., Nybing, J. D., Laustrup, L. C., Rasmussen, F., Nielsen, O. W., Boesen, M., & Andersen, M. B. (2023) · Radiology · doi:10.1148/radiol.222268 · Cited by 95
Tang, S., Ghorbani, A., Yamashita, R., Rehman, S., Dunnmon, J. A., Zou, J., & Rubin, D. L. (2021) · Scientific Reports · doi:10.1038/s41598-021-87762-2 · Cited by 76
Self-citationAutomated abnormality detection in lower extremity radiographs using deep learning
Varma, M., Lu, M., Gardner, R., Dunnmon, J., Khandwala, N., Rajpurkar, P., Long, J., Beaulieu, C. F., Shpanskaya, K. S., Fei-Fei, L., et al. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0126-0 · Cited by 99
“DenseNet models have been shown to outperform ResNet models on the ImageNet validation dataset as well as on the abnormality identification task for chest radiographs 25 .”
BackgroundMethodSelf-citationPark, S., Lee, S. M., Lee, K. H., Jung, K.-H., Bae, W., Choe, J., & Seo, J. B. (2019) · European Radiology · doi:10.1007/s00330-019-06532-x · Cited by 85
Desai, K. T., Befano, B., Xue, Z., Kelly, H. A., Campos, N. G., Egemen, D., Gage, J. C., Rodríguez, A. C., Sahasrabuddhe, V. V., Levitz, D., et al. (2021) · International Journal of Cancer · doi:10.1002/ijc.33879 · Cited by 81
“an accurate yet generalizable AVE algorithm via DL approach, can be assumed to be hundreds or greater for each target class to achieve satisfactory disease discrimination.(54,55) It is worth recalling that, even in high-burden settings, cervical precancer is relatively uncommon(52); thus, ethical acquisition(35) of ac…”
MethodUse of deep learning to detect cardiomegaly on thoracic radiographs in dogs
Burti, S., Osti, V. L., Zotti, A., & Banzato, T. (2020) · The Veterinary Journal · doi:10.1016/j.tvjl.2020.105505 · Cited by 95
Deep Sequential Feature Learning in Clinical Image Classification of Infectious Keratitis
Xu, Y., Kong, M. C., Xie, W., Duan, R., Fang, Z., Lin, Y., Zhu, Q., Tang, S., Wu, F. Q., & Yao, Y. (2020) · Engineering · doi:10.1016/j.eng.2020.04.012 · Cited by 78
“By using these big data, the recent CNN-based AI algorithms can perform anatomical structure segmentation on CT images [24], classify normal or abnormal findings of chest radiographs [25], perform screening for lung or breast cancer [26,27], detect critical findings in head CT scans [28], classify liver lesion using G…”
MethodA disentangled generative model for disease decomposition in chest X-rays via normal image synthesis
Tang, Y., Tang, Y., Zhu, Y., Xiao, J., & Summers, R. M. (2021) · Medical Image Analysis · doi:10.1016/j.media.2020.101839 · Cited by 66
Kim, H., Lee, D., Cho, W. S., Lee, J. C., Goo, J. M., Kim, H. C., & Park, C. M. (2020) · European Radiology · doi:10.1007/s00330-019-06628-4 · Cited by 69
Kim, Y., Kim, S., Cho, C. E., Song, I. H., Lee, H. J., Ahn, S., Park, S. Y., Gong, G., & Kim, N. (2020) · Scientific Reports · doi:10.1038/s41598-020-78129-0 · Cited by 70
Deep learning applied to automatic disease detection using chest X‐rays
Moses, D. A. (2021) · Journal of Medical Imaging and Radiation Oncology · doi:10.1111/1754-9485.13273 · Cited by 74
Low-Memory Neural Network Training: A Technical ReportPreprint
Sohoni, N. S., Aberger, C. R., Leszczynski, M., Zhang, J., & Ré, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.10631 · Cited by 118
Self-citationWolf, D., Payer, T., Lisson, C. S., Lisson, C. G., Beer, M., GOETZ, M. A., & Ropinski, T. (2023) · Scientific Reports · doi:10.1038/s41598-023-46433-0 · Cited by 89
“Deep learning models trained on medical images have recently demonstrated diagnostic accuracy comparable to human experts for narrow clinical tasks 2–5 .”
Cross-Modal Data Programming Enables Rapid Medical Machine Learning
Dunnmon, J. A., Ratner, A. J., Saab, K., Khandwala, N., Markert, M., Sagreiya, H., Goldman, R., Lee-Messer, C., Lungren, M. P., Rubin, D. L., et al. (2020) · Patterns · doi:10.1016/j.patter.2020.100019 · Cited by 57
“The image model used for this task was an 18-layer Residual Network (ResNet-18) with a sigmoid nonlinearity on top of a single-neuron final layer, which yields near-state-of-the-art results on this dataset at a modest computational cost.(7,9) The model was implemented using the PyTorch software framework and was initi…”
BackgroundMethodResultInfluentialSelf-citationPan, I., Thodberg, H. H., Halabi, S. S., Kalpathy–Cramer, J., & Larson, D. B. (2019) · Radiology Artificial Intelligence · doi:10.1148/ryai.2019190053 · Cited by 65
“Human readers can be considered as unique models in their own right, and ultimately, we believe the most robust solutions will combine automated models with human assessment (12–15).”
BackgroundA Practical Guide to Artificial Intelligence–Based Image Analysis in Radiology
Weikert, T., Cyriac, J., Yang, S., Nešić, I., Parmar, V., & Stieltjes, B. (2019) · Investigative Radiology · doi:10.1097/rli.0000000000000600 · Cited by 59
Nabulsi, Z., Sellergren, A., Jamshy, S., Lau, C. T., Santos, E. R. G., Kiraly, A. P., Ye, W., Yang, J., Pilgrim, R., Kazemzadeh, S., et al. (2021) · Scientific Reports · doi:10.1038/s41598-021-93967-2 · Cited by 74
Chang, K., Beers, A. L., Brink, L., Patel, J., Singh, P., Arun, N., Hoebel, K., Gaw, N. B., Shah, M., Pisano, E. D., et al. (2020) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2020.05.015 · Cited by 58
A new paradigm for accelerating clinical data science at Stanford Medicine
Datta, S., Posada, J., Olson, G., Li, W., O’Reilly, C., Balraj, D., Mesterhazy, J., Pallas, J. I., Desai, P., & Shah, N. H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.10534 · Cited by 29
Liu, G. S., Yang, A., Kim, D., Hojel, A., Voevodsky, D., Wang, J., Tong, C. C. L., Ungerer, H., Palmer, J. N., Kohanski, M. A., et al. (2022) · International Forum of Allergy & Rhinology · doi:10.1002/alr.22958 · Cited by 47
Fan, W., Yang, Y., Qi, J., Zhang, Q., Liao, C., Li, W., Wang, S., Wang, G., Yu, X., Wu, Q., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-45599-z · Cited by 64
Resilience to diabetic retinopathy
Serikbaeva, A., Li, Y., Ma, S., Yi, D., & Kazlauskas, A. (2024) · Progress in Retinal and Eye Research · doi:10.1016/j.preteyeres.2024.101271 · Cited by 51
“As with in several other medical image classification tasks (Dunnmon et al., 2019; Esteva et al., 2017; Park et al., 2019), this proved to be a more efficient means of learning to detect DR stages from fundus photographs.”
Self-citationUse of Deep Learning in Forensic Sex Estimation of Virtual Pelvic Models from the Han Population
Cao, Y.-J., Ma, Y., Yang, X., Xiong, J., Wang, Y., Zhang, J., Qin, Z., Chen, Y., Vieira, D. N., Chen, F., et al. (2022) · Forensic Sciences Research · doi:10.1080/20961790.2021.2024369 · Cited by 52
Murphy, Z. R., Venkatesh, K., Sulam, J., & Yi, P. H. (2022) · Radiology Artificial Intelligence · doi:10.1148/ryai.220012 · Cited by 59
Hong, G.-S., Jang, M., Kyung, S., Cho, K., Jeong, J., Lee, G. Y., Shin, K., Kim, K., Ryu, S. M., Seo, J. B., et al. (2023) · Korean Journal of Radiology · doi:10.3348/kjr.2023.0393 · Cited by 57
Deep learning to automate Brasfield chest radiographic scoring for cystic fibrosis
Zucker, E. J., Barnes, Z. A., Lungren, M. P., Shpanskaya, K. S., Seekins, J. M., Halabi, S. S., & Larson, D. B. (2019) · Journal of Cystic Fibrosis · doi:10.1016/j.jcf.2019.04.016 · Cited by 49
Self-citationFederated Learning for Multicenter Collaboration in Ophthalmology
Lu, C., Hanif, A., Singh, P., Chang, K., Coyner, A. S., Brown, J. M., Ostmo, S. R., Chan, R. V. P., Rubin, D. L., Chiang, M. F., et al. (2022) · Ophthalmology Retina · doi:10.1016/j.oret.2022.02.015 · Cited by 51
Self-citationJiang, S., Gu, Y., & Kumar, E. (2023) · Cloud Computing and Data Science · doi:10.37256/ccds.4220232740 · Cited by 55
Deep Learning to Quantify Pulmonary Edema in Chest Radiographs
Horng, S., Liao, R., Wang, X., Dalal, S., Golland, P., & Berkowitz, S. J. (2021) · Radiology Artificial Intelligence · doi:10.1148/ryai.2021190228 · Cited by 62
Classification of Color-Coded Scheimpflug Camera Corneal Tomography Images Using Deep Learning
Abdelmotaal, H., Mostafa, M. M., Mostafa, A. N. R., Mohamed, A.-S. A., & Abdelazeem, K. (2020) · Translational Vision Science & Technology · doi:10.1167/tvst.9.13.30 · Cited by 43
Dopamine: Differentially Private Federated Learning on Medical Data
Malekzadeh, M., Hasırcıoglu, B., Mital, N., Katarya, K., Özfatura, E., & Gündüz, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.11693 · Cited by 38
“Medical images are processed with DNNs for faster diagnosis of skin disease (Esteva et al. 2017), lung cancer (Dunnmon et al. 2019), or diabetic retinopathy (DR) (Gulshan et al. 2016).”
BackgroundThyreau, B., & Taki, Y. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101639 · Cited by 40
Eyuboglu, S., Angus, G., Patel, B. N., Pareek, A., Davidzon, G., Long, J., Dunnmon, J., & Lungren, M. P. (2021) · Nature Communications · doi:10.1038/s41467-021-22018-1 · Cited by 56
Self-citationGanesan, P., Rajaraman, S., Long, L. R., Ghoraani, B., & Antani, S. (2019) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2019.8857516 · Cited by 47
“This observation is coherent with various other studies [13], [5] which also emphasizes that augmentation intends to reduce model overfitting and improve generalization by introducing diversity in the training set distribution, hence improving the classifier performance.”
ResultEngle, E., Gabrielian, A., Long, A., Hurt, D. E., & Rosenthal, A. (2020) · PLOS ONE · doi:10.1371/journal.pone.0224445 · Cited by 37
Effect of image resolution on automated classification of chest X-rays
Haque, M. I. U., Dubey, A., Danciu, I., Justice, A. C., Ovchinnikova, O. S., & Hinkle, J. D. (2023) · Journal of medical imaging · doi:10.1117/1.jmi.10.4.044503 · Cited by 55
“Accurate automated analysis of CXRs would allow radiologists to improve the efficiency of their workflow on assessing abnormalities in CXRs and help extend this capability to under-served regions.(3,4) In recent years, deep learning (DL)(5) has seen remarkable success in medical image analysis.”
MethodKoyyada, S. P., & Singh, T. P. (2023) · Healthcare Analytics · doi:10.1016/j.health.2023.100206 · Cited by 56
Jin, K. N., Kim, E. Y., Kim, Y. J., Lee, G. P., Kim, H., Oh, S., Kim, Y., Han, J., & Cho, Y. J. (2022) · European Radiology · doi:10.1007/s00330-021-08397-5 · Cited by 37
“This could be the reason for our notably poor stand-alone performance (AUC of 0.867) in comparison with previous reports showing excellent AI performance (AUC of 0.96–0.99) in identifying multiple abnormalities on chest radiographs [25, 26].”
ResultInfluentialDeep Learning to Quantify Pulmonary Edema in Chest RadiographsPreprint
Horng, S., Liao, R., Wang, X., Dalal, S., Golland, P., & Berkowitz, S. J. (2021) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 38
New trend in artificial intelligence-based assistive technology for thoracic imaging
Yanagawa, M., Ito, R., Nozaki, T., Fujioka, T., Yamada, A., Fujita, S., Kamagata, K., Fushimi, Y., Tsuboyama, T., Matsui, Y., et al. (2023) · La radiologia medica · doi:10.1007/s11547-023-01691-w · Cited by 36
“CNNs trained on a modest collection of prospectively labeled data might achieve high diagnostic performance in classifying their data as normal or abnormal [110].”
BackgroundUetani, H., Nakaura, T., Kitajima, M., Yamashita, Y., Hamasaki, T., Tateishi, M., Morita, K., Sasao, A., Oda, S., Ikeda, O., et al. (2020) · Neuroradiology · doi:10.1007/s00234-020-02513-w · Cited by 33
Thian, Y. L., Ng, D. W., Hallinan, J. T. P. D., Jagmohan, P., Sia, S. Y., Adnan, J. S., Quek, S. T., & Feng, M. (2022) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-022-00594-y · Cited by 41
“observed in their training of a binary classifier of chest radiographs into normal and abnormal categories that the AUC of a model trained with 200,000 images was not significantly different from a model trained with 20,000 images [25].”
MethodMir, A. N., & Rizvi, D. R. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.111413 · Cited by 45
Thrall, J. H., Fessell, D. P., & Pandharipande, P. V. (2021) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2020.07.010 · Cited by 30
Zhou, Q.-Q., Hu, Z.-C., Tang, W., Xia, Z., Wang, J., Zhang, R., Li, X., Chen, C., Zhang, B., Ling-quan, L., et al. (2021) · Clinical Imaging · doi:10.1016/j.clinimag.2021.09.010 · Cited by 30
Observational Supervision for Medical Image Classification Using Gaze Data
Saab, K. K., Hooper, S., Sohoni, N. S., Parmar, J., Pogatchnik, B., Wu, S., Dunnmon, J., Zhang, H. R., Rubin, D. L., & Ré, C. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87196-3_56 · Cited by 33
“We consider three datasets: classifying chest X-rays for pneumothorax (CXR-P), classifying chest X-rays for a general abnormality (CXR-A), and classifying brain MRI slices for metastasis (METS) (all binary image classification).”
BackgroundMethodInfluentialSelf-citationShetty, S., Ananthanarayana, V. S., & Mahale, A. R. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-14940-x · Cited by 37
“2 The grid of normal (No diseases) and abnormal (pulmonary diseases) CXR from Indiana University dataset Initially, the CNNs were widely applicable in medical image analysis, including CXRs, diabetic retinopathy, and skin cancer [20, 27, 30, 36].”
BackgroundMethodExplainable-AI in Automated Medical Report Generation Using Chest X-ray Images
Ahmed, S. B., Solis-Oba, R., & Ilie, L. (2022) · Applied Sciences · doi:10.3390/app122211750 · Cited by 36
Performance of AI to exclude normal chest radiographs to reduce radiologists' Workload
Schalekamp, S., van Leeuwen, K., Calli, E., Murphy, K., Rutten, M., Geurts, B., Peters-Bax, L., van Ginneken, B., & Prokop, M. (2024) · European Radiology · doi:10.1007/s00330-024-10794-5 · Cited by 28
Curation of the CANDID-PTX Dataset with Free-Text Reports
Feng, S., Azzollini, D., Kim, J. S., Jin, C.-K., Gordon, S. P., Yeoh, J., Kim, E., Han, M., Lee, A., Patel, A., et al. (2021) · Radiology Artificial Intelligence · doi:10.1148/ryai.2021210136 · Cited by 49
Use of artificial intelligence in triaging of chest radiographs to reduce radiologists’ workload
Yoon, S. H., Park, S.-Y., Jang, S., Kim, J., Lee, K. W., Lee, W., Lee, S., Yun, G., & Lee, K. H. (2023) · European Radiology · doi:10.1007/s00330-023-10124-1 · Cited by 35
“Previous studies have reported the feasibility of automated triaging of normal chest radiographs using artificial intelligence systems [8–10].”
State-of-the-Art in Responsible, Explainable, and Fair AI for Medical Image Analysis
Amirian, S., Gao, F., Littlefield, N. G., Hill, J. H., Yates, A. J., Plate, J. F., Pantanowitz, L., Rashidi, H. H., & Tafti, A. P. (2025) · IEEE Access · doi:10.1109/access.2025.3555543 · Cited by 37
“…imaging, including cancer identification and tumor classification, by visualizing salient regions in images of the bladder, brain, breast, skin, cardiovascular, chest, gastrointestinal, and thyroid scans [42], [60], [212], [215], [216], [217], [218], [219], [220], [221], [222], [223], [224], [225].”
Metaheuristics role in image processing and computer vision applications: a comprehensive review
Şahin, M. F., & Anka, F. (2025) · Cluster Computing · doi:10.1007/s10586-025-05610-8 · Cited by 34
“In another detection method, X-ray imaging continues to face the critical issue of needing a sufficient amount of training data, and successful classification requires feature extraction [86].”
Reducing Errors Resulting From Commonly Missed Chest Radiography Findings
Gefter, W. B., & Hatabu, H. (2022) · CHEST Journal · doi:10.1016/j.chest.2022.12.003 · Cited by 38
Yuan, Y., Sun, C., Tang, X., Cheng, C., Mombaerts, L., Wang, M., Hu, T., Sun, C., Guo, Y., Li, X., et al. (2020) · Engineering · doi:10.1016/j.eng.2020.10.013 · Cited by 30
Xin, K. Z., Li, D., & Yi, P. H. (2021) · Emergency Radiology · doi:10.1007/s10140-021-01954-x · Cited by 33
Kim, H. W., Heo, S., Kim, M., Lee, J., Park, K. H., Lee, G., Baeg, S. I., Kwon, Y. E., Choi, H. M., Oh, D., et al. (2022) · Frontiers in Medicine · doi:10.3389/fmed.2022.878858 · Cited by 35
A machine learning proposal method to detect milk tainted with cheese whey
Lima, J. S., Ribeiro, D. C., Neto, H. A., Campos, S., Leite, M. O., Fortini, M. E. D. R., de Carvalho, B. P. M., Almeida, M. V. O., & Fonseca, L. M. (2022) · Journal of Dairy Science · doi:10.3168/jds.2021-21380 · Cited by 29
“980 for the test data, this receiver operating characteristics curve indicates an optimal classification performance of the model which may be applied for prediction purposes, because the model presents high levels of correct predictions for each class (Dunnmon et al., 2019; Neto et al., 2019).”
MethodFang, Z., Kuang, K., Lin, Y., Wu, F. Q., & Yao, Y. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413557 · Cited by 36
“These CNN-based AI algorithms can perform anatomical structure segmentation on CT images [33], classify normal or abnormal findings of chest radiographs [7], perform screening for lung [1] or breast cancer [25], detect critical findings in head CT scans [5], classify liver lesion [9] and detect lymph node metastases i…”
MethodChest X-Ray Image Preprocessing for Disease Classification
Caseneuve, G. M., Valova, I., LeBlanc, N., & Thibodeau, M. (2021) · Procedia Computer Science · doi:10.1016/j.procs.2021.08.068 · Cited by 26
Artificial Intelligence in Paediatric Tuberculosis
Naidoo, J., Shelmerdine, S. C., Ugas-Charcape, C. F., & Sodhi, A. S. (2023) · Pediatric Radiology · doi:10.1007/s00247-023-05606-9 · Cited by 33
“As a subset of AI, machine learning uses statistical methods to enable machines to improve with and evolve as a result of that experience [44].”
MethodYi, P. H., Lin, A., Wei, J., Yu, A. C., Sair, H. I., Hui, F. K., Hager, G. D., & Harvey, S. C. (2019) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-019-00244-w · Cited by 25
Clinical Artificial Intelligence Applications in Radiology
Lee, S., & Summers, R. M. (2021) · Radiologic Clinics of North America · doi:10.1016/j.rcl.2021.07.001 · Cited by 28
Hwang, E. J., Goo, J. M., Nam, J. G., Park, C. M., Hong, K. J., & Kim, K. H. (2023) · Korean Journal of Radiology · doi:10.3348/kjr.2022.0651 · Cited by 28
Filice, R. W. (2019) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2019.05.007 · Cited by 24
Deep Learning for Chest Radiology: A Review
Feng, Y., Teh, H. S., & Cai, Y. (2019) · Current Radiology Reports · doi:10.1007/s40134-019-0333-9 · Cited by 30
Doubly Weak Supervision of Deep Learning Models for Head CT
Saab, K. K., Dunnmon, J., Goldman, R. E., Ratner, A., Sagreiya, H., Ré, C., & Rubin, D. L. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32248-9_90 · Cited by 26
Self-citationKim, Y.-G., Lee, S. M., Lee, K. H., Jang, R., Seo, J. B., & Kim, N. (2020) · European Radiology · doi:10.1007/s00330-020-06892-9 · Cited by 27
“Nam et al [3] used a variable input matrix size between 500~1000 3 to preserve the aspect ratio in order to develop a CAD system for classifying abnormalities.”
BackgroundMethodInfluentialExploiting Cascaded Ensemble of Features for the Detection of Tuberculosis Using Chest Radiographs
Saif, A. F. M., Imtiaz, T., Shahnaz, C., Zhu, W., & Ahmad, M. O. (2021) · IEEE Access · doi:10.1109/access.2021.3102077 · Cited by 25
“However, recently neural network-especially the convolutional neural network (CNN) has brought significant improvement in computer-aided image classification tasks and provided more significant results than conventional machine learning approaches [15], [16].”
Result
Metadata
- Authors
- Dunnmon, J. A., Yi, D., Langlotz, C. P., Ré, C., Rubin, D. L., & Lungren, M. P.
- Affiliations
- Stanford University (US)
- ORCID
- Venue
- Radiology
- Published
- 2019-02
- Volume
- 290
- Issue
- 2
- Pages
- 537-544
- ISSN
- 0033-8419 (ISSN-L)
- 1527-1315
- PMC
- PMC6358056
- OpenAlex
- W2901782819
- Semantic Scholar
- 3f3ecb0b178b2df70c3f1e90684895ef82a3f836
- Cited by
- 206
- References
- 24
- Citations by year
- 2026: 4 · 2025: 16 · 2024: 24 · 2023: 38 · 2022: 26 · 2021: 39 · 2020: 37 · 2019: 20
- FWCI
- 16.73
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Radiology (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Intelligence Community Postdoctoral Research Fellowship Program
- National Institutes of Health (1U01CA187947, 1U01CA190214, U01CA142555)
- National Cancer Institute
- Office of Naval Research (N000141712266)
- Defense Advanced Research Projects Agency (FA87501720095)
- Topics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- convolutional neural networks, chest radiograph, automatic classification, normal vs abnormal classification, training data size, network architecture, activation visualization, error analysis, chest diseases
- MeSH terms
- Radiologists, Female, Humans, Lung, Male, Radiographic Image Interpretation, Computer-Assisted, Retrospective Studies, ROC Curve, Radiography, Thoracic, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.