Chest Radiograph Interpretation with Deep Learning Models: Assessment with Radiologist-adjudicated Reference Standards and Population-adjusted Evaluation
The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
Background Deep learning has the potential to augment the use of chest radiography in clinical radiology, but challenges include poor generalizability, spectrum bias, and difficulty comparing across studies. Purpose To develop and evaluate deep learning models for chest radiograph interpretation by using radiologist-adjudicated reference standards. Materials and Methods Deep learning models were developed to detect four findings (pneumothorax, opacity, nodule or mass, and fracture) on frontal chest radiographs. This retrospective study used two data sets. Data set 1 (DS1) consisted of 759 611 images from a multicity hospital network and ChestX-ray14 is a publicly available data set with 112 120 images. Natural language processing and expert review of a subset of images provided labels for 657 954 training images. Test sets consisted of 1818 and 1962 images from DS1 and ChestX-ray14, respectively. Reference standards were defined by radiologist-adjudicated image review. Performance was evaluated by area under the receiver operating characteristic curve analysis, sensitivity, specificity, and positive predictive value. Four radiologists reviewed test set images for performance comparison. Inverse probability weighting was applied to DS1 to account for positive radiograph enrichment and estimate population-level performance. Results In DS1, population-adjusted areas under the receiver operating characteristic curve for pneumothorax, nodule or mass, airspace opacity, and fracture were, respectively, 0.95 (95% confidence interval [CI]: 0.91, 0.99), 0.72 (95% CI: 0.66, 0.77), 0.91 (95% CI: 0.88, 0.93), and 0.86 (95% CI: 0.79, 0.92). With ChestX-ray14, areas under the receiver operating characteristic curve were 0.94 (95% CI: 0.93, 0.96), 0.91 (95% CI: 0.89, 0.93), 0.94 (95% CI: 0.93, 0.95), and 0.81 (95% CI: 0.75, 0.86), respectively. Conclusion Expert-level models for detecting clinically relevant chest radiograph findings were developed for this study by using adjudicated reference standards and with population-level performance estimation. Radiologist-adjudicated labels for 2412 ChestX-ray14 validation set images and 1962 test set images are provided. © RSNA, 2019 Online supplemental material is available for this article. See also the editorial by Chang in this issue.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Only eight studies in ophthalmology imaging14,21,32,33,43,55,108,109, ten studies in respiratory imaging64,66,70,72,75,79,82,87,89,110 and six studies in breast imaging62,91,97,104,106,111 mentioned adherence to the STARD-2015 guidelines or had a STARD flow diagram in the manuscript.”
From Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis (2021) · cites this paper as [82]
Citations
Cites
Fleischner Society: Glossary of Terms for Thoracic Imaging
Hansell, D. M., Bankier, A. A., MacMahon, H., McLoud, T. C., Müller, N. L., & Remy, J. (2008) · Radiology · doi:10.1148/radiol.2462070712 · Cited by 4,198 · cited 1×
“Clinical definitions for these categories were on the basis of the Fleischner Society Glos-sary of Terms for Thoracic Imaging, except for osseous fracture, which was defined as visible rib, clavicle, humeral, or vertebral body fractures (20).”
MethodArmato, S. G., McLennan, G., Bidaut, L., McNitt‐Gray, M. F., Meyer, C. R., Reeves, A. P., Zhao, B., Aberle, D. R., Henschke, C. I., Hoffman, E. A., et al. (2011) · Medical Physics · doi:10.1118/1.3528204 · Cited by 2,887
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
Sun, C., Shrivastava, A., Singh, S., & Gupta, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.97 · Cited by 2,484 · cited 1×
“We used Xception (22) as the convolutional neural network architecture that was pretrained on 300 000 000 natural images (23).”
MethodRajpurkar, P., Irvin, J., Ball, R. L., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D., Bagul, A., Langlotz, C. P., et al. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002686 · Cited by 1,449 · cited 2×
“Recent advances in deep learning offer promise for improving chest radiograph interpretation (1–4) and there are several recent reports of machine learning models achieving radiologist-level performance for different chest radiograph findings (5–7).”
BackgroundVery Deep Convolutional Networks for Text Classification
Conneau, A., Schwenk, H., Barrault, L., & Lecun, Y. (2017) · Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers · doi:10.18653/v1/e17-1104 · Cited by 1,046
Park, S. H., & Han, K. (2018) · Radiology · doi:10.1148/radiol.2017171920 · Cited by 894 · cited 1×
“However, because such data do not necessarily reflect real-world prevalence or diversity (33,34), such enrichment can also prevent meaningful clinical interpretation of diagnostic performance.”
BackgroundKrause, J., Gulshan, V., Rahimy, E., Karth, P., Widner, K., Corrado, G. S., Peng, L., & Webster, D. R. (2018) · Ophthalmology · doi:10.1016/j.ophtha.2018.01.034 · Cited by 564 · cited 1×
“…adjudicated labels, extensive work has been performed on the value of multiple interpretations in mammography (30) and other clinical settings (31,32). the most obviously positive and negative for pneumothorax, opacity, nodule or mass, or fracture for training, which can fail to address the…”
BackgroundSelf-citationMazurowski, M. A., Buda, M., Saha, A., & Bashir, M. R. (2019) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.26534 · Cited by 554 · cited 1×
“Recent advances in deep learning offer promise for improving chest radiograph interpretation (1–4) and there are several recent reports of machine learning models achieving radiologist-level performance for different chest radiograph findings (5–7).”
BackgroundNam, J. G., Park, S., Hwang, E. J., Lee, J. H., Jin, K.-N., Lim, K. Y., Vu, T. H., Sohn, J. H., Hwang, S., Goo, J. M., et al. (2019) · Radiology · doi:10.1148/radiol.2018180237 · Cited by 539 · cited 1×
“Recent advances in deep learning offer promise for improving chest radiograph interpretation (1–4) and there are several recent reports of machine learning models achieving radiologist-level performance for different chest radiograph findings (5–7).”
BackgroundMcBee, M. P., Awan, O. A., Colucci, A. T., Ghobadi, C. W., Kadom, N., Kansagra, A. P., Tridandapani, S., & Auffermann, W. F. (2018) · Academic Radiology · doi:10.1016/j.acra.2018.02.018 · Cited by 496
Research works printed but not linked: 3
- Auffermann WF, 2019, AJR Am J Roentgenol, 1
- Summers RM, NIH Chest X-ray Dataset of 14 Common Thorax Disease Categories
- Chollet F. Xception: Deep Learning with Depthwise Separable Convolutions. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, July 21–26, 2017. Piscataway, NJ: IEEE, 2017; 1800–1807.
Cited by
Items 101–200 of 337
Use of artificial intelligence in triaging of chest radiographs to reduce radiologists’ workload
Yoon, S. H., Park, S.-Y., Jang, S., Kim, J., Lee, K. W., Lee, W., Lee, S., Yun, G., & Lee, K. H. (2023) · European Radiology · doi:10.1007/s00330-023-10124-1 · Cited by 35
“The sensitivity of chest radiographs is highly variable, ranging from 30 to 89% [3–6 , 11– 13].”
Sufian, M. A., Hamzi, W., Sharifi, T., Zaman, S., Alsadder, L., Lee, E., Hakim, A., & Hamzi, B. (2024) · Journal of Personalized Medicine · doi:10.3390/jpm14080856 · Cited by 35
Value of Using a Generative AI Model in Chest Radiography Reporting: A Reader Study
Hong, E. K., Roh, B., Park, B., Jo, J.-B., Bae, W., Park, J. S., & Sung, D.-W. (2025) · Radiology · doi:10.1148/radiol.241646 · Cited by 35
Zhou, L., Yin, X., Zhang, T., Feng, Y., Zhao, Y. R., Jin, M., Peng, M., Xing, C., Li, F., Wang, Z., et al. (2021) · Radiology Artificial Intelligence · doi:10.1148/ryai.2021200172 · Cited by 36
“Additional models have been developed to detect 14 different pathologic abnormalities on chest radiographs (19) as well as to distinguish the presence or absence of fracture, nodule or mass, pneumothorax, or opacity at radiologist-level performance (20,21).”
MethodSelf-citationLuo, L., Chen, H., Xiao, Y., Zhou, Y., Wang, X., Vardhanabhuti, V., Wu, M., Han, C., Liu, Z., Fang, B., et al. (2022) · Radiology Artificial Intelligence · doi:10.1148/ryai.210299 · Cited by 36
“Each radiograph was labeled with yes or no findings of airspace opacity, fracture, mass or nodule, or pneumothorax by at least three radiologists from Google Health (7).”
BackgroundAdvancing Multimodal Medical Capabilities of Gemini
Yang, L., Xu, S., Sellergren, A., Kohlberger, T., Zhou, Y., Ktena, I., Kiraly, A., Ahmed, F., Hormozdiari, F., Jaroensri, T., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.03162 · Cited by 34
Self-citationChen, Y., Wan, Y., & Pan, F. (2023) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-023-00801-4 · Cited by 34
“5), which negatively affects the performance of the models and the ability to utilize different datasets effectively [36, 37].”
BackgroundAutomatic Scan Range Delimitation in Chest CT Using Deep Learning
Demircioğlu, A., Kim, M., Stein, M. C., Guberina, N., Umutlu, L., & Naßenstein, K. (2021) · Radiology Artificial Intelligence · doi:10.1148/ryai.2021200211 · Cited by 34
“Subgroup analysis showed highest examples (9), whereas the ChestNet has been trained on a dataset with more than 650 000 images for a network to achieve accuracy similar to that of a human radiologist (17).”
MethodDeep Learning for Detection of Pulmonary Metastasis on Chest Radiographs
Hwang, E. J., Lee, J. S., Lee, J. H., Lim, W. H., Kim, J. H., Choi, K. S., Choi, T. W., Kim, T., Goo, J. M., & Park, C. M. (2021) · Radiology · doi:10.1148/radiol.2021210578 · Cited by 35
Govindarajan, A., Govindarajan, A., Tanamala, S., Chattoraj, S., Reddy, B., Agrawal, R., Iyer, D., Srivastava, A., Kumar, P., & Putha, P. (2022) · Diagnostics · doi:10.3390/diagnostics12112724 · Cited by 34
“Sometimes, well-trained radiologists find it challenging to differentiate between the lesions or correctly identify very obscure pulmonary nodules [6].”
Automated image label extraction from radiology reports — A review
Pereira, S. C., Mendonça, A. M., Campilho, A., Sousa, P., & Lopes, C. T. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102814 · Cited by 33
Chest X-ray abnormalities localization via ensemble of deep convolutional neural networks
Pham, V. T., Tran, C.-M., Zheng, S., Minh, V. T., & Nath, S. (2021) · International Conference on Advanced Technologies for Communications · doi:10.1109/atc52653.2021.9598342 · Cited by 33
Pul, U., Tichý, A., Pitchika, V., & Schwendicke, F. (2025) · Journal of Dentistry · doi:10.1016/j.jdent.2025.105868 · Cited by 33
Kuo, P., Tsai, C. C., López, D. M., Karargyris, A., Pollard, T., Johnson, A. E. W., & Celi, L. A. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00393-9 · Cited by 32
Artificial Intelligence for Early Detection of Chest Nodules in X-ray Images
Chiu, H., Peng, R. H.-T., Lin, Y.-C., Wang, T., Yang, Y., Chen, Y., Wu, M., Shiao, T., Chao, H., Chen, Y., et al. (2022) · Biomedicines · doi:10.3390/biomedicines10112839 · Cited by 32
“For example, Majkowska and colleagues used two large datasets containing more than 800,000 images to train an AI model to detect pneumothorax, opacity, nodules, masses, and fractures [29].”
Osteo fracture identification using deep learning techniques
Kishor, K., Sengar, A., Sharma, A. J., & Gautam, D. (2025) · Health Services and Outcomes Research Methodology · doi:10.1007/s10742-025-00340-1 · Cited by 30
Deep-Learning-Based Diagnosis of Bedside Chest X-ray in Intensive Care and Emergency Medicine
Niehues, S. M., Adams, L. C., Gaudin, R., Erxleben, C., Keller, S., Makowski, M. R., Vahldiek, J. L., & Bressem, K. K. (2021) · Investigative Radiology · doi:10.1097/rli.0000000000000771 · Cited by 30
Nguyen, N. H., Nguyen, H. Q., Nguyen, N. T., Nguyễn, T. V., Pham, H., & Nguyen, T. N.-M. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.890759 · Cited by 30
“For example, several deep learning models were developed (3, 5) on the ChestX-ray14 dataset (14) for classifying 14 common thoracic pathologies.”
MethodHofmeister, J., Garin, N., Montet, X., Scheffler, M., Platon, A., Poletti, P., Stirnemann, J., Debray, M., Claessens, Y.-É., Duval, X., et al. (2024) · European Radiology Experimental · doi:10.1186/s41747-023-00416-y · Cited by 30
Hemmer, P., Schellhammer, S., Vössing, M., Jakubik, J., & Satzger, G. (2022) · Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/344 · Cited by 29
Hwang, E. J., Goo, J. M., Nam, J. G., Park, C. M., Hong, K. J., & Kim, K. H. (2023) · Korean Journal of Radiology · doi:10.3348/kjr.2022.0651 · Cited by 28
DeepSDM: Boundary-aware pneumothorax segmentation in chest X-ray images
Wang, Y., Wang, K., Peng, X., Shi, L., Sun, J., Zheng, S., Shan, F., Shi, W., & Liu, L. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.05.029 · Cited by 28
Clinical Artificial Intelligence Applications in Radiology
Lee, S., & Summers, R. M. (2021) · Radiologic Clinics of North America · doi:10.1016/j.rcl.2021.07.001 · Cited by 28
Improving reference standards for validation of AI-based radiography
Duggan, G. E., Reicher, J. J., Liu, Y., Tse, D., & Shetty, S. (2021) · British Journal of Radiology · doi:10.1259/bjr.20210435 · Cited by 27
Self-citationBoMD: Bag of Multi-label Descriptors for Noisy Chest X-ray Classification
Chen, Y., Liu, F., Wang, H., Wang, C., Liu, Y., Tian, Y., & Carneiro, G. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01946 · Cited by 27
“Each image is manually re-labelled by at least three certified radiologists [42] with final label pooled from an adjudication process.”
MethodSelf-citationNVUM: Non-volatile Unbiased Memory for Robust Medical Image Classification
Liu, F., Chen, Y., Tian, Y., Liu, Y., Wang, C., Belagiannis, V., & Carneiro, G. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16437-8_52 · Cited by 26
“Furthermore, the clean test set from [19] is much smaller than OPI and PDC with only two classes available, so we consider results in Tab.”
MethodInfluentialSelf-citationLee, H. W., Jin, K. N., Oh, S., Kang, S., Lee, S. M., Jeong, I. B., Son, J. W., Han, J., Heo, E. Y., Lee, J. G., et al. (2023) · Annals of the American Thoracic Society · doi:10.1513/annalsats.202206-481oc · Cited by 27
Artificial Intelligence in Lung Imaging
Choe, J., Lee, S. M., Hwang, H. J., Yun, J., Kim, N., & Seo, J. B. (2022) · Seminars in Respiratory and Critical Care Medicine · doi:10.1055/s-0042-1755571 · Cited by 26
A systematic approach to deep learning-based nodule detection in chest radiographs
Behrendt, F., Bengs, M., Bhattacharya, D., Krüger, J., Opfer, R., & Schlaefer, A. (2023) · Scientific Reports · doi:10.1038/s41598-023-37270-2 · Cited by 26
The present and future role of artificial intelligence and machine learning in anesthesiology
Alexander, J. C., Romito, B. T., & Çobanoğlu, M. C. (2020) · International Anesthesiology Clinics · doi:10.1097/aia.0000000000000294 · Cited by 25
“26 The attributes of ML algorithms that provide such an advantage in besting humans in games with structured rules and well-de fi ned winning conditions do not carry over to many health care tasks as there is substantial nuance in image analysis and “ winning ” is not an easily de fi ned state.”
BackgroundUllah, N., Khan, J. A., Almakdi, S., Alshehri, M. S., Qathrady, M. A., Anwar, M. S., & Syed, I. (2024) · Biochemistry and Cell Biology · doi:10.1139/bcb-2023-0265 · Cited by 24
AI for Manufacturing and Healthcare: a chemistry and engineering perspectivePreprint
Chen, J., Yuan, Y., Ziabari, A., Xu, X., Zhang, H., Christakopoulos, P., Bonnesen, P. V., Ivanov, I. N., Ganesh, P., Chen, W., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.01520 · Cited by 24
Cho, Y., Kim, J. S., Lim, T. H., Lee, I., & Choi, J. (2021) · Scientific Reports · doi:10.1038/s41598-021-92523-2 · Cited by 23
Mosquera, C., Díaz, F. N., Binder, F., Rabellino, J. M., Benítez, S., Beresñak, A. D., Seehaus, A., Ducrey, G., Ocantos, J., & Luna, D. R. (2021) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2021.106130 · Cited by 22
Levels of Autonomous Radiology
Ghuwalewala, S., Kulkarni, V., Pant, R., & Kharat, A. T. (2022) · Interactive Journal of Medical Research · doi:10.2196/38655 · Cited by 22
“The problem of ambiguous ground truth can be mitigated by using expert adjudication [21] or multiphasic review [22] to create high-quality labels, which may help yield better models than other approaches in improving model performance on original labels [20].”
BackgroundShakya, K. S., Alavi, A., Porteous, J., Priti, K., Laddi, A., & Jaiswal, M. (2024) · Information · doi:10.3390/info15050246 · Cited by 20
Fathi, M., Eshraghi, R., Behzad, S., Tavasol, A., Bahrami, A., Tafazolimoghadam, A., Bhatt, V., Ghadimi, D. J., & Gholamrezanezhad, A. (2024) · Emergency Radiology · doi:10.1007/s10140-024-02278-2 · Cited by 16
“Additionally, in a study by Majkowska et al., a deep learning (DL) model was used to distinguish between acute, subacute, and chronic fracture lesions in radiographs at various locations such as the spine, ribs, shoulder, and clavicle [49].”
Generalized Radiographic View Identification with Deep Learning
Fang, X., Harris, L., Zhou, W., & Huo, D. (2020) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-020-00408-z · Cited by 20
“Deep learning has been demonstrated to be capable of performing image classification tasks [4, 13, 14] at a level of performance comparable with humans.”
BackgroundDual-model approach for accurate chest disease detection using GViT and swin transformer V2
Ahmad, K., Rehman, H. U., Shah, B., Ali, F., & Hussain, I. (2025) · Scientific Reports · doi:10.1038/s41598-025-16422-6 · Cited by 19
Roy, A., & Satija, U. (2023) · IEEE International Symposium on Medical Measurements and Applications (MeMeA) · doi:10.1109/memea57477.2023.10171911 · Cited by 19
Radiological Analysis of COVID-19 Using Computational Intelligence: A Broad Gauge Study
Ligi, S. V., Kundu, S. S., Kumar, R., Narayanamoorthi, R., Lai, K. W., & Dhanalakshmi, S. (2022) · Journal of Healthcare Engineering · doi:10.1155/2022/5998042 · Cited by 19
Kim, H., Lee, K. H., Han, K., Lee, J. W., Kim, J. Y., Im, D. J., Hong, Y. J., Choi, B. W., & Hur, J. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.53820 · Cited by 18
Yu, S., & Zhou, P. (2025) · PLoS ONE · doi:10.1371/journal.pone.0323239 · Cited by 17
“The NIH Chest X-ray dataset [36] is a large publicly available dataset for chest X-ray (CXR) image analysis, primarily used in the research of deep learning models for medical image classification and disease detection.”
Nguyễn, N. H., Pham, H., Tran, T. T., Nguyen, T. N., & Nguyen, H. Q. (2022) · medRxiv · doi:10.1101/2022.03.04.22271937 · Cited by 17
Wollek, A., Willem, T., Ingrisch, M., Sabel, B. O., & Lasser, T. (2023) · Medical Physics · doi:10.1002/mp.16790 · Cited by 17
Efficiency and safety of automated label cleaning on multimodal retinal images
Lin, T., Wang, M., Lin, A., Mai, X., Liang, H., Tham, Y. C., & Chen, H. (2025) · npj Digital Medicine · doi:10.1038/s41746-024-01424-x · Cited by 16
MacPherson, M., Hutchinson, C. E., Horst, C., Goh, V., & Montana, G. (2023) · Radiology Artificial Intelligence · doi:10.1148/ryai.230019 · Cited by 16
K-Diag: Knowledge-enhanced Disease Diagnosis in Radiographic ImagingPreprint
Wu, C., Zhang, X., Wang, Y., Zhang, Y., & Xie, W. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.11557 · Cited by 16
“In this paper, we also construct a dataset by assembling 11 public datasets, including ChestXray-14 [33] GoogleNIH [22] Covid-CXR2 [29] CheXpert [15] Object-CXR [14] NLM-TB [16] RSNA [32] SIIM-ACR [11] VinDR [24] OpenI [7] MIMIC-CXR [19], termed as CXR-Mix .”
MethodAutomatic Diagnosis of Pneumothorax From Chest Radiographs: A Systematic Literature Review
Iqbal, T., Shaukat, A., Akram, U., Mustansar, Z., & Khan, A. (2021) · IEEE Access · doi:10.1109/access.2021.3122998 · Cited by 16
“In [122], deep learning model was trained in order to detect four pathologies including pneumothorax, nodule, airspace opacity and fracture.”
BackgroundMethodKao, C.-Y., Lin, C., Chao, C.-C., Huang, H.-S., Lee, H.-Y., Chang, C., Sung, K., Chen, T., Chiang, P., Huang, L.-T., et al. (2021) · Diagnostics · doi:10.3390/diagnostics11071182 · Cited by 15
“Secondly, the datasets for training, validation, and testing were relatively small compared to other studies [6,8,15,16,20].”
ResultSelf-citationA Novel Deep Learning Model to Distinguish Malignant Versus Benign Solid Lung Nodules
Wang, S., Zhou, L., Li, X., Tang, J., Wu, J., Yin, X., Chen, Y., & Ling-quan, L. (2022) · Medical Science Monitor · doi:10.12659/msm.936830 · Cited by 15
“Lesion segmentation is the most challenging aspect of model evaluation [49,50], and we used expert image annotation, which can be an effective way to optimize the model.”
MethodHuh, J. E., Lee, J. H., Hwang, E. J., & Park, C. M. (2023) · Korean Journal of Radiology · doi:10.3348/kjr.2022.0548 · Cited by 15
“For example: 1) DL models detecting malignant pulmonary nodules, which were set based on pathologically proven lesions [16,19] or partially clinically diagnosed malignancies [14,15]; 2) DL models detecting clinically relevant diseases on chest radiographs based on the disease-specific gold standard [8] or expert inter…”
MethodDiseases Classification Utilizing Tooth X-ray Images Based On Convolutional Neural Network
Deng, L. Y., Ho, S. S., & Lim, X. Y. (2020) · International Symposium on Computer, Consumer and Control · doi:10.1109/is3c50286.2020.00084 · Cited by 15
“In December 2019, the Google Artificial Intelligence team published a model of lung medical images with four categories using Machine Learning(ML), including pneumothorax, nodular mass, fracture, and pneumothorax shadow [3].”
MethodMoving Artificial Intelligence from Feasible to Real: Time to Drill for Gas and Build Roads
Chang, P. J. (2019) · Radiology · doi:10.1148/radiol.2019192527 · Cited by 15
On the Composition and Limitations of Publicly Available COVID-19 X-Ray Imaging DatasetsPreprint
Cruz, B. G. S., Sölter, J., Bossa, M. N., & Husch, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.11572 · Cited by 15
“2 ChestXray-NIH Google 14 [26] re-annotated subset from ChestXray-NIH”
MethodCurrent Status and Future Direction of Artificial Intelligence in Healthcare and Medical Education
Jung, J. S. (2020) · Korean Medical Education Review · doi:10.17496/kmer.2020.22.2.99 · Cited by 14
Irvin, J., Pareek, A., Long, J., Rajpurkar, P., Eng, D. K.-M., Khandwala, N., Haug, P. J., Jephson, A. R., Conner, K. E., Gordon, B. H., et al. (2021) · Journal of Thoracic Imaging · doi:10.1097/rti.0000000000000622 · Cited by 14
A ResNet-Based Audio-Visual Fusion Model for Piano Skill Evaluation
Zhao, X., Wang, Y., & Cai, X. (2023) · Applied Sciences · doi:10.3390/app13137431 · Cited by 13
“segmentation [17,18] and medical image analysis [19,20].”
MIDAS: a new platform for quality-graded health data for AI-enabled healthcare in India
Maity, D., Satish, R., Jadeja, D. A., Dharmaraju, R., Chandru, V., Sundaresan, R., Singh, H., & Pal, D. (2024) · Nature Medicine · doi:10.1038/s41591-024-03198-x · Cited by 13
Misztal, K., Pocha, A., Durak-Kozica, M., Wątor, M., Kubica-Misztal, A., & Hartel, M. (2020) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.104092 · Cited by 13
Schulz, D., Rasch, S., Heilmaier, M., Abbassi, R., Poszler, A., Ulrich, J. D., Steinhardt, M., Kaissis, G. A., Schmid, R. M., Braren, R., et al. (2023) · Critical Care · doi:10.1186/s13054-023-04426-5 · Cited by 13
“For example, Majkowska et al. use a machine learning approach to automatically detect four abnormal findings in X-ray images [17].”
BackgroundArzamasov, K. M., Vasilev, Y. A., Vladzymyrskyy, A. V., Omelyanskaya, O. V., Михайлович, Ш. И., Kozhikhina, D. D., Goncharova, I., Gelezhe, P., Kirpichev, Y., Bobrovskaya, T., et al. (2023) · Healthcare · doi:10.3390/healthcare11121684 · Cited by 13
“Several studies have demonstrated the high diagnostic accuracy of AI algorithms for screening studies such as mammography [13,30–32] and chest X-ray [7,18,33].”
BackgroundSingh, A., Gorade, V., & Mishra, D. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3455337 · Cited by 13
“[24], [25] Most existing SSL methods successfully capture either global or fine-grained features; however, they often struggle to integrate both [26], [27].”
Chen, C.-C., Wu, C., Chen, C. P. C., Chung, C.-Y., Chen, S.-C., Lee, M. S.-S., Cheng, C., & Liao, C. (2023) · JMIR Formative Research · doi:10.2196/42788 · Cited by 12
Predictive performance of radiomic models based on features extracted from pretrained deep networks
Demircioğlu, A. (2022) · Insights into Imaging · doi:10.1186/s13244-022-01328-y · Cited by 12
Investigating the impact of class-dependent label noise in medical image classification
Khanal, B., Hasan, S. M. K., Khanal, B., & Linte, C. A. (2023) · SPIE Medical Imaging: Image Processing · doi:10.1117/12.2654420 · Cited by 12
“The label noise can be more severe when efforts are made to reduce the cost of annotation, such as using non-experts or automated methods to generate labels from medical reports 4,5 .”
MethodLi, X., Zeng, L., Lu, X., Chen, K., Yu, M., Wang, B., & Zhao, M. (2023) · Brain Sciences · doi:10.3390/brainsci13071056 · Cited by 12
“Additionally, it is anticipated to meet the clinician’s demands for greater efficacy and precision when evaluating the rupture probability of cerebral aneurysms [11,12].”
BackgroundLourenço, C. D. S., Tjepkema‐Cloostermans, M. C., & van Putten, M. J. A. M. (2023) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2023.101318 · Cited by 12
Learning to complement with multiple humans
Zhang, Z., Nguyen, C. C., Wells, K., Do, T.-T., & Carneiro, G. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.112376 · Cited by 12
“The multi-rater NIH Chest X-ray dataset [91, 92] contains an average of 3 manual labels per image for four radiographic findings on 4,374 chest X-ray images [91] from the ChestX-ray8 dataset [92].”
MethodThree Reasons Why Artificial Intelligence Might Be the Radiologist’s Best Friend
van Rijn, R. R., & Luca, A. D. (2020) · Radiology · doi:10.1148/radiol.2020200855 · Cited by 12
Keski-Filppula, T., Nikki, M. J., Haapea, M., Ramanauskas, N., & Tervonen, O. A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.08123 · Cited by 12
“Currently, the most promising results against a radiologist are on detecting isolated findings, such as lung nodules, or pneumothoraces [2, 6].”
BackgroundLNDb v4: pulmonary nodule annotation from medical reports
Ferreira, C. A., Sousa, C., Marques, I. D., de Sousa, P. H. P., Ramos, I. M., Coimbra, M. T., & Campilho, A. (2024) · Scientific Data · doi:10.1038/s41597-024-03345-6 · Cited by 12
“Nevertheless, it is uncommon that these differences lead to significant discrepancies in diagnosis 22,23 .”
BackgroundA Dataset for Understanding Radiologist-Artificial Intelligence Collaboration
Moehring, A., Kutwal, M., Huang, R., Banerjee, O., Jacobi, A., Eber, C. D., Mendoza, D. P., Chung, M., Dayan, E., Gupta, Y., et al. (2025) · Scientific Data · doi:10.1038/s41597-025-05054-0 · Cited by 12
An open chest X-ray dataset with benchmarks for automatic radiology report generation in French
Metmer, H., & Yang, X. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128478 · Cited by 11
Augmenting Chest X-ray Datasets with Non-Expert Annotations
Cheplygina, V., Damgaard, C., Eriksen, T. N., Juodelyte, D., & Jiménez-Sánchez, A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98688-8_10 · Cited by 11
“For NIH-CXR14 multiple additional label sets exist, we investigate the following four sets: BBox dataset [29] released alongside NIH-CXR14; two GCS datasets [19, 18] collected as part of two independent studies; and the RSNA set [27] released to explore performance on pneumonia.”
MethodSeedat, N., Imrie, F., & van der Schaar, M. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2403.04551 · Cited by 11
Learning to Defer with Limited Expert Predictions
Hemmer, P., Thede, L., Vössing, M., Jakubik, J., & Kühl, N. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i5.25742 · Cited by 10
Hwang, S. H., Shin, H. J., Kim, E., Lee, E. H., & Lee, M. (2023) · Scientific Reports · doi:10.1038/s41598-023-47194-6 · Cited by 10
Chattoraj, S., Reddy, B., Tadepalli, M., & Putha, P. (2024) · Indian Journal of Tuberculosis · doi:10.1016/j.ijtb.2024.05.008 · Cited by 10
CheXNet: Combing Transformer and CNN for Thorax Disease Diagnosis from Chest X-ray Images
Wu, X., Feng, Y., Xu, H. L., Lin, Z., Li, S., Qiu, S., Liu, Q., & Ma, Y. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-8558-6_7 · Cited by 10
A clinical validation of VinDr-CXR, an AI system for detecting abnormal chest radiographsPreprint
Nguyen, N. H., Nguyen, H. Q., Nguyen, N. T., Nguyễn, T. V., Pham, H., & Nguyen, T. N.-M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.02256 · Cited by 10
“For example, several deep learning models were developed [3,5] on the ChestX-ray14 dataset [13] for classifying 14 common thoracic pathologies.”
MethodImplementation of Artificial Intelligence–Assisted Chest X-ray Interpretation: It Is About Time
Ram, S., & Bodduluri, S. (2023) · Annals of the American Thoracic Society · doi:10.1513/annalsats.202303-195ed · Cited by 10
Assessment of germinal matrix hemorrhage on head ultrasound with deep learning algorithms
Kim, K. Y., Nowrangi, R., McGehee, A., Joshi, N., & Acharya, P. T. (2022) · Pediatric Radiology · doi:10.1007/s00247-021-05239-w · Cited by 9
German CheXpert Chest X-ray Radiology Report Labeler
Wollek, A., Hyska, S., Sedlmeyr, T., Haitzer, P., Rueckel, J., Sabel, B. O., Ingrisch, M., & Lasser, T. (2024) · RöFo - Fortschritte auf dem Gebiet der Röntgenstrahlen und der bildgebenden Verfahren · doi:10.1055/a-2234-8268 · Cited by 9
“Recently, deep learning models used in decision support systems have achieved performance levels in chest X-ray diagnosis of pathologies like pneumonia that are comparable to those of radiologists [6, 7].”
Kromrey, M., Steiner, L., Schön, F., Gamain, J., Roller, C., & Malsch, C. (2024) · Healthcare · doi:10.3390/healthcare12222225 · Cited by 9
Rahman, M. M., Gao, J., Carey, K. A., Edelson, D. P., Afshar, A. S., Garrett, J. W., Chen, G., Afshar, M., & Churpek, M. M. (2025) · JMIR AI · doi:10.2196/67144 · Cited by 9
“Specifically, the radiograph datasets used for pretraining consisted of the following datasets: NIH aka Chest X-ray14 [34], PC aka PadChest [35], CheX aka CheXpert [36], MIMIC-CXR [37], OpenI [38], Google [39], and RSNA Pneumonia Detection Challenge [40].”
Nguyen, T. N., Vo, T. M., Nguyễn, T. V., Pham, H., & Nguyen, H. Q. (2022) · PLoS ONE · doi:10.1371/journal.pone.0276545 · Cited by 9
“The earlier dataset ChestX-ray14 [17], an expansion of ChestX-ray8 [17], including the same set of 14 findings has been used to develop deep learning models [36, 37].”
BackgroundFan, Y.-J., Tzeng, I., Huang, Y., Hsu, Y., Wei, B., Hung, S.-T., & Cheng, Y. (2023) · Biomedicines · doi:10.3390/biomedicines11030760 · Cited by 9
“Previous studies have focused on diseases which require label annotation from radiologists on chest X-rays compared to the result of machine reading [11,12,26].”
ResultLee, K. H., Lee, J. H., Park, S., Jeon, Y. K., Chung, D. H., Kim, Y. T., Goo, J. M., & Kim, H. (2023) · Annals of the American Thoracic Society · doi:10.1513/annalsats.202210-895oc · Cited by 9
Five principles for the intelligent use of AI in medical imaging
Colak, E., Moreland, R., & Ghassemi, M. (2021) · Intensive Care Medicine · doi:10.1007/s00134-020-06316-8 · Cited by 9
Kazemzadeh, S., Yu, J., Jamshy, S., Pilgrim, R., Nabulsi, Z., Chen, C., Beladia, N., Lau, C. T., McKinney, S. M., Hughes, T., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.07540 · Cited by 9
Self-citationDao, L., Harmouch, S. S., Chin, A. S.-L., Dao, T., Thian, Z., Chartrand‐Lefebvre, C., & Cohen, J. (2022) · medRxiv · doi:10.1101/2022.07.11.22277278 · Cited by 9
“…discrepancy between the radiologist’s, clinician’s, and automatic labeller’s understanding of a radiology report (35); errors in labelling due in part to automatic labellers (36,37); framing bias or limitations in objectivity (33,38,39); and interobserver variability, (40) to only name a few.”
BackgroundLee, S. B. (2025) · Scientific Reports · doi:10.1038/s41598-025-02294-3 · Cited by 7
Liao, H.-C., Lin, C. S., Wang, C., & Fang, W. (2023) · Digital Health · doi:10.1177/20552076231191055 · Cited by 8
Vontetsianos, A., Güven, D. K., Betka, S., Souto‐Miranda, S., Marillier, M., Price, O. J., Hui, C. Y., Sivapalan, P., Jácome, C. I. O., Aliverti, A., et al. (2023) · ERJ Open Research · doi:10.1183/23120541.00194-2023 · Cited by 8
Was there COVID-19 back in 2012? Challenge for AI in Diagnosis with Similar IndicationsPreprint
Banerjee, I., Sinha, P., Purkayastha, S., Mashhaditafreshi, N., Tariq, A., Jeong, J., Trivedi, H., & Gichoya, J. W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.13262 · Cited by 8
Explainable Artificial Intelligence Technique in Deep Learning–Based Medical Image Analysis
Gupta, B., Malviya, R., Sundram, S., & Sridhar, S. B. (2025) · Wiley eBooks · doi:10.1002/9781394302444.ch7 · Cited by 7
Wu, Y., Fong, S. J., & Liu, L. (2025) · Soft Computing · doi:10.1007/s00500-025-10644-6 · Cited by 7
“Majkowska et al. (2019) introduced deep learning models for interpreting chest radiographs and validated them using XAI techniques.”
A 178-clinical-center experiment of integrating AI solutions for lung pathology diagnosis
Ibragimov, B., Arzamasov, K. M., Maksudov, B., Kiselev, S., Mongolin, A., Mustafaev, T., Ibragimova, D., Evteeva, K., Андрейченко, А. Е., & Morozov, S. P. (2023) · Scientific Reports · doi:10.1038/s41598-023-27397-7 · Cited by 7
Yoon, M. S., Kwon, G., Oh, J., Ryu, J., Lim, J., Kang, B.-K., Lee, J., & Han, D.-K. (2023) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-022-00772-y · Cited by 7
“Majkowska et al. reported that the performance of pneumothorax detection among four classifications was 0.940 ~ 0.950 of AUC in a test with National Institutes of Health (NIH) ChestX-ray14 after training with 1818 images from one center [15].”
MethodInfluential
Metadata
- Authors
- Majkowska, A., Mittal, S., Steiner, D. F., Reicher, J. J., McKinney, S. M., Duggan, G. E., Eswaran, K., Cameron Chen, P.-H., Liu, Y., Kalidindi, S. R., Ding, A., Corrado, G. S., Tse, D., & Shetty, S.
- Affiliations
- Stanford Health Care (US)
- From Google Health, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (A.M., S.M., D.F.S., S.M.M., G.E.D., K.E., P.H.C.C., Y.L., G.S.C., D.T., S.S.); Stanford Healthcare and Palo Alto Veterans Affairs, Palo Alto, Calif (J.J.R.); Apollo Radiology International, Hyderabad, India (S.R.K.); and California Advanced Imaging, Novato, Calif (A.D.).
- ORCID
- Venue
- Radiology
- Published
- 2020-02
- Volume
- 294
- Issue
- 2
- Pages
- 421-431
- ISSN
- 0033-8419 (ISSN-L)
- 1527-1315
- OpenAlex
- W2991676991
- Cited by
- 277
- References
- 30
- Citations by year
- 2026: 21 · 2025: 37 · 2024: 36 · 2023: 48 · 2022: 48 · 2021: 52 · 2020: 34
- FWCI
- 22.28
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Radiology (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- chest X-ray interpretation, deep learning, pneumothorax detection, lung nodule detection, lung opacity, rib fracture detection, inverse probability weighting, spectrum bias, generalizability, ChestX-ray14, positive predictive value, sensitivity and specificity
- MeSH terms
- Radiologists, Deep Learning, Adolescent, Adult, Aged, Aged, 80 and over, Child, Child, Preschool, Female, Humans, Infant, Male, Middle Aged, Pneumothorax, Radiographic Image Interpretation, Computer-Assisted, Reference Standards, Respiratory Tract Diseases, Retrospective Studies, Sensitivity and Specificity, Thoracic Injuries, Radiography, Thoracic, Reproducibility of Results, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.