Deep Learning for Chest Radiograph Diagnosis in the Emergency Department
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
Background The performance of a deep learning (DL) algorithm should be validated in actual clinical situations, before its clinical implementation. Purpose To evaluate the performance of a DL algorithm for identifying chest radiographs with clinically relevant abnormalities in the emergency department (ED) setting. Materials and Methods This single-center retrospective study included consecutive patients who visited the ED and underwent initial chest radiography between January 1 and March 31, 2017. Chest radiographs were analyzed with a commercially available DL algorithm. The performance of the algorithm was evaluated by determining the area under the receiver operating characteristic curve (AUC), sensitivity, and specificity at predefined operating cutoffs (high-sensitivity and high-specificity cutoffs). The sensitivities and specificities of the algorithm were compared with those of the on-call radiology residents who interpreted the chest radiographs in the actual practice by using McNemar tests. If there were discordant findings between the algorithm and resident, the residents reinterpreted the chest radiographs by using the algorithm’s output. Results A total of 1135 patients (mean age, 53 years ± 18; 582 men) were evaluated. In the identification of abnormal chest radiographs, the algorithm showed an AUC of 0.95 (95% confidence interval [CI]: 0.93, 0.96), a sensitivity of 88.7% (227 of 256 radiographs; 95% CI: 84.1%, 92.3%), and a specificity of 69.6% (612 of 879 radiographs; 95% CI: 66.5%, 72.7%) at the high-sensitivity cutoff and a sensitivity of 81.6% (209 of 256 radiographs; 95% CI: 76.3%, 86.2%) and specificity of 90.3% (794 of 879 radiographs; 95% CI: 88.2%, 92.2%) at the high-specificity cutoff. Radiology residents showed lower sensitivity (65.6% [168 of 256 radiographs; 95% CI: 59.5%, 71.4%], P < .001) and higher specificity (98.1% [862 of 879 radiographs; 95% CI: 96.9%, 98.9%], P < .001) compared with the algorithm. After reinterpretation of chest radiographs with use of the algorithm’s outputs, the sensitivity of the residents improved (73.4% [188 of 256 radiographs; 95% CI: 68.0%, 78.8%], P = .003), whereas specificity was reduced (94.3% [829 of 879 radiographs; 95% CI: 92.8%, 95.8%], P < .001). Conclusion A deep learning algorithm used with emergency department chest radiographs showed diagnostic performance for identifying clinically relevant abnormalities and helped improve the sensitivity of radiology residents’ evaluation. Published under a CC BY 4.0 license. Online supplemental material is available for this article. See also the editorial by Munera and Infante in this issue.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“13 (refs. 63–75) studies validated algorithms on external data”
From Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis (2021) · cites this paper as [65]
In the citing paper
“Twenty-one54,63–67,70,72,76–88 studies compared algorithm performance against healthcare professionals.”
From Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis (2021) · cites this paper as [65]
Citations
Cites
DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988) · Biometrics · doi:10.2307/2531595 · Cited by 22,649
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697
Ehteshami Bejnordi, B., Veta, M., Johannes van Diest, P., van Ginneken, B., Karssemeijer, N., Litjens, G., van der Laak, J. A. W. M., and the CAMELYON16 Consortium, Hermsen, M., Manson, Q. F., et al. (2017) · JAMA · doi:10.1001/jama.2017.14585 · Cited by 3,293
Ting, D. S. W., Cheung, C. Y.-L., Lim, G., Tan, G. S. W., Quang, N. D., Gan, A., Hamzah, H., Garcia-Franco, R., San Yeo, I. Y., Lee, S. Y., et al. (2017) · JAMA · doi:10.1001/jama.2017.18152 · Cited by 2,402
Park, S. H., & Han, K. (2018) · Radiology · doi:10.1148/radiol.2017171920 · Cited by 894
Self-citationNam, J. G., Park, S., Hwang, E. J., Lee, J. H., Jin, K.-N., Lim, K. Y., Vu, T. H., Sohn, J. H., Hwang, S., Goo, J. M., et al. (2019) · Radiology · doi:10.1148/radiol.2018180237 · Cited by 539
Self-citationHwang, E. J., Park, S., Jin, K.-N., Kim, J. I., Choi, S. Y., Lee, J. H., Goo, J. M., Aum, J., Yim, J.-J., Cohen, J. G., et al. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.1095 · Cited by 447
Self-citationComparisons of Predictive Values of Binary Medical Diagnostic Tests for Paired Designs
Leisenring, W., Alono, T., & Pepe, M. S. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00345.x · Cited by 355
Hwang, E. J., Park, S., Jin, K.-N., Kim, J. I., Choi, S. Y., Lee, J. H., Goo, J. M., Aum, J., Yim, J.-J., Park, C. M., et al. (2019) · Clinical Infectious Diseases · doi:10.1093/cid/ciy967 · Cited by 241
Self-citationEng, J., Mysko, W. K., Weller, G. E. R., Renard, R., Gitlin, J. N., Bluemke, D. A., Magid, D., Kelen, G. D., & Scott, W. W. (2000) · American Journal of Roentgenology · doi:10.2214/ajr.175.5.1751233 · Cited by 124
Cited by
Preparing Medical Imaging Data for Machine Learning
Willemink, M. J., Koszek, W. A., Hardell, C., Wu, J., Fleischmann, D., Harvey, H., Folio, L. R., Summers, R. M., Rubin, D. L., & Lungren, M. P. (2020) · Radiology · doi:10.1148/radiol.2020192224 · Cited by 1,020
Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis
Aggarwal, R., Sounderajah, V., Martin, G., Ting, D. S. W., Karthikesalingam, A., King, D., Ashrafian, H., & Darzi, A. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00438-z · Cited by 946
Deep learning for chest X-ray analysis: A survey
Çallı, E., Sogancioglu, E., van Ginneken, B., van Leeuwen, K. G., & Murphy, K. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102125 · Cited by 506
Seah, J. C. Y., Tang, C. H. M., Buchlak, Q. D., Holt, X. G., Wardman, J. B., Aimoldin, A., Esmaili, N., Ahmad, H., Pham, H., Lambert, J. F., et al. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00106-0 · Cited by 229
COVID-19 on Chest Radiographs: A Multireader Evaluation of an Artificial Intelligence System
Murphy, K., Smits, H., Knoops, A. J. G., Korst, M. B. J. M., Samson, T., Scholten, E. T., Schalekamp, S., Schaefer-Prokop, C. M., Philipsen, R. H. H. M., Meijers, A., et al. (2020) · Radiology · doi:10.1148/radiol.2020201874 · Cited by 214
“An artificial intelligence (AI) system may be a helpful tool as an adjunct for radiologists or, in the common case that radiologic expertise is not available, for the medical team (7,8).”
BackgroundVentilator-associated pneumonia: pathobiological heterogeneity and diagnostic challenges
Howroyd, F., Chacko, C., MacDuff, A., Gautam, N., Pouchet, B., Tunnicliffe, B., Weblin, J., Gao-Smith, F., Ahmed, Z., Duggal, N. A., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-50805-z · Cited by 162
“AI canpotentially expedite and enhanceCT or CXR interpretation and aid decision-making without microbial sampling, yet further validation is required in clinical practice models 152 – 156 .”
BackgroundArtificial intelligence in emergency medicine: A scoping review
Kirubarajan, A., Taher, A., Khan, S., & Masood, S. (2020) · JACEP Open · doi:10.1002/emp2.12277 · Cited by 127
Yoo, H., Kim, K. H., Singh, R., Digumarthy, S. R., & Kalra, M. K. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.17135 · Cited by 122
Self-citationA Review of Recent Advances in Deep Learning Models for Chest Disease Detection Using Radiography
Ait Nasser, A., & Akhloufi, M. A. (2023) · Diagnostics · doi:10.3390/diagnostics13010159 · Cited by 121
Nam, J. G., Kim, M., Park, J., Hwang, E. J., Lee, J. H., Hong, J. H., Goo, J. M., & Park, C. M. (2021) · European Respiratory Journal · doi:10.1183/13993003.03061-2020 · Cited by 103
“Among the 1455 chest radiographs from 1178 patients, 202 chest radiographs from 202 patients (95 males and 107 females; mean± SD age 57.6±17.9 years) were selected to match the previously reported disease prevalence of patients visiting the emergency department [24].”
ResultInfluentialSelf-citationApplications of artificial intelligence and machine learning in respiratory medicine
Gonem, S., Janssens, W., Das, N., & Topalovic, M. (2020) · Thorax · doi:10.1136/thoraxjnl-2020-214556 · Cited by 99
“29 However, the DNN was not trained to interpret radiographs with multiple pathologies, nor to interpret images in the context of background clinical information.”
A scalable physician-level deep learning algorithm detects universal trauma on pelvic radiographs
Cheng, C., Wang, Y., Chen, H., Hsiao, P.-M., Yeh, C., Hsieh, C., Miao, S., Xiao, J. Z., Liao, C., & Lü, L. (2021) · Nature Communications · doi:10.1038/s41467-021-21311-3 · Cited by 113
Using AI to Improve Radiologist Performance in Detection of Abnormalities on Chest Radiographs
Bennani, S., Regnard, N.-E., Ventre, J., Lassalle, L., Nguyen, T., Ducarouge, A., Dargent, L., Guillo, E., Gouhier, E., Zaimi, S.-H., et al. (2023) · Radiology · doi:10.1148/radiol.230860 · Cited by 120
Autonomous Chest Radiograph Reporting Using AI: Estimation of Clinical Impact
Plesner, L. L., Müller, F. C., Nybing, J. D., Laustrup, L. C., Rasmussen, F., Nielsen, O. W., Boesen, M., & Andersen, M. B. (2023) · Radiology · doi:10.1148/radiol.222268 · Cited by 95
Deep learning applications in pulmonary medical imaging: recent updates and insights on COVID-19
Farhat, H. J., Sakr, G. E., & Kilany, R. (2020) · Machine Vision and Applications · doi:10.1007/s00138-020-01101-5 · Cited by 93
“Many clinical applications of deep learning-based models have started to take their place, as that by Hwang et al. [241], and Foch Hospital in Suresnes, France [242].”
BackgroundHwang, E. J., & Park, C. M. (2020) · Korean Journal of Radiology · doi:10.3348/kjr.2019.0821 · Cited by 71
“Indeed, in our recent study (57), a DL algorithm to identify CXRs with major thoracic disease showed a decreased performance in a diagnostic cohort comprising patients from the emergency department (AUC, 0.”
MethodSelf-citationAI in radiology: From promise to practice − A guide to effective integration
Katal, S., York, B. R., & Gholamrezanezhad, A. (2024) · European Journal of Radiology · doi:10.1016/j.ejrad.2024.111798 · Cited by 83
Anderson, P. G., Tarder-Stoll, H., Alpaslan, M., Keathley, N., Levin, D. L., Venkatesh, S., Bartel, E., Sicular, S., Howell, S., Lindsey, R. V., et al. (2024) · Scientific Reports · doi:10.1038/s41598-024-76608-2 · Cited by 101
Artificial Intelligence-Based Detection of Pneumonia in Chest Radiographs
Becker, J., Decker, J. A., Römmele, C., Kahn, M., Messmann, H., Wehler, M., Schwarz, F., Kroencke, T. J., & Scheurig‐Muenkler, C. (2022) · Diagnostics · doi:10.3390/diagnostics12061465 · Cited by 82
“Other studies have proved that this program’s performance in detecting the aforementioned imaging patterns is excellent [11,31,32].”
BackgroundDeep learning applied to automatic disease detection using chest X‐rays
Moses, D. A. (2021) · Journal of Medical Imaging and Radiation Oncology · doi:10.1111/1754-9485.13273 · Cited by 74
Salloch, S., & Eriksen, A. (2024) · The American Journal of Bioethics · doi:10.1080/15265161.2024.2353800 · Cited by 51
“Radiology (Hwang et al. 2019), pathology (Kiani et al. 2020; Bulten et al. 2022) and ophthalmology (Gulshan et al. 2016; Ting et al. 2017) are among the pioneering disciplines in the introduction of ML_CDSS because they deal with large quantities of visual data.”
Nabulsi, Z., Sellergren, A., Jamshy, S., Lau, C. T., Santos, E. R. G., Kiraly, A. P., Ye, W., Yang, J., Pilgrim, R., Kazemzadeh, S., et al. (2021) · Scientific Reports · doi:10.1038/s41598-021-93967-2 · Cited by 74
Ullah, N., Khan, J. A., Almakdi, S., Khan, M. S., Alshehri, M., Alboaneen, D., & Raza, A. (2022) · Applied Sciences · doi:10.3390/app12126269 · Cited by 53
“When experienced radiologists are unavailable, these computer-aided detection (CADs) systems can help with imaging-based patient classification in resource-constrained situations and increase practitioner’s chest radiograph interpretation accuracy and inter-reader variability [28,29].”
BackgroundDeep Learning-Assisted Diagnosis of Pediatric Skull Fractures on Plain Radiographs
Choi, J. W., Cho, Y. J., Ha, J. Y., Lee, Y. Y., Koh, S. Y., Seo, J. Y., Choi, Y. H., Cheon, J., Phi, J. H., Kim, I., et al. (2022) · Korean Journal of Radiology · doi:10.3348/kjr.2021.0449 · Cited by 57
“Several previous studies have suggested the potential role of a deep learning model as a second reader for inexperienced radiologists or physicians [38,39].”
Hwang, E. J., Kim, H., Yoon, S. H., Goo, J. M., & Park, C. M. (2020) · Korean Journal of Radiology · doi:10.3348/kjr.2020.0536 · Cited by 49
“4%) after review of the CAD results (28).”
BackgroundSelf-citationFan, W., Yang, Y., Qi, J., Zhang, Q., Liao, C., Li, W., Wang, S., Wang, G., Yu, X., Wu, Q., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-45599-z · Cited by 64
“A few studies 12,13 have tried to localize abnormalities with the heatmaps generated by deep classi fi cation models.”
BackgroundArtificial Intelligence-Based Diagnosis of Cardiac and Related Diseases
Arsalan, M., Owais, M., Mahmood, T., Choi, J., & Park, K. R. (2020) · Journal of Clinical Medicine · doi:10.3390/jcm9030871 · Cited by 52
Hwang, E. J., Kim, K. B., Kim, J. Y., Lim, J., Nam, J. G., Choi, H., Kim, H., Yoon, S. H., Goo, J. M., & Park, C. M. (2021) · PLoS ONE · doi:10.1371/journal.pone.0252440 · Cited by 51
“These computer-aided detection systems (CADs) may help imaging-based triage of patients in resource-constrained environments where expert radiologists are not available, and improve accuracy and inter-reader variability of physicians’ CXR interpretations [25, 26].”
BackgroundSelf-citationValidation study of machine-learning chest radiograph software in primary and emergency medicine
van Beek, E. J. R., Ahn, J. S., Kim, M. J., & Murchison, J. T. (2022) · Clinical Radiology · doi:10.1016/j.crad.2022.08.129 · Cited by 51
Hong, W., Hwang, E. J., Lee, J. H., Park, J., Goo, J. M., & Park, C. M. (2022) · Radiology · doi:10.1148/radiol.211706 · Cited by 49
Self-citationCheXGAT: A disease correlation-aware network for thorax disease diagnosis from chest X-ray images
Lee, Y., Huang, S.-K., & Chang, R. (2022) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2022.102382 · Cited by 50
Jones, C. M., Danaher, L., Milne, M. R., Tang, C., Seah, J., Oakden-Rayner, L., Johnson, A., Buchlak, Q. D., & Esmaili, N. (2021) · BMJ Open · doi:10.1136/bmjopen-2021-052902 · Cited by 45
Wang, H., Gu, H., Qin, P., & Wang, J. (2020) · PLoS ONE · doi:10.1371/journal.pone.0242013 · Cited by 49
“The deep learning aided quick diagnose can also be applied in the emergency department [35].”
BackgroundKim, J. H., Kim, J. Y., Kim, G. H., Kang, D., Kim, I. J., Seo, J., Andrews, J. R., & Park, C. M. (2020) · Journal of Clinical Medicine · doi:10.3390/jcm9061981 · Cited by 42
Self-citationChest radiographs and machine learning – Past, present and future
Jones, C. M., Buchlak, Q. D., Oakden‐Rayner, L., Milne, M., Seah, J. C. Y., Esmaili, N., & Hachey, B. (2021) · Journal of Medical Imaging and Radiation Oncology · doi:10.1111/1754-9485.13274 · Cited by 58
“One study assessed a machine learning model used to assist radiology residents, demonstrating improved performance in interpreting chest radiographs obtained in the emergency department setting across a range of findings.(29) Most recently, the model developed by Seah et al.”
BackgroundDoherty, G., McLaughlin, L., Hughes, C., McConnell, J., Bond, R., & McFadden, S. (2024) · Radiography · doi:10.1016/j.radi.2023.12.019 · Cited by 38
Kim, J. H., Han, S. G., Cho, A., Shin, H. J., & Baek, S.-E. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01679-4 · Cited by 38
“Moreover, chest radiography tends to be performed to assess whether further imaging studies are required to confirm the diagnosis in the ED [17]; as such, using DLCR, which helps screen for abnormalities rather than establish a diagnosis, is practical.”
BackgroundResultJin, K. N., Kim, E. Y., Kim, Y. J., Lee, G. P., Kim, H., Oh, S., Kim, Y., Han, J., & Cho, Y. J. (2022) · European Radiology · doi:10.1007/s00330-021-08397-5 · Cited by 37
“For the analysis of sensitivity and specificity, the threshold for the output of the AI solution was defined as 15%; the value had been validated by Lunit’s variable internal datasets and previous literature [11, 14].”
MethodInfluentialJafar, A., Hameed, M. T., Akram, N., Waqas, U., Kim, H. S., & Naqvi, R. A. (2022) · Journal of Personalized Medicine · doi:10.3390/jpm12060988 · Cited by 37
“…groups the similar features. images are Medical images are complex, and this pixel-based semantic segmentation help with efficiently locating the infected areas in an image [12]. complex, and this pixel-based semantic segmentation help with efficiently locating the infected areas in an image [12].”
BackgroundAn adversarial approach for the robust classification of pneumonia from chest radiographs
Janizek, J. D., Erion, G., DeGrave, A. J., & Lee, S. (2020) · ACM Conference on Health, Inference, and Learning (CHIL) · doi:10.1145/3368555.3384458 · Cited by 46
“View is additionally an important confounder to control because commerically-available chest radiograph algorithms are currently designed to accept both AP and PA view radiographs as input [12].”
BackgroundMethodKim, C., Yang, Z., Park, S. H., Hwang, S. H., Oh, Y., Kang, E.-Y., & Yong, H. S. (2023) · European Radiology · doi:10.1007/s00330-022-09315-z · Cited by 38
“There have been studies in the form of diagnostic cohorts for validation of the Lunit INSIGHT for CXR [16, 18, 25].”
MethodSelf-citationGipson, J., Tang, V. D., Seah, J. C. Y., Kavnoudias, H., Zia, A., Lee, R., Mitra, B., & Clements, W. (2022) · British Journal of Radiology · doi:10.1259/bjr.20210979 · Cited by 38
“Hwang et al 13 previously assessed an algorithm which could classify multiple thoracic diseases, however it appears each radiograph only had one pathology.”
BackgroundDiagnostic performance of artificial intelligence model for pneumonia from chest radiography
Kwon, T., Lee, S. P., Kim, D., Jang, J., Lee, M., Kang, S. U., Kim, H. J., Oh, K.-Y., On, J., Kim, Y. J., et al. (2021) · PLoS ONE · doi:10.1371/journal.pone.0249399 · Cited by 37
Self-citationKobayashi, M., Ishioka, J., Matsuoka, Y., Fukuda, Y., Kohno, Y., Kawano, K., Morimoto, S., Muta, R., Fujiwara, M., Kawamura, N., et al. (2021) · BMC Urology · doi:10.1186/s12894-021-00874-9 · Cited by 32
Deep Learning Analysis of Chest Radiographs to Triage Patients with Acute Chest Pain Syndrome
Kolossváry, M., Raghu, V. K., Nagurney, J. T., Hoffmann, U., & Lu, M. T. (2023) · Radiology · doi:10.1148/radiol.221926 · Cited by 38
Buchlak, Q. D., Tang, C. H. M., Seah, J. C. Y., Johnson, A., Holt, X., Bottrell, G., Wardman, J. B., Samarasinghe, G., Pinheiro, L., Xia, H., et al. (2023) · European Radiology · doi:10.1007/s00330-023-10074-8 · Cited by 37
“CNNs have been applied successfully to many domains of medicine [8, 9] and have demonstrated strong image classification performance in radiology [10, 11].”
BackgroundKim, Y., Park, J. Y., Hwang, E. J., Lee, S. M., & Park, C. M. (2021) · Journal of Thoracic Disease · doi:10.21037/jtd-21-1342 · Cited by 31
“21037/jtd-21-1342 thoracic diseases on chest radiographs (CXR) and computed tomography (CT) (9-27).”
BackgroundSelf-citationUse of artificial intelligence in triaging of chest radiographs to reduce radiologists’ workload
Yoon, S. H., Park, S.-Y., Jang, S., Kim, J., Lee, K. W., Lee, W., Lee, S., Yun, G., & Lee, K. H. (2023) · European Radiology · doi:10.1007/s00330-023-10124-1 · Cited by 35
“The combination of these abnormal patterns enables the algorithms to cover most of the clinically relevant diseases on chest radiographs and identify normal radiographs that are unlikely to contain abnormal findings [7, 8].”
Self-citationPerformance of AI to exclude normal chest radiographs to reduce radiologists' Workload
Schalekamp, S., van Leeuwen, K., Calli, E., Murphy, K., Rutten, M., Geurts, B., Peters-Bax, L., van Ginneken, B., & Prokop, M. (2024) · European Radiology · doi:10.1007/s00330-024-10794-5 · Cited by 28
Review on chest pathogies detection systems using deep learning techniques
Rehman, A., Khan, A., Fatima, G., Naz, S., & Razzak, I. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10457-9 · Cited by 32
Hwang, E. J., Goo, J. M., Nam, J. G., Park, C. M., Hong, K. J., & Kim, K. H. (2023) · Korean Journal of Radiology · doi:10.3348/kjr.2022.0651 · Cited by 28
“2-4) [11-19].”
InfluentialSelf-citationDeep-Learning-Based Diagnosis of Bedside Chest X-ray in Intensive Care and Emergency Medicine
Niehues, S. M., Adams, L. C., Gaudin, R., Erxleben, C., Keller, S., Makowski, M. R., Vahldiek, J. L., & Bressem, K. K. (2021) · Investigative Radiology · doi:10.1097/rli.0000000000000771 · Cited by 30
Yoo, H., Kim, E. Y., Kim, H., Choi, Y. R., Kim, M. Y., Hwang, S. H., Kim, Y. J., Cho, Y. J., & Jin, K. N. (2022) · Korean Journal of Radiology · doi:10.3348/kjr.2022.0189 · Cited by 27
Artificial intelligence decision points in an emergency department
Chang, H., & Cha, W. C. (2022) · Clinical and Experimental Emergency Medicine · doi:10.15441/ceem.22.366 · Cited by 26
Clinical Artificial Intelligence Applications in Radiology
Lee, S., & Summers, R. M. (2021) · Radiologic Clinics of North America · doi:10.1016/j.rcl.2021.07.001 · Cited by 28
Miró Catalina, Q., Vidal-Alaball, J., Fuster-Casanovas, A., Escalé-Besa, A., Ruiz Comellas, A., & Solé-Casals, J. (2024) · Scientific Reports · doi:10.1038/s41598-024-55792-1 · Cited by 23
“These findings align with the results of a study that performed an external validation of a similar algorithm in an emergency department 35 .”
ResultUllah, N., Khan, J. A., Almakdi, S., Alshehri, M. S., Qathrady, M. A., Anwar, M. S., & Syed, I. (2024) · Biochemistry and Cell Biology · doi:10.1139/bcb-2023-0265 · Cited by 24
Lobo, M. D. (2023) · Advances in medical education, research, and ethics (AMERE) book series · doi:10.4018/978-1-6684-7164-7.ch004 · Cited by 23
Hwang, E. J., Park, J., Hong, W., Lee, H., Choi, H., Kim, H., Nam, J. G., Goo, J. M., Yoon, S. H., Lee, C. H., et al. (2022) · European Radiology · doi:10.1007/s00330-022-08593-x · Cited by 23
Self-citationJang, S. B., Lee, S. H., Lee, D. E., Park, S.-Y., Kim, J. K., Cho, J. W., Cho, J., Kim, K. B., Park, B., Park, J., et al. (2020) · PLoS ONE · doi:10.1371/journal.pone.0242759 · Cited by 23
Self-citationZhang, Y., Liu, M., Zhang, L., Wang, L., Zhao, K., Hu, S., Chen, X., & Xie, X. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.55113 · Cited by 20
Choi, S. Y., Park, S., Kim, M., Park, J., Choi, Y. R., & Jin, K. N. (2021) · Medicine · doi:10.1097/md.0000000000025663 · Cited by 27
“…markers, physicians can examine patients suspicious of serious illnesses fi rst. Additionally, the ef fi cacy of our algorithm as the primary reader in the clinical setting has been validated by the recent research of Hwang et al, [23] which tested the same deep learning algorithm used in our study.”
MethodSelf-citationDyer, T., Chawda, S. J., Alkilani, R., Morgan, T. N., Hughes, M., & Rasalingham, S. (2021) · Neuroradiology · doi:10.1007/s00234-021-02826-4 · Cited by 21
Kim, N. J., Oh, J., Kim, S., Cha, J., Choi, J., Im, S. G., Choi, S., & Jang, B. C. (2024) · Advanced Functional Materials · doi:10.1002/adfm.202305136 · Cited by 21
Fernández‐Miranda, P. M., Fraguela, E. M., de Linera-Alperi, M. Á., Cobo, M., del Barrio, A. P., Rodriguez, D., Vega, J. A., & Iglesias, L. L. (2024) · Scientific Reports · doi:10.1038/s41598-024-64941-5 · Cited by 23
Fanni, S. C., Marcucci, A., Volpi, F., Valentino, S., Neri, E., & Romei, C. (2023) · Diagnostics · doi:10.3390/diagnostics13122020 · Cited by 25
“[51].”
MultiCOVID: a multi modal deep learning approach for COVID-19 diagnosis
Hardy-Werbin, M., Maiques, J. M., Busto, M., Cirera, I., Aguirre, A., García-Gisbert, N., Zuccarino, F., Carbullanca, S., Bellido, L. A. D. C., Ramal, D., et al. (2023) · Scientific Reports · doi:10.1038/s41598-023-46126-8 · Cited by 19
“Moreover, although a huge amount of literature about COVID-19 diagnosis and prognosis has been published using only blood tests 17–20 or CXR 21–28 this is the first study that combines both parameters and compares its prediction capacity at diagnosis.”
BackgroundArtificial Intelligence in Emergency Trauma Care: A Preliminary Scoping Review
Ventura, C., Denton, E. E., & David, J. A. (2024) · Medical Devices Evidence and Research · doi:10.2147/mder.s467146 · Cited by 29
Cho, S. Y., Lee, J. H., Ryu, J. M., Lee, J. E., Cho, E. Y., Ahn, C. H., Paeng, K., Yoo, I., Ock, C., & Song, S. Y. (2021) · Scientific Reports · doi:10.1038/s41598-021-96855-x · Cited by 18
Kim, U.-H., Kim, M. Y., Park, E., Lee, W., Lim, W., Kim, H., Oh, S., & Jin, K. N. (2021) · Korean Journal of Radiology · doi:10.3348/kjr.2021.0201 · Cited by 16
“The main reason for this could be that the model group identification had performed an external test for an independent imbalanced dataset (Supplementary Table 1) with multiclass classification for “26 subgroups” compared with the other two tasks [28-30].”
MethodSelf-citationShinohara, Y., Takahashi, N., Lee, Y., Ohmura, T., Umetsu, A., Kinoshita, F., Kuya, K., Kato, A., & Kinoshita, T. (2020) · Japanese Journal of Radiology · doi:10.1007/s11604-020-00986-6 · Cited by 18
Development and operation of a digital platform for sharing pathology image data
Kang, Y., Kim, Y. J., Park, S., Ro, G., Hong, C., Jang, H., Cho, S., Hong, W., Kang, D. U., Chun, J., et al. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01466-1 · Cited by 19
Self-citationWen, R., Xu, P., Cai, Y., Wang, F., Li, M., Zeng, X., & Liu, C. (2023) · Infection and Drug Resistance · doi:10.2147/idr.s404786 · Cited by 17
“Compared with this algorithm, radiology residents showed lower sensitivity but higher specificity.(16) However, most of the current studies are based on the differential diagnosis between viral pneumonia, especially COVID-19, and other types of pneumonia, such as interstitial pneumonia and community-acquired pneumonia.”
ResultArzamasov, K. M., Vasilev, Y. A., Zelenova, M. A., Дмитриевич, П. Л., Busygina, Y., Bobrovskaya, T., Четвериков, С. Ф., Shikhmuradov, D. U., Pankratov, A., Kirpichev, Y., et al. (2024) · Quantitative Imaging in Medicine and Surgery · doi:10.21037/qims-24-160 · Cited by 16
Multi-Label Generalized Zero Shot Learning for the Classification of Disease in Chest RadiographsPreprint
Hayat, N., Lashen, H., & Shamout, F. E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.06563 · Cited by 29
“Deep learning has enabled the development of computer aided diagnostic systems that can classify diseases in medical images with human level precision (Qin et al., 2018; Rajpurkar et al., 2018; Hwang et al., 2019).”
BackgroundHwang, E. J., Kim, H., Lee, J. H., Goo, J. M., & Park, C. M. (2020) · European Radiology · doi:10.1007/s00330-020-07062-7 · Cited by 15
“The study population (1135/1135) was reported previously [19].”
MethodResultInfluentialSelf-citationImage Quality Control in Lumbar Spine Radiography Using Enhanced U-Net Neural Networks
Chen, X., Deng, Q., Wang, Q., Liu, X., Chen, L., Liu, J., Li, S., Wang, M., & Cao, G. (2022) · Frontiers in Public Health · doi:10.3389/fpubh.2022.891766 · Cited by 16
“For example, in Hwang’s and Annarumma’s studies (31, 32), deep learning algorithms were used to classify the urgency of abnormal chest radiographs, so that abnormal cases could seek expert opinions as soon as possible.”
MethodAlcaraz, J. M. L., Bouma, H., & Strodthoff, N. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110196 · Cited by 13
Bettinger, H., Lenczner, G., Guigui, J., Rotenberg, L., Zerbib, E., Attia, A., Vidal, J., & Beaumel, P. (2024) · Diagnostics · doi:10.3390/diagnostics14111183 · Cited by 14
Current Status and Future Direction of Artificial Intelligence in Healthcare and Medical Education
Jung, J. S. (2020) · Korean Medical Education Review · doi:10.17496/kmer.2020.22.2.99 · Cited by 14
Huh, J. E., Lee, J. H., Hwang, E. J., & Park, C. M. (2023) · Korean Journal of Radiology · doi:10.3348/kjr.2022.0548 · Cited by 15
“As less experienced radiologists tend to be more strongly influenced by the DLAD model [8,34], more studies that include these groups are warranted.”
BackgroundSelf-citationEvolving and Novel Applications of Artificial Intelligence in Thoracic Imaging
Chang, J. Y., & Makary, M. S. (2024) · Diagnostics · doi:10.3390/diagnostics14131456 · Cited by 15
“Hwang et al. demonstrated that a commercialized DL algorithm achieved an AUC of 0.95 in interpreting patient CXRs in the ED, showing higher sensitivity but lower specificity compared to radiology residents [20].”
MethodInfluentialAutomated Abnormality Classification of Chest Radiographs using MobileNetV2
Genc, S., Akpınar, K. N., & Karagöl, S. (2020) · International Congress on Human-Computer Interaction, Optimization and Robotic Applications (ICHORA) · doi:10.1109/hora49412.2020.9152607 · Cited by 13
“investigated the performance of DCNN in medical diagnosis and disease prediction [6].”
BackgroundLee, S. Y., Ha, S., Jeon, M. G., Li, H., Choi, H., Kim, H. P., Choi, Y. R., Hoseok, I., Jeong, Y. J., Park, Y. H., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00658-x · Cited by 16
“Second, most assessments of commercialized CADs have been limited to detecting a single type of abnormality (e.g., pneumothorax vs. normal) 39 – 41 or any presence of abnormality (i.e., abnormal vs. normal) 42 – 45 .”
BackgroundSelf-citationIrvin, J., Pareek, A., Long, J., Rajpurkar, P., Eng, D. K.-M., Khandwala, N., Haug, P. J., Jephson, A. R., Conner, K. E., Gordon, B. H., et al. (2021) · Journal of Thoracic Imaging · doi:10.1097/rti.0000000000000622 · Cited by 14
“24 Another study investigated the performance of a deep learning model for detecting lung opacities in intensive care unit supine chest radiographs and showed the model achieved diagnostic accuracy similar to board-certi fi ed radiologists.”
Parpaleix, A., Parsy, C., Cordari, M., & Mejdoubi, M. (2023) · European Journal of Radiology Open · doi:10.1016/j.ejro.2023.100482 · Cited by 13
Deep Learning Models to Predict Fatal Pneumonia Using Chest X-Ray Images
Anai, S., Hisasue, J., Takaki, Y., & Hara, N. (2022) · Canadian Respiratory Journal · doi:10.1155/2022/8026580 · Cited by 12
Deep Learning for Automated Triaging of Stable Chest Radiographs in a Follow-up Setting
Yun, J., Ahn, Y., Cho, K., Oh, S. Y., Lee, S. M., Kim, N., & Seo, J. B. (2023) · Radiology · doi:10.1148/radiol.230606 · Cited by 14
The role of information systems in emergency department decision-making—a literature review
Born, C., Schwarz, R., Böttcher, T. P., Hein, A., & Krcmar, H. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae096 · Cited by 13
“Hwang et al 129 and Kim et al 130 highlighted radiographic abnormalities, improving their detection.”
BackgroundLung Disease Classification Using Deep Learning and ROI-Based Chest X-Ray Images
Nadal-Martínez, A., Talavera-Martínez, L., Munar, M., & González-Hidalgo, M. (2025) · Technologies · doi:10.3390/technologies14010001 · Cited by 11
Huynh, J., Masoudi, S., Noorbakhsh, A., Mahmoodi, A., Kligerman, S., Yen, A., Jacobs, K. E., Hahn, L. D., Hasenstab, K. A., Pazzani, M. J., et al. (2022) · IEEE Access · doi:10.1109/access.2022.3172706 · Cited by 11
“Recently, several groups [14]–[16] have reported the application of deep convolutional neural networks (CNNs) to classify chest radiographs for various pathologies, including pneumonia, pulmonary edema, pneumothorax, and many others.”
BackgroundHa, E.-G., Jeon, K. J., Lee, C., Kim, H.-S., & Han, S. (2023) · Dentomaxillofacial Radiology · doi:10.1259/dmfr.20220413 · Cited by 11
Arzamasov, K. M., Vasilev, Y. A., Vladzymyrskyy, A. V., Omelyanskaya, O. V., Михайлович, Ш. И., Kozhikhina, D. D., Goncharova, I., Gelezhe, P., Kirpichev, Y., Bobrovskaya, T., et al. (2023) · Healthcare · doi:10.3390/healthcare11121684 · Cited by 13
Kim, S., Cho, S., Cho, K., Seo, J., Nam, Y., Park, J., Kim, K., Kim, D. E., Hwang, J., Yun, J., et al. (2021) · Korean Journal of Radiology · doi:10.3348/kjr.2021.0170 · Cited by 10
“Studies across multiple medical specialties have employed AI to mimic the diagnostic abilities at or above the level of human experts [2-6].”
BackgroundSelf-citationImportance of Artificial Intelligence and Machine Learning in fighting with COVID-19 Epidemic
Solanki, V., Solanki, U., Baliyan, A., Kukreja, V., Lamba, V. K., & Sahoo, B. K. (2021) · International Conference on Information Systems and Computer Networks (ISCON) · doi:10.1109/iscon52037.2021.9702316 · Cited by 10
Heo, S., Ha, J., Jung, W., Yoo, S., Song, Y., Kim, T., & Cha, W. C. (2021) · Scientific Reports · doi:10.1038/s41598-022-16313-0 · Cited by 15
Self-citationHong, S., Hwang, E. J., Kim, S., Song, J., Lee, T., Jo, G. D., Choi, Y., Park, C. M., & Goo, J. M. (2023) · Diagnostics · doi:10.3390/diagnostics13061089 · Cited by 9
“However, the purpose of previous studies was to evaluate the performance of an AI-CAD to identify clinically significant abnormalities among chest radiographs from patients in the emergency department [12] and to evaluate the calibration of the AI-CAD [13], which was entirely different from that of the present study.”
ResultSelf-citationKromrey, M., Steiner, L., Schön, F., Gamain, J., Roller, C., & Malsch, C. (2024) · Healthcare · doi:10.3390/healthcare12222225 · Cited by 9
Efficacy of a deep learning-based software for chest X-ray analysis in an emergency department
Selvam, S., Peyrony, O., Elezi, A., Bragança, A., Zagdanski, A., Biard, L., Assouline, J., Chassagnon, G., Mulier, G., & de Margerie‐Mellon, C. (2025) · Diagnostic and Interventional Imaging · doi:10.1016/j.diii.2025.03.007 · Cited by 9
the MultiomIcs claSSIfier for pulmOnary Nodules (MISSION) Collaborative Group, Deng, J., Zhao, M., Li, Q., Zhang, Y., Ma, M., Li, C., Wang, J., She, Y., Jiang, Y., et al. (2021) · Translational Lung Cancer Research · doi:10.21037/tlcr-21-971 · Cited by 9
“To date, limited studies have investigated the benefits of an AI system in assessing invasiveness (7,20,39,40).”
Background
Metadata
- Authors
- Hwang, E. J., Nam, J. G., Lim, W. H., Park, S. J., Jeong, Y. S., Kang, J. H., Hong, E. K., Kim, T. M., Goo, J. M., Park, S., Kim, K. H., & Park, C. M.
- Affiliations
- Seoul National University (KR)
- ORCID
- Venue
- Radiology
- Published
- 2019-12
- Volume
- 293
- Issue
- 3
- Pages
- 573-580
- ISSN
- 0033-8419 (ISSN-L)
- 1527-1315
- OpenAlex
- W2982071196
- Cited by
- 164
- References
- 21
- Citations by year
- 2026: 8 · 2025: 14 · 2024: 29 · 2023: 28 · 2022: 25 · 2021: 34 · 2020: 25 · 2019: 1
- FWCI
- 14.14
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Radiology (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Seoul National University Open Repository (Seoul National University) (submittedVersion, cc-by)
- Funders
- Seoul National University Hospital (06-2016-3000)
- Topics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- deep learning, chest X-ray interpretation, emergency department, clinical implementation of AI, radiology workflow, chest radiography
- MeSH terms
- Deep Learning, Adult, Aged, Clinical Competence, Diagnosis, Differential, Emergency Service, Hospital, Female, Humans, Male, Middle Aged, Retrospective Studies, Sensitivity and Specificity, Radiography, Thoracic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.