The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Deep Learning for Chest Radiograph Diagnosis in the Emergency Department
Open accessAbstract
Background The performance of a deep learning (DL) algorithm should be validated in actual clinical situations, before its clinical implementation. Purpose To evaluate the performance of a DL algorithm for identifying chest radiographs with clinically relevant abnormalities in the emergency department (ED) setting. Materials and Methods This single-center retrospective study included consecutive patients who visited the ED and underwent initial chest radiography between January 1 and March 31, 2017. Chest radiographs were analyzed with a commercially available DL algorithm. The performance of the algorithm was evaluated by determining the area under the receiver operating characteristic curve (AUC), sensitivity, and specificity at predefined operating cutoffs (high-sensitivity and high-specificity cutoffs). The sensitivities and specificities of the algorithm were compared with those of the on-call radiology residents who interpreted the chest radiographs in the actual practice by using McNemar tests. If there were discordant findings between the algorithm and resident, the residents reinterpreted the chest radiographs by using the algorithm’s output. Results A total of 1135 patients (mean age, 53 years ± 18; 582 men) were evaluated. In the identification of abnormal chest radiographs, the algorithm showed an AUC of 0.95 (95% confidence interval [CI]: 0.93, 0.96), a sensitivity of 88.7% (227 of 256 radiographs; 95% CI: 84.1%, 92.3%), and a specificity of 69.6% (612 of 879 radiographs; 95% CI: 66.5%, 72.7%) at the high-sensitivity cutoff and a sensitivity of 81.6% (209 of 256 radiographs; 95% CI: 76.3%, 86.2%) and specificity of 90.3% (794 of 879 radiographs; 95% CI: 88.2%, 92.2%) at the high-specificity cutoff. Radiology residents showed lower sensitivity (65.6% [168 of 256 radiographs; 95% CI: 59.5%, 71.4%], P < .001) and higher specificity (98.1% [862 of 879 radiographs; 95% CI: 96.9%, 98.9%], P < .001) compared with the algorithm. After reinterpretation of chest radiographs with use of the algorithm’s outputs, the sensitivity of the residents improved (73.4% [188 of 256 radiographs; 95% CI: 68.0%, 78.8%], P = .003), whereas specificity was reduced (94.3% [829 of 879 radiographs; 95% CI: 92.8%, 95.8%], P < .001). Conclusion A deep learning algorithm used with emergency department chest radiographs showed diagnostic performance for identifying clinically relevant abnormalities and helped improve the sensitivity of radiology residents’ evaluation. Published under a CC BY 4.0 license. Online supplemental material is available for this article. See also the editorial by Munera and Infante in this issue.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“13 (refs. 63–75) studies validated algorithms on external data”
From Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis (2021) · cites this paper as [65]
In the citing paper
“Twenty-one54,63–67,70,72,76–88 studies compared algorithm performance against healthcare professionals.”
From Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis (2021) · cites this paper as [65]
Citations
Cites
DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988) · Biometrics · doi:10.2307/2531595 · Cited by 22,649
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697
Ehteshami Bejnordi, B., Veta, M., Johannes van Diest, P., van Ginneken, B., Karssemeijer, N., Litjens, G., van der Laak, J. A. W. M., and the CAMELYON16 Consortium, Hermsen, M., Manson, Q. F., et al. (2017) · JAMA · doi:10.1001/jama.2017.14585 · Cited by 3,293
Ting, D. S. W., Cheung, C. Y.-L., Lim, G., Tan, G. S. W., Quang, N. D., Gan, A., Hamzah, H., Garcia-Franco, R., San Yeo, I. Y., Lee, S. Y., et al. (2017) · JAMA · doi:10.1001/jama.2017.18152 · Cited by 2,402
Park, S. H., & Han, K. (2018) · Radiology · doi:10.1148/radiol.2017171920 · Cited by 894
Self-citationNam, J. G., Park, S., Hwang, E. J., Lee, J. H., Jin, K.-N., Lim, K. Y., Vu, T. H., Sohn, J. H., Hwang, S., Goo, J. M., et al. (2019) · Radiology · doi:10.1148/radiol.2018180237 · Cited by 539
Self-citationHwang, E. J., Park, S., Jin, K.-N., Kim, J. I., Choi, S. Y., Lee, J. H., Goo, J. M., Aum, J., Yim, J.-J., Cohen, J. G., et al. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.1095 · Cited by 447
Self-citationComparisons of Predictive Values of Binary Medical Diagnostic Tests for Paired Designs
Leisenring, W., Alono, T., & Pepe, M. S. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00345.x · Cited by 355
Hwang, E. J., Park, S., Jin, K.-N., Kim, J. I., Choi, S. Y., Lee, J. H., Goo, J. M., Aum, J., Yim, J.-J., Park, C. M., et al. (2019) · Clinical Infectious Diseases · doi:10.1093/cid/ciy967 · Cited by 241
Self-citationEng, J., Mysko, W. K., Weller, G. E. R., Renard, R. L., Gitlin, J. N., Bluemke, D. A., Magid, D., Kelen, G. D., & Scott, W. W. (2000) · American Journal of Roentgenology · doi:10.2214/ajr.175.5.1751233 · Cited by 126
Cited by
Preparing Medical Imaging Data for Machine Learning
Willemink, M. J., Koszek, W. A., Hardell, C., Wu, J., Fleischmann, D., Harvey, H., Folio, L. R., Summers, R. M., Rubin, D. L., & Lungren, M. P. (2020) · Radiology · doi:10.1148/radiol.2020192224 · Cited by 1,020
Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis
Aggarwal, R., Sounderajah, V., Martin, G., Ting, D. S. W., Karthikesalingam, A., King, D., Ashrafian, H., & Darzi, A. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00438-z · Cited by 946
Deep learning for chest X-ray analysis: A survey
Çallı, E., Sogancioglu, E., van Ginneken, B., van Leeuwen, K. G., & Murphy, K. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102125 · Cited by 506
Seah, J. C. Y., Tang, C. H. M., Buchlak, Q. D., Holt, X. G., Wardman, J. B., Aimoldin, A., Esmaili, N., Ahmad, H., Pham, H., Lambert, J. F., et al. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00106-0 · Cited by 229
COVID-19 on Chest Radiographs: A Multireader Evaluation of an Artificial Intelligence System
Murphy, K., Smits, H., Knoops, A. J. G., Korst, M. B. J. M., Samson, T., Scholten, E. T., Schalekamp, S., Schaefer-Prokop, C. M., Philipsen, R. H. H. M., Meijers, A., et al. (2020) · Radiology · doi:10.1148/radiol.2020201874 · Cited by 214
Metadata
- Authors
- Hwang, E. J., Nam, J. G., Lim, W. H., Park, S. J., Jeong, Y. S., Kang, J. H., Hong, E. K., Kim, T. M., Goo, J. M., Park, S., Kim, K. H., & Park, C. M.
- Affiliations
- Seoul National University (KR)
- ORCID
- Venue
- Radiology
- Published
- 2019-12
- Volume
- 293
- Issue
- 3
- Pages
- 573-580
- ISSN
- 0033-8419 (ISSN-L)
- 1527-1315
- OpenAlex
- W2982071196
- Cited by
- 164
- References
- 21
- Citations by year
- 2026: 8 · 2025: 14 · 2024: 29 · 2023: 28 · 2022: 25 · 2021: 34 · 2020: 25 · 2019: 1
- FWCI
- 14.14
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Radiology (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Seoul National University Open Repository (Seoul National University) (submittedVersion, cc-by)
- Funders
- Seoul National University Hospital (06-2016-3000)
- Topics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- deep learning, chest X-ray interpretation, emergency department, clinical implementation of AI, radiology workflow, chest radiography
- MeSH terms
- Deep Learning, Adult, Aged, Clinical Competence, Diagnosis, Differential, Emergency Service, Hospital, Female, Humans, Male, Middle Aged, Retrospective Studies, Sensitivity and Specificity, Radiography, Thoracic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Deep Learning for Chest Radiograph Diagnosis in the Emergency Department. https://app.cochat.ai/papers/10.1148/radiol.2019191225
- The paper
- Hwang, E. J., Nam, J. G., Lim, W. H., Park, S. J., Jeong, Y. S., Kang, J. H., Hong, E. K., Kim, T. M., Goo, J. M., Park, S., Kim, K. H., & Park, C. M. (2019). Deep Learning for Chest Radiograph Diagnosis in the Emergency Department. Radiology. https://doi.org/10.1148/radiol.2019191225
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.