Development and Validation of Deep Learning–based Automatic Detection Algorithm for Malignant Pulmonary Nodules on Chest Radiographs
The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Abstract
Purpose To develop and validate a deep learning-based automatic detection algorithm (DLAD) for malignant pulmonary nodules on chest radiographs and to compare its performance with physicians including thoracic radiologists. Materials and Methods For this retrospective study, DLAD was developed by using 43 292 chest radiographs (normal radiograph-to-nodule radiograph ratio, 34 067:9225) in 34 676 patients (healthy-to-nodule ratio, 30 784:3892; 19 230 men [mean age, 52.8 years; age range, 18-99 years]; 15 446 women [mean age, 52.3 years; age range, 18-98 years]) obtained between 2010 and 2015, which were labeled and partially annotated by 13 board-certified radiologists, in a convolutional neural network. Radiograph classification and nodule detection performances of DLAD were validated by using one internal and four external data sets from three South Korean hospitals and one U.S. hospital. For internal and external validation, radiograph classification and nodule detection performances of DLAD were evaluated by using the area under the receiver operating characteristic curve (AUROC) and jackknife alternative free-response receiver-operating characteristic (JAFROC) figure of merit (FOM), respectively. An observer performance test involving 18 physicians, including nine board-certified radiologists, was conducted by using one of the four external validation data sets. Performances of DLAD, physicians, and physicians assisted with DLAD were evaluated and compared. Results According to one internal and four external validation data sets, radiograph classification and nodule detection performances of DLAD were a range of 0.92-0.99 (AUROC) and 0.831-0.924 (JAFROC FOM), respectively. DLAD showed a higher AUROC and JAFROC FOM at the observer performance test than 17 of 18 and 15 of 18 physicians, respectively (P < .05), and all physicians showed improved nodule detection performances with DLAD (mean JAFROC FOM improvement, 0.043; range, 0.006-0.190; P < .05). Conclusion This deep learning-based automatic detection algorithm outperformed physicians in radiograph classification and nodule detection performance for malignant pulmonary nodules on chest radiographs, and it enhanced physicians' performances when used as a second reader. © RSNA, 2018 Online supplemental material is available for this article.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“13 (refs. 63–75) studies validated algorithms on external data”
From Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis (2021) · cites this paper as [69]
In the citing paper
“Therefore, possibilities for developing computerized auxiliary diagnostic tools have become almost unlimited(39,40).”
From Artificial intelligence, machine learning, computer-aided diagnosis, and radiomics: advances in imaging towards to precision medicine (2019) · cites this paper as [39]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 2×
“3 (n = 11) 73 (8/11) [43, 91] 27 (9/33) [15, 44] 74 (49/66) [64, 83] 58 (32/55) [45, 70] 75 (33/44) [60, 86]”
DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988) · Biometrics · doi:10.2307/2531595 · Cited by 22,649
Representation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001
Multiple significance tests: the Bonferroni method
Bland, J. M., & Altman, D. G. (1995) · BMJ · doi:10.1136/bmj.310.6973.170 · Cited by 3,341
Effective Doses in Radiology and Diagnostic Nuclear Medicine: A Catalog
Mettler, F. A., Huda, W., Yoshizumi, T. T., & Mahesh, M. (2008) · Radiology · doi:10.1148/radiol.2481071451 · Cited by 2,008
Benefits and Harms of CT Screening for Lung Cancer: A Systematic Review
Bach, P. B., Mirkin, J. N., Oliver, T. K., Azzoli, C. G., Berry, D. A., Brawley, O. W., Byers, T., Colditz, G. A., Gould, M. K., Jett, J. R., et al. (2012) · JAMA · doi:10.1001/jama.2012.5521 · Cited by 1,244
Lung Nodule Classification Using Deep Features in CT Images
Kumar, D., Wong, A., & Clausi, D. A. (2015) · 2015 12th Conference on Computer and Robot Vision · doi:10.1109/crv.2015.25 · Cited by 398
Cited by
High-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450
Key challenges for delivering clinical impact with artificial intelligence
Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G., & King, D. (2019) · BMC Medicine · doi:10.1186/s12916-019-1426-2 · Cited by 3,136
Liu, X., Faes, L., Kale, A. U., Wagner, S. K., Fu, D. J., Bruynseels, A., Mahendiran, T., Moraes, G., Shamdas, M., Kern, C., et al. (2019) · The Lancet Digital Health · doi:10.1016/s2589-7500(19)30123-2 · Cited by 1,883
Big data and machine learning algorithms for health-care delivery
Ngiam, K. Y., & Khor, I. W. (2019) · The Lancet Oncology · doi:10.1016/s1470-2045(19)30149-4 · Cited by 1,424
A Survey of Deep Active Learning
Ren, P., Xiao, Y., Chang, X., Huang, P.-Y., Li, Z., Gupta, B. B., Chen, X., & Wang, X. (2022) · ACM Computing Surveys · doi:10.1145/3472291 · Cited by 1,169
Metadata
- Authors
- Nam, J. G., Park, S., Hwang, E. J., Lee, J. H., Jin, K.-N., Lim, K. Y., Vu, T. H., Sohn, J. H., Hwang, S., Goo, J. M., & Park, C. M.
- Affiliations
- University of California, San Francisco (US)
- ORCID
- Venue
- Radiology
- Published
- 2019-01
- Volume
- 290
- Issue
- 1
- Pages
- 218-228
- ISSN
- 0033-8419 (ISSN-L)
- 1527-1315
- OpenAlex
- W2894319790
- Cited by
- 539
- References
- 28
- Citations by year
- 2026: 29 · 2025: 37 · 2024: 81 · 2023: 88 · 2022: 80 · 2021: 113 · 2020: 67 · 2019: 38
- FWCI
- 31.42
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Radiology (publishedVersion)
- PubMed (publishedVersion)
- Seoul National University Open Repository (Seoul National University) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Health Sciences › Medicine › Pulmonary and Respiratory Medicine › Lung Cancer Diagnosis and Treatment
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- lung cancer, chest radiograph, convolutional neural networks, computer-aided detection, radiograph classification, observer performance, radiologist comparison, external validation, area under the receiver operating characteristic, lung nodule detection, thoracic radiology, lung cancer screening, lung cancer staging
- MeSH terms
- Deep Learning, Adolescent, Adult, Aged, Aged, 80 and over, Algorithms, Female, Humans, Lung Neoplasms, Male, Middle Aged, Radiographic Image Interpretation, Computer-Assisted, Retrospective Studies, Radiography, Thoracic, Reproducibility of Results, Multiple Pulmonary Nodules, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Development and Validation of Deep Learning–based Automatic Detection Algorithm for Malignant Pulmonary Nodules on Chest Radiographs. https://app.cochat.ai/papers/10.1148/radiol.2018180237
- The paper
- Nam, J. G., Park, S., Hwang, E. J., Lee, J. H., Jin, K.-N., Lim, K. Y., Vu, T. H., Sohn, J. H., Hwang, S., Goo, J. M., & Park, C. M. (2019). Development and Validation of Deep Learning–based Automatic Detection Algorithm for Malignant Pulmonary Nodules on Chest Radiographs. Radiology. https://doi.org/10.1148/radiol.2018180237
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.