CoChat

Development and Validation of a Deep Learning–based Automatic Detection Algorithm for Active Pulmonary Tuberculosis on Chest Radiographs

Open access
Hwang, Eui Jin, Park, Sunggyun, Jin, Kwang-Nam, Kim, Jung Im, Choi, So Young, Lee, Jong Hyuk, Goo, Jin Mo, Aum, Jaehong, Yim, Jae-Joon, Park, Chang Min, Deep Learning-Based Automatic Detection Algorithm Development and Evaluation Group, Kim, Dong Hyeon, Woo, Woo, Choi, Choi, Hwang, In Pyung, Song, Yong Sub, Lim, Lim, Kim, Kim, Wi, Jae Yeon, Oh, Su Suk, Kang, Mi-Jin · 2019 · Clinical Infectious Diseases

The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.

Abstract

Background: Detection of active pulmonary tuberculosis on chest radiographs (CRs) is critical for the diagnosis and screening of tuberculosis. An automated system may help streamline the tuberculosis screening process and improve diagnostic performance. Methods: We developed a deep learning–based automatic detection (DLAD) algorithm using 54c221 normal CRs and 6768 CRs with active pulmonary tuberculosis that were labeled and annotated by 13 board-certified radiologists. The performance of DLAD was validated using 6 external multicenter, multinational datasets. To compare the performances of DLAD with physicians, an observer performance test was conducted by 15 physicians including nonradiology physicians, board-certified radiologists, and thoracic radiologists. Image-wise classification and lesion-wise localization performances were measured using area under the receiver operating characteristic (ROC) curves and area under the alternative free-response ROC curves, respectively. Sensitivities and specificities of DLAD were calculated using 2 cutoffs (high sensitivity [98%] and high specificity [98%]) obtained through in-house validation. Results: DLAD demonstrated classification performance of 0.977–1.000 and localization performance of 0.973–1.000. Sensitivities and specificities for classification were 94.3%–100% and 91.1%–100% using the high-sensitivity cutoff and 84.1%–99.0% and 99.1%–100% using the high-specificity cutoff. DLAD showed significantly higher performance in both classification (0.993 vs 0.746–0.971) and localization (0.993 vs 0.664–0.925) compared to all groups of physicians. Conclusions: Our DLAD demonstrated excellent and consistent performance in the detection of active pulmonary tuberculosis on CR, outperforming physicians, including thoracic radiologists.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/cid/ciy967

Cites

31 of 31 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1. Performance of deep learning–based automatic detection algorithm (DLAD) at in-house validation and external validation. Original (a) and zoomed (b) receiver operating characteristic (ROC) curves for DLAD in in-house validation and external validation datasets. The DLAD showed consistently high performance in image-wise classification, not only in the internal validation dataset but also in the 6 external validation datasets; AUROC values ranged from 0.977 to 1.000. For lesion-wise localization performance assessed by jackknife alternative free-response ROC (c, d), DLAD showed consistently high performance in different datasets; AUAFROC ranged from 0.973 to 1.000. Abbreviations: AUAFROC, area under the alternative free-response receiver operating characteristic curves; BMC, Boramae Medical Center; DEMC, Daejeon Eulji Medical Center; KUHG, Kyunghee University Hospital at Gangdong; SNUH, Seoul National University Hospital.
Figure 1. Performance of deep learning–based automatic detection algorithm (DLAD) at in-house validation and external validation. Original (a) and zoomed (b) receiver operating characteristic (ROC) curves for DLAD in in-house validation and external validation datasets. The DLAD showed consistently high performance in image-wise classification, not only in the internal validation dataset but also in the 6 external validation datasets; AUROC values ranged from 0.977 to 1.000. For lesion-wise localization performance assessed by jackknife alternative free-response ROC (c, d), DLAD showed consistently high performance in different datasets; AUAFROC ranged from 0.973 to 1.000. Abbreviations: AUAFROC, area under the alternative free-response receiver operating characteristic curves; BMC, Boramae Medical Center; DEMC, Daejeon Eulji Medical Center; KUHG, Kyunghee University Hospital at Gangdong; SNUH, Seoul National University Hospital.CC BY · PubMed Central
Figure 2. Comparison of diagnostic performance between deep learning–based automatic detection algorithm (DLAD) and physician groups. The DLAD showed significantly higher performance than all reader groups both in terms of image-wise classification (a) and lesion-wise localization (b) in the observer performance test. Abbreviations: AUAFROC, area under the alternative free-response receiver operating characteristic curves; AUROC, area under the receiver operating characteristic curve.
Figure 2. Comparison of diagnostic performance between deep learning–based automatic detection algorithm (DLAD) and physician groups. The DLAD showed significantly higher performance than all reader groups both in terms of image-wise classification (a) and lesion-wise localization (b) in the observer performance test. Abbreviations: AUAFROC, area under the alternative free-response receiver operating characteristic curves; AUROC, area under the receiver operating characteristic curve.CC BY · PubMed Central
Figure 3. Representative case from the observer performance test. Chest radiograph of a 25-year-old woman shows a cavitary mass with multiple satellite nodules in the right upper lung field (a), which corresponded well with computed tomography images. These radiologic findings are typical for active pulmonary tuberculosis (b). Deep learning–based automatic detection algorithm provided a probability value of 0.9663 for active pulmonary tuberculosis in this case, and the classification activation map correctly localized the lesion in the right upper lung field (c).
Figure 3. Representative case from the observer performance test. Chest radiograph of a 25-year-old woman shows a cavitary mass with multiple satellite nodules in the right upper lung field (a), which corresponded well with computed tomography images. These radiologic findings are typical for active pulmonary tuberculosis (b). Deep learning–based automatic detection algorithm provided a probability value of 0.9663 for active pulmonary tuberculosis in this case, and the classification activation map correctly localized the lesion in the right upper lung field (c).CC BY · PubMed Central
Figure 4. Representative case from the observer performance test. Chest radiograph of a 59-year-old female patient revealed nodular infiltrations at both lung apices (a), with a corresponding computed tomography image (b) that was initially missed by 2 readers (nonradiology physicians). Deep learning–based automatic detection algorithm (DLAD) provided a probability value of 0.9526, with a corresponding classification activation map (c). Readers who initially misclassified the chest radiograph corrected their classification after checking the results of DLAD.
Figure 4. Representative case from the observer performance test. Chest radiograph of a 59-year-old female patient revealed nodular infiltrations at both lung apices (a), with a corresponding computed tomography image (b) that was initially missed by 2 readers (nonradiology physicians). Deep learning–based automatic detection algorithm (DLAD) provided a probability value of 0.9526, with a corresponding classification activation map (c). Readers who initially misclassified the chest radiograph corrected their classification after checking the results of DLAD.CC BY · PubMed Central
Figure 5. Representative case from the observer performance test. Chest radiograph of a 35-year-old female patient revealed a subtle nodular infiltration in the right upper lung field (a), with a corresponding computed tomography image (b). Deep learning–based automatic detection algorithm (DLAD) provided a probability value of 0.1625 and correctly localized the lesion (c). Seven of the 15 readers had initially missed the lesion; however, 2 readers corrected their reading after reviewing the results of DLAD.
Figure 5. Representative case from the observer performance test. Chest radiograph of a 35-year-old female patient revealed a subtle nodular infiltration in the right upper lung field (a), with a corresponding computed tomography image (b). Deep learning–based automatic detection algorithm (DLAD) provided a probability value of 0.1625 and correctly localized the lesion (c). Seven of the 15 readers had initially missed the lesion; however, 2 readers corrected their reading after reviewing the results of DLAD.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 3

  1. 2017., Global tuberculosis report 2017
  2. 2013., Systematic screening for active tuberculosis: principles and recommendations
  3. Zhai, RJafroc: analysis of data acquired using the receiver operating characteristic paradigm and its extensions

Cited by

See all 266

Metadata

Authors
Hwang, E. J., Park, S., Jin, K.-N., Kim, J. I., Choi, S. Y., Lee, J. H., Goo, J. M., Aum, J., Yim, J.-J., Park, C. M., Deep Learning-Based Automatic Detection Algorithm Development and Evaluation Group, Kim, D. H., Woo, W., Choi, C., Hwang, I. P., Song, Y. S., Lim, L., Kim, K., Wi, J. Y., Oh, S. S., & Kang, M.-J.
Affiliations
  • Seoul National University (KR)
  • Kyung Hee University Hospital at Gangdong (KR)
  • Daejeon Eulji Medical Center, Eulji University (KR)
Corresponding author
Park, Chang Min
Venue
Clinical Infectious Diseases
Published
2018-11-08
Volume
69
Issue
5
Pages
739-747
ISSN
  • 1058-4838 (ISSN-L)
  • 1537-6591
OpenAlex
W2900683726
Cited by
241
References
31
Citations by year
2026: 23 · 2025: 26 · 2024: 28 · 2023: 38 · 2022: 45 · 2021: 40 · 2020: 27 · 2019: 14
FWCI
13.97
Type
journal article
Language
en
Open access
Locations
  • Clinical Infectious Diseases (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Seoul National University Open Repository (Seoul National University) (submittedVersion)
Funders
  • Seoul National University Hospital Research (04-2016-3000)
  • Seoul Research & Business Development Program (FI170002)
Topics
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
Keywords
active pulmonary tuberculosis, chest radiograph, deep learning, lesion localization, observer performance comparison, tuberculosis screening, diagnostic accuracy
MeSH terms
Deep Learning, Adult, Aged, Algorithms, Automation, Female, Humans, Male, Middle Aged, Radiography, ROC Curve, Sensitivity and Specificity, Thorax, Tuberculosis, Pulmonary

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.