Chest X-ray Analysis With Deep Learning-Based Software as a Triage Test for Pulmonary Tuberculosis: An Individual Patient Data Meta-Analysis of Diagnostic Accuracy
Open accessAbstract
Background: Automated radiologic analysis using computer-aided detection software (CAD) could facilitate chest X-ray (CXR) use in tuberculosis diagnosis. There is little to no evidence on the accuracy of commercially available deep learning-based CAD in different populations, including patients with smear-negative tuberculosis and people living with human immunodeficiency virus (HIV, PLWH). Methods: We collected CXRs and individual patient data (IPD) from studies evaluating CAD in patients self-referring for tuberculosis symptoms with culture or nucleic acid amplification testing as the reference. We reanalyzed CXRs with three CAD programs (CAD4TB version (v) 6, Lunit v3.1.0.0, and qXR v2). We estimated sensitivity and specificity within each study and pooled using IPD meta-analysis. We used multivariable meta-regression to identify characteristics modifying accuracy. Results: We included CXRs and IPD of 3727/3967 participants from 4/7 eligible studies. 17% (621/3727) were PLWH. 17% (645/3727) had microbiologically confirmed tuberculosis. Despite using the same threshold score for classifying CXR in every study, sensitivity and specificity varied from study to study. The software had similar unadjusted accuracy (at 90% pooled sensitivity, pooled specificities were: CAD4TBv6, 56.9% [95% confidence interval {CI}: 51.7–61.9]; Lunit, 54.1% [95% CI: 44.6–63.3]; qXRv2, 60.5% [95% CI: 51.7–68.6]). Adjusted absolute differences in pooled sensitivity between PLWH and HIV-uninfected participants were: CAD4TBv6, −13.4% [−21.1, −6.9]; Lunit, +2.2% [−3.6, +6.3]; qXRv2: −13.4% [−21.5, −6.6]; between smear-negative and smear-positive tuberculosis was: were CAD4TBv6, −12.3% [−19.5, −6.1]; Lunit, −17.2% [−24.6, −10.5]; qXRv2, −16.6% [−24.4, −9.9]. Accuracy was similar to human readers. Conclusions: For CAD CXR analysis to be implemented as a high-sensitivity tuberculosis rule-out test, users will need threshold scores identified from their own patient populations and stratified by HIV and smear status.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Some CAD developers recommend a universal threshold score, while others recommend setting-specific thresholds due to variation in CAD abnormality scores between software and populations.
“Some developers recommend a threshold score for universal use, above which an individual would be classified as having abnormalities suggestive of tuberculosis [6, 7]. Other developers, such as CAD4TB (Delft Imaging Systems, Netherlands), recommend scores tailored to each setting due to variation in CAD abnormality scores between different software and populations [5, 6, 8–13]”
From Computer-aided detection thresholds for digital chest radiography interpretation in tuberculosis diagnostic algorithms. (2024) · cites this paper as [8]
Methods (Data Analysis; page not supplied) says Lunit and qXRv2 have developer-recommended universal thresholds, whereas CAD4TBv6 does not. The Discussion and Results (pages not supplied) support selecting thresholds for particular populations because performance varies between studies. The paper does not say that CAD4TB’s developer recommends setting-specific thresholds; that recommendation is presented as the authors’ implementation advice. (LLM, conf=0.84)
Experimental — accuracy not yet measuredThe majority of studies reporting on CAD threshold determination only included individuals who met criteria for microbiological testing.
“The majority of studies reporting on CAD threshold determination only included individuals who met some criteria for microbiological testing, such as having symptoms and/or abnormal chest radiographs [8, 16–27].”
From Computer-aided detection thresholds for digital chest radiography interpretation in tuberculosis diagnostic algorithms. (2024) · cites this paper as [8]
The Methods section says the included studies enrolled people self-referring with tuberculosis symptoms and used microbiological testing as the reference standard. The Results report study-specific CAD thresholds for those studies. This supports the cited paper as an example, but it does not establish that a majority of all studies reporting CAD threshold determination used such eligibility criteria. (LLM, conf=0.83)
Experimental — accuracy not yet measuredContext-specific CAD threshold determination is often not done due to logistical and financial constraints.
“yet context-specific determination is often not done due to logistical and financial constraints [8, 14, 15].”
From Computer-aided detection thresholds for digital chest radiography interpretation in tuberculosis diagnostic algorithms. (2024) · cites this paper as [8]
The Discussion says setting-specific threshold selection requires local patient data, resources, protocols, and tools, and calls this an important implementation challenge. It does not say that such determination is often omitted or identify financial constraints as a reason. (LLM, conf=0.83)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Reitsma, J. B., Glas, A. S., Rutjes, A. W. S., Scholten, R. J. P. M., Bossuyt, P. M., & Zwinderman, A. H. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.02.022 · Cited by 3,369 · see citing works
Stewart, L. A., Clarke, M., Rovers, M., Riley, R. D., Simmonds, M., Stewart, G., & Tierney, J. F. (2015) · JAMA · doi:10.1001/jama.2015.3656 · Cited by 2,134 · see citing works
van Cleeff, M., Kivihya-Ndugga, L., Meme, H., Odhiambo, J., & Klatser, P. (2005) · BMC Infectious Diseases · doi:10.1186/1471-2334-5-111 · Cited by 199 · see citing works
Melendez, J., Sánchez, C. I., Philipsen, R. H. H. M., Maduskar, P., Dawson, R., Theron, G., Dheda, K., & van Ginneken, B. (2016) · Scientific Reports · doi:10.1038/srep25265 · Cited by 152 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 37
- Williams, 1907, The Use of X-Ray examinations in pulmonary tuberculosis, Boston Med Surg J, 157, 850, 10.1056/NEJM190712261572602 · DOI
- McAdams, 2006, Recent advances in chest radiography, Radiology, 241, 663, 10.1148/radiol.2413051535 · DOI
- Hosny, 2018, Artificial intelligence in radiology, Nat Rev Cancer, 18, 500, 10.1038/s41568-018-0016-5 · DOI · Paper page
- Harris, 2019, A systematic review of the diagnostic accuracy of artificial intelligence-based computer programs to analyze chest x-rays for pulmonary tuberculosis, PLoS One, 14, e0221339, 10.1371/journal.pone.0221339 · DOI
- Pande, 2016, Computer-aided detection of pulmonary tuberculosis on digital chest radiographs: a systematic review, Int J Tuberc Lung Dis, 20, 1226, 10.5588/ijtld.15.0926 · DOI
- Ahmad Khan, 2020, Deep learning-based chest X-ray analysis software as triage tests for pulmonary tuberculosis: a prospective study of diagnostic accuracy for culture-confirmed disease, Lancet Digit Health, 2, e573, 10.1016/S2589-7500(20)30221-1 · DOI · Paper page
- World Health Organization., 2016, Chest Radiography in Tuberculosis Detection - Summary of current WHO recommendations and guidance on programmatic approaches, 2016
- Whiting, 2011, QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies, Ann Intern Med, 155, 529, 10.7326/0003-4819-155-8-201110180-00009 · DOI · Paper page
- Steinhauser, 2016, Modelling multiple thresholds in meta-analysis of diagnostic test accuracy studies, BMC Med Res Methodol, 16, 97, 10.1186/s12874-016-0196-1 · DOI
- Rücker, 2019, diagmeta: meta-analysis of diagnostic accuracy studies with several R packages
- World Health Organization., 2014, High-priority target product profiles for new tuberculosis diagnostics: report of a consensus meeting
- Doebler, 2015, Meta-analysis of diagnostic accuracy with mada. R Packages
- World Health Organization., 2013, Systematic screening for active tuberculosis: principles and recommendations
- SAS Institute Inc., 2016, Base SAS® 9.4 procedures guide: statistical procedures, 6th ed
- R Foundation for Statistical Computing., 2013, R: a language and environment for statistical computing
- Breuninger, 2014, Diagnostic accuracy of computer-aided detection of pulmonary tuberculosis in chest radiographs: a validation study from sub-Saharan Africa, PLoS One, 9, e106381, 10.1371/journal.pone.0106381 · DOI · Paper page
- Muyoyeta, 2014, The sensitivity and specificity of using a computer aided diagnosis program for automatically scoring chest X-rays of presumptive TB patients compared with Xpert MTB/RIF in Lusaka Zambia, PLoS One, 9, e93757, 10.1371/journal.pone.0093757 · DOI · Paper page
- Philipsen, 2015, Automated chest-radiography as a triage for Xpert testing in resource-constrained settings: a prospective study of diagnostic accuracy and costs, Sci Rep, 5, 12215, 10.1038/srep12215 · DOI · Paper page
- Qin, 2019, Using artificial intelligence to read chest radiographs for tuberculosis detection: a multi-site evaluation of the diagnostic accuracy of three deep learning systems, Sci Rep, 9, 15000, 10.1038/s41598-019-51503-3 · DOI · Paper page
- Cohen, 1996, The validity of classic symptoms and chest radiographic configuration in predicting pulmonary tuberculosis, Chest, 109, 420, 10.1378/chest.109.2.420 · DOI
- Maduskar, 2013, Detection of tuberculosis using digital chest radiography: automated reading vs interpretation by clinical officers, Int J Tuberc Lung Dis, 17, 1613, 10.5588/ijtld.13.0325 · DOI · Paper page
- Pinto, 2013, Scoring systems using chest radiographic features for the diagnosis of pulmonary tuberculosis in adults: a systematic review, Eur Respir J, 42, 480, 10.1183/09031936.00107412 · DOI
- Rahman, 2017, An evaluation of automated chest radiography reading software for tuberculosis screening among public- and private-sector patients, Eur Respir J, 49, 10.1183/13993003.02159-2016 · DOI · Paper page
- Tamhane, 2009, Predictors of smear-negative pulmonary tuberculosis in HIV-infected patients, Battambang, Cambodia, Int J Tuberc Lung Dis, 13, 347
- Wilcke, 1997, Diagnostic strategy for pulmonary tuberculosis in a low-incidence country: results of chest X-ray and sputum cultured for Mycobacterium tuberculosis, Respir Med, 91, 281, 10.1016/S0954-6111(97)90031-7 · DOI
- Graham, 2002, Chest radiograph abnormalities associated with tuberculosis: reproducibility and yield of active cases, Int J Tuberc Lung Dis, 6, 137
- Balabanova, 2005, Variability in interpretation of chest radiographs among Russian clinicians and implications for screening programmes: observational study, BMJ, 331, 379, 10.1136/bmj.331.7513.379 · DOI
- Koppaka, 2005, How reliable is chest radiography?, 302
- Kosack, 2017, Evaluation of a chest radiograph reading and recording system for tuberculosis in a HIV-positive cohort, Clin Radiol, 72, 519 e1, 10.1016/j.crad.2017.01.008 · DOI
- Pinto, 2013, Development of a simple reliable radiographic scoring system to aid the diagnosis of pulmonary tuberculosis, PLoS One, 8, e54235, 10.1371/journal.pone.0054235 · DOI
- Ralph, 2010, A simple, valid, numerical score for grading chest x-ray severity in adult smear-positive pulmonary tuberculosis, Thorax, 65, 863, 10.1136/thx.2010.136242 · DOI
- Zellweger, 2006, Intra-observer and overall agreement in the radiological assessment of tuberculosis, Int J Tuberc Lung Dis, 10, 1123
- Bruno, 2015, Understanding and confronting our mistakes: the epidemiology of error in radiology and strategies for error reduction, Radiographics, 35, 1668, 10.1148/rg.2015150023 · DOI
- Waite, 2017, Tired in the reading room: the influence of fatigue in radiology, J Am Coll Radiol, 14, 191, 10.1016/j.jacr.2016.10.009 · DOI
- Ahmad Khan, 2017, Computer-aided reading of tuberculosis chest radiography: moving the research agenda forward to inform policy, Eur Respir J, 50, 10.1183/13993003.00953-2017 · DOI · Paper page
- Zaidi, 2018, Evaluation of the diagnostic accuracy of computer-aided detection of tuberculosis on chest radiography among private sector patients in Pakistan, Sci Rep, 8, 12339, 10.1038/s41598-018-30810-1 · DOI
- Qin, 2020, Can artificial intelligence (AI) be used to accurately detect tuberculosis (TB) from chest x-ray? a multiplatform evaluation of five AI products used for TB screening in a high TB-burden setting, arXiv
Cited by
Sorted by citations
Geric, C., Qin, Z. Z., Denkinger, C. M., Kik, S. V., Marais, B., Anjos, A., David, P.-M., Ahmad Khan, F., & Trajman, A. (2023) · The International Journal of Tuberculosis and Lung Disease · doi:10.5588/ijtld.22.0687 · Cited by 61 · see citing works
Self-citationVanobberghen, F., Keter, A. K., Jacobs, B. K. M., Glass, T. R., Lynen, L., Law, I., Murphy, K., van Ginneken, B., Ayakaka, I., van Heerden, A., et al. (2024) · ERJ Open Research · doi:10.1183/23120541.00508-2023 · Cited by 13 · see citing works
“…CAD4TB (Delft Imaging Systems, Netherlands), recommend scores tailored to each setting due to variation in CAD abnormality scores between different software and populations [5, 6, 8 – 13], yet context-specific determination is often not done due to logistical and financial constraints [8, 14, 15].”
BackgroundResultInfluentialSelf-citation
- Authors
- Tavaziva, G., Harris, M., Abidi, S. K., Geric, C., Breuninger, M., Dheda, K., Esmail, A., Muyoyeta, M., Reither, K., Majidulla, A., Khan, A. J., Campbell, J. R., David, P.-M., Denkinger, C., Miller, C., Nathavitharana, R., Pai, M., Benedetti, A., & Ahmad Khan, F.
- Affiliations
- Research Institute of the McGill University Health Centre (CA)
- Boston University (US)
- University of Cologne (DE)
- University of Cape Town (ZA)
- Centre for Infectious Disease Research in Zambia (ZM)
- Swiss Tropical and Public Health Institute (CH)
- Interactive Research and Development (PK)
- Institut de Recherche pour le Développement (FR)
- Université de Montréal (CA)
- University Hospital Heidelberg (DE)
- World Health Organization (CH)
- Beth Israel Deaconess Medical Center (US)
- McGill International TB Centre, Research Institute of the McGill University Health Centre, Montreal, Canada
- ORCID
- Venue
- Clinical Infectious Diseases
- Publisher
- Oxford University Press (OUP)
- Published
- 2021-07-21
- Volume
- 74
- Issue
- 8
- Pages
- 1390-1400
- ISSN
- 1058-4838 (ISSN-L)
- 1537-6591
- PMC
- PMC9049274
- OpenAlex
- W3184797773
- Cited by
- 79
- References
- 4
- Citations by year
- 2026: 14 · 2025: 17 · 2024: 15 · 2023: 18 · 2022: 12 · 2021: 3
- FWCI
- 1.40
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Clinical Infectious Diseases (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Kölner Universitäts PublikationsServer (Universität zu Köln) (publishedVersion)
- edoc (University of Basel) (acceptedVersion)
- LSHTM Research Online (London School of Hygiene and Tropical Medicine) (acceptedVersion)
- Funders
- World Health Organization
- McGill University
- Lunit
- Canadian Institutes of Health Research
- National Institute on Drug Abuse (R25DA037756)
- Topics
- Health Sciences › Medicine › Infectious Diseases › Tuberculosis Research and Epidemiology
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Keywords
- pulmonary tuberculosis, chest radiography, computer-aided detection, deep learning, diagnostic accuracy, triage, HIV coinfection, smear-negative tuberculosis, CAD4TB, sensitivity and specificity, tuberculosis screening, people living with HIV, smear-positive tuberculosis
- MeSH terms
- Deep Learning, Humans, Sensitivity and Specificity, Software, Triage, Tuberculosis, Tuberculosis, Pulmonary, X-Rays, HIV Infections
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
