CoChat
Sign Up
Cited by 106COVID-19 diagnosis using AITuberculosis Research and EpidemiologyAI in cancer detection

Independent evaluation of 12 artificial intelligence solutions for the detection of tuberculosis

Open access
Codlin, Andrew J., Dao, Thang Phuoc, Vo, Luan Nguyen Quang, Forse, Rachel J., Van Truong, Vinh, Dang, Ha Minh, Nguyen, Lan Huu, Nguyen, Hoa Binh +5 more·2021·Scientific Reports
DOI 10.1038/s41598-021-03265-0PMC PMC8668935
Open this paper in CoChat — free accountOpen PDF

Abstract

There have been few independent evaluations of computer-aided detection (CAD) software for tuberculosis (TB) screening, despite the rapidly expanding array of available CAD solutions. We developed a test library of chest X-ray (CXR) images which was blindly re-read by two TB clinicians with different levels of experience and then processed by 12 CAD software solutions. Using Xpert MTB/RIF results as the reference standard, we compared the performance characteristics of each CAD software against both an Expert and Intermediate Reader, using cut-off thresholds which were selected to match the sensitivity of each human reader. Six CAD systems performed on par with the Expert Reader (Qure.ai, DeepTek, Delft Imaging, JF Healthcare, OXIPIT, and Lunit) and one additional software (Infervision) performed on par with the Intermediate Reader only. Qure.ai, Delft Imaging and Lunit were the only software to perform significantly better than the Intermediate Reader. The majority of these CAD software showed significantly lower performance among participants with a past history of TB. The radiography equipment used to capture the CXR image was also shown to affect performance for some CAD software. TB program implementers now have a wide selection of quality CAD software solutions to utilize in their CXR screening initiatives.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41598-021-03265-0

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Specificity was poorer in those with history of tuberculosis, as expected and corresponding to findings in other settings.

    “Lastly, specificity was poorer in those with history of tuberculosis, as expected and corresponding to findings in other settings [11, 17, 23, 43].”

    From Computer-aided detection thresholds for digital chest radiography interpretation in tuberculosis diagnostic algorithms. (2024) · cites this paper as [43]

    The Discussion reports significantly lower CAD ROC AUC in people with a history of TB for six of seven top-performing solutions. It also explains that residual abnormalities can produce high CAD scores despite negative Xpert results, consistent with poorer specificity. The paper does not directly report a specificity comparison by TB history or establish the claim about findings in other settings. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 38

  1. [3]Creswell J Sahu S Blok L Bakker MI Stevens R Ditiu L A multi-site evaluation of innovative approaches to increase tuberculosis case notification: Summary results PLoS ONE 2014 9 4 e94465 24722399 PMC3983196
  2. [4]Miller C Lonnroth K Sotgiu G Migliori GB The long and winding road of chest radiography for tuberculosis detection Eur. Respir. J. 2017 49 5 1700364 28529204
  3. [5]World Health Organization WHO Consolidated Guidelines on Tuberculosis Module 2: Screening—Systematic Screening for Tuberculosis Disease 2021 World Health Organization 33822560
  4. [6]Morishita F Garfin AMCG Lew W Oh KH Yadav R-P Reston JC Bringing state-of-the-art diagnostics to vulnerable populations: The use of a mobile screening unit in active case finding for tuberculosis in Palawan, the Philippines PLoS ONE 2017 12 2 e0171310 28152082 PMC5289556
  5. [7]Camelique O Scholtissen S Dousset J-P Bonnet M Bastard M Hewison C Mobile community-based active case-finding for tuberculosis among older populations in rural Cambodia Int. J. Tuberc. Lung Dis. 2019 23 10 1107 1114 31627776
  6. [8]Madhani F Maniar RA Burfat A Ahmed M Farooq S Sabir A Automated chest radiography and mass systematic screening for tuberculosis Int. J. Tuberc. Lung Dis. 2020 24 7 665 673 32718398
  7. [9]Nguyen LH Codlin AJ Vo LNQ Dao T Tran D Forse RJ An evaluation of programmatic community-based chest X-ray screening for tuberculosis in Ho Chi Minh City Vietnam. Trop. Med. Infect. Dis. 2020 5 4 185 PMC7768495 33321696
  8. [10]Codlin AJ Monyrath C Ky M Gerstel L Creswell J Eang MT Results from a roving, active case finding initiative to improve tuberculosis detection among older people in rural Cambodia using the Xpert MTB/RIF assay and chest X-ray J. Clin. Tuberc. Other Mycobact Dis. 2018 13 22 27 31720408 PMC6830163
  9. [11]Fehr J Konigorski S Olivier S Gunda R Surujdeen A Gareta D Computer-aided interpretation of chest radiography to detect TB in rural South Africa MedRxiv. 2020 10.1101/2020.09.04.20188045 · DOI
  10. [13]Pande T Pai M Khan FA Denkinger CM Use of chest radiography in the 22 highest tuberculosis burden countries Eur. Respir. J. 2015 46 6 1816 1819 26405288
  11. [15]Liang G Fan W Luo H Zhu X The emerging roles of artificial intelligence in cancer drug development and precision therapy Biomed. Pharmacother. 2020 128 110255 32446113
  12. [16]Paul D Sanap G Shenoy S Kalyane D Kalia K Tekade RK Artificial intelligence in drug discovery and development Drug Discov. Today. 2021 26 1 80 93 33099022 PMC7577280
  13. [17]Mohapatra S Swarnkar T Mishra D Buyya R Mohapatra P Patnaik S Artificial intelligence for smart healthcare management: Brief study Intelligent and Cloud Computing 2021 Springer 365 373
  14. [18]Shen J Zhang CJP Jiang B Chen J Song J Liu Z Artificial intelligence versus clinicians in disease diagnosis: Systematic review JMIR Med. Inform. 2019 7 3 e10010 31420959 PMC6716335
  15. [19]Kulkarni S Jha S Artificial intelligence, radiology, and tuberculosis: A review Acad. Radiol. 2020 27 1 71 75 31759796
  16. [20]Maduskar P Muyoyeta M Ayles H Hogeweg L Peters-Bax L van Ginneken B Detection of tuberculosis using digital chest radiography: Automated reading vs interpretation by clinical officers Int. J. Tuberc. Lung Dis. 2013 17 12 1613 1620 24200278
  17. [21]Breuninger M van Ginneken B Philipsen RHHM Mhimbira F Hella JJ Lwilla F Diagnostic accuracy of computer-aided detection of pulmonary tuberculosis in chest radiographs: A validation study from sub-Saharan Africa PLoS ONE 2014 9 9 e106381 25192172 PMC4156349
  18. [22]Philipsen RHHM Sánchez CI Maduskar P Melendez J Peters-Bax L Peter JG Automated chest-radiography as a triage for Xpert testing in resource-constrained settings: A prospective study of diagnostic accuracy and costs Sci. Rep. 2015 5 12215 26212560 PMC4515744
  19. [23]Rahman MT Codlin AJ Rahman MM Nahar A Reja M Islam T An evaluation of automated chest radiography reading software for tuberculosis screening among public- and private-sector patients Eur. Respir. J. 2017 49 5 1602159 28529202 PMC5460641
  20. [24]Harris M Qi A Jeagal L Torabi N Menzies D Korobitsyn A A systematic review of the diagnostic accuracy of artificial intelligence-based computer programs to analyze chest x-rays for pulmonary tuberculosis PLoS ONE 2019 14 9 e0221339 31479448 PMC6719854
  21. [25]Pande T Cohen C Pai M Ahmad KF Computer-aided detection of pulmonary tuberculosis on digital chest radiographs: A systematic review Int. J. Tuberc. Lung Dis. 2016 20 9 1226 1230 27510250
  22. [26]Khan FA Majidulla A Tavaziva G Nazish A Abidi SK Benedetti A Chest X-ray analysis with deep learning-based software as a triage test for pulmonary tuberculosis: A prospective study of diagnostic accuracy for culture-confirmed disease Lancet Digit. Health 2020 2 11 e573 e581 33328086
  23. [27]Qin ZZ Sander MS Rai B Titahong CN Sudrungrot S Laah SN Using artificial intelligence to read chest radiographs for tuberculosis detection: A multi-site evaluation of the diagnostic accuracy of three deep learning systems Sci. Rep. 2019 9 1 1 10 31628424 PMC6802077
  24. [28]Nash M Kadavigere R Andrade J Sukumar CA Chawla K Shenoy VP Deep learning, computer-aided radiography reading for tuberculosis: A diagnostic accuracy study from a tertiary hospital in India Sci. Rep. 2020 10 1 1 10 31937802 PMC6959311
  25. [29]Qin ZZ Ahmed S Sarker MS Paul K Adel ASS Naheyan T Tuberculosis detection from chest x-rays for triaging in a high tuberculosis-burden setting: An evaluation of five artificial intelligence algorithms Lancet Digit Health. 2021 3 9 e543 e554 34446265
  26. [31]Law I Floyd K National tuberculosis prevalence surveys in Africa, 2008–2016: An overview of results and lessons learned Trop. Med. Int. Health 2020 25 11 1308 1327 32910557 PMC8043149
  27. [32]Onozaki I Law I Sismanidis C Zignol M Glaziou P Floyd K National tuberculosis prevalence surveys in Asia, 1990–2012: An overview of results and lessons learned Trop Med Int Health 2015 20 9 1128 1145 25943163
  28. [33]Vo LNQ Forse RJ Codlin AJ Vu TN Le GT Do GC A comparative impact evaluation of two human resource models for community-based active tuberculosis case finding in Ho Chi Minh City, Viet Nam BMC Public Health 2020 20 1 934 32539700 PMC7296629
  29. [34]Mac TH Phan TH Nguyen VV Dong TTT Le HV Nguyen QD Optimizing active tuberculosis case finding: Evaluating the impact of community referral for chest X-ray screening and Xpert testing on case notifications in two cities in Viet Nam Trop. Med. Infect. Dis. 2020 5 4 181 PMC7709663 33265972
  30. [35]Theron G Venter R Smith L Esmail A Randall P Sood V False-positive Xpert MTB/RIF results in retested patients with previous tuberculosis: Frequency, profile, and prospective clinical outcomes J. Clin. Microbiol. 2018 10.1128/JCM.01696-17 29305538 PMC5824043 · DOI
  31. [36]Nyboe J Results of the international study on X-ray classification Bull. Int. Union Tuberc. 1968 41 115 124 5710244
  32. [37]Kosack CS Spijker S Halton J Bonnet M Nicholas S Chetcuti K Evaluation of a chest radiograph reading and recording system for tuberculosis in a HIV-positive cohort Clin. Radiol. 2017 72 6 e1 e9 28236438
  33. [40]Stec N Arje D Moody AR Krupinski EA Tyrrell PN A systematic review of fatigue in radiology: Is it a problem? Am. J. Roentgenol. 2018 210 4 799 806 29446673
  34. [41]Dembrower K Wåhlin E Liu Y Salim M Smith K Lindholm P Effect of artificial intelligence-based triaging of breast cancer screening mammograms on cancer detection and radiologist workload: A retrospective simulation study Lancet Digit. Health 2020 2 9 e468 e474 33328114
  35. [42]McKinney SM Sieniek M Godbole V Godwin J Antropova N Ashrafian H International evaluation of an AI system for breast cancer screening Nature 2020 577 7788 89 94 31894144
  36. [43]Horne DJ Kohli M Zifodya JS Schiller I Dendukuri N Tollefson D Xpert MTB/RIF and Xpert MTB/RIF Ultra for pulmonary tuberculosis and rifampicin resistance in adults Cochrane Database Syst. Rev. 2019 6 CD009593 31173647 PMC6555588
  37. [44]Sander M Laah S Titahong C Lele C Kinge T de Jong B Systematic screening for tuberculosis among hospital outpatients in Cameroon: The role of screening and testing algorithms to improve case detection J. Clin. Tuberc. Other Mycobact. Dis. 2019 15 100095 31720422 PMC6830146
  38. [45]Vo LNQ Codlin AJ Forse RJ Nguyen NT Vu TN Le GT Evaluating the yield of systematic screening for tuberculosis among three priority groups in Ho Chi Minh City, Viet Nam Infect. Dis. Poverty 2020 9 1 166 33292638 PMC7724701

Cited by

Sorted by citations

Authors
Codlin, A. J., Dao, T. P., Vo, L. N. Q., Forse, R. J., Van Truong, V., Dang, H. M., Nguyen, L. H., Nguyen, H. B., Nguyen, N. V., Sidney-Annerstedt, K., Squire, B., Lönnroth, K., & Caws, M.
Affiliations
  • Institut de Recherche pour le Développement (FR)
  • Karolinska Institutet (SE)
  • Pham Ngoc Thach Hospital (VN)
  • Hanoi Lung Hospital (VN)
  • Liverpool School of Tropical Medicine (GB)
Corresponding author
Codlin, Andrew J.
ORCID
Venue
Scientific Reports
Publisher
Springer Science and Business Media LLC
Published
2021-12-13
Volume
11
Issue
1
Pages
23895-23895
ISSN
2045-2322 (ISSN-L)
DOI
10.1038/s41598-021-03265-0
PMC
PMC8668935
OpenAlex
W4200587732
Cited by
106
References
3
Citations by year
2026: 22 · 2025: 22 · 2024: 24 · 2023: 28 · 2022: 10
FWCI
8.19
Type
journal article
Language
en
Open access
Licence
Locations
  • Scientific Reports (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • LSTM Online Archive (Liverpool School of Tropical Medicine) (acceptedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • European Commission (HORIZON2020, 733174)
  • Global Affairs Canada
Topics
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
  • Health Sciences › Medicine › Infectious Diseases › Tuberculosis Research and Epidemiology
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
Keywords
tuberculosis detection, computer-aided detection, chest X-ray screening, Xpert MTB/RIF, reader experience, radiographic equipment
MeSH terms
Machine Learning, Adolescent, Adult, Female, Humans, Male, Middle Aged, Radiographic Image Interpretation, Computer-Assisted, Software, Radiography, Thoracic, Tuberculosis, Pulmonary

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review