Student or educator? Sign up with your school email and get 6 months of Pro free.
Cited by 229

The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.

Effect of a comprehensive deep-learning model on the accuracy of chest x-ray interpretation by radiologists: a retrospective, multireader multicase study

Open access
Seah, Jarrel C Y, Tang, Cyril H M, Buchlak, Quinlan D, Holt, Xavier G, Wardman, Jeffrey B, Aimoldin, Anuar, Esmaili, Nazanin, Ahmad, Hassan, Pham, Hung, Lambert, John F, Hachey, Ben, Hogg, Stephen J F, Johnston, Benjamin P, Bennett, Christine, Oakden-Rayner, Luke, Brotchie, Peter, Jones, Catherine M·2021·The Lancet Digital Health

Abstract

BACKGROUND: Chest x-rays are widely used in clinical practice; however, interpretation can be hindered by human error and a lack of experienced thoracic radiologists. Deep learning has the potential to improve the accuracy of chest x-ray interpretation. We therefore aimed to assess the accuracy of radiologists with and without the assistance of a deep-learning model. METHODS: In this retrospective study, a deep-learning model was trained on 821 681 images (284 649 patients) from five data sets from Australia, Europe, and the USA. 2568 enriched chest x-ray cases from adult patients (≥16 years) who had at least one frontal chest x-ray were included in the test dataset; cases were representative of inpatient, outpatient, and emergency settings. 20 radiologists reviewed cases with and without the assistance of the deep-learning model with a 3-month washout period. We assessed the change in accuracy of chest x-ray interpretation across 127 clinical findings when the deep-learning model was used as a decision support by calculating area under the receiver operating characteristic curve (AUC) for each radiologist with and without the deep-learning model. We also compared AUCs for the model alone with those of unassisted radiologists. If the lower bound of the adjusted 95% CI of the difference in AUC between the model and the unassisted radiologists was more than -0·05, the model was considered to be non-inferior for that finding. If the lower bound exceeded 0, the model was considered to be superior. FINDINGS: Unassisted radiologists had a macroaveraged AUC of 0·713 (95% CI 0·645-0·785) across the 127 clinical findings, compared with 0·808 (0·763-0·839) when assisted by the model. The deep-learning model statistically significantly improved the classification accuracy of radiologists for 102 (80%) of 127 clinical findings, was statistically non-inferior for 19 (15%) findings, and no findings showed a decrease in accuracy when radiologists used the deep-learning model. Unassisted radiologists had a macroaveraged mean AUC of 0·713 (0·645-0·785) across all findings, compared with 0·957 (0·954-0·959) for the model alone. Model classification alone was significantly more accurate than unassisted radiologists for 117 (94%) of 124 clinical findings predicted by the model and was non-inferior to unassisted radiologists for all other clinical findings. INTERPRETATION: This study shows the potential of a comprehensive deep-learning model to improve chest x-ray interpretation across a large breadth of clinical practice. FUNDING: Annalise.ai.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/s2589-7500(21)00106-0

Cites

13 of 13 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 13 references

Research works printed but not linked: 38

  1. Lee, 2013, Cognitive and system factors contributing to diagnostic errors in radiology, AJR Am J Roentgenol, 201, 611, 10.2214/AJR.12.10375 · DOI · Paper page
  2. Del Ciello, 2017, Missed lung cancer: when, where, and why?, Diagn Interv Radiol, 23, 118, 10.5152/dir.2016.16187 · DOI · Paper page
  3. Brady, 2017, Error and discrepancy in radiology: inevitable or avoidable?, Insights Imaging, 8, 171, 10.1007/s13244-016-0534-1 · DOI · Paper page
  4. Rajpurkar, 2017, Chexnet: radiologist-level pneumonia detection on chest x-rays with deep learning, arXiv
  5. Wu, 2020, Comparison of chest radiograph interpretations by artificial intelligence algorithm vs radiology residents, JAMA Netw Open, 3, 10.1001/jamanetworkopen.2020.22779 · DOI
  6. Wang, 2018, Chestnet: a deep neural network for classification of thoracic diseases on chest radiography, arXiv
  7. Moffett, 2016, Comparing posteroanterior with lateral and anteroposterior chest radiography in the initial detection of parapneumonic effusions, Am J Emerg Med, 34, 2402, 10.1016/j.ajem.2016.09.021 · doi:10.1016/j.ajem.2016.09.021
  8. Gordienko Y, Gang P, Hui J, et al. Deep learning with lung segmentation and bone shadow exclusion techniques for chest x-ray analysis of lung cancer. International Conference on Computer Science, Engineering and Education Applications; Kiev, Ukraine; Jan 18–20, 2018 (638–47). · DOI
  9. Qin, 2019, Using artificial intelligence to read chest radiographs for tuberculosis detection: a multi-site evaluation of the diagnostic accuracy of three deep learning systems, Sci Rep, 9, 10.1038/s41598-019-51503-3 · DOI · Paper page
  10. Kim, 2020, Clinical validation of a deep learning algorithm for detection of pneumonia on chest radiographs in emergency department patients with acute febrile respiratory illness, J Clin Med, 9
  11. Hurt, 2020, Augmenting interpretation of chest radiographs with deep learning probability maps, J Thorac Imaging, 35, 285, 10.1097/RTI.0000000000000505 · DOI
  12. Bassi, 2020, A deep convolutional neural network for COVID-19 detection using chest x-rays, arXiv
  13. Hwang, 2020, Deep learning algorithm for surveillance of pneumothorax after lung biopsy: a multicenter diagnostic cohort study, Eur Radiol, 30, 3660, 10.1007/s00330-020-06771-3 · DOI
  14. Wang, 2020, Potential of deep learning in assessing pneumoconiosis depicted on digital chest radiography, Occup Environ Med, 77, 597, 10.1136/oemed-2019-106386 · DOI
  15. Jang, 2020, Deep learning-based automatic detection algorithm for reducing overlooked lung cancers on chest radiographs, Radiology, 296, 652, 10.1148/radiol.2020200165 · DOI
  16. Singh, 2019, Assessment of critical feeding tube malpositions on radiographs using deep learning, J Digit Imaging, 32, 651, 10.1007/s10278-019-00229-9 · DOI
  17. Nam, 2019, Development and validation of deep learning-based automatic detection algorithm for malignant pulmonary nodules on chest radiographs, Radiology, 290, 218, 10.1148/radiol.2018180237 · DOI · Paper page
  18. Cicero, 2017, Training and validating a deep convolutional neural network for computer-aided detection and classification of abnormalities on frontal chest radiographs, Invest Radiol, 52, 281, 10.1097/RLI.0000000000000341 · DOI · Paper page
  19. Tschandl, 2020, Human-computer collaboration for skin cancer recognition, Nat Med, 26, 1229, 10.1038/s41591-020-0942-0 · DOI · Paper page
  20. Nash, 2020, Deep learning, computer-aided radiography reading for tuberculosis: a diagnostic accuracy study from a tertiary hospital in India, Sci Rep, 10, 210, 10.1038/s41598-019-56589-3 · DOI · Paper page
  21. Park, 2020, Deep learning-based detection system for multiclass lesions on chest radiographs: comparison with observer readings, Eur Radiol, 30, 1359, 10.1007/s00330-019-06532-x · DOI
  22. Johnson, 2019, MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports, Sci Data, 6, 317, 10.1038/s41597-019-0322-0 · DOI · Paper page
  23. Irvin J, Rajpurkar P, Ko M, et al. Chexpert: a large chest radiograph dataset with uncertainty labels and expert comparison. Association for the Advancement of Artificial Intelligence Conference on Artificial Intelligence; Honolulu, HI, USA; Jan 27 to Feb 1, 2019 (590–97). · DOI · Paper page
  24. Chinchor N, Sundheim BM. MUC-5 evaluation metrics. In: Fifth Message Understanding Conference (MUC-5): Proceedings of a Conference Held in Baltimore, MD, USA; Aug 25–27, 1993. · doi:10.3115/1072017.1072023
  25. Gallas, 2014, Generalized Roe and Metz receiver operating characteristic model: analytic link between simulated decision scores and empirical AUC variances and covariances, J Med Imaging, 1, 10.1117/1.JMI.1.3.031006 · DOI
  26. Benjamini, 1995, Controlling the false discovery rate: a practical and powerful approach to multiple testing, J R Stat Soc B, 57, 289, 10.1111/j.2517-6161.1995.tb02031.x · DOI · Paper page
  27. Baldi, 2000, Assessing the accuracy of prediction algorithms for classification: an overview, Bioinformatics, 16, 412, 10.1093/bioinformatics/16.5.412 · DOI · Paper page
  28. Obuchowski, 2019, Statistical considerations for testing an AI algorithm used for prescreening lung CT images, Contemp Clin Trials Commun, 16, 10.1016/j.conctc.2019.100434 · DOI
  29. Gennaro, 2018, The “perfect” reader study, Eur J Radiol, 103, 139, 10.1016/j.ejrad.2018.03.014 · DOI
  30. Yates, 2018, Machine learning “red dot”: open-source, cloud, deep convolutional neural networks in chest radiograph binary normality classification, Clin Radiol, 73, 827, 10.1016/j.crad.2018.05.015 · DOI
  31. Elkin, 2008, NLP-based identification of pneumonia cases from free-text radiological reports, AMIA Annu Symp Proc, 2008, 172
  32. Berbaum, 2015, Satisfaction of search in chest radiography 2015, Acad Radiol, 22, 1457, 10.1016/j.acra.2015.07.011 · DOI
  33. Shiraishi J, Abe H, Li F, Engelmann R, MacMahon H, Doi K. Computer-aided diagnosis for the detection and classification of lung cancers on chest radiographs: ROC analysis of radiologists' performance. Academic Radiol 13: 995–1003. · DOI
  34. Dendumrongsup, 2014, Multi-reader multi-case studies using the area under the receiver operator characteristic curve as a measure of diagnostic accuracy: systematic review with a focus on quality of data reporting, PLoS One, 9, 10.1371/journal.pone.0116018 · DOI
  35. Buchlak, 2020, Ethical thinking machines in surgery and the requirement for clinical leadership, Am J Surg, 220, 1372, 10.1016/j.amjsurg.2020.06.073 · DOI
  36. Adil, 2021, Predicting the individual treatment effect of neurosurgery for TBI patients in the low resource setting: a machine learning approach in Uganda, J Neurotrauma, 38, 928, 10.1089/neu.2020.7262 · DOI
  37. Mould, 1995, The early history of x-ray diagnosis with emphasis on the contributions of physics 1895-1915, Phys Med Biol, 40, 1741, 10.1088/0031-9155/40/11/001 · doi:10.1088/0031-9155/40/11/001
  38. Garland, 1959, Studies on the accuracy of diagnostic procedures, Am J Roentgenol Radium Ther Nucl Med, 82, 25

Cited by

See all 229

Metadata

Authors
Seah, J. C. Y., Tang, C. H. M., Buchlak, Q. D., Holt, X. G., Wardman, J. B., Aimoldin, A., Esmaili, N., Ahmad, H., Pham, H., Lambert, J. F., Hachey, B., Hogg, S. J. F., Johnston, B. P., Bennett, C., Oakden-Rayner, L., Brotchie, P., & Jones, C. M.
Venue
The Lancet Digital Health
Publisher
Elsevier BV
Published
2021-08
Volume
3
Issue
8
Pages
e496-e506
ISSN
2589-7500 (ISSN-L)
OpenAlex
W3177044444
Cited by
229
References
13
FWCI
8.33
Type
journal article
Language
en
Open access
gold
Licence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Effect of a comprehensive deep-learning model on the accuracy of chest x-ray interpretation by radiologists: a retrospective, multireader multicase study. https://app.cochat.ai/papers/10.1016/s2589-7500(21)00106-0
The paper
Seah, J. C. Y., Tang, C. H. M., Buchlak, Q. D., Holt, X. G., Wardman, J. B., Aimoldin, A., Esmaili, N., Ahmad, H., Pham, H., Lambert, J. F., Hachey, B., Hogg, S. J. F., Johnston, B. P., Bennett, C., Oakden-Rayner, L., Brotchie, P., & Jones, C. M. (2021). Effect of a comprehensive deep-learning model on the accuracy of chest x-ray interpretation by radiologists: a retrospective, multireader multicase study. The Lancet Digital Health. https://doi.org/10.1016/s2589-7500(21)00106-0

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.