The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Effect of a comprehensive deep-learning model on the accuracy of chest x-ray interpretation by radiologists: a retrospective, multireader multicase study
Open accessAbstract
BACKGROUND: Chest x-rays are widely used in clinical practice; however, interpretation can be hindered by human error and a lack of experienced thoracic radiologists. Deep learning has the potential to improve the accuracy of chest x-ray interpretation. We therefore aimed to assess the accuracy of radiologists with and without the assistance of a deep-learning model. METHODS: In this retrospective study, a deep-learning model was trained on 821 681 images (284 649 patients) from five data sets from Australia, Europe, and the USA. 2568 enriched chest x-ray cases from adult patients (≥16 years) who had at least one frontal chest x-ray were included in the test dataset; cases were representative of inpatient, outpatient, and emergency settings. 20 radiologists reviewed cases with and without the assistance of the deep-learning model with a 3-month washout period. We assessed the change in accuracy of chest x-ray interpretation across 127 clinical findings when the deep-learning model was used as a decision support by calculating area under the receiver operating characteristic curve (AUC) for each radiologist with and without the deep-learning model. We also compared AUCs for the model alone with those of unassisted radiologists. If the lower bound of the adjusted 95% CI of the difference in AUC between the model and the unassisted radiologists was more than -0·05, the model was considered to be non-inferior for that finding. If the lower bound exceeded 0, the model was considered to be superior. FINDINGS: Unassisted radiologists had a macroaveraged AUC of 0·713 (95% CI 0·645-0·785) across the 127 clinical findings, compared with 0·808 (0·763-0·839) when assisted by the model. The deep-learning model statistically significantly improved the classification accuracy of radiologists for 102 (80%) of 127 clinical findings, was statistically non-inferior for 19 (15%) findings, and no findings showed a decrease in accuracy when radiologists used the deep-learning model. Unassisted radiologists had a macroaveraged mean AUC of 0·713 (0·645-0·785) across all findings, compared with 0·957 (0·954-0·959) for the model alone. Model classification alone was significantly more accurate than unassisted radiologists for 117 (94%) of 124 clinical findings predicted by the model and was non-inferior to unassisted radiologists for all other clinical findings. INTERPRETATION: This study shows the potential of a comprehensive deep-learning model to improve chest x-ray interpretation across a large breadth of clinical practice. FUNDING: Annalise.ai.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.324 · Cited by 30,581
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072
Comparison of the predicted and observed secondary structure of T4 phage lysozyme
Matthews, B. W. (1975) · Biochimica et Biophysica Acta (BBA) - Protein Structure · doi:10.1016/0005-2795(75)90109-9 · Cited by 5,399
Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., & Summers, R. M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.369 · Cited by 4,616
Class-Balanced Loss Based on Effective Number of Samples
Cui, Y., Jia, M., Lin, T.-Y., Song, Y., & Belongie, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00949 · Cited by 3,094
Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm
Dawid, A. P., & Skene, A. M. (1979) · Applied Statistics · doi:10.2307/2346806 · Cited by 1,692
Machine learning for medical imaging
Erickson, B. J., Korfiatis, P., Akkus, Z., & Kline, T. L. (2017) · RadioGraphics · doi:10.1148/rg.2017160130 · Cited by 1,672
PadChest: A large chest x-ray image dataset with multi-label annotated reports
Bustos, A., Pertusa, A., Salinas, J.-M., & de la Iglesia-Vayá, M. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101797 · Cited by 691
Hidden stratification causes clinically meaningful failures in machine learning for medical imaging
Oakden-Rayner, L., Dunnmon, J., Carneiro, G., & Re, C. (2020) · Proceedings of the ACM Conference on Health, Inference, and Learning · doi:10.1145/3368555.3384468 · Cited by 411
Self-citation
Research works printed but not linked: 38
- Lee, 2013, Cognitive and system factors contributing to diagnostic errors in radiology, AJR Am J Roentgenol, 201, 611, 10.2214/AJR.12.10375 · DOI · Paper page
- Del Ciello, 2017, Missed lung cancer: when, where, and why?, Diagn Interv Radiol, 23, 118, 10.5152/dir.2016.16187 · DOI · Paper page
- Brady, 2017, Error and discrepancy in radiology: inevitable or avoidable?, Insights Imaging, 8, 171, 10.1007/s13244-016-0534-1 · DOI · Paper page
- Rajpurkar, 2017, Chexnet: radiologist-level pneumonia detection on chest x-rays with deep learning, arXiv
- Wu, 2020, Comparison of chest radiograph interpretations by artificial intelligence algorithm vs radiology residents, JAMA Netw Open, 3, 10.1001/jamanetworkopen.2020.22779 · DOI
- Wang, 2018, Chestnet: a deep neural network for classification of thoracic diseases on chest radiography, arXiv
- Moffett, 2016, Comparing posteroanterior with lateral and anteroposterior chest radiography in the initial detection of parapneumonic effusions, Am J Emerg Med, 34, 2402, 10.1016/j.ajem.2016.09.021 · doi:10.1016/j.ajem.2016.09.021
- Gordienko Y, Gang P, Hui J, et al. Deep learning with lung segmentation and bone shadow exclusion techniques for chest x-ray analysis of lung cancer. International Conference on Computer Science, Engineering and Education Applications; Kiev, Ukraine; Jan 18–20, 2018 (638–47). · DOI
- Qin, 2019, Using artificial intelligence to read chest radiographs for tuberculosis detection: a multi-site evaluation of the diagnostic accuracy of three deep learning systems, Sci Rep, 9, 10.1038/s41598-019-51503-3 · DOI · Paper page
- Kim, 2020, Clinical validation of a deep learning algorithm for detection of pneumonia on chest radiographs in emergency department patients with acute febrile respiratory illness, J Clin Med, 9
- Hurt, 2020, Augmenting interpretation of chest radiographs with deep learning probability maps, J Thorac Imaging, 35, 285, 10.1097/RTI.0000000000000505 · DOI
- Bassi, 2020, A deep convolutional neural network for COVID-19 detection using chest x-rays, arXiv
- Hwang, 2020, Deep learning algorithm for surveillance of pneumothorax after lung biopsy: a multicenter diagnostic cohort study, Eur Radiol, 30, 3660, 10.1007/s00330-020-06771-3 · DOI
- Wang, 2020, Potential of deep learning in assessing pneumoconiosis depicted on digital chest radiography, Occup Environ Med, 77, 597, 10.1136/oemed-2019-106386 · DOI
- Jang, 2020, Deep learning-based automatic detection algorithm for reducing overlooked lung cancers on chest radiographs, Radiology, 296, 652, 10.1148/radiol.2020200165 · DOI
- Singh, 2019, Assessment of critical feeding tube malpositions on radiographs using deep learning, J Digit Imaging, 32, 651, 10.1007/s10278-019-00229-9 · DOI
- Nam, 2019, Development and validation of deep learning-based automatic detection algorithm for malignant pulmonary nodules on chest radiographs, Radiology, 290, 218, 10.1148/radiol.2018180237 · DOI · Paper page
- Cicero, 2017, Training and validating a deep convolutional neural network for computer-aided detection and classification of abnormalities on frontal chest radiographs, Invest Radiol, 52, 281, 10.1097/RLI.0000000000000341 · DOI · Paper page
- Tschandl, 2020, Human-computer collaboration for skin cancer recognition, Nat Med, 26, 1229, 10.1038/s41591-020-0942-0 · DOI · Paper page
- Nash, 2020, Deep learning, computer-aided radiography reading for tuberculosis: a diagnostic accuracy study from a tertiary hospital in India, Sci Rep, 10, 210, 10.1038/s41598-019-56589-3 · DOI · Paper page
- Park, 2020, Deep learning-based detection system for multiclass lesions on chest radiographs: comparison with observer readings, Eur Radiol, 30, 1359, 10.1007/s00330-019-06532-x · DOI
- Johnson, 2019, MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports, Sci Data, 6, 317, 10.1038/s41597-019-0322-0 · DOI · Paper page
- Irvin J, Rajpurkar P, Ko M, et al. Chexpert: a large chest radiograph dataset with uncertainty labels and expert comparison. Association for the Advancement of Artificial Intelligence Conference on Artificial Intelligence; Honolulu, HI, USA; Jan 27 to Feb 1, 2019 (590–97). · DOI · Paper page
- Chinchor N, Sundheim BM. MUC-5 evaluation metrics. In: Fifth Message Understanding Conference (MUC-5): Proceedings of a Conference Held in Baltimore, MD, USA; Aug 25–27, 1993. · doi:10.3115/1072017.1072023
- Gallas, 2014, Generalized Roe and Metz receiver operating characteristic model: analytic link between simulated decision scores and empirical AUC variances and covariances, J Med Imaging, 1, 10.1117/1.JMI.1.3.031006 · DOI
- Benjamini, 1995, Controlling the false discovery rate: a practical and powerful approach to multiple testing, J R Stat Soc B, 57, 289, 10.1111/j.2517-6161.1995.tb02031.x · DOI · Paper page
- Baldi, 2000, Assessing the accuracy of prediction algorithms for classification: an overview, Bioinformatics, 16, 412, 10.1093/bioinformatics/16.5.412 · DOI · Paper page
- Obuchowski, 2019, Statistical considerations for testing an AI algorithm used for prescreening lung CT images, Contemp Clin Trials Commun, 16, 10.1016/j.conctc.2019.100434 · DOI
- Gennaro, 2018, The “perfect” reader study, Eur J Radiol, 103, 139, 10.1016/j.ejrad.2018.03.014 · DOI
- Yates, 2018, Machine learning “red dot”: open-source, cloud, deep convolutional neural networks in chest radiograph binary normality classification, Clin Radiol, 73, 827, 10.1016/j.crad.2018.05.015 · DOI
- Elkin, 2008, NLP-based identification of pneumonia cases from free-text radiological reports, AMIA Annu Symp Proc, 2008, 172
- Berbaum, 2015, Satisfaction of search in chest radiography 2015, Acad Radiol, 22, 1457, 10.1016/j.acra.2015.07.011 · DOI
- Shiraishi J, Abe H, Li F, Engelmann R, MacMahon H, Doi K. Computer-aided diagnosis for the detection and classification of lung cancers on chest radiographs: ROC analysis of radiologists' performance. Academic Radiol 13: 995–1003. · DOI
- Dendumrongsup, 2014, Multi-reader multi-case studies using the area under the receiver operator characteristic curve as a measure of diagnostic accuracy: systematic review with a focus on quality of data reporting, PLoS One, 9, 10.1371/journal.pone.0116018 · DOI
- Buchlak, 2020, Ethical thinking machines in surgery and the requirement for clinical leadership, Am J Surg, 220, 1372, 10.1016/j.amjsurg.2020.06.073 · DOI
- Adil, 2021, Predicting the individual treatment effect of neurosurgery for TBI patients in the low resource setting: a machine learning approach in Uganda, J Neurotrauma, 38, 928, 10.1089/neu.2020.7262 · DOI
- Mould, 1995, The early history of x-ray diagnosis with emphasis on the contributions of physics 1895-1915, Phys Med Biol, 40, 1741, 10.1088/0031-9155/40/11/001 · doi:10.1088/0031-9155/40/11/001
- Garland, 1959, Studies on the accuracy of diagnostic procedures, Am J Roentgenol Radium Ther Nucl Med, 82, 25
Cited by
Rajpurkar, P., Chen, E., Banerjee, O., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-021-01614-0 · Cited by 3,212
The current and future state of AI interpretation of medical images
Rajpurkar, P., & Lungren, M. P. (2023) · New England Journal of Medicine · doi:10.1056/nejmra2301725 · Cited by 443
Benchmarking saliency methods for chest X-ray interpretation
Saporta, A., Gui, X., Agrawal, A., Pareek, A., Truong, S. Q. H., Nguyen, C. D. T., Ngo, V.-D., Seekins, J., Blankenberg, F. G., Ng, A. Y., et al. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00536-x · Cited by 270
Robust and data-efficient generalization of self-supervised machine learning for diagnostic imaging
Azizi, S., Culp, L., Freyberg, J., Mustafa, B., Baur, S., Kornblith, S., Chen, T., Tomasev, N., Mitrović, J., Strachan, P., et al. (2023) · Nature Biomedical Engineering · doi:10.1038/s41551-023-01049-7 · Cited by 250
Heterogeneity and predictors of the effects of AI assistance on radiologists
Yu, F., Moehring, A., Banerjee, O., Salz, T., Agarwal, N., & Rajpurkar, P. (2024) · Nature Medicine · doi:10.1038/s41591-024-02850-w · Cited by 200
Metadata
- Authors
- Seah, J. C. Y., Tang, C. H. M., Buchlak, Q. D., Holt, X. G., Wardman, J. B., Aimoldin, A., Esmaili, N., Ahmad, H., Pham, H., Lambert, J. F., Hachey, B., Hogg, S. J. F., Johnston, B. P., Bennett, C., Oakden-Rayner, L., Brotchie, P., & Jones, C. M.
- Venue
- The Lancet Digital Health
- Publisher
- Elsevier BV
- Published
- 2021-08
- Volume
- 3
- Issue
- 8
- Pages
- e496-e506
- ISSN
- 2589-7500 (ISSN-L)
- OpenAlex
- W3177044444
- Cited by
- 229
- References
- 13
- FWCI
- 8.33
- Type
- journal article
- Language
- en
- Open access
- gold
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- creativecommons.org/licenses/by/4.0/ (vor)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Effect of a comprehensive deep-learning model on the accuracy of chest x-ray interpretation by radiologists: a retrospective, multireader multicase study. https://app.cochat.ai/papers/10.1016/s2589-7500(21)00106-0
- The paper
- Seah, J. C. Y., Tang, C. H. M., Buchlak, Q. D., Holt, X. G., Wardman, J. B., Aimoldin, A., Esmaili, N., Ahmad, H., Pham, H., Lambert, J. F., Hachey, B., Hogg, S. J. F., Johnston, B. P., Bennett, C., Oakden-Rayner, L., Brotchie, P., & Jones, C. M. (2021). Effect of a comprehensive deep-learning model on the accuracy of chest x-ray interpretation by radiologists: a retrospective, multireader multicase study. The Lancet Digital Health. https://doi.org/10.1016/s2589-7500(21)00106-0
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.