Assessment of the effect of a comprehensive chest radiograph deep learning model on radiologist reports and patient outcomes: a real-world observational study
Open accessThe DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
Objectives: Artificial intelligence (AI) algorithms have been developed to detect imaging features on chest X-ray (CXR) with a comprehensive AI model capable of detecting 124 CXR findings being recently developed. The aim of this study was to evaluate the real-world usefulness of the model as a diagnostic assistance device for radiologists. Design: This prospective real-world multicentre study involved a group of radiologists using the model in their daily reporting workflow to report consecutive CXRs and recording their feedback on level of agreement with the model findings and whether this significantly affected their reporting. Setting: The study took place at radiology clinics and hospitals within a large radiology network in Australia between November and December 2020. Participants: Eleven consultant diagnostic radiologists of varying levels of experience participated in this study. Primary and secondary outcome measures: Proportion of CXR cases where use of the AI model led to significant material changes to the radiologist report, to patient management, or to imaging recommendations. Additionally, level of agreement between radiologists and the model findings, and radiologist attitudes towards the model were assessed. Results: Of 2972 cases reviewed with the model, 92 cases (3.1%) had significant report changes, 43 cases (1.4%) had changed patient management and 29 cases (1.0%) had further imaging recommendations. In terms of agreement with the model, 2569 cases showed complete agreement (86.5%). 390 (13%) cases had one or more findings rejected by the radiologist. There were 16 findings across 13 cases (0.5%) deemed to be missed by the model. Nine out of 10 radiologists felt their accuracy was improved with the model and were more positive towards AI poststudy. Conclusions: Use of an AI model in a real-world reporting environment significantly improved radiologist reporting and showed good agreement with radiologists, highlighting the potential for AI diagnostic support to improve clinical practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Controlling the false discovery rate in behavior genetics research
Benjamini, Y., Drai, D., Elmer, G., Kafkafi, N., & Golani, I. (2001) · Behavioural Brain Research · doi:10.1016/s0166-4328(01)00297-2 · Cited by 4,146
Artificial intelligence in radiology
Hosny, A., Parmar, C., Quackenbush, J., Schwartz, L. H., & Aerts, H. J. W. L. (2018) · Nature Reviews Cancer · doi:10.1038/s41568-018-0016-5 · Cited by 3,803
Lakhani, P., & Sundaram, B. (2017) · Radiology · doi:10.1148/radiol.2017162326 · Cited by 1,775
Machine learning for medical imaging
Erickson, B. J., Korfiatis, P., Akkus, Z., & Kline, T. L. (2017) · RadioGraphics · doi:10.1148/rg.2017160130 · Cited by 1,672
Deep learning-enabled medical computer vision
Esteva, A., Chou, K., Yeung, S., Naik, N., Madani, A., Mottaghi, A., Liu, Y., Topol, E., Dean, J., & Socher, R. (2021) · npj Digital Medicine · doi:10.1038/s41746-020-00376-2 · Cited by 1,406
Hwang, E. J., Park, S., Jin, K.-N., Kim, J. I., Choi, S. Y., Lee, J. H., Goo, J. M., Aum, J., Yim, J.-J., Cohen, J. G., et al. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.1095 · Cited by 447
Cognitive and System Factors Contributing to Diagnostic Errors in Radiology
Lee, C. S., Nagy, P. G., Weaver, S. J., & Newman-Toker, D. E. (2013) · American Journal of Roentgenology · doi:10.2214/ajr.12.10375 · Cited by 415
Kim, J. Y., Choe, P. G., Oh, Y., Oh, K. J., Kim, J., Park, S. J., Park, J. H., Na, H. K., & Oh, M.-D. (2020) · Journal of Korean Medical Science · doi:10.3346/jkms.2020.35.e61 · Cited by 404
Qin, Z. Z., Sander, M. S., Rai, B., Titahong, C. N., Sudrungrot, S., Laah, S. N., Adhikari, L. M., Carter, E. J., Puri, L., Codlin, A. J., et al. (2019) · Scientific Reports · doi:10.1038/s41598-019-51503-3 · Cited by 289
Sim, Y., Chung, M. J., Kotter, E., Yune, S., Kim, M., Do, S., Han, K., Kim, H., Yang, S., Lee, D.-J., et al. (2020) · Radiology · doi:10.1148/radiol.2019182465 · Cited by 248
Deep learning in chest radiography: Detection of findings and presence of change
Singh, R., Kalra, M. K., Nitiwarangkul, C., Patti, J. A., Homayounieh, F., Padole, A., Rao, P., Putha, P., Muse, V. V., Sharma, A., et al. (2018) · PLOS ONE · doi:10.1371/journal.pone.0204155 · Cited by 195
Deep Learning for Chest Radiograph Diagnosis in the Emergency Department
Hwang, E. J., Nam, J. G., Lim, W. H., Park, S. J., Jeong, Y. S., Kang, J. H., Hong, E. K., Kim, T. M., Goo, J. M., Park, S., et al. (2019) · Radiology · doi:10.1148/radiol.2019191225 · Cited by 164
Khan, F. A., Majidulla, A., Tavaziva, G., Nazish, A., Abidi, S. K., Benedetti, A., Menzies, D., Johnston, J. C., Khan, A. J., & Saeed, S. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30221-1 · Cited by 134
Nash, M., Kadavigere, R., Andrade, J., Sukumar, C. A., Chawla, K., Shenoy, V. P., Pande, T., Huddart, S., Pai, M., & Saravu, K. (2020) · Scientific Reports · doi:10.1038/s41598-019-56589-3 · Cited by 86
Park, S., Lee, S. M., Kim, N., Choe, J., Cho, Y., Do, K.-H., & Seo, J. B. (2019) · European Radiology · doi:10.1007/s00330-019-06130-x · Cited by 71
Liang, C.-H., Liu, Y.-C., Wu, M.-T., Garcia-Castro, F., Alberich-Bayarri, A., & Wu, F.-Z. (2020) · Clinical Radiology · doi:10.1016/j.crad.2019.08.005 · Cited by 69
Cited by
Heterogeneity and predictors of the effects of AI assistance on radiologists
Yu, F., Moehring, A., Banerjee, O., Salz, T., Agarwal, N., & Rajpurkar, P. (2024) · Nature Medicine · doi:10.1038/s41591-024-02850-w · Cited by 200
Artificial intelligence for diagnostics in radiology practice: a rapid systematic scoping review
Lawrence, R., Dodsworth, E., Massou, E., Sherlaw-Johnson, C., Ramsay, A. I. G., Walton, H., O'Regan, T., Gleeson, F., Crellin, N., Herbert, K., et al. (2025) · eClinicalMedicine · doi:10.1016/j.eclinm.2025.103228 · Cited by 69
Shelmerdine, S. C., Togher, D., Rickaby, S., & Dean, G. (2024) · Clinical Radiology · doi:10.1016/j.crad.2024.05.018 · Cited by 17
Storey, M., Packer, J., Grandal, N. S., Rhodes, A., Moon, J., Togher, D., Abubacker, Z., Rickaby, S., Dean, G., & Shelmerdine, S. C. (2026) · NEJM AI · doi:10.1056/aics2500539 · Cited by 4
Metadata
- Authors
- Jones, C. M., Danaher, L., Milne, M. R., Tang, C., Seah, J., Oakden-Rayner, L., Johnson, A., Buchlak, Q. D., & Esmaili, N.
- Venue
- BMJ Open
- Publisher
- BMJ
- Published
- 2021-12
- Volume
- 11
- Issue
- 12
- Pages
- e052902
- ISSN
- 2044-6055 (ISSN-L)
- OpenAlex
- W4200486940
- Cited by
- 45
- References
- 17
- FWCI
- 1.53
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.