CoChat

Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis

Open access

The DOI matches its Crossref record; 143 of the 143 references it cites were found in a registry or an index; 36 of the 128 sources its claims cite were checked for support.

Abstract

Deep learning (DL) has the potential to transform medical diagnostics. However, the diagnostic accuracy of DL is uncertain. Our aim was to evaluate the diagnostic accuracy of DL algorithms to identify pathology in medical imaging. Searches were conducted in Medline and EMBASE up to January 2020. We identified 11,921 studies, of which 503 were included in the systematic review. Eighty-two studies in ophthalmology, 82 in breast disease and 115 in respiratory disease were included for meta-analysis. Two hundred twenty-four studies in other specialities were included for qualitative review. Peer-reviewed studies that reported on the diagnostic accuracy of DL algorithms to identify pathology using medical imaging were included. Primary outcomes were measures of diagnostic accuracy, study design and reporting standards in the literature. Estimates were pooled using random-effects meta-analysis. In ophthalmology, AUC’s ranged between 0.933 and 1 for diagnosing diabetic retinopathy, age-related macular degeneration and glaucoma on retinal fundus photographs and optical coherence tomography. In respiratory imaging, AUC’s ranged between 0.864 and 0.937 for diagnosing lung nodules or lung cancer on chest X-ray or CT scan. For breast imaging, AUC’s ranged between 0.868 and 0.909 for diagnosing breast cancer on mammogram, ultrasound, MRI and digital breast tomosynthesis. Heterogeneity was high between studies and extensive variation in methodology, terminology and outcome measures was noted. This can lead to an overestimation of the diagnostic accuracy of DL algorithms on medical imaging. There is an immediate need for the development of artificial intelligence-specific EQUATOR guidelines, particularly STARD, in order to provide guidance around key issues in this field.

Overview

CoChat Summary

TL;DR

This systematic review and meta-analysis found that deep learning algorithms generally achieve high diagnostic accuracy in medical imaging, with pooled AUCs ranging from 0.864 to 1 across ophthalmology, respiratory, and breast imaging. However, high heterogeneity and widespread methodological/reporting deficiencies mean these estimates are uncertain and may overestimate true accuracy.

Key findings

  • Across 503 included studies, pooled AUCs were high: ophthalmology 0.933–1 for diabetic retinopathy, age-related macular degeneration, and glaucoma on retinal fundus photographs or OCT; respiratory 0.864–0.937 for lung nodules or lung cancer on chest X-ray or CT; breast 0.868–0.909 for breast cancer on mammogram, ultrasound, MRI, or digital breast tomosynthesis.
  • Heterogeneity was high between studies, with many I² values above 75% and often near 99% (e.g., I² 99.9% for diabetic retinopathy on retinal fundus photographs; 99.6% for tuberculosis on X-ray).
  • QUADAS-2 assessment classified a majority as high or unclear risk: patient selection 59/82 (72%) ophthalmic, 89/115 (77%) respiratory, 62/82 (76%) breast; flow/timing 66/82 (80%), 93/115 (81%), 70/82 (85%); reference standard applicability concerns 60/82 (73%), 104/115 (90%), 78/82 (95%).
  • Only 8 ophthalmology, 10 respiratory, and 6 breast studies mentioned STARD-2015 adherence or included a STARD flow diagram; no studies provided a prespecified sample size calculation, and few used prospective data (8 ophthalmology, 2 respiratory, 0 breast) or external validation (29 ophthalmology, 13 respiratory, 8 breast).

Methods

  • Systematic review and meta-analysis conducted per PRISMA-DTA; Medline and EMBASE searched up to 3 January 2020; peer-reviewed English studies reporting diagnostic accuracy of deep learning algorithms on medical imaging in human populations were included.
  • Searches identified 11,921 abstracts; 503 studies fulfilled inclusion criteria for the systematic review, with meta-analysis in ophthalmology (82 studies), respiratory medicine (115), and breast cancer (82), plus qualitative synthesis of 224 studies in other specialities.
  • Data were extracted independently by two investigators; risk of bias/applicability assessed with QUADAS-2 by three investigators; bivariate models and DerSimonian and Laird random-effects meta-analysis were used, with I² for heterogeneity and SROC where raw data were available; analyses used Stata version 15; PROSPERO CRD42020167503.

Limitations

  • The paper states that many included studies had methodological deficiencies or poor reporting, so the pooled diagnostic performance estimates are uncertain and may represent an overestimation of true accuracy; it also states that no quality assessment for transparency of reporting was conducted because STARD-2015 was not designed for deep learning studies.
  • The paper states that classical statistical comparison between imaging modalities was not possible, that each modality could not be separated into subsets to break down heterogeneity, and that findings may not generalise to other forms of medical imaging or AI studies.
  • Evident from the text: heterogeneity was high across studies (I² often >75%, frequently >99%), most studies were at high or unclear risk of bias on QUADAS-2, and few studies were prospective, externally validated, or had prespecified sample size calculations.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41746-021-00438-z

Cites

143 of 143 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 PRISMA flow diagram of included studies. PRISMA (preferred reporting items for systematic reviews and meta-analyses) flow diagram of included studies.
Fig. 1 PRISMA flow diagram of included studies. PRISMA (preferred reporting items for systematic reviews and meta-analyses) flow diagram of included studies.CC BY · PubMed Central
Fig. 2 QUADAS-2 summary plots. Risk of bias and applicability concerns summary about each QUADAS-2 domain presented as percentages across the 82 included studies in ophthalmic imaging (a), 115 in respiratory imaging (b) and 82 in breast imaging (c).
Fig. 2 QUADAS-2 summary plots. Risk of bias and applicability concerns summary about each QUADAS-2 domain presented as percentages across the 82 included studies in ophthalmic imaging (a), 115 in respiratory imaging (b) and 82 in breast imaging (c).CC BY · PubMed Central

Verification

Claims the paper makes

11 Valid · 3 Not valid · 32 Not checked · 165 Own results and interpretations

131 more claims · 3 with their cited sources checked

Create a free account to see all 211 claims and to verify the 92 references still open.

See all claims — free account
  • Probably not a substantial claim. 38
  • It may state two separate claims as one. 46
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 143 references

Cited by

See all 1,000

Metadata

Authors
Aggarwal, R., Sounderajah, V., Martin, G., Ting, D. S. W., Karthikesalingam, A., King, D., Ashrafian, H., & Darzi, A.
Affiliations
  • Imperial College London (GB)
  • Singapore National Eye Center (SG)
Corresponding author
Ashrafian, Hutan
Venue
npj Digital Medicine
Published
2021-04-07
Volume
4
Issue
1
Pages
65-65
ISSN
2398-6352 (ISSN-L)
OpenAlex
W3150212014
Cited by
946
References
143
Citations by year
2026: 188 · 2025: 259 · 2024: 196 · 2023: 166 · 2022: 104 · 2021: 31
FWCI
10.26
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Spiral (Imperial College London) (submittedVersion, cc-by)
Funders
  • National Institute for Health and Care Research (NF-SI-0510-10186, NF-SI-0617-10038, CL-2019-21-503, NIHR-INF-0738)
  • NIHR Imperial Biomedical Research Centre
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Retinal Imaging and Analysis
Keywords
deep learning, diagnostic accuracy, diagnostic imaging, artificial intelligence in healthcare, diabetic retinopathy, age-related macular degeneration, glaucoma, retinal fundus photographs, optical coherence tomography, pulmonary nodules, lung cancer, breast cancer, mammography, digital breast tomosynthesis, STARD, reporting guidelines

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis. https://app.cochat.ai/papers/10.1038/s41746-021-00438-z
The paper
Aggarwal, R., Sounderajah, V., Martin, G., Ting, D. S. W., Karthikesalingam, A., King, D., Ashrafian, H., & Darzi, A. (2021). Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis. npj Digital Medicine. https://doi.org/10.1038/s41746-021-00438-z

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.