Radiology artificial intelligence: a systematic review and evaluation of methods (RAISE)
Open accessThe DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
Objective: There has been a large amount of research in the field of artificial intelligence (AI) as applied to clinical radiology. However, these studies vary in design and quality and systematic reviews of the entire field are lacking.This systematic review aimed to identify all papers that used deep learning in radiology to survey the literature and to evaluate their methods. We aimed to identify the key questions being addressed in the literature and to identify the most effective methods employed. Methods: We followed the PRISMA guidelines and performed a systematic review of studies of AI in radiology published from 2015 to 2019. Our published protocol was prospectively registered. Results: Our search yielded 11,083 results. Seven hundred sixty-seven full texts were reviewed, and 535 articles were included. Ninety-eight percent were retrospective cohort studies. The median number of patients included was 460. Most studies involved MRI (37%). Neuroradiology was the most common subspecialty. Eighty-eight percent used supervised learning. The majority of studies undertook a segmentation task (39%). Performance comparison was with a state-of-the-art model in 37%. The most used established architecture was UNet (14%). The median performance for the most utilised evaluation metrics was Dice of 0.89 (range .49–.99), AUC of 0.903 (range 1.00–0.61) and Accuracy of 89.4 (range 70.2–100). Of the 77 studies that externally validated their results and allowed for direct comparison, performance on average decreased by 6% at external validation (range increase of 4% to decrease 44%). Conclusion: This systematic review has surveyed the major advances in AI as applied to clinical radiology. Key Points: • While there are many papers reporting expert-level results by using deep learning in radiology, most apply only a narrow range of techniques to a narrow selection of use cases. • The literature is dominated by retrospective cohort studies with limited external validation with high potential for bias. • The recent advent of AI extensions to systematic reporting guidelines and prospective trial registration along with a focus on external validation and explanations show potential for translation of the hype surrounding AI from code to clinic.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“AI research in radiology is an active field in healthcare owing to its affinity for imaging [14], with the number of AI-related publications and medical device certifications increasing annually [15–17].”
From Fairness of artificial intelligence in healthcare: review and recommendations (2024) · cites this paper as [17]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Checklist for Artificial Intelligence in Medical Imaging (CLAIM): a guide for authors and reviewers
Mongan, J., Moy, L., & Kahn, C. E. (2020) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2020200029 · Cited by 1,443 · cited 1×
“The quality of research design and reporting varies, and guidelines such as those issued by CONSORT [26] and the RSNA [27] should be followed to improve the quality of research overall.”
DARPA's Explainable Artificial Intelligence Program
Gunning, D., & Aha, D. W. (2019) · AI Magazine · doi:10.1609/aimag.v40i2.2850 · Cited by 1,360 · cited 1×
“We reviewed included papers for a variety of established methods of explanation [23] including examples, visualisations, natural language and counterfactuals.”
BackgroundThrall, J. H., Li, X., Li, Q., Cruz, C., Do, S., Dreyer, K., & Brink, J. (2018) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2017.12.026 · Cited by 710
Liu, X., Rivera, S. C., Moher, D., Calvert, M. J., & Denniston, A. K. (2020) · BMJ · doi:10.1136/bmj.m3164 · Cited by 702
Muehlematter, U. J., Daniore, P., & Vokinger, K. N. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30292-2 · Cited by 665
Canadian Association of Radiologists White Paper on Artificial Intelligence in Radiology
Tang, A., Tam, R., Cadrin-Chênevert, A., Guest, W., Chong, J., Barfett, J., Chepelev, L., Cairns, R., Mitchell, J. R., Cicero, M. D., et al. (2018) · Canadian Association of Radiologists Journal · doi:10.1016/j.carj.2018.02.002 · Cited by 522
Meaningful information and the right to explanation
Selbst, A. D., & Powles, J. (2017) · International Data Privacy Law · doi:10.1093/idpl/ipx022 · Cited by 516
Differential Data Augmentation Techniques for Medical Imaging Classification Tasks.
Hussain, Z., Gimenez, F., Yi, D., & Rubin, D. L. (2017) · PubMed · Cited by 469
Research works printed but not linked: 1
- Kelleher DJ, Namee BM, D’arcy A (2020) Fundamentals of machine learning for predictive data analytics: algorithms, worked examples, and case studies (2nd edn). MIT Press, p 856
Cited by
Fairness of artificial intelligence in healthcare: review and recommendations
Ueda, D., Kakinuma, T., Fujita, S., Kamagata, K., Fushimi, Y., Ito, R., Matsui, Y., Nozaki, T., Nakaura, T., Fujima, N., et al. (2024) · Japanese Journal of Radiology · doi:10.1007/s11604-023-01474-3 · Cited by 607
“AI research in radiology is an active field in healthcare owing to its affinity for imaging [14], with the number of AI‑related publications and medical device certifications increasing annually [15–17].”
BackgroundCellViT: Vision Transformers for precise cell segmentation and classification
Hörst, F., Rempe, M., Heine, L., Seibold, C., Keyl, J., Baldini, G., Ugurel, S., Siveke, J., Grünwald, B., Egger, J., et al. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103143 · Cited by 338
AI in imaging: the regulatory landscape
Hill, D. L. G. (2024) · British Journal of Radiology · doi:10.1093/bjr/tqae002 · Cited by 52
“23 There are increasing numbers of papers that highlight the risks that many AI products may not be sufficiently safe and effective for clinical use, and encourage the development of suitable “ comprehensive guidelines for their implementation ” .”
BackgroundAgentic AI and large language models in radiology: opportunities and hallucination challenges
Salehi, S., Singh, Y., Horst, K. K., Hathaway, Q. A., & Erickson, B. J. (2025) · Bioengineering · doi:10.3390/bioengineering12121303 · Cited by 32
“Limitation: No standardized protocols exist for evaluating multi-agent systems against single-agent baselines in prospective trials [17].”
Artificial Intelligence in Radiology: Opportunities and Challenges
Flory, M. N., Napel, S., & Tsai, E. B. (2024) · Seminars in Ultrasound, CT and MRI · doi:10.1053/j.sult.2024.02.004 · Cited by 29
Metadata
- Authors
- Kelly, B. S., Judge, C., Bollard, S. M., Clifford, S. M., Healy, G. M., Aziz, A., Mathur, P., Islam, S., Yeom, K. W., Lawlor, A., & Killeen, R. P.
- Affiliations
- University College Dublin (IE)
- Lucile Packard Children's Hospital (US)
- St. Vincent's University Hospital (IE)
- Hammersmith Hospital (GB)
- Ollscoil na Gaillimhe – University of Galway (IE)
- Corresponding author
- Kelly, Brendan S.
- ORCID
- Venue
- European Radiology
- Published
- 2022-04-14
- Volume
- 32
- Issue
- 11
- Pages
- 7998-8007
- ISSN
- 0938-7994 (ISSN-L)
- 1432-1084
- PMC
- PMC9668941
- OpenAlex
- W4223974852
- Cited by
- 252
- References
- 28
- Citations by year
- 2026: 54 · 2025: 85 · 2024: 71 · 2023: 37 · 2022: 4
- FWCI
- 3.40
- Type
- journal article
- Language
- en
- Update notices
- correction (2022-05-20)
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- European Radiology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- Wellcome Trust
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- artificial intelligence in radiology, deep learning, medical image analysis, external validation, study design quality, segmentation, U-Net, supervised learning, magnetic resonance imaging, neuroradiology, reporting guidelines, clinical translation of AI
- MeSH terms
- Artificial Intelligence, Humans, Prospective Studies, Radiography, Radiology, Retrospective Studies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Radiology artificial intelligence: a systematic review and evaluation of methods (RAISE). https://app.cochat.ai/papers/10.1007/s00330-022-08784-6
- The paper
- Kelly, B. S., Judge, C., Bollard, S. M., Clifford, S. M., Healy, G. M., Aziz, A., Mathur, P., Islam, S., Yeom, K. W., Lawlor, A., & Killeen, R. P. (2022). Radiology artificial intelligence: a systematic review and evaluation of methods (RAISE). European Radiology. https://doi.org/10.1007/s00330-022-08784-6
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.