The R-AI-DIOLOGY checklist: a practical checklist for evaluation of artificial intelligence tools in clinical neuroradiology
The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
Artificial intelligence (AI)-based tools are gradually blending into the clinical neuroradiology practice. Due to increasing complexity and diversity of such AI tools, it is not always obvious for the clinical neuroradiologist to capture the technical specifications of these applications, notably as commercial tools very rarely provide full details. The clinical neuroradiologist is thus confronted with the increasing dilemma to base clinical decisions on the output of AI tools without knowing in detail what is happening inside the "black box" of those AI applications. This dilemma is aggravated by the fact that currently, no established and generally accepted rules exist concerning best clinical practice and scientific and clinical validation nor for the medico-legal consequences in cases of wrong diagnoses. The current review article provides a practical checklist of essential points, intended to aid the user to identify and double-check necessary aspects, although we are aware that not all this information may be readily available at this stage, even for certified and commercially available AI tools. Furthermore, we therefore suggest that the developers of AI applications provide this information.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“unlike the R-AI-DIOLOGY checklist, which, apart from being focused explicitly on AI systems in radiology, only addresses the technical aspects of the design and development phases [33].”
From Checklist Approach to Developing and Implementing AI in Clinical Settings: Instrument Development Study (2025) · cites this paper as [33]
Citations
Cites
High-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450 · cited 1×
“Artificial intelligence (AI) and big data analyses are considered the driving forces of the fourth industrial revolution and many hopes have been raised regarding the application of AI in healthcare, particularly medical imaging [1].”
BackgroundDeep learning in medical image analysis
Shen, D., Wu, G., & Suk, H.-I. (2017) · Annual Review of Biomedical Engineering · doi:10.1146/annurev-bioeng-071516-044442 · Cited by 4,968 · cited 1×
“The performance of AI algorithms may be hindered by the so-called domain shift, which is rooted in differences between training data used during algorithm development and test data used either in a clinical study or in clinical practice [11].”
MethodConvolutional neural networks: an overview and application in radiology
Yamashita, R., Nishio, M., Do, R. K. G., & Togashi, K. (2018) · Insights into Imaging · doi:10.1007/s13244-018-0639-9 · Cited by 4,682
Big data and machine learning algorithms for health-care delivery
Ngiam, K. Y., & Khor, I. W. (2019) · The Lancet Oncology · doi:10.1016/s1470-2045(19)30149-4 · Cited by 1,424
Deep Learning: A Primer for Radiologists
Chartrand, G., Cheng, P. M., Vorontsov, E., Drozdzal, M., Turcotte, S., Pal, C. J., Kadoury, S., & Tang, A. (2017) · RadioGraphics · doi:10.1148/rg.2017170077 · Cited by 1,115 · cited 1×
“…modern and clinically available AI tools for neuroradiology, prior to their use in our daily routine Focusing on the presented clinical checklist, we did not aim to replicate already existing excellent review articles concerning in particular deep learning, included here for examples [8, 9].”
MethodPreparing Medical Imaging Data for Machine Learning
Willemink, M. J., Koszek, W. A., Hardell, C., Wu, J., Fleischmann, D., Harvey, H., Folio, L. R., Summers, R. M., Rubin, D. L., & Lungren, M. P. (2020) · Radiology · doi:10.1148/radiol.2020192224 · Cited by 1,020 · cited 1×
“For example, AI-based applications concerning stroke require faster turna-round times than volumetry-based methods in the context of neurodegeneration [25–27].”
MethodPark, S. H., & Han, K. (2018) · Radiology · doi:10.1148/radiol.2017171920 · Cited by 894 · cited 1×
“This is commonly referred to as spectrum bias or reference dataset bias, which results from an unrealistic distribution of disease severity in a training dataset [13, 14].”
BackgroundPesapane, F., Codari, M., & Sardanelli, F. (2018) · European Radiology Experimental · doi:10.1186/s41747-018-0061-6 · Cited by 833
Muehlematter, U. J., Daniore, P., & Vokinger, K. N. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30292-2 · Cited by 665 · cited 1×
“After an exponential increase of scientific publications regarding AI in (neuro)radiology in the last decade, AI-based applications for clinical use have been developed and cleared for clinical purposes [2–4].”
Backgroundvan Leeuwen, K. G., Schalekamp, S., Rutten, M. J. C. M., van Ginneken, B., & de Rooij, M. (2021) · European Radiology · doi:10.1007/s00330-021-07892-z · Cited by 461 · cited 1×
“Investigating the scientific evidence of 100 CE-marked and commercially available AI tools in radiology, the authors found that for the majority of tools (64/100), peer-reviewed evidence on its efficacy is lacking and only the minority (18/100) have demonstrated (potential) clinical impact [33].”
BackgroundSica, G. T. (2006) · Radiology · doi:10.1148/radiol.2383041109 · Cited by 331 · cited 1×
“This is commonly referred to as spectrum bias or reference dataset bias, which results from an unrealistic distribution of disease severity in a training dataset [13, 14].”
BackgroundHuisman, M., Ranschaert, E., Parker, W., Mastrodicasa, D., Koci, M., Pinto de Santos, D., Coppola, F., Morozov, S., Zins, M., Bohyn, C., et al. (2021) · European Radiology · doi:10.1007/s00330-021-07781-5 · Cited by 243 · cited 1×
“However, those with knowledge of AI were more likely to view the technology positively [6].”
BackgroundIdentification of Anonymous MRI Research Participants with Face-Recognition Software
Schwarz, C. G., Kremers, W. K., Therneau, T. M., Sharp, R. R., Gunter, J. L., Vemuri, P., Arani, A., Spychalla, A. J., Kantarci, K., Knopman, D. S., et al. (2019) · New England Journal of Medicine · doi:10.1056/nejmc1908881 · Cited by 215 · cited 1×
“, name and date of birth is removed, uploading a 3D dataset without removing the face area may be problematic regarding patient’s privacy [23, 24].”
Radiomics in Glioblastoma: Current Status and Challenges Facing Clinical Implementation
Chaddad, A., Kucharczyk, M. J., Daniel, P., Sabri, S., Jean-Claude, B. J., Niazi, T., & Abdulkarim, B. (2019) · Frontiers in Oncology · doi:10.3389/fonc.2019.00374 · Cited by 184 · cited 1×
“Another example is the reliability of segmentation of brain tumors in neuro-oncology, as new methods are being published trying to improve accuracy and reproducibility [19].”
MethodDeep Learning: An Update for Radiologists
Cheng, P. M., Montagnon, E., Yamashita, R., Pan, I., Cadrin-Chênevert, A., Perdigón Romero, F., Chartrand, G., Kadoury, S., & Tang, A. (2021) · RadioGraphics · doi:10.1148/rg.2021200210 · Cited by 166 · cited 1×
“…modern and clinically available AI tools for neuroradiology, prior to their use in our daily routine Focusing on the presented clinical checklist, we did not aim to replicate already existing excellent review articles concerning in particular deep learning, included here for examples [8, 9].”
MethodA methodology for the pseudonymization of medical data
Neubauer, T., & Heurix, J. (2010) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2010.10.016 · Cited by 146
Transfer learning in magnetic resonance brain imaging: a systematic review
Valverde, J. M., Imani, V., Abdollahzadeh, A., De Feo, R., Prakash, M., Ciszek, R., & Tohka, J. (2021) · Journal of Imaging · doi:10.3390/jimaging7040066 · Cited by 137 · cited 2×
“Artificial intelligence (AI) and big data analyses are considered the driving forces of the fourth industrial revolution and many hopes have been raised regarding the application of AI in healthcare, particularly medical imaging [1].”
BackgroundFree DICOM de-identification tools in clinical research: functioning and safety of patient privacy
Aryanto, K. Y. E., Oudkerk, M., & van Ooijen, P. M. A. (2015) · European Radiology · doi:10.1007/s00330-015-3794-0 · Cited by 109
Pemberton, H., Zaki, L. A. M., Goodkin, O., Das, R., Steketee, R. M. E., Barkhof, F., & Vernooij, M. W. (2021) · Neuroradiology · doi:10.1007/s00234-021-02746-3 · Cited by 90 · cited 1×
“Another recent study reviewed technical and clinical validation of brain morphometry for dementia and found that from 17 products, 11 companies published some form of technical validation, but only 4 used a dementia population [34].”
BackgroundSchwarz, C. G., Kremers, W. K., Wiste, H. J., Gunter, J. L., Vemuri, P., Spychalla, A. J., Kantarci, K., Schultz, A. P., Sperling, R. A., Knopman, D. S., et al. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.117845 · Cited by 88
Molina, D., Pérez-Beteta, J., Martínez‐González, A., Martino, J., Velásquez, C., Arana, E., & Pérez‐García, V. M. (2017) · PLoS ONE · doi:10.1371/journal.pone.0178843 · Cited by 70 · cited 1×
“This compromises the pooling and the reproducibility of published data, when using independent imaging sets [15].”
MethodArtificial Intelligence in Neuroradiology: Current Status and Future Directions.
Lui, Y. W., Chang, P., Zaharchuk, G., Barboriak, D. P., Flanders, A. E., Wintermark, M., Hess, C. P., & Filippi, C. G. (2020) · PubMed Central · doi:10.3174/ajnr.a6681 · Cited by 68 · cited 1×
“On the other hand, the neuroradiologist is confronted with an ever-increasing number of radiological exams [7] and oftentimes increased financial pressure due to reduced reimbursement in most countries.”
BackgroundImaging Quality Control in the Era of Artificial Intelligence
Larson, D. B., & Boland, G. W. (2019) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2019.05.048 · Cited by 45 · cited 1×
“Those tools without data quality check can still process the data in the “black box,” yet this might result potentially erroneous outputs and predictions [20].”
BackgroundHedderich, D. M., Keicher, M., Wiestler, B., Gruber, M. J., Burwinkel, H., Hinterwimmer, F., Czempiel, T., Spiro, J. E., Pinto dos Santos, D., Heim, D., et al. (2021) · Healthcare · doi:10.3390/healthcare9101278 · Cited by 42 · cited 1×
“However, their clinical implementation is still lagging behind, which may—at least in part—be due to a lack of training and knowledge of neuroradiologists with respect to these emerging tools and implications of their use in clinical practice [5].”
BackgroundSelf-citationBasic MR sequence parameters systematically bias automated brain volume estimation
Haller, S., Falkovskiy, P., Meuli, R. A., Thiran, J., Krüger, G., Lövblad, K., Kober, T., Roche, A., & Maréchal, B. (2016) · Neuroradiology · doi:10.1007/s00234-016-1737-3 · Cited by 31
Self-citationCopelan, A. Z., Smith, E. R., Drocton, G., Narsinh, K. H., Murph, D., Khangura, R. S., Hartley, Z. J., Abla, A. A., Dillon, W. P., Dowd, C. F., et al. (2020) · American Journal of Neuroradiology · doi:10.3174/ajnr.a6908 · Cited by 22
Research works printed but not linked: 1
- Biberacher V, Schmidt P, Keshavan A, Boucard CC, Righart R, Sämann P, Preibisch C, Fröbel D, Aly L, Hemmer B, Zimmer C, Henry RG, Mühlau M (2016) Intra- and interscanner variability of magnetic resonance imaging based volumetry in multiple sclerosis. Neuroimage 14:2188–197
Cited by
Reporting guidelines in medical artificial intelligence: a systematic review and meta-analysis
Kolbinger, F. R., Veldhuizen, G. P., Zhu, J., Truhn, D., & Kather, J. N. (2024) · Communications Medicine · doi:10.1038/s43856-024-00492-0 · Cited by 160
Jacob, C., Brasier, N., Laurenzi, E., Heuss, S., Mougiakakou, S. G., Çöltekin, A., & Peter, M. K. (2025) · Journal of Medical Internet Research · doi:10.2196/67485 · Cited by 145
“Although regulatory bodies like the FDA in the United States and CE marking in the EU set minimum safety and efficacy standards, there are significant gaps between legal certification and real-world clinical validation, workflow integration, and ongoing use [21,45].”
InfluentialGeneralizability of FDA-Approved AI-Enabled Medical Devices for Clinical Use
Windecker, D., Baj, G., Shiri, I., Kazaj, P. M., Kaesmacher, J., Gräni, C., & Siontis, G. C. M. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.8052 · Cited by 142
Artificial intelligence tools in clinical neuroradiology: essential medico-legal aspects
Hedderich, D. M., Weisstanner, C., Cauter, S. V., Federau, C., Edjlali, M., Radbruch, A., Gerke, S., & Haller, S. (2023) · Neuroradiology · doi:10.1007/s00234-023-03152-7 · Cited by 44
“As state-of-the-art CDS tools will (at least in the foreseeable future) not work in a fully automated way, the ultimate responsibility for a diagnostic or therapeutic decision will likely remain with the physician, who has to validate the results of the CDS tool [3, 5, 6].”
BackgroundSelf-citationDeveloping and deploying deep learning models in brain magnetic resonance imaging: A review
Aggarwal, K., Jimeno, M. M., Ravi, K. S., González, R. G., & Geethanath, S. (2023) · NMR in Biomedicine · doi:10.1002/nbm.5014 · Cited by 40
Metadata
- Authors
- Haller, S., Van Cauter, S., Federau, C., Hedderich, D. M., & Edjlali, M.
- Affiliations
- University of Geneva (CH)
- University of Zurich (CH)
- TUM Klinikum (DE)
- Centre National de la Recherche Scientifique (FR)
- Corresponding author
- Haller, Sven
- Venue
- Neuroradiology
- Published
- 2022-01-31
- Volume
- 64
- Issue
- 5
- Pages
- 851-864
- ISSN
- 0028-3940 (ISSN-L)
- 1432-1920
- OpenAlex
- W4210778361
- Cited by
- 33
- References
- 26
- Citations by year
- 2026: 4 · 2025: 11 · 2024: 9 · 2023: 8 · 2022: 1
- FWCI
- 0.50
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.springer.com/tdm (tdm)
- www.springer.com/tdm (vor)
- Locations
- Neuroradiology (publishedVersion)
- PubMed (publishedVersion)
- Document Server@UHasselt (UHasselt) (submittedVersion)
- mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) (submittedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiology practices and education
- Keywords
- AI tools, AI validation, black box problem, medicolegal liability, misdiagnosis, clinical decision-making
- MeSH terms
- Artificial Intelligence, Humans, Checklist
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.