Cited by 352Dementia and Cognitive Impairment ResearchMedical Image Segmentation TechniquesArtificial Intelligence in Healthcare

Standardized evaluation of algorithms for computer-aided diagnosis of dementia based on structural MRI: the CADDementia challenge

Open access
Bron, Esther E., Smits, Marion, van der Flier, Wiesje M., Vrenken, Hugo, Barkhof, Frederik, Scheltens, Philip, Papma, Janne M., Steketee, Rebecca M.E. +42 more·2015·NeuroImage

Abstract

Algorithms for computer-aided diagnosis of dementia based on structural MRI have demonstrated high performance in the literature, but are difficult to compare as different data sets and methodology were used for evaluation. In addition, it is unclear how the algorithms would perform on previously unseen data, and thus, how they would perform in clinical practice when there is no real opportunity to adapt the algorithm to the data at hand. To address these comparability, generalizability and clinical applicability issues, we organized a grand challenge that aimed to objectively compare algorithms based on a clinically representative multi-center data set. Using clinical practice as the starting point, the goal was to reproduce the clinical diagnosis. Therefore, we evaluated algorithms for multi-class classification of three diagnostic groups: patients with probable Alzheimer's disease, patients with mild cognitive impairment and healthy controls. The diagnosis based on clinical criteria was used as reference standard, as it was the best available reference despite its known limitations. For evaluation, a previously unseen test set was used consisting of 354 T1-weighted MRI scans with the diagnoses blinded. Fifteen research teams participated with a total of 29 algorithms. The algorithms were trained on a small training set (n=30) and optionally on data from other sources (e.g., the Alzheimer's Disease Neuroimaging Initiative, the Australian Imaging Biomarkers and Lifestyle flagship study of aging). The best performing algorithm yielded an accuracy of 63.0% and an area under the receiver-operating-characteristic curve (AUC) of 78.8%. In general, the best performances were achieved using feature extraction based on voxel-based morphometry or a combination of features that included volume, cortical thickness, shape and intensity. The challenge is open for new submissions via the web-based framework: http://caddementia.grand-challenge.org.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.neuroimage.2015.01.048

Cites

68 of 68 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Traditional machine learning methods are considerably useful for analyzing unified data modalities such as biomarkers, genetic information, neuroimaging, and connected speech.

    “traditional ML paradigms have considerable utility in analyzing unified data modalities, including biomarkers,16 genetic information (including emerging polygenic risk scores),17 neuroimaging,18 and connected speech.19”
  • Many AI techniques need sophisticated computational infrastructure that may not be available in typical clinical settings.

    “Many AI techniques require sophisticated computational infrastructure that may not be readily available in typical clinical settings.16”

    From Review of Artificial Intelligence for Clinical Use in Alzheimer's Disease and Related Dementias (2026) · cites this paper as [16]

    Section 3.2 reports computation times for individual dementia-diagnosis algorithms, but does not say they require sophisticated infrastructure or that such infrastructure is unavailable in typical clinics. The Introduction and section 5.1 discuss barriers to clinical implementation involving generalizability, training data, and algorithm availability instead. Searches for hardware, computing resources, and infrastructure found no passage addressing the attributed claim; page numbers were not available in our copy. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 12

  1. Abdulkadir A Peter J Brox T Ronneberger O Klöppel S Voxel-based multi-class classification of AD, MCI, and elderly controls: Blind evaluation on an independent test set Proc MICCAI workshop Challenge on Computer-Aided Diagnosis of Dementia Based on Structural MRI Data 2014 8 15
  2. Amoroso N Errico R Bellotti R PRISMA-CAD: Fully automated method for Computer-Aided Diagnosis of Dementia based on structural MRI data Proc MICCAI workshop Challenge on Computer-Aided Diagnosis of Dementia Based on Structural MRI Data 2014 16 23
  3. Bron EE Smits M van Swieten JC Niessen WJ Klein S Proc MICCAI workshop Challenge on Computer-Aided Diagnosis of Dementia Based on Structural MRI Data 2014
  4. Cárdenas-Peña D Álvarez Meza A Castellanos-Dominguez G CADDementia based on structural MRI using Supervised Kernel-based Representations Proc MICCAI workshop Challenge on Computer-Aided Diagnosis of Dementia Based on Structural MRI Data 2014 24 30
  5. Dolph CV Samad MD Iftekharuddin KM Classification of Alzheimer’s disease using structural MRI Proc MICCAI workshop Challenge on Computer-Aided Diagnosis of Dementia Based on Structural MRI Data 2014 31 7
  6. Franke K Gaser C Dementia classification based on brain age estimation Proc MICCAI workshop Challenge on Computer-Aided Diagnosis of Dementia Based on Structural MRI Data 2014 48 54
  7. Ledig C Guerrero R Tong T Gray K Makropoulos A Heckemann RA Rueckert D Alzheimer’s disease state classification using structural volumetry, cortical thickness and intensity features Proc MICCAI workshop Challenge on Computer-Aided Diagnosis of Dementia Based on Structural MRI Data 2014 55 64
  8. Moradi E Gaser C Huttunen H Tohka J MRI based dementia classification using semi-supervised learning and domain adaptation Proc MICCAI workshop Challenge on Computer-Aided Diagnosis of Dementia Based on Structural MRI Data 2014 65 73
  9. Sarica A di Fatta G Smith G Cannataro M Saddy JD Advanced Feature Selection in Multinominal Dementia Classification from Structural MRI Data Proc MICCAI workshop Challenge on Computer-Aided Diagnosis of Dementia Based on Structural MRI Data 2014 82 91
  10. Sensi F Rei L Gemme G Bosco P Chincarini A Global Disease Index, a novel tool for MTL atrophy assessment Proc MICCAI workshop Challenge on Computer-Aided Diagnosis of Dementia Based on Structural MRI Data 2014 92 100
  11. Tangaro S Inglese P Maglietta R Tateo A MIND-BA: Fully automated method for Computer-Aided Diagnosis of Dementia based on structural MRI data Proc MICCAI workshop Challenge on Computer-Aided Diagnosis of Dementia Based on Structural MRI Data 2014 119 28
  12. Wachinger C Batmanghelich K Golland P Reuter M BrainPrint in the Computer-Aided Diagnosis of Alzheimer’s Disease Proc MICCAI workshop Challenge on Computer-Aided Diagnosis of Dementia Based on Structural MRI Data 2014a 129 38

Cited by

Sorted by citations

Authors
Bron, E. E., Smits, M., van der Flier, W. M., Vrenken, H., Barkhof, F., Scheltens, P., Papma, J. M., Steketee, R. M. E., Méndez Orellana, C., Meijboom, R., Pinto, M., Meireles, J. R., Garrett, C., Bastos-Leite, A. J., Abdulkadir, A., Ronneberger, O., Amoroso, N., Bellotti, R., Cárdenas-Peña, D., Álvarez-Meza, A. M., Dolph, C. V., Iftekharuddin, K. M., Eskildsen, S. F., Coupé, P., Fonov, V. S., Franke, K., Gaser, C., Ledig, C., Guerrero, R., Tong, T., Gray, K. R., Moradi, E., Tohka, J., Routier, A., Durrleman, S., Sarica, A., Di Fatta, G., Sensi, F., Chincarini, A., Smith, G. M., Stoyanov, Z. V., Sørensen, L., Nielsen, M., Tangaro, S., Inglese, P., Wachinger, C., Reuter, M., van Swieten, J. C., Niessen, W. J., & Klein, S.
Affiliations
  • Erasmus MC (NL)
  • Amsterdam Neuroscience (NL)
  • Hospital de São João (PT)
  • Universidade do Porto (PT)
  • University of Freiburg (DE)
  • Istituto Nazionale di Fisica Nucleare, Sezione di Bari (IT)
  • Universidad Nacional de Colombia (CO)
  • Old Dominion University (US)
  • Aarhus University (DK)
  • Montreal Neurological Institute and Hospital (CA)
  • Jena University Hospital (DE)
  • Imperial College London (GB)
  • Tampere University of Technology (FI)
  • Centre National de la Recherche Scientifique (FR)
  • Magna Graecia University (IT)
  • Istituto Nazionale di Fisica Nucleare (IT)
  • University of Reading (GB)
  • University of Copenhagen (DK)
  • Harvard University (US)
Corresponding author
Bron, Esther E.
Venue
NeuroImage
Publisher
Elsevier BV
Published
2015-05
Volume
111
Pages
562-579
ISSN
1053-8119 (ISSN-L)
OpenAlex
W2135011268
Cited by
352
References
68
Citations by year
2026: 21 · 2025: 30 · 2024: 22 · 2023: 29 · 2022: 25 · 2021: 32 · 2020: 28 · 2019: 46 · 2018: 31 · 2017: 39 · 2016: 39 · 2015: 10
FWCI
32.16
Type
journal article
Language
en
Open access
Locations
  • NeuroImage (publishedVersion)
  • PubMed (publishedVersion)
  • Amsterdam UMC (VU Amsterdam) - Institutional Repository (publishedVersion)
  • EUR Research Repository (Erasmus University Rotterdam) (publishedVersion)
  • PubMed Central (acceptedVersion)
  • CentAUR (University of Reading) (submittedVersion)
  • EUR Research Repository (Erasmus University Rotterdam) (submittedVersion)
  • RePub (Erasmus University Rotterdam) (submittedVersion, other-oa)
  • HAL (submittedVersion, other-oa)
  • UCL Discovery (University College London) (submittedVersion)
  • Pure Amsterdam UMC (submittedVersion)
  • Research at the University of Copenhagen (University of Copenhagen) (submittedVersion)
  • CINECA IRIS Institutional Research Information System (University of Bari Aldo Moro) (submittedVersion, cc-by-nc-nd)
  • Spiral (Imperial College London) (submittedVersion, cc-by-nc-nd)
  • Open Research (University of Surrey) (submittedVersion)
  • INFM-OAR (INFN Catania) (submittedVersion, other-oa)
  • Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
  • Data Archiving and Networked Services (DANS) (submittedVersion)
  • Istituto Nazionale di Fisica Nucleare. Centro Nazionale Analisi Fotogrammi (cc-by)
Funders
  • Alzheimer's Disease Neuroimaging Initiative
  • Commonwealth Scientific and Industrial Research Organisation
  • National Institutes of Health (U01 AG024904)
  • U.S. Department of Defense (W81XWH-12-2-0012)
Topics
  • Health Sciences › Medicine › Psychiatry and Mental health › Dementia and Cognitive Impairment Research
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
computer-aided diagnosis, dementia, Alzheimer's disease, mild cognitive impairment, structural MRI, societal challenges, algorithm comparison, generalizability, multi-center data, voxel-based morphometry, cortical thickness, T1-weighted imaging
MeSH terms
Aged, Aged, 80 and over, Algorithms, Alzheimer Disease, Diagnosis, Computer-Assisted, Female, Humans, Image Interpretation, Computer-Assisted, Magnetic Resonance Imaging, Male, Middle Aged, Sensitivity and Specificity, Cognitive Dysfunction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review