A plea for confidence intervals and consideration of generalizability in diagnostic studies
Open accessThe DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.
Abstract
Sir, Many thanks for letting us respond to the interesting letter concerning our recent paper. We are grateful for the chance to clarify the points raised, which suggest our conclusions were too optimistic. In our paper (Kloppel et al., 2008), we used MRI scans from pathologically proven cases of Alzheimer's disease and frontotemporal lobar degeneration (FTLD) to validate trained sets for a machine learning-based support vector machine (SVM) approach to the categorization of structural scans from normal and each other. This rigorous approach substantially limited the number of available subjects, which we made perfectly clear in our article, but which was unavoidable given our novel approach. Frost and colleagues are right to point out that such low numbers result in larger confidence intervals than if we were able to include more scans. This is an object of our further empirical studies—what is the improvement in classification gained using this technique with greater numbers of scans in the trained set? The graph below (Fig. 1) illustrates diagnostic accuracy when the whole brain grey matter segment is used to separate probable Alzheimer's disease patients from all clinical stages (MMSE range of 3 to 30; defined clinically in the same way as group III, in our original paper) from controls. Classification is performed repeatedly and after removing one Alzheimer's disease patient and one control each time. Results are fairly stable but accuracy becomes more variable until a steep decline occurs when less than around 20 subjects per group are included. Suffice it to say we were surprised how well Alzheimer's disease was distinguished from FTLD given the even smaller numbers of validated scans we had available for that classification. To clarify these issues, we provide a table that supplements our data with CIs. Further, we found very similar results using two completely independent datasets and the CIs become relatively small when data from the first two datasets are combined. So, although we agree with the question posed theoretically, practically the results stand as proof of principle. Alzheimer's disease patients from all clinical stages (including all subjects from group III) and an equal number of age and sex matched cognitively normal controls are separated repeatedly. Before each classification, one patient and one control are removed to illustrate the robustness of classification with shrinking group size. Demographic information on groups I, II and IV with post-mortem confirmation of Alzheimer's disease obtained at different centres *MMSE scores obtained around the time of scanning only available from 12 subjects; **groups are age matched (P = 0.1). The first and third image sets are from a largely community-based sample, whereas subjects from sample II tended to be younger. No strong family history was present in any of the subjects. FTLD = frontotemporal lobar degeneration; MMSE = Folstein Mini Mental State Examination. (Source: Kloppel et al., 2008). Results of SVM classification using grey matter from the whole brain for image analysis aConsidering a correctly identified Alzheimer's disease case as a true positive. 95% CIs are calculated according to the efficient-score method (Newcombe, 1998; http://faculty.vassar.edu/lowry/clin1.html). Results of SVM classification using only grey matter of antero-medial temporal lobe structures for analysis aConsidering a correctly identified Alzheimer's disease case as a true positive. 95% CIs computed as above. We also agree that some statements found on the BBC's website (BBC, 2008) are misleading. Specifically, we show that SVMs provide a much faster classification than full clinical workup. Where the website misleads is in implying that they detect early degeneration faster, which is clearly beyond the scope of the current article and again a subject of ongoing study. It is important to emphasize that such multivariate methods generalize to new data. Figure 1 in our original paper illustrates that during training, samples from those individual subjects (i.e. normalized grey matter segments from either the whole brain or the hippocampus area), which best separate the two groups define the decision boundary. The figure is an example with two dimensions but in reality, the number of dimensions equals the number of voxels used. If a classifier generalizes well, a new scan will be assigned to the same side of the decision boundary as the rest of a diagnostic group. It is a critical part of our results that the decision boundary defined by data from one imaging centre using different hardware and sequences is sufficiently general to separate data accurately from other imaging centres. This ability is of great practical relevance as a library of well-defined cases can be made available to referral centres as a general trained set to diagnose scans collected there. While our results are promising, as we pointed out in our article, ‘a formal comparison with mod
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Automatic classification of MR scans in Alzheimer’s disease.
Kloppel, S., Stonnington, C. M., Chu, C., Draganski, B., Scahill, R. I., Rohrer, J. D., Fox, N. C., Jack, C. R., Ashburner, J., & Frackowiak, R. S. J. (2008) · Brain · doi:10.1093/brain/awm319 · Cited by 1,170
Self-citation
Cited by
Machine learning in major depression: from classification to treatment outcome prediction
Gao, S., Calhoun, V. D., & Sui, J. (2018) · CNS Neuroscience & Therapeutics · doi:10.1111/cns.13048 · Cited by 370
Cui, Z., & Gong, G. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2018.06.001 · Cited by 370
Automated detection of brain atrophy patterns based on MRI for the prediction of Alzheimer's disease
Plant, C., Teipel, S. J., Oswald, A., Böhm, C., Meindl, T., Mourao-Miranda, J., Bokde, A. W., Hampel, H., & Ewers, M. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2009.11.046 · Cited by 343
Wolfers, T., Buitelaar, J. K., Beckmann, C. F., Franke, B., & Marquand, A. F. (2015) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2015.08.001 · Cited by 290
Diagnostic neuroimaging across diseases
Klöppel, S., Abdulkadir, A., Jack, C. R., Koutsouleris, N., Mourão-Miranda, J., & Vemuri, P. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2011.11.002 · Cited by 279
Self-citation
Metadata
- Authors
- Kloppel, S., Stonnington, C. M., Chu, C., Draganski, B., Scahill, R. I., Rohrer, J. D., Fox, N. C., Ashburner, J., & Frackowiak, R. S. J.
- Venue
- Brain
- Publisher
- Oxford University Press (OUP)
- Published
- 2008-05-21
- Volume
- 132
- Issue
- 4
- Pages
- e102-e102
- ISSN
- 0006-8950 (ISSN-L)
- 1460-2156
- OpenAlex
- W1991527314
- Cited by
- 20
- References
- 1
- FWCI
- 1.67
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.