CoChat

Reanalysis of “Bedside detection of awareness in the vegetative state: a cohort study”

Open access
Goldfine, Andrew M, Bardin, Jonathan C, Noirhomme, Quentin, Fins, Joseph J, Schiff, Nicholas D, Victor, Jonathan D · 2013 · The Lancet
Cited by 98

The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.

Abstract

Cruse and colleagues reported1 that a new electroencephalography (EEG)-based tool was able to show that 3 out of 16 vegetative state (VS) patients performed a motor imagery task requiring language and short-term memory. This finding, if confirmed, has major implications for diagnosis and care of severely brain-injured patients. We were concerned about the method’s validity because of the difficulty of the task, and its critical reliance on certain statistical assumptions. To allow us to test the validity of the method, Cruse and colleagues graciously supplied their data and analysis software. Below we show that the patient data do not meet the statistical assumptions made in Cruse et al., likely because of the presence of various artifacts (Table). We then show that when the data are re-analyzed by methods that do not depend on these model assumptions, there is no evidence for task performance in the patients. Table Overview of analyses and findings. To begin, we examine the EEG data itself. The normals have findings typical of healthy adults (Figure 1A, left): rhythmicity in the alpha range (~10 Hz) with minimal eye-blink and muscle artifact. In contrast, the patients’ EEG (Figure 1A, right) is dominated by 1–4 Hz activity, as is typical of severe brain dysfunction, deep sleep or anesthesia2. Frequency-domain representation (Figure 1B) confirms these findings. It also reveals that the patient’s EEG has significant muscle artifact3 that fluctuates block-to-block. Figure 1 Time and frequency domain representations of the EEG of a typical normal (N2) and patient (P13) who had similar classification rates in Cruse et al. (75% and 78%, respectively; Webappendix for methods). A. Laplacian-montaged EEG of the first trial of ... To determine whether subjects performed motor imagery, Cruse and colleagues used a multivariate method (Support Vector Machine; SVM)4,5 to differentiate EEG signals recorded while subjects were asked to imagine moving their hand, vs. their toes. SVM is a powerful technique, but, without a gold-standard for task performance, the validity hinges on the appropriateness of the statistical model.6 As detailed below, the statistical model used in Cruse et al. did not account for relationships between adjacent blocks, or correlations between trials within a block. For calculation of accuracy (how often the SVM correctly classified trials as “hand” vs. “toe”), the Cruse et al. methods did not take into account the possibility of slow variations across blocks, as their approach always classified pairs of neighbouring blocks (e.g., hand and toe block 1, but never hand block 1 and toe block 4). We modified their analysis to use these alternative pairings for cross-validation6 (Webappendix). In two of the positive patients (Webappendix Figure 1), accuracy decreased to chance (P1), or worse-than-chance (P12) as the test-block-pairs were further apart. This drop in accuracy implies that idiosyncratic relationships between adjacent blocks contributed substantially to SVM performance in these subjects. For calculation of significance, Cruse and colleagues calculated p-values using a binomial distribution for the number of correct trials, an approach that assumes that each trial is an independent assay. We found that this assumption does not hold in the patients. First, frequency domain representation of the EEG (Figure 1B; Webappendix) reveals a lack of independence: data from individual trials are more nearly matched within a block than across blocks. Second, we applied the Cruse et al. analysis separately to all time points of the trials. For patients, we found that worse-than-chance classification occurred substantially more often than expected from binomial statistics. This excess of outliers implies that trials are correlated (Webappendix and Webappendix Figure 2). We next show that when the SVM results are re-analyzed with a statistical approach that takes into account the correlations mentioned above (Webappendix and Webappendix Table 1 for full details), there is no statistical evidence of a task-related signal. To take into account correlations between blocks, we defined accuracy using all block-pairs as test components6, rather than restricting consideration to adjacent block pairs. To account for dependence among trials, we determined significance via a permutation test that recognized the block design. With this approach, positive normals remained significant, but only one patient (P13) remained significant (p=0·0286; lowest possible p-value with 4 blocks). We further note that even for random data, a classifier would be expected to yield 1 in 20 positive subjects at p≤0.05. We therefore corrected for multiple comparisons via the False-Discovery Rate (FDR)7; normals remained significant but none of the patients were significant at p≤0·05. Finally, we applied an independent approach that asked whether there was a significant difference between task and rest periods, using univariate statistics (i.e., separate tests for each frequency and channel of the EEG; methods in Webappendix and8; Webappendix Figures 3 and 4). Normals showed the expected task-related changes in motor imagery tasks (decreases in EEG power from 7–30 Hz, especially over the motor cortices contralateral to the imagined limb movement; p≤0. 05 after FDR correction)9,10. None of the 16 patients had significant changes identified by this measure. This emphasizes that even if we were to accept the ‘positive’ patient classifications of Cruse et al. as different from chance, the EEG signals lack the expected physiological changes associated with motor imagery (in contrast to the suggestion made by Cruse and colleagues in connection with their Figure 2). In sum, we found that the method of Cruse et al. is not valid because the patient data do not meet the assumptions of their statistical model. Specifically, the model does not allow for correlations between nearby trials and blocks, which are likely induced by fluctuating artifact and arousal state; when these factors are taken into account, there is no statistical evidence for task performance in patients. Importantly, the model of Cruse et al. generally suffices for normals, where there is minimal artifact contamination. These findings cast doubt about conclusions drawn from this method, both in Cruse et al., and a more recent study11. SVM and related methods are useful tools, particularly in EEG analysis for Brain-Computer Interface (BCI)10,12. In BCI applications, subjects can confirm task performance and the consequences of classifier failure are limited to reduced device performance. But in the diagnostic setting (e.g., determination of consciousness, genomic diagnosis of cancer13,14), classifier failure can misinform clinical decision making, with major consequences for patients and families. Given this, and the ease of dissemination of EEG technology, standards of demonstration of validity need to be high. Our analysis suggests that the approach of Cruse et al. falls short of this standard. Finally, we wish to emphasize the importance of data sharing. This analysis would not have been possible without full access to the original data and code.15

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/s0140-6736(13)60125-7

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 4 references

Cited by

See all 98

Metadata

Authors
Goldfine, A. M., Bardin, J. C., Noirhomme, Q., Fins, J. J., Schiff, N. D., & Victor, J. D.
Venue
The Lancet
Publisher
Elsevier BV
Published
2013-01
Volume
381
Issue
9863
Pages
289-291
ISSN
  • 0099-5355 (ISSN-L)
  • 0140-6736
OpenAlex
W1965624581
Cited by
98
References
4
FWCI
12.32
Type
journal article
Language
en
Open access
green

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Reanalysis of “Bedside detection of awareness in the vegetative state: a cohort study”. https://app.cochat.ai/papers/10.1016/s0140-6736(13)60125-7
The paper
Goldfine, A. M., Bardin, J. C., Noirhomme, Q., Fins, J. J., Schiff, N. D., & Victor, J. D. (2013). Reanalysis of “Bedside detection of awareness in the vegetative state: a cohort study”. The Lancet. https://doi.org/10.1016/s0140-6736(13)60125-7

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.