Student or educator? Sign up with your school email and get 6 months of Pro free.

The DOI matches its Crossref record; 96 of the 99 references it cites were found in a registry or an index; 11 of the 17 sources its claims cite were checked for support.

Data leakage in deep learning studies of translational EEG

Open access

Published version of a preprint

Abstract

A growing number of studies apply deep neural networks (DNNs) to recordings of human electroencephalography (EEG) to identify a range of disorders. In many studies, EEG recordings are split into segments, and each segment is randomly assigned to the training or test set. As a consequence, data from individual subjects appears in both the training and the test set. Could high test-set accuracy reflect data leakage from subject-specific patterns in the data, rather than patterns that identify a disease? We address this question by testing the performance of DNN classifiers using segment-based holdout (in which segments from one subject can appear in both the training and test set), and comparing this to their performance using subject-based holdout (where all segments from one subject appear exclusively in either the training set or the test set). In two datasets (one classifying Alzheimer's disease, and the other classifying epileptic seizures), we find that performance on previously-unseen subjects is strongly overestimated when models are trained using segment-based holdout. Finally, we survey the literature and find that the majority of translational DNN-EEG studies use segment-based holdout. Most published DNN-EEG studies may dramatically overestimate their classification performance on new subjects.

Overview

CoChat Summary

TL;DR

This paper shows that segment-based train/test splits in deep learning EEG studies leak subject-specific information and inflate test accuracy. In two reproduced CNN case studies, performance on unseen subjects dropped sharply under subject-based holdout, and a literature review found most translational DNN-EEG studies used segment-based holdout.

Key findings

  • Alzheimer’s disease classification: segment-based holdout reached 99.8% test accuracy (99.1–100.0%), but subject-based holdout fell to 53.0% (43.1–64.8%), near chance; the difference was significant (Wilcoxon T = 0.0, p = 0.002).
  • Seizure detection: segment-based holdout reached 79.1% (78.8–79.4%), while subject-based holdout fell to 65.1% (61.3–69.1%); the difference was significant (Wilcoxon T = 0.0, p = 0.0001).
  • In a literature review of 63 relevant papers, only 17 (27.0%) unambiguously avoided subject-specific data leakage; the majority used segment-based holdout or were unclear.

Methods

  • Reproduced two published CNN architectures without modification: a 1D CNN for Alzheimer’s disease diagnosis from eyes-open resting-state EEG (49 AD, 49 SCI; 250 Hz; 2-s segments) and a 2D CNN for seizure detection from the Siena Scalp EEG Database (14 epilepsy patients; 47 seizures; spectrogram images).
  • Compared segment-based holdout, where EEG segments are split without regard to subject ID, against subject-based holdout, where all segments from a subject appear only in training or test.
  • Surveyed 63 papers published since 2018 using DNNs with EEG for Alzheimer’s, Parkinson’s, ADHD, depression, schizophrenia, or seizures, classifying each study’s train-test split as segments, subjects, both, or unclear.

Limitations

  • The paper states that the precise amount of data leakage and bias likely differs by experiment details, including fewer participants possibly producing stronger bias and model architecture possibly influencing how much subject-specific representation is learned.
  • Evident from the text: the literature review was non-exhaustive, included 63 papers, and relied on whether papers explicitly reported subject-based splitting; some studies were labeled “unclear.”
  • Evident from the text: the empirical demonstration is limited to two datasets and two reused CNN architectures, with no hyperparameter tuning or model selection, so the exact magnitude of inflation may not generalize to all DNN-EEG studies.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3389/fnins.2024.1373515

Cites

96 of 96 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Illustration of segment-based and subject-based holdout. This example shows cross-validation with three participants, each of whom have three segments of data, and 3-fold cross-validation (CV). Each row shows a separate CV fold. Each square illustrates a single EEG segment, with blue squares indicating observations in the training set and red squares indicating observations in the test set. Gray rectangles are drawn around observations from the same subject.
Figure 1 Illustration of segment-based and subject-based holdout. This example shows cross-validation with three participants, each of whom have three segments of data, and 3-fold cross-validation (CV). Each row shows a separate CV fold. Each square illustrates a single EEG segment, with blue squares indicating observations in the training set and red squares indicating observations in the test set. Gray rectangles are drawn around observations from the same subject.CC BY · PubMed Central
Figure 2 Test-set accuracy of CNN models predicting held-out data, plotted separately for segment-based holdout and subject-based holdout. (A) Accuracy for models trained to classify Alzheimer's disease in individual subjects. Boxes show the inter-quartile range, dark lines show the median, and whiskers extend to the minimum and maximum points. (B) Accuracy for models trained to identify seizures in segments of EEG data. Details as in (A).
Figure 2 Test-set accuracy of CNN models predicting held-out data, plotted separately for segment-based holdout and subject-based holdout. (A) Accuracy for models trained to classify Alzheimer's disease in individual subjects. Boxes show the inter-quartile range, dark lines show the median, and whiskers extend to the minimum and maximum points. (B) Accuracy for models trained to identify seizures in segments of EEG data. Details as in (A).CC BY · PubMed Central
Figure 3 Test-set accuracy of CNN models plotted as a function of the training epoch. Gray lines show accuracy in individual cross-validation folds, and red lines show the average across folds. (A) Accuracy for models trained to classify Alzheimer's disease using segment-based holdout. (B) Accuracy for models trained to classify Alzheimer's disease using subject-based holdout. (C) Accuracy for models trained to identify seizures using segment-based holdout. (D) Accuracy for models trained to identify seizures using subject-based holdout.
Figure 3 Test-set accuracy of CNN models plotted as a function of the training epoch. Gray lines show accuracy in individual cross-validation folds, and red lines show the average across folds. (A) Accuracy for models trained to classify Alzheimer's disease using segment-based holdout. (B) Accuracy for models trained to classify Alzheimer's disease using subject-based holdout. (C) Accuracy for models trained to identify seizures using segment-based holdout. (D) Accuracy for models trained to identify seizures using subject-based holdout.CC BY · PubMed Central
Figure 4 Number of studies using each type of test-split. “Segments”: Segments of EEG data were assigned to the training and test sets without regard to subject; this approach leads to data leakage. “Subjects”: Each subject's data appeared in only the training set or the test set. “Both”: Both the Subjects and Segments approaches were used in different analyses. “Unclear”: We could not determine which approach was used for train-test splits.
Figure 4 Number of studies using each type of test-split. “Segments”: Segments of EEG data were assigned to the training and test sets without regard to subject; this approach leads to data leakage. “Subjects”: Each subject's data appeared in only the training set or the test set. “Both”: Both the Subjects and Segments approaches were used in different analyses. “Unclear”: We could not determine which approach was used for train-test splits.CC BY · PubMed Central

Verification

Claims the paper makes

6 Valid · 0 Not valid · 4 Not checked · 54 Own results and interpretations

41 more claims

Create a free account to see all 64 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 14
  • Probably not a substantial claim. 5

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Cited works that were retracted

Works this paper cites that were later retracted, from the retraction notices publishers deposit. Major when the notice came out before this paper was published.

1 of 1

Citations

Cites

1 retracted

Show all 99 references

Cited by

See all 103

Metadata

Authors
Brookshire, G., Kasper, J., Blauch, N. M., Wu, Y. C., Glatt, R., Merrill, D. A., Gerrol, S., Yoder, K. J., Quirk, C., & Lucero, C.
Affiliations
  • Carnegie Mellon University (US)
  • University of California, Los Angeles (US)
Corresponding author
Brookshire, Geoffrey
Venue
Frontiers in Neuroscience
Published
2024-05-03
Volume
18
Pages
1373515-1373515
ISSN
1662-453X (ISSN-L)
OpenAlex
W4396619951
Cited by
103
References
99
Citations by year
2026: 81 · 2025: 17 · 2024: 5
FWCI
24.95
Type
journal article
Language
en
Open access
Locations
  • Frontiers in Neuroscience (publishedVersion, cc-by)
  • medRxiv (acceptedVersion, cc-by-nd)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
Keywords
data leakage, deep neural network, cross-subject generalization, Alzheimer's disease classification, epileptic seizure classification

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Data leakage in deep learning studies of translational EEG. https://app.cochat.ai/papers/10.3389/fnins.2024.1373515
The paper
Brookshire, G., Kasper, J., Blauch, N. M., Wu, Y. C., Glatt, R., Merrill, D. A., Gerrol, S., Yoder, K. J., Quirk, C., & Lucero, C. (2024). Data leakage in deep learning studies of translational EEG. Frontiers in Neuroscience. https://doi.org/10.3389/fnins.2024.1373515

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.