The DOI matches its Crossref record; 96 of the 99 references it cites were found in a registry or an index; 11 of the 17 sources its claims cite were checked for support.
Data leakage in deep learning studies of translational EEG
Open accessPublished version of a preprint
Abstract
A growing number of studies apply deep neural networks (DNNs) to recordings of human electroencephalography (EEG) to identify a range of disorders. In many studies, EEG recordings are split into segments, and each segment is randomly assigned to the training or test set. As a consequence, data from individual subjects appears in both the training and the test set. Could high test-set accuracy reflect data leakage from subject-specific patterns in the data, rather than patterns that identify a disease? We address this question by testing the performance of DNN classifiers using segment-based holdout (in which segments from one subject can appear in both the training and test set), and comparing this to their performance using subject-based holdout (where all segments from one subject appear exclusively in either the training set or the test set). In two datasets (one classifying Alzheimer's disease, and the other classifying epileptic seizures), we find that performance on previously-unseen subjects is strongly overestimated when models are trained using segment-based holdout. Finally, we survey the literature and find that the majority of translational DNN-EEG studies use segment-based holdout. Most published DNN-EEG studies may dramatically overestimate their classification performance on new subjects.
Overview
CoChat Summary
TL;DR
This paper shows that segment-based train/test splits in deep learning EEG studies leak subject-specific information and inflate test accuracy. In two reproduced CNN case studies, performance on unseen subjects dropped sharply under subject-based holdout, and a literature review found most translational DNN-EEG studies used segment-based holdout.
Key findings
- Alzheimer’s disease classification: segment-based holdout reached 99.8% test accuracy (99.1–100.0%), but subject-based holdout fell to 53.0% (43.1–64.8%), near chance; the difference was significant (Wilcoxon T = 0.0, p = 0.002).
- Seizure detection: segment-based holdout reached 79.1% (78.8–79.4%), while subject-based holdout fell to 65.1% (61.3–69.1%); the difference was significant (Wilcoxon T = 0.0, p = 0.0001).
- In a literature review of 63 relevant papers, only 17 (27.0%) unambiguously avoided subject-specific data leakage; the majority used segment-based holdout or were unclear.
Methods
- Reproduced two published CNN architectures without modification: a 1D CNN for Alzheimer’s disease diagnosis from eyes-open resting-state EEG (49 AD, 49 SCI; 250 Hz; 2-s segments) and a 2D CNN for seizure detection from the Siena Scalp EEG Database (14 epilepsy patients; 47 seizures; spectrogram images).
- Compared segment-based holdout, where EEG segments are split without regard to subject ID, against subject-based holdout, where all segments from a subject appear only in training or test.
- Surveyed 63 papers published since 2018 using DNNs with EEG for Alzheimer’s, Parkinson’s, ADHD, depression, schizophrenia, or seizures, classifying each study’s train-test split as segments, subjects, both, or unclear.
Limitations
- The paper states that the precise amount of data leakage and bias likely differs by experiment details, including fewer participants possibly producing stronger bias and model architecture possibly influencing how much subject-specific representation is learned.
- Evident from the text: the literature review was non-exhaustive, included 63 papers, and relied on whether papers explicitly reported subject-based splitting; some studies were labeled “unclear.”
- Evident from the text: the empirical demonstration is limited to two datasets and two reused CNN architectures, with no hyperparameter tuning or model selection, so the exact magnitude of inflation may not generalize to all DNN-EEG studies.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
6 Valid · 0 Not valid · 4 Not checked · 54 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Segments of EEG from one subject are more similar to each other than to segments from different subjects (Demuru and Fraschini, 2020).”
§ Data leakage in deep learning studies of translational EEG > 1 Introduction
What the sources say
- Partly supportedcites Demuru and Fraschini, 2020 sourceExperimental — accuracy not yet measured
The Discussion reports strong subject-specific properties in EEG’s aperiodic power-spectrum features, and the Performance evaluation compares similarity scores for feature vectors from genuine and impostor pairs. This supports greater within-subject similarity for those features, but the claim generalizes the finding to EEG segments as a whole. (LLM, conf=0.85)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“For example, if a study trains a classifier to identify individual subjects with a disease, then there may be stronger bias when the study involves fewer participants (Saeb et al., 2017).”
§ Data leakage in deep learning studies of translational EEG > 4 Discussion > 4.4 Caveats
What the sources say
- Supportedcites Saeb et al., 2017 source
In “Simulated clinical prediction > Analyses,” the difference between record-wise and subject-wise classification errors was largest with fewer subjects and decreased as the number increased. The paper identifies this as bias from record-wise cross-validation in a classifier distinguishing diseased from healthy subjects; the claim’s “may” is consistent with that specific finding. (LLM, conf=0.88)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This conclusion mirrors results from studies examining a variety of other types of data and classifier models (Saeb et al., 2017; Wen et al., 2018, 2020; Chaibub Neto et al., 2019; Tougui et al., 2021; Tampu et al., 2022).”
§ Data leakage in deep learning studies of translational EEG > 4 Discussion > 4.4 Caveats
What the sources say
- Supportedcites Saeb et al., 2017 source
In Human activity recognition (Data description and methods; Analyses), Saeb et al. compare record-wise and subject-wise cross-validation using smartphone sensor data and a random-forest classifier. They find that record-wise splitting mixes subjects across training and test folds and significantly inflates estimated accuracy for prediction on new subjects. This supports the cited paper as an example of the analogous leakage problem; it does not need to establish the results of the other studies cited alongside it. (LLM, conf=0.88)
- Could not verifycites Wen et al., 2018 sourceExperimental — accuracy not yet measured
The 'Other deep learning methods' section discusses overfitting from small training samples, and the Conclusion discusses limitations of deep learning, but neither addresses leakage caused by segment-based cross-validation. Searches for segment-based cross-validation and for training and testing on the same subjects found no passage in our copy supporting that point; page numbers were not available. (LLM, conf=0.74)
- Supportedcites Chaibub Neto et al., 2019 source
The Discussion reports identity confounding when voice and tapping classifiers were evaluated with record-wise splits, and recommends subject-wise splits instead. The Results report the same issue across three digital-health datasets. This supports citing the study as an example of the leakage problem in other data, without implying that this individual study tested multiple classifier models. (LLM, conf=0.90)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Similar patterns of “identity confounding” data leakage have been documented in studies using functional (Wen et al., 2018) and anatomical (Wen et al., 2020) MRI, optical coherence tomography (OCT) (Tampu et al., 2022), accelerometer and gyroscope recordings from smartphones (Saeb et al., 2017), audio voice recordings (Chaibub Neto et al., 2019; Tougui et al., 2021), and performance on motor tasks”
§ Data leakage in deep learning studies of translational EEG > 4 Discussion > 4.3 Data leakage in other methods
What the sources say
- Could not verifycites Wen et al., 2018 sourceExperimental — accuracy not yet measured
The 'Other deep learning methods' section warns that small fMRI samples may lead to overfitting, but it does not document identity confounding or data leakage in functional MRI studies. Searches for identity confounding, leakage, and subject-level train/test splitting found no passage addressing that specific claim. (LLM, conf=0.75)
- Not checked — no full text of the sourcecites Wen et al., 2020 source
- Supportedcites Tampu et al., 2022 source
The Tampu citation is attached to the OCT example, not the other data types in the sentence. The Introduction describes leakage when images from the same subject appear in both training and testing sets, and the Discussion reports inflated performance from such overlap in OCT datasets. Page numbers are not provided in our copy. (LLM, conf=0.93)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Models trained using segment-based holdout, however, strongly overestimate their ability to perform this task.”
§ Data leakage in deep learning studies of translational EEG > 4 Discussion
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
41 more claims
Create a free account to see all 64 claims and every source verdict.
- It may state two separate claims as one. 14
- Probably not a substantial claim. 5
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Data leakage in deep learning studies of translational EEG > 2 Method > 2.5 Literature review
“This non-exhaustive search included 63 papers, all of which were published since 2018 and used deep learning to study one of the conditions named above.”
§ Data leakage in deep learning studies of translational EEG > 3 Results > 3.2 Data leakage in published EEG studies
“| Zhao and He (2015) | Alzheimer's | Segments |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Data leakage in deep learning studies of translational EEG > 2 Method > 2.5 Literature review
“This non-exhaustive search included 63 papers, all of which were published since 2018 and used deep learning to study one of the conditions named above.”
§ Data leakage in deep learning studies of translational EEG > 3 Results > 3.2 Data leakage in published EEG studies
“| Kim et al. (2018) | Alcoholism | Segments |”
Cited works that were retracted
Works this paper cites that were later retracted, from the retraction notices publishers deposit. Major when the notice came out before this paper was published.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“Zhou, 2022, Deep learning enabled diagnosis of children's ADHD based on the big data of video screen long-range EEG, J. Healthc. Eng, 2022, 5222136, 10.1155/2022/5222136”
Cited work 10.1155/2022/5222136 · retracted 2023-12-06 · notice 10.1155/2023/9856982 · this paper published 2024-05-03
Citations
Cites
1 retracted
Zhou, D., Liao, Z., & Chen, R. (2022) · Journal of Healthcare Engineering · doi:10.1155/2022/5222136 · Cited by 17
RetractedFolstein, M. F., Folstein, S. E., & McHugh, P. R. (1975) · Journal of Psychiatric Research · doi:10.1016/0022-3956(75)90026-6 · Cited by 88,282 · cited 1×
“Diagnoses were performed using standard clinical methods on the basis of neurological examinations, cognitive testing (MMSE Folstein et al., 1975 or MoCA Nasreddine et al., 2005), clinical history (e.g., hypertension, diabetes, head injury, depression), and laboratory results (e.g., vitamin B-12…”
MethodThe Montreal Cognitive Assessment, MoCA: a brief screening tool for mild cognitive impairment
Nasreddine, Z. S., Phillips, N. A., Bédirian, V., Charbonneau, S., Whitehead, V., Collin, I., Cummings, J. L., & Chertkow, H. (2005) · Journal of the American Geriatrics Society · doi:10.1111/j.1532-5415.2005.53221.x · Cited by 25,553 · cited 4×
“Diagnoses were performed using standard clinical methods on the basis of neurological examinations, cognitive testing (MMSE Folstein et al., 1975 or MoCA Nasreddine et al., 2005), clinical history (e.g., hypertension, diabetes, head injury, depression), and laboratory results (e.g., vitamin B-12…”
MethodInfluentialHirono, N., Mori, E., Ikejiri, Y., Imamura, T., Shimomura, T., Hashimoto, M., Yamashita, H., & Ikeda, M. (1997) · Brain and nerve · Cited by 21,012
Kok, J. S., Berg, I. J., & Scherder, E. J. A. (2013) · Dementia and Geriatric Cognitive Disorders Extra · doi:10.1159/000353441 · Cited by 20,589
Older Adults' Reasons for Using Technology while Aging in Place
Peek, S. T. M., Luijkx, K. G., Rijnaard, M. D., Nieboer, M. E., van der Voort, C. S., Aarts, S., van Hoof, J., Vrijhoef, H. J. M., & Wouters, E. J. M. (2016) · Gerontology · doi:10.1159/000430949 · Cited by 19,690
Sensitivity and False Alarm Rate of a Fall Sensor in Long-Term Fall Detection in the Elderly
Kangas, M., Korpelainen, R., Vikman, I., Nyberg, L., & Jämsä, T. (2015) · Gerontology · doi:10.1159/000362720 · Cited by 18,909
Goldberger, A. L., Amaral, L. A. N., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark, R. G., Mietus, J. E., Moody, G. B., Peng, C.-K., & Stanley, H. E. (2000) · Circulation · doi:10.1161/01.cir.101.23.e215 · Cited by 14,809
Artificial intelligence in radiology
Hosny, A., Parmar, C., Quackenbush, J., Schwartz, L. H., & Aerts, H. J. W. L. (2018) · Nature Reviews Cancer · doi:10.1038/s41568-018-0016-5 · Cited by 3,803 · cited 10×
“Our Alzheimer’s CNN classifier appeared to have an accuracy of above 99% when using segment-based holdout, but its true performance on previously-unseen subjects was indistinguishable from chance.”
InfluentialAcharya, U. R., Oh, S. L., Hagiwara, Y., Tan, J. H., & Adeli, H. (2018) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2017.09.017 · Cited by 1,647
Cited by
Don’t push the button! Exploring data leakage risks in machine learning and transfer learning
Apicella, A., Isgrò, F., & Prevete, R. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11326-3 · Cited by 190
Rokhva, S., Teimourpour, B., & Soltani, A. H. (2024) · Food and Humanity · doi:10.1016/j.foohum.2024.100378 · Cited by 57
How EEG preprocessing shapes decoding performance
Kessler, R., Enge, A., & Skeide, M. A. (2025) · Communications Biology · doi:10.1038/s42003-025-08464-3 · Cited by 58
“Recent studies have investigated how leakage is introduced when decoding is performed across participants with segment-based instead of subject-based holdout sets during cross-validation 73 .”
Pup, F. D., Zanola, A., Tshimanga, L. F., Bertoldo, A., Finos, L., & Atzori, M. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110608 · Cited by 57
Sadeq, M. A., Ghorab, R. M. F., Ashry, M. H., Abozaid, A. M., Banihani, H. A., Salem, M., Aisheh, M. T. A., Abuzahra, S., Mourid, M. R., Assker, M. M., et al. (2024) · Scientific Reports · doi:10.1038/s41598-024-68996-2 · Cited by 34
Metadata
- Authors
- Brookshire, G., Kasper, J., Blauch, N. M., Wu, Y. C., Glatt, R., Merrill, D. A., Gerrol, S., Yoder, K. J., Quirk, C., & Lucero, C.
- Affiliations
- Carnegie Mellon University (US)
- University of California, Los Angeles (US)
- Corresponding author
- Brookshire, Geoffrey
- ORCID
- Venue
- Frontiers in Neuroscience
- Publisher
- Frontiers Media SA
- Published
- 2024-05-03
- Volume
- 18
- Pages
- 1373515-1373515
- ISSN
- 1662-453X (ISSN-L)
- PMC
- PMC11099244
- OpenAlex
- W4396619951
- Semantic Scholar
- d59558f683e7d6e02aedc0b8510e7a601c103054
- Cited by
- 103
- References
- 99
- Citations by year
- 2026: 81 · 2025: 17 · 2024: 5
- FWCI
- 24.95
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Frontiers in Neuroscience (publishedVersion, cc-by)
- medRxiv (acceptedVersion, cc-by-nd)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Keywords
- data leakage, deep neural network, cross-subject generalization, Alzheimer's disease classification, epileptic seizure classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Data leakage in deep learning studies of translational EEG. https://app.cochat.ai/papers/10.3389/fnins.2024.1373515
- The paper
- Brookshire, G., Kasper, J., Blauch, N. M., Wu, Y. C., Glatt, R., Merrill, D. A., Gerrol, S., Yoder, K. J., Quirk, C., & Lucero, C. (2024). Data leakage in deep learning studies of translational EEG. Frontiers in Neuroscience. https://doi.org/10.3389/fnins.2024.1373515
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.