The DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
How to get statistically significant effects in any ERP experiment (and why you shouldn't)
Open accessAbstract
ERP experiments generate massive datasets, often containing thousands of values for each participant, even after averaging. The richness of these datasets can be very useful in testing sophisticated hypotheses, but this richness also creates many opportunities to obtain effects that are statistically significant but do not reflect true differences among groups or conditions (bogus effects). The purpose of this paper is to demonstrate how common and seemingly innocuous methods for quantifying and analyzing ERP effects can lead to very high rates of significant but bogus effects, with the likelihood of obtaining at least one such bogus effect exceeding 50% in many experiments. We focus on two specific problems: using the grand‐averaged data to select the time windows and electrode sites for quantifying component amplitudes and latencies, and using one or more multifactor statistical analyses. Reanalyses of prior data and simulations of typical experimental designs are used to show how these problems can greatly increase the likelihood of significant but bogus results. Several strategies are described for avoiding these problems and for increasing the likelihood that significant effects actually reflect true differences among groups or conditions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2025) · doi:10.32614/r.manuals · Cited by 352,181
Oostenveld, R., Fries, P., Maris, E., & Schoffelen, J.-M. (2011) · Computational Intelligence and Neuroscience · doi:10.1155/2011/156869 · Cited by 10,323
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Nonparametric statistical testing of EEG- and MEG-data
Maris, E., & Oostenveld, R. (2007) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2007.03.024 · Cited by 8,496
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 1×
“Estimating how often this happens is difficult, but there is growing evidence that many published results in psychology (Open Science Collaboration, 2015), neuroscience (Button et al., 2013), and oncology (Prinz, Schlange, & Asadullah, 2011) are not replicable.”
BackgroundSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
(2005) · Cited by 5,977
The Oxford Handbook of Event-Related Potential Components
(2011) · doi:10.1093/oxfordhb/9780195374148.001.0001 · Cited by 2,263
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888
McCarthy, G., & Wood, C. C. (1985) · Electroencephalography and Clinical Neurophysiology/Evoked Potentials Section · doi:10.1016/0168-5597(85)90015-2 · Cited by 1,756
Electrophysiological correlates of feature analysis during visual search
LUCK, S. J., & HILLYARD, S. A. (1994) · Psychophysiology · doi:10.1111/j.1469-8986.1994.tb02218.x · Cited by 1,485
Self-citationPuzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition
Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2009.01125.x · Cited by 1,475
On Quantifying Surprise: The Variation of Event‐Related Potentials With Subjective Probability
Duncan‐Johnson, C. C., & Donchin, E. (1977) · Psychophysiology · doi:10.1111/j.1469-8986.1977.tb01312.x · Cited by 1,224
Mass univariate analysis of event‐related brain potentials/fields I: A critical tutorial review
Groppe, D. M., Urbach, T. P., & Kutas, M. (2011) · Psychophysiology · doi:10.1111/j.1469-8986.2011.01273.x · Cited by 1,169
Spatial filtering during visual search: Evidence from human electrophysiology.
Luck, S. J., & Hillyard, S. A. (1994) · Journal of Experimental Psychology: Human Perception and Performance · doi:10.1037//0096-1523.20.5.1000 · Cited by 1,121
Self-citationVogel, E. K., Luck, S. J., & Shapiro, K. L. (1998) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037//0096-1523.24.6.1656 · Cited by 763
Self-citationThe mismatch negativity (MMN): towards the optimal paradigm
Näätänen, R., Pakarinen, S., Rinne, T., & Takegata, R. (2004) · Clinical Neurophysiology · doi:10.1016/j.clinph.2003.04.001 · Cited by 721
Keil, A., Debener, S., Gratton, G., Junghöfer, M., Kappenman, E. S., Luck, S. J., Luu, P., Miller, G. A., & Yee, C. M. (2014) · Psychophysiology · doi:10.1111/psyp.12147 · Cited by 699
Self-citationIs the Replicability Crisis Overblown? Three Arguments Examined
Pashler, H., & Harris, C. R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463401 · Cited by 697
The ɛ‐Adjustment Procedure for Repeated‐Measures Analyses of Variance
Richard Jennings, J., & Wood, C. C. (1976) · Psychophysiology · doi:10.1111/j.1469-8986.1976.tb00116.x · Cited by 490
Electrophysiological Evidence of the Capture of Visual Attention
Hickey, C., McDonald, J. J., & Theeuwes, J. (2006) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2006.18.4.604 · Cited by 475
A critique of functional localisers
Friston, K. J., Rotshtein, P., Geng, J. J., Sterzer, P., & Henson, R. N. (2006) · NeuroImage · doi:10.1016/j.neuroimage.2005.08.012 · Cited by 424
Hidden multiplicity in exploratory multiway ANOVA: Prevalence and remedies
Cramer, A. O. J., van Ravenzwaaij, D., Matzke, D., Steingroever, H., Wetzels, R., Grasman, R. P. P. P., Waldorp, L. J., & Wagenmakers, E.-J. (2016) · Psychonomic Bulletin & Review · doi:10.3758/s13423-015-0913-5 · Cited by 416
Eimer, M., & Kiss, M. (2008) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2008.20099 · Cited by 332
A Common Neural Mechanism for Preventing and Terminating the Allocation of Attention
Sawaki, R., Geng, J. J., & Luck, S. J. (2012) · Journal of Neuroscience · doi:10.1523/jneurosci.1864-12.2012 · Cited by 265
Self-citationClayson, P. E., Baldwin, S. A., & Larson, M. J. (2013) · Psychophysiology · doi:10.1111/psyp.12001 · Cited by 249
Javitt, D. C., Grochowski, S., Shelley, A.-M., & Ritter, W. (1998) · Electroencephalography and Clinical Neurophysiology/Evoked Potentials Section · doi:10.1016/s0168-5597(97)00073-7 · Cited by 241
P300 and Stimulus Categorization: Two Plus One is not so Different from One Plus One
Johnson, R., & Donchin, E. (1980) · Psychophysiology · doi:10.1111/j.1469-8986.1980.tb00131.x · Cited by 235
Mass univariate analysis of event‐related brain potentials/fields II: Simulation studies
Groppe, D. M., Urbach, T. P., & Kutas, M. (2011) · Psychophysiology · doi:10.1111/j.1469-8986.2011.01272.x · Cited by 212
The Lateralized Readiness Potential
Smulders, F. T. Y., & Miller, J. O. (2011) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780195374148.013.0115 · Cited by 132
Luck, S. J., Kappenman, E. S., Fuller, R. L. M., Robinson, B. M., Summerfelt, A. T., & Gold, J. M. (2009) · Psychophysiology · doi:10.1111/j.1469-8986.2009.00817.x · Cited by 110
Self-citationEvent-related brain activity associated with auditory pattern processing
Alain, C., Cortese, F., & Picton, T. W. (1998) · Neuroreport · doi:10.1097/00001756-199810260-00037 · Cited by 75
What ERPs Can Tell Us aboutWorking Memory
Perez, V. B., & Vogel, E. K. (2011) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780195374148.013.0180 · Cited by 39
The Allocation of Attention and Working Memory in Visual Crowding
Bacigalupo, F., & Luck, S. J. (2014) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_00771 · Cited by 36
Self-citation
Cited by
Rubin, M. (2021) · Synthese · doi:10.1007/s11229-021-03276-4 · Cited by 371
Large-scale replication study reveals a limit on probabilistic prediction in language comprehension
Nieuwland, M. S., Politzer-Ahles, S., Heyselaar, E., Segaert, K., Darley, E., Kazanina, N., Von Grebmer Zu Wolfsthurn, S., Bartolozzi, F., Kogan, V., Ito, A., et al. (2018) · eLife · doi:10.7554/elife.33468 · Cited by 299
Coll, M.-P., Hobson, H., Bird, G., & Murphy, J. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2020.12.012 · Cited by 262
ERP CORE: An open resource for human event-related potential research
Kappenman, E. S., Farrens, J. L., Zhang, W., Stewart, A. X., & Luck, S. J. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2020.117465 · Cited by 223
Self-citationUnderstanding neural signals of post-decisional performance monitoring: An integrative review
Desender, K., Ridderinkhof, K. R., & Murphy, P. R. (2021) · eLife · doi:10.7554/elife.67556 · Cited by 77
Metadata
- Authors
- Luck, S. J., & Gaspelin, N.
- Venue
- Psychophysiology
- Publisher
- Wiley
- Published
- 2016-12-20
- Volume
- 54
- Issue
- 1
- Pages
- 146-157
- ISSN
- 0048-5772 (ISSN-L)
- 1469-8986
- OpenAlex
- W2562953409
- Cited by
- 1,195
- References
- 35
- FWCI
- 41.92
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.