Moderators of the internal consistency of error‐related negativity scores: A meta‐analysis of internal consistency estimates
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
To ensure adequate reliability (i.e., internal consistency), it is common in studies using event‐related brain potentials (ERPs) to exclude participants for having too few trials. This practice is particularly relevant for error‐related ERPs, such as error‐related negativity (ERN), where the number of recorded ERN trials is not entirely under the researcher's control. Furthermore, there is a widespread practice of inferring reliability based on published psychometric research, which assumes that internal consistency is a universal property of ERN. The present, preregistered reliability generalization study examined whether there is heterogeneity in internal consistency estimates of ERN scores and whether contextual factors moderate reliability. A total of 189 internal consistency estimates from 68 samples nested within 43 studies ( n = 4,499 total participants) were analyzed. There was substantial heterogeneity in ERN score internal consistency, which was partially moderated by the type of paradigm (e.g., Stroop, flanker), the clinical status of the sample, the ocular artifact correction procedure, measurement sensors (single versus cluster), and the approach to scoring and estimating reliability, suggesting that contextual factors impact internal consistency at the individual study level. Age, sex, year of publication, artifact rejection procedure, acquisition system, sample type (undergraduate versus community), and length of mean amplitude window did not significantly moderate reliability. Notably, the overall estimated reliability of ERN scores was below established standards. Recommendations for improving ERN score reliability are provided, but the routine failure of most ERN studies to report internal consistency represents a substantial barrier to understanding the factors that impact reliability.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A new method for off-line removal of ocular artifact
Gratton, G., Coles, M. G. H., & Donchin, E. (1983) · Electroencephalography and Clinical Neurophysiology · doi:10.1016/0013-4694(83)90135-9 · Cited by 5,055
Toward the future of psychiatric diagnosis: the seven pillars of RDoC
Cuthbert, B. N., & Insel, T. R. (2013) · BMC Medicine · doi:10.1186/1741-7015-11-126 · Cited by 3,053
Outlier and influence diagnostics for meta-analysis
Viechtbauer, W., & Cheung, M. W.-L. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.11 · Cited by 2,949
Statistical methods in psychology journals: Guidelines and explanations.
Wilkinson, L., & Task Force on Statistical Inference (1999) · American Psychologist · doi:10.1037/0003-066x.54.8.594 · Cited by 2,791
Improved tests for a random effects meta‐regression with a single covariate
Knapp, G., & Hartung, J. (2003) · Statistics in Medicine · doi:10.1002/sim.1482 · Cited by 2,075
Falkenstein, M., Hohnsbein, J., Hoormann, J., & Blanke, L. (1991) · Electroencephalography and Clinical Neurophysiology · doi:10.1016/0013-4694(91)90062-9 · Cited by 1,706
The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences
Hedge, C., Powell, G., & Sumner, P. (2018) · Behavior Research Methods · doi:10.3758/s13428-017-0935-1 · Cited by 1,457
Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors
Gelman, A., & Carlin, J. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614551642 · Cited by 1,343
Cheung, M. W.-L. (2014) · Psychological Methods · doi:10.1037/a0032968 · Cited by 1,342
Fitting three-level meta-analytic models in R: A step-by-step tutorial
Assink, M., & Wibbelink, C. J. M. (2016) · The Quantitative Methods for Psychology · doi:10.20982/tqmp.12.3.p154 · Cited by 1,330
Nieuwenhuis, S., Ridderinkhof, K. R., Blom, J., Band, G. P. H., & Kok, A. (2001) · Psychophysiology · doi:10.1111/1469-8986.3850752 · Cited by 1,170
Lack of group-to-individual generalizability is a threat to human subjects research
Fisher, A. J., Medaglia, J. D., & Jeronimus, B. F. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1711978115 · Cited by 1,099
Intercoder Reliability and Validity of WebPlotDigitizer in Extracting Graphed Data
Drevon, D., Fursa, S. R., & Malcolm, A. L. (2017) · Behavior Modification · doi:10.1177/0145445516673998 · Cited by 1,006
The NIMH research domain criteria initiative: Background, issues, and pragmatics
Kozak, M. J., & Cuthbert, B. N. (2016) · Psychophysiology · doi:10.1111/psyp.12518 · Cited by 655
Theta lingua franca: A common mid‐frontal substrate for action monitoring processes
Cavanagh, J. F., Zambrano‐Vazquez, L., & Allen, J. J. B. (2012) · Psychophysiology · doi:10.1111/j.1469-8986.2011.01293.x · Cited by 651
Measurement error and the replication crisis
Loken, E., & Gelman, A. (2017) · Science · doi:10.1126/science.aal3618 · Cited by 628
Fixed effects and variance components estimation in three-level meta-analysis
Konstantopoulos, S. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.35 · Cited by 521
Making sense of all the conflict: A theoretical review and critique of conflict-related ERPs
Larson, M. J., Clayson, P. E., & Clawson, A. (2014) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2014.06.007 · Cited by 454
Self-citationThe stability of error‐related brain activity with increasing trials
Olvet, D. M., & Hajcak, G. (2009) · Psychophysiology · doi:10.1111/j.1469-8986.2009.00848.x · Cited by 453
Making Reliability Reliable: A Systematic Approach to Reliability Coefficients
Cho, E. (2016) · Organizational Research Methods · doi:10.1177/1094428116656239 · Cited by 378
Infantolino, Z. P., Luking, K. R., Sauder, C. L., Curtin, J. J., & Hajcak, G. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2018.02.024 · Cited by 108
Reliability of the ERN across multiple tasks as a function of increasing errors
Meyer, A., Riesel, A., & Proudfit, G. H. (2013) · Psychophysiology · doi:10.1111/psyp.12132 · Cited by 107
Random measurement error: Why worry? An example of cardiovascular risk factors
Brakenhoff, T. B., van Smeden, M., Visseren, F. L. J., & Groenwold, R. H. H. (2018) · PLOS ONE · doi:10.1371/journal.pone.0192298 · Cited by 52
Valadez, E. A., & Simons, R. F. (2018) · Psychophysiology · doi:10.1111/psyp.13010 · Cited by 41
Brand, A., & Bradley, M. T. (2016) · Psychological Reports · doi:10.1177/0033294115625265 · Cited by 25
Cited by
Tiwari, C. K., Bhat, M. A., Khan, S. T., Subramaniam, R., & Khan, M. A. I. (2024) · Interactive Technology and Smart Education · doi:10.1108/itse-04-2023-0061 · Cited by 315
Clayson, P. E., Baldwin, S. A., & Larson, M. J. (2021) · Psychophysiology · doi:10.1111/psyp.13762 · Cited by 84
Self-citationClayson, P. E., Kappenman, E. S., Gehring, W. J., Miller, G. A., & Larson, M. J. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.117932 · Cited by 28
Self-citationLoTemplio, S. B., Lopes, C. L., McDonnell, A. S., Scott, E. E., Payne, B. R., & Strayer, D. L. (2023) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2023.1150244 · Cited by 25
Clayson, P. E., Mcdonald, J. B., Park, B., Holbrook, A., Baldwin, S. A., Riesel, A., & Larson, M. J. (2025) · Psychophysiology · doi:10.1111/psyp.14496 · Cited by 21
Self-citation
Metadata
- Authors
- Clayson, P. E.
- Venue
- Psychophysiology
- Publisher
- Wiley
- Published
- 2020-04-23
- Volume
- 57
- Issue
- 8
- Pages
- e13583-e13583
- ISSN
- 0048-5772 (ISSN-L)
- 1469-8986
- OpenAlex
- W3018604527
- Cited by
- 73
- References
- 25
- FWCI
- 7.17
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.