The psychometric upgrade psychophysiology needs
Open accessThe DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Abstract
Although biological measurements are constrained by the same fundamental psychometric principles as self‐report measurements, these essential principles are often neglected in most fields of neuroscience, including psychophysiology. Potential reasons for this neglect could include a lack of understanding of appropriate measurement theory or a lack of accessible software for psychometric analysis. Generalizability theory is a flexible and multifaceted measurement theory that is well suited to handling the nuances of psychophysiological data, such as the often unbalanced number of trials and intraindividual variability of scores of event‐related brain potential (ERP) data. The ERP Reliability Analysis Toolbox (ERA Toolbox) was designed for psychophysiologists and is tractable software that can support the routine evaluation of psychometrics using generalizability theory. Psychometrics can guide task refinement, data‐processing decisions, and selection of candidate biomarkers for clinical trials. The present review provides an extensive treatment of additional psychometric characteristics relevant to studies of psychophysiology, including validity and validation, standardization, dimensionality, and measurement invariance. Although the review focuses on ERPs, the discussion applies broadly to psychophysiological measures and beyond. The tools needed to rigorously assess psychometric reliability and validate psychophysiological measures are now readily available. With the profound implications that psychophysiological research can have on understanding brain‐behavior relationships and the identification of biomarkers, there is simply too much at stake to ignore the crucial processes of evaluating psychometric reliability and validity.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Putnick, D. L., & Bornstein, M. H. (2016) · Developmental Review · doi:10.1016/j.dr.2016.06.004 · Cited by 4,369
The neural basis of error detection: Conflict monitoring and the error-related negativity
Yeung, N., Botvinick, M. M., & Cohen, J. D. (2004) · Psychological Review · doi:10.1037/0033-295x.111.4.931 · Cited by 1,924
Falkenstein, M., Hohnsbein, J., Hoormann, J., & Blanke, L. (1991) · Electroencephalography and Clinical Neurophysiology · doi:10.1016/0013-4694(91)90062-9 · Cited by 1,706
The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences
Hedge, C., Powell, G., & Sumner, P. (2018) · Behavior Research Methods · doi:10.3758/s13428-017-0935-1 · Cited by 1,457
Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors
Gelman, A., & Carlin, J. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614551642 · Cited by 1,343
Why We (Usually) Don't Have to Worry About Multiple Comparisons
Gelman, A., Hill, J., & Yajima, M. (2012) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2011.618213 · Cited by 1,274
Nieuwenhuis, S., Ridderinkhof, K. R., Blom, J., Band, G. P. H., & Kok, A. (2001) · Psychophysiology · doi:10.1111/1469-8986.3850752 · Cited by 1,170
Brennan, R. L. (2001) · doi:10.1007/978-1-4757-3456-0 · Cited by 1,121
Meaning and Values in Test Validation: The Science and Ethics of Assessment
Messick, S. (1989) · Educational Researcher · doi:10.3102/0013189x018002005 · Cited by 948
THEORY OF GENERALIZABILITY: A LIBERALIZATION OF RELIABILITY THEORY†
Cronbach, L. J., Rajaratnam, N., & Gleser, G. C. (1963) · British Journal of Statistical Psychology · doi:10.1111/j.2044-8317.1963.tb00206.x · Cited by 737
The NIMH research domain criteria initiative: Background, issues, and pragmatics
Kozak, M. J., & Cuthbert, B. N. (2016) · Psychophysiology · doi:10.1111/psyp.12518 · Cited by 655
Understanding the Impact of Convergent Validity on Research Results
Carlson, K. D., & Herdman, A. O. (2012) · Organizational Research Methods · doi:10.1177/1094428110392383 · Cited by 642
Measurement error and the replication crisis
Loken, E., & Gelman, A. (2017) · Science · doi:10.1126/science.aal3618 · Cited by 628
Overbeek, T. J. M., Nieuwenhuis, S., & Ridderinkhof, K. R. (2005) · Journal of Psychophysiology · doi:10.1027/0269-8803.19.4.319 · Cited by 600
Making sense of all the conflict: A theoretical review and critique of conflict-related ERPs
Larson, M. J., Clayson, P. E., & Clawson, A. (2014) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2014.06.007 · Cited by 454
Self-citationSzucs, D., & Ioannidis, J. P. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.117164 · Cited by 453
Making Reliability Reliable: A Systematic Approach to Reliability Coefficients
Cho, E. (2016) · Organizational Research Methods · doi:10.1177/1094428116656239 · Cited by 378
The Error-Related Negativity (ERN/Ne)
Gehring, W. J., Liu, Y., Orr, J. M., & Carp, J. (2011) · Oxford Handbooks Online · doi:10.1093/oxfordhb/9780195374148.013.0120 · Cited by 331
Error awareness and the error-related negativity: evaluating the first decade of evidence
Wessel, J. R. (2012) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2012.00088 · Cited by 215
Error-related brain activity in the age of RDoC: A review of the literature
Weinberg, A., Dieterich, R., & Riesel, A. (2015) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2015.02.029 · Cited by 179
The erring brain: Error‐related negativity as an endophenotype for OCD—A review and meta‐analysis
Riesel, A. (2019) · Psychophysiology · doi:10.1111/psyp.13348 · Cited by 170
Psychological measurement and the replication crisis: Four sacred cows
Lilienfeld, S. O., & Strother, A. N. (2020) · Canadian Psychology / Psychologie canadienne · doi:10.1037/cap0000236 · Cited by 144
Infantolino, Z. P., Luking, K. R., Sauder, C. L., Curtin, J. J., & Hajcak, G. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2018.02.024 · Cited by 108
Error awareness as evidence accumulation: effects of speed-accuracy trade-off on error signaling
Steinhauser, M., & Yeung, N. (2012) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2012.00240 · Cited by 96
Clayson, P. E., Baldwin, S. A., & Larson, M. J. (2021) · Psychophysiology · doi:10.1111/psyp.13762 · Cited by 84
Self-citationMathews, C. A., Perez, V. B., Delucchi, K. L., & Mathalon, D. H. (2012) · Biological Psychology · doi:10.1016/j.biopsycho.2011.12.018 · Cited by 78
Lutz, M. C., Kok, R., Verveer, I., Malbec, M., Koot, S., van Lier, P. A. C., & Franken, I. H. A. (2021) · Journal of Psychiatry and Neuroscience · doi:10.1503/jpn.200031 · Cited by 44
Boen, R., Quintana, D. S., Ladouceur, C. D., & Tamnes, C. K. (2022) · Psychophysiology · doi:10.1111/psyp.14003 · Cited by 40
Clayson, P. E., Mcdonald, J. B., Park, B., Holbrook, A., Baldwin, S. A., Riesel, A., & Larson, M. J. (2025) · Psychophysiology · doi:10.1111/psyp.14496 · Cited by 21
Self-citation
Cited by
Clayson, P. E., Mcdonald, J. B., Park, B., Holbrook, A., Baldwin, S. A., Riesel, A., & Larson, M. J. (2025) · Psychophysiology · doi:10.1111/psyp.14496 · Cited by 21
Self-citationHolbrook, A., Park, B., Baldwin, S. A., Riesel, A., Larson, M. J., & Clayson, P. (2025) · Psychophysiology · doi:10.1111/psyp.70042 · Cited by 6
Self-citationLoTemplio, S., Silcox, J., Strayer, D. L., & Payne, B. R. (2026) · Psychophysiology · doi:10.1111/psyp.70216 · Cited by 3
Metadata
- Authors
- Clayson, P. E.
- Venue
- Psychophysiology
- Publisher
- Wiley
- Published
- 2024-01-16
- Volume
- 61
- Issue
- 3
- Pages
- e14522-e14522
- ISSN
- 0048-5772 (ISSN-L)
- 1469-8986
- OpenAlex
- W4390929125
- Cited by
- 39
- References
- 29
- FWCI
- 11.50
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.