CoChat
Sign Up
Cited by 84Psychometric Methodologies and TestingPsychological Testing and AssessmentNeural and Behavioral Psychology Studies

Evaluating the internal consistency of subtraction‐based and residualized difference scores: Considerations for psychometric reliability analyses of event‐related potentials

Open access
Clayson, Peter E., Baldwin, Scott A., Larson, Michael J.·2021·Psychophysiology
DOI 10.1111/psyp.13762

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

In studies of event‐related brain potentials (ERPs), difference scores between conditions in a task are frequently used to isolate neural activity for use as a dependent or independent variable. Adequate score reliability is a prerequisite for studies examining relationships between ERPs and external correlates, but there is no extensive treatment on the suitability of the various available approaches to estimating difference score reliability that focus on ERP research. In the present study, we provide formulas from classical test theory and generalizability theory for estimating the internal consistency of subtraction‐based and residualized difference scores. These formulas are then applied to error‐related negativity (ERN) and reward positivity (RewP) difference scores from the same sample of 117 participants. Analyses demonstrate that ERN difference scores can be reliable, which supports their use in studies of individual differences. However, RewP difference scores yielded poor reliability due to the high correlation between the constituent reward and non‐reward ERPs. Findings emphasize that difference score reliability largely depends on the internal consistency of constituent scores and the correlation between those scores. Furthermore, generalizability theory yields more suitable estimates of internal consistency for subtraction‐based difference scores than classical test theory. We conclude that ERP difference scores can show adequate reliability and be useful for isolating neural activity in studies of individual differences.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1111/psyp.13762

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The explicit modeling of the group × response type interaction was chosen over subtraction-based difference scores because subtraction-based scores compound the imperfect reliabilities of the two constituent scores, especially when the constituent scores are strongly correlated.

    “This explicit modeling of the interaction was chosen over the more common subtraction-based difference scores because subtraction-based scores compound the imperfect reliabilities of the two constituent scores, especially when the constituent scores are strongly correlated (Clayson et al., 2021; Meyer et al., 2017) as is typically the case with the ERN and CRN.”

    From Foster care intervention protects cognitive control skills in young adults exposed to institutional rearing (2026) · cites this paper as Clayson et al., 2021

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 76

  1. Allison, P. D. (1990). Change scores as dependent variables in regression analysis. Social Methodology, 20, 93-114. doi:10.2307/271083 · DOI
  2. Baldwin, S. A. (2017). Improving the rigor of psychophysiology research. Int J Psychophysiol, 111, 5-16. doi:10.1016/j.ijpsycho.2016.04.006 · DOI
  3. Baldwin, S. A., Larson, M. J., & Clayson, P. E. (2015). The dependability of electrophysiological measurements of performance monitoring in a clinical sample: A generalizability and decision analysis of the ERN and Pe. Psychophysiology, 52, 790- 800. doi:https://doi.org/10.1111/psyp.12401 · DOI
  4. Bereiter, C. (1963). Some persisting dilemmas in the measurement of change. In C. W. Harris (Ed.), Problems in measuring change (Vol. 2). Madison, WI: University of Wisconsin Press.
  5. Boudewyn, M. A., Luck, S. J., Farrens, J. L., & Kappenman, E. S. (2017). How many trials does it take to get a significant ERP effect? It depends. Psychophysiology, 14, e13049. doi:10.1111/psyp.13049 · DOI
  6. Brennan, R. L. (2001). Generalizability theory: Statistics for social science and public policy. New York, NY: Springer-Verlag.
  7. Carbine, K. A., Clayson, P. E., Baldwin, S. A., LeCheminant, J., & Larson, M. J. (2020). Using generalizability theory and the ERP Reliability Analysis (ERA) Toolbox for assessing test-retest reliability of ERP scores Part 2: Application to food-based tasks and stimuli. PsyArXiv. doi:10.31234/osf.io/zuc8g · DOI
  8. Charter, R. A. (2001). It is time to bury the Spearman-Brown "prophecy" formula for some common applications. Educational and Psychological Measurement, 61, 690-696. doi:10.1177/00131640121971446 · DOI
  9. Clayson, P. E. (2020). Moderators of the internal consistency of error-related negativity scores: A meta-analysis of internal consistency estimates. Psychophysiology, 57, e13583. doi:https://doi.org/10.1111/psyp.13583 · DOI
  10. Clayson, P. E., Brush, C. J., & Hajcak, G. (2020). Data quality and reliability metrics for event- related potentials (ERPs): The utility of subject-level reliability. PsyArXiv. doi:10.31234/osf.io/ja6bw · DOI
  11. Clayson, P. E., Carbine, K. A., Baldwin, S. A., Olsen, J. A., & Larson, M. J. (2020). Using generalizability theory and the ERP Reliability Analysis (ERA) Toolbox for assessing test-retest reliability of ERP scores Part 1: Algorithms, framework, and implementation. PsyArXiv. doi:10.31234/osf.io/kcven · DOI
  12. Clayson, P. E., Carbine, K. A., & Larson, M. J. (2020). A registered report of error-related negativity and reward positivity as biomarkers of depression: P-curving the evidence. International Journal of Psychophysiology, 150, 50-72. doi:https://doi.org/10.1016/j.ijpsycho.2020.01.005 · DOI
  13. Clayson, P. E., Joshi, Y. B., Thomas, M. L., Tarasenko, M., Bismark, A., Sprock, J., . . . Light, G. A. (2020). The viability of the frequency following response characteristics for use as biomarkers of cognitive therapeutics in schizophrenia. PsyArXiv. doi:10.31234/osf.io/fgevt · DOI
  14. Clayson, P. E., & Larson, M. J. (2013). Adaptation to emotional conflict: Evidence from a novel face emotion paradigm. PLoS ONE, 8, e75776. doi:10.1371/journal.pone.0075776 · DOI
  15. Clayson, P. E., & Miller, G. A. (2017a). ERP Reliability Analysis (ERA) Toolbox: An open- source toolbox for analyzing the reliability of event-related potentials. International Journal of Psychophysiology, 111, 68-79. doi:https://doi.org/10.1016/j.ijpsycho.2016.10.012 · DOI
  16. Clayson, P. E., & Miller, G. A. (2017b). Psychometric considerations in the measurement of event-related brain potentials: Guidelines for measurement and reporting. International Journal of Psychophysiology, 111, 57-67. doi:https://doi.org/10.1016/j.ijpsycho.2016.09.005 · DOI
  17. Clayson, P. E., Wynn, J. K., Green, M. F., & Horan, W. P. (2018). Cognitive correlates of performance monitoring ERPs in schizophrenia. Psychophysiology, 55, S34-S136. doi:10.1111/psyp.13264 · DOI
  18. Clayson, P. E., Wynn, J. K., Infantolino, Z. P., Hajcak, G., Green, M. F., & Horan, W. P. (2019). Reward processing in certain versus uncertain contexts in schizophrenia: An event- related potential (ERP) study. Journal of Abnormal Psychology, 128, 867-880. doi:10.1037/abn0000469 · DOI
  19. Crane, N. A., Funkhouser, C. J., Burkhouse, K. L., Klumpp, H., Phan, K. L., & Shankman, S. A. (2021). Cannabis users demonstrate enhanced neural reactivity to reward: An event- related potential and time-frequency EEG study. Addictive Behaviors, 113, 106669. doi:https://doi.org/10.1016/j.addbeh.2020.106669 · DOI
  20. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334. doi:10.1007/BF02310555 · DOI · Paper page
  21. Cronbach, L. J., & Furby, L. (1970). How we should measure "change": Or should we? Psychological Bulletin, 74, 68-80.
  22. Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnum, N. (1972). The dependability of behavioral measures: Theory of generalizability for scores and profiles. New York, NY: John Wiley.
  23. De Cesarei, A., & Codispoti, M. (2011). Affective modulation of the LPP and α-ERD during picture viewing. Psychophysiology, 48, 1397-1404. doi:10.1111/j.1469- 8986.2011.01204.x · DOI
  24. Draheim, C., Mashburn, C. A., Martin, J. D., & Engle, R. W. (2019). Reaction time in differential and developmental research: A review and commentary on the problems and alternatives. Psychological Bulletin, 145, 508-535. doi:10.1037/bul0000192 · DOI · Paper page
  25. Feldt, L. S., & Charter, R. A. (2003). Estimating the reliability of a test split into two parts of equal or unequal length. Psychological Methods, 8, 102-109. doi:10.1037/1082- 989X.8.1.102 · DOI
  26. Fischer, A. G., Klein, T. A., & Ullsperger, M. (2017). Comparing the error-related negativity across groups: The impact of error-and trial-number differences. Psychophysiology, 54, 998-1009. doi:10.1111/psyp.12863 · DOI
  27. Flegal, K. M., Kit, B. K., & Graubard, B. I. (2017). Bias in hazard ratios arising From misclassification according to self-reported weight and height in observational studies of body mass index and mortality. American Journal of Epidemiology, 187, 125-134. doi:10.1093/aje/kwx193 · DOI
  28. Gelman, A., & Carlin, J. (2014). Beyond power calculations: Assessing type S (sign) and type M (magnitude) errors. Perspectives on Psychological Science, 9, 641-651. doi:10.1177/1745691614551642 · DOI · Paper page
  29. Hajcak, G., Klawohn, J., & Meyer, A. (2019). The utility of event-related potentials in clinical psychology. Annual Review of Clinical Psychology, 15, 71-95. doi:https://doi.org/10.1146/annurev-clinpsy-050718-095457 · DOI
  30. Hanna, G. L., Liu, Y., Rough, H. E., Surapaneni, M., Hanna, B. S., Arnold, P. D., & Gehring, W. J. (2020). A diagnostic biomarker for pediatric generalized anxiety disorder using the error-related negativity. Child Psychiatry & Human Development, 51, 827-838. doi:10.1007/s10578-020-01021-5 · DOI · Paper page
  31. Hochberger, W. C., Joshi, Y. B., Thomas, M. L., Zhang, W., Bismark, A. W., Treichler, E. B. H., . . . Light, G. A. (2019). Neurophysiologic measures of target engagement predict response to auditory-based cognitive training in treatment refractory schizophrenia. Neuropsychopharmacology, 44, 606-612. doi:10.1038/s41386-018-0256-9 · DOI
  32. Holroyd, C. B., Pakzad-Vaezi, K. L., & Krigolson, O. E. (2008). The feedback correct-related positivity: Sensitivity of the event-related brain potential to unexpected positive feedback. Psychophysiology, 45, 688-697. doi:10.1111/j.1469-8986.2008.00668.x · DOI
  33. Jennings, J. R. (1987). Editorial policy on analyses of variance with repeated measures. Psychophysiology, 24, 474-475. doi:10.1111/j.1469-8986.1987.tb00320.x · DOI
  34. Jennings, M., & Cribbie, R. A. (2016). Comparing pre-post change across groups: Guidelines for choosing between difference scores, ANCOVA, and residual change scores. Journal of Data Science, 14, 205-230.
  35. Kappenman, E. S., & Luck, S. J. (2012). ERP components: The ups and downs of brainwave recordings. In S. J. Luck & E. S. Kappenman (Eds.), The Oxford handbook of event- related potential components (pp. 3-30). New York, NY: Oxford University Press, Inc.
  36. Kenny, D. A. (1975). A quasi-experimental approach to assessing treatment effects in the nonequivalent control group design. Psychological Bulletin, 82, 345-362. doi:10.1037/0033-2909.82.3.345 · DOI
  37. Klawohn, J., Meyer, A., Weinberg, A., & Hajcak, G. (2020). Methodological choices in event- related potential (ERP) research and their impact on internal consistency reliability and individual differences: An examination of the error-related negativity (ERN) and anxiety. Journal of Abnormal Psychology, 129, 29-37. doi:https://doi.org/10.1037/abn0000458 · DOI
  38. Kolossa, A., & Kopp, B. (2018). Data quality over data quantity in computational cognitive neuroscience. NeuroImage, 172, 775-785. doi:10.1016/j.neuroimage.2018.01.005 · DOI
  39. Kujawa, A., Smith, E., Luhmann, C., & Hajcak, G. (2013). The feedback negativity reflects favorable compared to nonfavorable outcomes based on global, not local, alternatives. Psychophysiology, 50, 134-138. doi:10.1111/psyp.12002 · DOI
  40. Linn, R. L., & Slinde, J. A. (1977). The determination of the significance of change between pre- and posttesting periods. Review of Educational Research, 47, 121-150. doi:10.3102/00346543047001121 · DOI
  41. Loken, E., & Gelman, A. (2017). Measurement error and the replication crisis. Science, 355, 584-585. doi:https://doi.org/10.1126/science.aal3618 · DOI · Paper page
  42. Lord, F. M. (1956). The measurement of growth. ETS Research Bulletin Series, 1956, i-22. doi:10.1002/j.2333-8504.1956.tb00058.x · DOI
  43. Lord, F. M. (1963). Elementary models for measuring change. In C. W. Harris (Ed.), Problems in measuring change (pp. 21-38). Madison, WI: University of Wisconsin Press.
  44. Luck, S. J. (2014). An introduction to the event-related potential technique (2nd ed.). Cambridge, MA: The MIT Press.
  45. Luck, S. J., & Gaspelin, N. (2017). How to get statistically significant effects in any ERP experiment (and why you shouldn't). Psychophysiology, 54, 146-157. doi:10.1111/psyp.12639 · DOI
  46. Luck, S. J., Stewart, A. X., Simmons, A. M., & Rhemtulla, M. (2020). Standardized measurement error: A universal measure of data quality for averaged event-related potentials. PsyArXiv. doi:10.31234/osf.io/dwm64 · DOI
  47. Macatee, R. J., Burkhouse, K. L., Afshar, K., Schroth, C., Aase, D. M., Greenstein, J. E., . . . Phan, K. L. (2020). Nonlinear relations between post-traumatic stress symptoms and electrocortical reactivity during emotional face processing in combat-exposed veterans. Psychophysiology, 57, e13423. doi:10.1111/psyp.13423 · DOI
  48. Meyer, A., Lerner, M. D., De Los Reyes, A., Laird, R. D., & Hajcak, G. (2017). Considering ERP difference scores as individual difference measures: Issues with subtraction and alternative approaches. Psychophysiology, 54, 114-122. doi:10.1111/psyp.12664 · DOI · Paper page
  49. Moran, T. P., Schroder, H. S., Kneip, C., & Moser, J. S. (2017). Meta-analysis and psychophysiology: A tutorial using depression and action-monitoring event-related potentials. International Journal of Psychophysiology, 111, 17-32. doi:10.1016/j.ijpsycho.2016.07.001 · DOI
  50. Moser, J., Clayson, P. E., Muir, A., Allen, W., Modersitzki, E., Louis, C., . . . Larson, M. J. (2019). A preliminary report on a multi-site effort to replicate the association between worry and enlarged actionam monitoring brain potentials (Moser et al., 2012). Psychophysiology, 56, S3-S39. doi:10.1111/psyp.13501 · DOI
  51. Muir, A. M., Carbine, K. A., Goodwin, J., Hedges-Muncy, A., Endrass, T., & Larson, M. J. (2019). Differentiating electrophysiological indices of internal and external performance monitoring: Relationship with perfectionism and locus of control. PloS one, 14, e0219883-0219821. doi:10.1371/journal.pone.0219883 · DOI
  52. Muir, A. M., Hedges-Muncy, A., Clawson, A., Carbine, K. A., & Larson, M. J. (2020). Dimensions of anxiety and depression and neurophysiological indicators of error- monitoring: Relationship with delta and theta oscillatory power and error-related negativity amplitude. Psychophysiology, 57, e13595. doi:10.1111/psyp.13595 · DOI
  53. Nieuwenhuis, S., Ridderinkhof, K. R., Blom, J., Band, G. P., & Kok, A. (2001). Error-related brain potentials are differentially related to awareness of response errors: Evidence from an antisaccade task. Psychophysiology, 38, 752-760. doi:10.1111/1469-8986.3850752 · DOI · Paper page
  54. O'Connor Jr, E. F. (1972). Extending classical test theory to the measurement of change. Journal of Educational and Behavioral Statistics, 42, 73-97. doi:10.3102/00346543042001073 · DOI
  55. Overall, J. E., & Woodward, J. A. (1975). Unreliability of difference scores: A paradox for measurement of change. Psychological Bulletin, 82, 85-86. doi:10.1037/h0076158 · DOI
  56. Rogosa, D., Brandt, D., & Zimowski, M. (1982). A growth curve approach to the measurement of change. Psychological Bulletin, 92, 726-748. doi:10.1037/0033-2909.92.3.726 · DOI
  57. Rulon, P. J. (1939). A simplified procedure for determining the reliability of a test by split- halves. Harvard Educational Review, 9, 99-103.
  58. Sandre, A., Banica, I., Riesel, A., Flake, J., Klawohn, J., & Weinberg, A. (2020). Comparing the effects of different methodological decisions on the error-related negativity and its association with behaviour and genders. International Journal of Psychophysiology, 156, 18-39. doi:https://doi.org/10.1016/j.ijpsycho.2020.06.016 · DOI
  59. Schönbrodt, F. D., & Perugini, M. (2013). At what sample size do correlations stabilize? Journal of Research in Personality, 47, 609-612. doi:10.1016/j.jrp.2013.05.009 · DOI · Paper page
  60. Schroder, H. S., Nickels, S., Cardenas, E., Breiger, M., Perlo, S., & Pizzagalli, D. A. (2020). Optimizing assessments of post-error slowing: A neurobehavioral investigation of a flanker task. Psychophysiology, 57, e13473. doi:10.1111/psyp.13473 · DOI
  61. Steinke, A., & Kopp, B. (2020). RELEX: An Excel-based software tool for sampling split-half reliability coefficients. Methods in Psychology, 2, 100023. doi:10.1016/j.metip.2020.100023 · DOI
  62. Suor, J. H., Granros, M., Kujawa, A., Fitzgerald, K. D., Monk, C. S., Phan, K. L., & Burkhouse, K. L. (in press). The moderating role of externalizing problems on the association between anxiety and the error-related negativity in youth. Developmental Psychobiology. doi:10.1002/dev.22023 · DOI
  63. Thomas, D. R., & Zumbo, B. D. (2011). Difference scores from the point of view of reliability and repeated-measures ANOVA. Educational and Psychological Measurement, 72, 37- 43. doi:10.1177/0013164411409929 · DOI
  64. Trafimow, D. (2015). A defense against the alleged unreliability of difference scores. Cogent Mathematics, 2, 3. doi:10.1080/23311835.2015.1064626 · DOI
  65. Traub, R. E. (1967). A note on the reliability of residual change scores. Journal of Educational Measurement, 4, 253-256.
  66. Urbach, T. P., & Kutas, M. (2006). Interpreting event-related brain potential (ERP) distributions: Implications of baseline potentials and variability with application to amplitude normalization by vector scaling. Biological Psychology, 72, 333-343. doi:10.1016/j.biopsycho.2005.11.012 · DOI
  67. Vidal, F., Hasbroucq, T., Grapperon, J., & Bonnet, M. (2000). Is the 'error negativity' specific to errors? Biological Psychology, 51, 109-128. doi:10.1016/s0301-0511(99)00032-0 · DOI · Paper page
  68. Vispoel, W. P., Morris, C. A., & Kilinc, M. (2018). Applications of generalizability theory and their relations to classical test theory and structural equation modeling. Psychological Methods, 23, 1-26. doi:https://doi.org/10.1037/met0000107 · DOI
  69. Warrens, M. J. (2015). A comparison of reliability coefficients for psychometric tests that consist of two parts. Advances in Data Analysis and Classification, 10, 71-84. doi:10.1007/s11634-015-0198-6 · DOI
  70. Williams, R. H., & Zimmerman, D. W. (1982). The comparative reliability of simple and residualized difference scores. The Journal of Experimental Education, 51, 94-97. doi:10.1080/00220973.1982.11011846 · DOI
  71. Williams, R. H., Zimmerman, D. W., Rich, J. M., & Steed, J. L. (1984). An empirical study of the relative error magnitude in three measures of change. The Journal of Experimental Education, 53, 55-57. doi:10.1080/00220973.1984.10806362 · DOI
  72. Yin, P., & Brennan, R. L. (2002). An investigation of difference scores for a grade-level testing program. International Journal of Testing, 2, 83-105. doi:10.1207/S15327574IJT0202_1 · DOI
  73. Zimmerman, D. W., & Williams, R. H. (1982a). Gain scores in research can be highly reliable. Journal of Educational Measurement, 19, 149-154. doi:10.2307/1434908 · DOI
  74. Zimmerman, D. W., & Williams, R. H. (1982b). A note on the correlation of gains and initial status. The Journal of General Psychology, 107, 203-207. doi:10.1080/00221309.1982.9709927 · DOI
  75. Zimmerman, D. W., & Williams, R. H. (1982c). The relative error magnitude in three measures of change. Psychometrika, 47, 141-147. doi:10.1007/BF02296271 · DOI
  76. Zimmerman, D. W., & Williams, R. H. (1998). Reliability of gain scores under realistic assumptions about properties of pre-test and post-test scores. British Journal of Mathematical and Statistical Psychology, 51, 343-351. doi:10.1111/j.2044- 8317.1998.tb00685.x · DOI · Paper page

Cited by

Sorted by citations

Authors
Clayson, P. E., Baldwin, S. A., & Larson, M. J.
Affiliations
  • University of South Florida (US)
  • Brigham Young University (US)
Corresponding author
Clayson, Peter E.
ORCID
Venue
Psychophysiology
Publisher
Wiley
Published
2021-01-21
Volume
58
Issue
4
Pages
e13762-e13762
ISSN
  • 0048-5772 (ISSN-L)
  • 1469-8986
DOI
10.1111/psyp.13762
OpenAlex
W3122346719
Cited by
84
References
9
Citations by year
2026: 15 · 2025: 17 · 2024: 20 · 2023: 8 · 2022: 8 · 2021: 15
FWCI
7.75
Type
journal article
Language
en
Open access
Licence
Locations
  • Psychophysiology (publishedVersion)
  • PubMed (publishedVersion)
  • PsyArXiv (OSF Preprints) (submittedVersion)
  • PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
Topics
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
  • Social Sciences › Psychology › Applied Psychology › Psychological Testing and Assessment
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
Keywords
event-related potentials, difference scores, internal consistency, reliability statistics, generalizability theory, classical test theory, error-related negativity, reward positivity, individual differences
MeSH terms
Biological Variation, Population, Adult, Electroencephalography, Evoked Potentials, Humans, Models, Theoretical, Psychometrics, Psychophysiology, Reproducibility of Results

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review