The DOI matches its Crossref record; 146 of the 150 references it cites were found in a registry or an index.
The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences
Open accessAbstract
Individual differences in cognitive paradigms are increasingly employed to relate cognition to brain structure, chemistry, and function. However, such efforts are often unfruitful, even with the most well established tasks. Here we offer an explanation for failures in the application of robust cognitive paradigms to the study of individual differences. Experimental effects become well established - and thus those tasks become popular - when between-subject variability is low. However, low between-subject variability causes low reliability for individual differences, destroying replicable correlations with other factors and potentially undermining published conclusions drawn from correlational relationships. Though these statistical issues have a long history in psychology, they are widely overlooked in cognitive psychology and neuroscience today. In three studies, we assessed test-retest reliability of seven classic tasks: Eriksen Flanker, Stroop, stop-signal, go/no-go, Posner cueing, Navon, and Spatial-Numerical Association of Response Code (SNARC). Reliabilities ranged from 0 to .82, being surprisingly low for most tasks given their common use. As we predicted, this emerged from low variance between individuals rather than high measurement variance. In other words, the very reason such tasks produce robust and easily replicable experimental effects - low between-participant variability - makes their use as correlational tools problematic. We demonstrate that taking such reliability estimates into account has the potential to qualitatively change theoretical conclusions. The implications of our findings are that well-established approaches in experimental psychology and neuropsychology may not directly translate to the study of individual differences in brain structure, chemistry, and function, and alternative metrics may be required.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This contrast was used (cf. anticipation of social reward minus neutral outcomes) because reliability of difference scores is always lower than reliability of their individual parts,(53) which has been shown to diminish reliability of task-based fMRI(54) (see also(55)) and hinder mechanistic research.(56)”
From Amplification of Positivity Treatment for Anxiety and Depression: A Randomized Experimental Therapeutics Trial Targeting Social Reward Sensitivity to Enhance Social Connectedness (2024) · cites this paper as [53]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For the Stroop task, reliability was ρ congruent = .86, ρ incongruent = .82, and ρ interference = .78.”
From Problematic social media use and cognitive performance in young adults: Associations with selective attention and inhibitory control (2026) · cites this paper as Hedge, Powell, & Sumner, 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Further, it is often the case that cognitive tasks that produce robust effects in experimental settings are poorly suited for individual differences studies due to low between-participant reliability (Hedge et al. 2018).”
From Practicing Retrieval Facilitates Learning (2021) · cites this paper as Hedge et al. 2018
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“For the Simon task, reliability was ρ_congruent = .84, ρ_incongruent = .81, and ρ_interference = .76.”
From Problematic social media use and cognitive performance in young adults: Associations with selective attention and inhibitory control (2026) · cites this paper as Hedge et al., 2018
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007) · Behavior Research Methods · doi:10.3758/bf03193146 · Cited by 65,971
Structural equation modeling in practice: A review and recommended two-step approach
Anderson, J. C., & Gerbing, D. W. (1988) · Psychological Bulletin · doi:10.1037/0033-2909.103.3.411 · Cited by 41,941
Statistical power analyses using G∗Power 3.1: tests for correlation and regression analyses
Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009) · Behavior Research Methods · doi:10.3758/brm.41.4.1149 · Cited by 37,615
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
Statistical Methods for Rates and Proportions
Fleiss, J. L., Levin, B., & Paik, M. C. (2003) · Wiley Series in Probability and Statistics · doi:10.1002/0471445428 · Cited by 18,646
Miyake, A., Friedman, N. P., Emerson, M. J., Witzki, A. H., Howerter, A., & Wager, T. D. (2000) · Cognitive Psychology · doi:10.1006/cogp.1999.0734 · Cited by 15,798
Review of Psychometric theory (2nd ed.).
SITGREAVES, R. (1979) · Contemporary Psychology: A Journal of Reviews · doi:10.1037/018882 · Cited by 11,425
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Posner, M. I. (1980) · Quarterly Journal of Experimental Psychology · doi:10.1080/00335558008248231 · Cited by 9,670 · cited 2×
“Study 3 Again, performance was comparable to previous reports in the literature (Navon, 1977; Posner, 1980; Rusconi et al., 2013).”
Research works printed but not linked: 7
- Crocker, L. M., & Algina, J. (1986). Introduction to Classicial and Modern Test Theory. New York: CBS College Publishing.
- Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundementals of Item Response Theory. Newbury Park: Sage.
- Lynam, D. R., Smith, G. T., Whiteside, S. P., & Cyders, M. A. (2006). The UPPS-P: Assessing give personality pathways to impulsive behavior (Technical Report). West Lafayette: Purdue University.
- Nunnally, J. C. (1978). Psychometric theory (2nd ed.). New York.: McGraw-Hill.
- Peirce, J. W. (2013). Introduction to Python and PsychoPy. Perception, 42, 2–3.
- Rogosa, D. (1995). Myths and methods: "Myths about longitudinal research" plus supplemental questions. In J. M. Gottman (Ed.), The analysis of change (pp. pp. 3–65). Hillsdale: Lawrence Ealbaum Associates.
- Zumbo, B. D. (1999). The simple difference score as an inherently poor measure of change: Some reality, much mythology. In B. Thompson (Ed.), Advances in Social Science Methodoloy (pp. pp. 269–304). Greenwich: JAI Press.
Cited by
Elliott, M. L., Knodt, A. R., Ireland, D., Morris, M. L., Poulton, R., Ramrakha, S., Sison, M. L., Moffitt, T. E., Caspi, A., & Hariri, A. R. (2020) · Psychological Science · doi:10.1177/0956797620916786 · Cited by 714
Why Are Self-Report and Behavioral Measures Weakly Correlated?
Dang, J., King, K. M., & Inzlicht, M. (2020) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2020.01.007 · Cited by 634
“Executive functions are measured by various behavioral tasks with low between-person variance [7].”
BackgroundLarge-scale analysis of test–retest reliabilities of self-regulation measures
Enkavi, A. Z., Eisenberg, I. W., Bissett, P. G., Mazza, G. L., MacKinnon, D. P., Marsch, L. A., & Poldrack, R. A. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1818430116 · Cited by 489
“(33) argue, behavioral tasks designed with the subtraction logic to isolate specific cognitive processes become well-established in the literature precisely for their low between-subject variability.”
BackgroundIndividual Differences in Language Acquisition and Processing
Kidd, E., Donnelly, S., & Christiansen, M. H. (2018) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2017.11.006 · Cited by 507
“An experimental task that is reliable at the group level is, often by design, an unreliable ID task, because experimental tasks aim to minimize between-participant variability, whereas ID tasks should aim to maximize it [139].”
BackgroundUncovering the structure of self-regulation through data-driven ontology discovery
Eisenberg, I. W., Bissett, P. G., Zeynep Enkavi, A., Li, J., MacKinnon, D. P., Marsch, L. A., & Poldrack, R. A. (2019) · Nature Communications · doi:10.1038/s41467-019-10301-1 · Cited by 474
Metadata
- Authors
- Hedge, C., Powell, G., & Sumner, P.
- Affiliations
- Cardiff University (GB)
- Corresponding author
- Hedge, Craig
- Venue
- Behavior Research Methods
- Published
- 2017-07-19
- Volume
- 50
- Issue
- 3
- Pages
- 1166-1186
- ISSN
- 1554-351X (ISSN-L)
- 1554-3528
- PMC
- PMC5990556
- OpenAlex
- W2724821146
- Cited by
- 1,457
- References
- 150
- Citations by year
- 2026: 165 · 2025: 168 · 2024: 178 · 2023: 207 · 2022: 188 · 2021: 192 · 2020: 175 · 2019: 111 · 2018: 61 · 2017: 9
- FWCI
- 125.87
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Behavior Research Methods (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- ORCA Online Research @Cardiff (Cardiff University) (acceptedVersion)
- Aston Publications Explorer (Aston University) (submittedVersion, cc-by)
- OSF Preprints (OSF Preprints) (submittedVersion)
- Funders
- Wellcome Trust (104943/Z/14/Z)
- Economic and Social Research Council (ES/K002325/1)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Cognitive Abilities and Testing
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Keywords
- individual differences, test-retest reliability, between-subject variability, measurement reliability, cognitive paradigm, flanker task, Stroop task, stop-signal task, go/no-go task, Navon task, SNARC effect, neuropsychology, brain structure and cognition
- MeSH terms
- Brain, Cognition, Cues, Female, Humans, Individuality, Male, Neuropsychology, Reaction Time, Task Performance and Analysis, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences. https://app.cochat.ai/papers/10.3758/s13428-017-0935-1
- The paper
- Hedge, C., Powell, G., & Sumner, P. (2018). The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences. Behavior Research Methods. https://doi.org/10.3758/s13428-017-0935-1
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.