Large-scale analysis of test–retest reliabilities of self-regulation measures
Open accessThe DOI matches its Crossref record; 52 of the 53 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
The ability to regulate behavior in service of long-term goals is a widely studied psychological construct known as self-regulation. This wide interest is in part due to the putative relations between self-regulation and a range of real-world behaviors. Self-regulation is generally viewed as a trait, and individual differences are quantified using a diverse set of measures, including self-report surveys and behavioral tasks. Accurate characterization of individual differences requires measurement reliability, a property frequently characterized in self-report surveys, but rarely assessed in behavioral tasks. We remedy this gap by ( i ) providing a comprehensive literature review on an extensive set of self-regulation measures and ( ii ) empirically evaluating test–retest reliability of this battery in a new sample. We find that dependent variables (DVs) from self-report surveys of self-regulation have high test–retest reliability, while DVs derived from behavioral tasks do not. This holds both in the literature and in our sample, although the test–retest reliability estimates in the literature are highly variable. We confirm that this is due to differences in between-subject variability. We also compare different types of task DVs (e.g., model parameters vs. raw response times) in their suitability as individual difference DVs, finding that certain model parameters are as stable as raw DVs. Our results provide greater psychometric footing for the study of self-regulation and provide guidance for future studies of individual differences in this domain.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Nonetheless, custom- designed behavioural tasks can be less effective at measuring typical rather than extraordinary performance 54 .”
From Predictors and consequences of intellectual humility (2022) · cites this paper as [54]
In the citing paper
“This has partially been done for the cognitive control example (Henderson et al. 2012), but in general, standard laboratory tasks are not well optimized to capture this kind of endophenotype (Enkavi et al. 2019).”
From Deep brain stimulation for psychiatric disorders: From focal brain targets to cognitive networks (2021) · cites this paper as Enkavi et al. 2019
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
Statistical Methods for Rates and Proportions
Fleiss, J. L., Levin, B., & Paik, M. C. (2003) · Wiley Series in Probability and Statistics · doi:10.1002/0471445428 · Cited by 18,646
Miyake, A., Friedman, N. P., Emerson, M. J., Witzki, A. H., Howerter, A., & Wager, T. D. (2000) · Cognitive Psychology · doi:10.1006/cogp.1999.0734 · Cited by 15,798
An Integrative Theory of Prefrontal Cortex Function
Miller, E. K., & Cohen, J. D. (2001) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.24.1.167 · Cited by 12,803 · cited 1×
“Within cognitive psychology and neuroscience, there has been particular interest in isolating mechanisms involved in “cognitive control” (13, 14).”
BackgroundAmazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
In search of how people change: Applications to addictive behaviors.
Prochaska, J. O., DiClemente, C. C., & Norcross, J. C. (1992) · American Psychologist · doi:10.1037/0003-066x.47.9.1102 · Cited by 8,596
The proof and measurement of association between two things
Spearman, C. (1904) · The American Journal of Psychology · doi:10.2307/1412159 · Cited by 8,039
Measures of Reliability in Sports Medicine and Science
Hopkins, W. G. (2000) · Sports Medicine · doi:10.2165/00007256-200030010-00001 · Cited by 4,635 · cited 3×
“The ICC, which ranges from −1 to 1, is a preferred metric for reliability and is not biased by sample size (32).”
BackgroundInfluentialRatcliff, R. (1978) · Psychological Review · doi:10.1037/0033-295x.85.2.59 · Cited by 4,513 · cited 1×
“We compare the reliability of raw DVs (response times and accuracies) to parameters of the DDM, a well-established model that addresses speed–accuracy trade-offs and offers interpretable latent variables (34, 35).”
MethodRunning experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767
Delay of gratification in children.
Mischel, W., Shoda, Y., & Rodriguez, M. L. (1989) · Science · doi:10.1126/science.2658056 · Cited by 3,397
(1981) · PubMed · Cited by 2,253
Heroin addicts have higher discount rates for delayed rewards than non-drug-using controls.
Kirby, K. N., Petry, N. M., & Bickel, W. K. (1999) · Journal of Experimental Psychology: General · doi:10.1037/0096-3445.128.1.78 · Cited by 2,171
Self-Control in Decision-Making Involves Modulation of the vmPFC Valuation System
Hare, T. A., Camerer, C. F., & Rangel, A. (2009) · Science · doi:10.1126/science.1168450 · Cited by 1,928
Measuring change over time: Assessing the usefulness of evaluative instruments
Guyatt, G., Walter, S., & Norman, G. (1987) · Journal of Chronic Diseases · doi:10.1016/0021-9681(87)90069-5 · Cited by 1,880
Diffusion decision model: Current issues and history
Ratcliff, R., Smith, P. L., Brown, S. D., & McKoon, G. (2016) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2016.01.007 · Cited by 1,630 · cited 1×
“We compare the reliability of raw DVs (response times and accuracies) to parameters of the DDM, a well-established model that addresses speed–accuracy trade-offs and offers interpretable latent variables (34, 35).”
MethodEvaluating amazon’s mechanical turk as a tool for experimental behavioral research
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057410 · Cited by 1,617
The Online Laboratory: Conducting Experiments in a Real Labor Market∗Preprint
Zeckhauser, R. J. (2016) · Cited by 1,530
The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences
Hedge, C., Powell, G., & Sumner, P. (2018) · Behavior Research Methods · doi:10.3758/s13428-017-0935-1 · Cited by 1,457 · cited 2×
“(33) argue, behavioral tasks designed with the subtraction logic to isolate specific cognitive processes become well-established in the literature precisely for their low between-subject variability.”
BackgroundHDDM: Hierarchical Bayesian estimation of the Drift-Diffusion Model in Python
Wiecki, T. V., Sofer, I., & Frank, M. J. (2013) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2013.00014 · Cited by 1,047 · cited 4×
“The HDDM uses hierarchical Bayesian modeling to allow simultaneous estimation of both group and individual subject parameters.”
MethodInfluentialA meta-analysis of the convergent validity of self-control measures
Duckworth, A. L., & Kern, M. L. (2011) · Journal of Research in Personality · doi:10.1016/j.jrp.2011.02.004 · Cited by 1,031
The proof and measurement of association between two things. By C. Spearman, 1904.
Spearman, C. (1987) · PubMed · Cited by 1,023
The viability of crowdsourcing for survey research
Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. N. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0081-0 · Cited by 1,018
Karr, J. E., Areshenkoff, C. N., Rast, P., Hofer, S. M., Iverson, G. L., & Garcia-Barrera, M. A. (2018) · Psychological Bulletin · doi:10.1037/bul0000160 · Cited by 637
Sharma, L., Markon, K. E., & Clark, L. A. (2014) · Psychological Bulletin · doi:10.1037/a0034418 · Cited by 630
Baker, F., Johnson, M. W., & Bickel, W. K. (2003) · Journal of Abnormal Psychology · doi:10.1037/0021-843x.112.3.382 · Cited by 602
The online laboratory: conducting experiments in a real labor market
Horton, J. J., Rand, D. G., & Zeckhauser, R. (2011) · Experimental Economics · doi:10.1007/s10683-011-9273-9 · Cited by 567
An EZ-diffusion model for response time and accuracy
Wagenmakers, E.-J., Van Der Maas, H. L. J., & Grasman, R. P. P. P. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194023 · Cited by 555
Why obese children cannot resist food: The role of impulsivity
Nederkoorn, C., Braet, C., Van Eijs, Y., Tanghe, A., & Jansen, A. (2006) · Eating Behaviors · doi:10.1016/j.eatbeh.2005.11.005 · Cited by 532
Cyders, M. A., & Coskunpinar, A. (2011) · Clinical Psychology Review · doi:10.1016/j.cpr.2011.06.001 · Cited by 531
Pathological gambling severity is associated with impulsivity in a delay discounting procedure
Alessi, S. M., & Petry, N. M. (2003) · Behavioural Processes · doi:10.1016/s0376-6357(03)00150-5 · Cited by 526
Reframing Health Behavior Change With Behavioral Economics
(2000) · Psychology Press eBooks · doi:10.4324/9781410605061 · Cited by 427
The Online Laboratory: Conducting Experiments in a Real Labor Market
Horton, J., Rand, D., & Zeckhauser, R. (2010) · National Bureau of Economic Research · doi:10.3386/w15961 · Cited by 388
Nederkoorn, C., Smulders, F. T. Y., Havermans, R. C., Roefs, A., & Jansen, A. (2006) · Appetite · doi:10.1016/j.appet.2006.05.008 · Cited by 374
Impulsivity and response inhibition in alcohol dependence and problem gambling
Lawrence, A. J., Luty, J., Bogdan, N. A., Sahakian, B. J., & Clark, L. (2009) · Psychopharmacology · doi:10.1007/s00213-009-1645-x · Cited by 329
Behavioral components of impulsivity.
Stahl, C., Voss, A., Schmitz, F., Nuszbaum, M., Tüscher, O., Lieb, K., & Klauer, K. C. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033981 · Cited by 319
Rey-Mermet, A., Gade, M., & Oberauer, K. (2018) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037/xlm0000450 · Cited by 310
Interpreting Reaction Time Measures in Between-Group Comparisons
Salthouse, T. A., & Hedden, T. (2002) · Journal of Clinical and Experimental Neuropsychology · doi:10.1076/jcen.24.7.858.8392 · Cited by 171
Paap, K. R., & Sawi, O. M. (2016) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2016.10.002 · Cited by 162
Hendrickson, K. L., & Rasmussen, E. B. (2013) · Behaviour Research and Therapy · doi:10.1016/j.brat.2013.04.002 · Cited by 161
Time Discounting Predicts Creditworthiness
Meier, S., & Sprenger, C. D. (2012) · Psychological Science · doi:10.1177/0956797611425931 · Cited by 156
Saunders, B., Milyavskaya, M., Etz, A., Randles, D., & Inzlicht, M. (2018) · Collabra: Psychology · doi:10.1525/collabra.134 · Cited by 132
Statistical Methods for Rates and Proportions.
Marshall, T. F. D. C., & Fleiss, J. L. (1976) · Journal of the Royal Statistical Society Series D (The Statistician) · doi:10.2307/2988144 · Cited by 132
Self-reported and neuropsychological measures of impulsivity in pathological gambling
FUENTES, D., TAVARES, H., ARTES, R., & GORENSTEIN, C. (2006) · Journal of the International Neuropsychological Society · doi:10.1017/s1355617706061091 · Cited by 125
Go–no-go performance in pathological gamblers
Kertzman, S., Lowengrub, K., Aizer, A., Vainder, M., Kotler, M., & Dannon, P. N. (2008) · Psychiatry Research · doi:10.1016/j.psychres.2007.06.026 · Cited by 103
The Experiment Factory: Standardizing Behavioral Experiments
Sochat, V. V., Eisenberg, I. W., Enkavi, A. Z., Li, J., Bissett, P. G., & Poldrack, R. A. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.00610 · Cited by 64
Self-citationStautz, K., Zupan, Z., Field, M., & Marteau, T. M. (2018) · Health Psychology Review · doi:10.1080/17437199.2017.1421477 · Cited by 64 · cited 1×
“Self-regulation is also thought to play a critical role in behavior change, bolstering the individual against temptations to revert to older behaviors (1, 10, 11), although its role as a moderator of behavior change has recently been challenged (12).”
BackgroundApplying novel technologies and methods to inform the ontology of self-regulation
Eisenberg, I. W., Bissett, P. G., Canning, J. R., Dallery, J., Enkavi, A. Z., Whitfield-Gabrieli, S., Gonzalez, O., Green, A. I., Greene, M. A., Kiernan, M., et al. (2018) · Behaviour Research and Therapy · doi:10.1016/j.brat.2017.09.014 · Cited by 61 · cited 1×
“Participants were a subset from a larger study (25) conducted on Amazon Mechanical Turk.”
BackgroundSelf-citationThe (In)significance of Executive Functions for the Trait of Self-Control: A Psychometric Study
Nęcka, E., Gruszka, A., Orzechowski, J., Nowak, M., & Wójcik, N. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.01139 · Cited by 44
Tiplady, B. (1992) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03203464 · Cited by 31
Caruso, J. C. (2004) · European Journal of Psychological Assessment · doi:10.1027/1015-5759.20.3.166 · Cited by 28
Behavior change and individual differences in self-control
Rozensky, R. H., & Bellack, A. S. (1974) · Behaviour Research and Therapy · doi:10.1016/0005-7967(74)90127-2 · Cited by 26
Uncovering mental structure through data-driven ontology discoveryPreprint
Eisenberg, I. W., Bissett, P. G., Enkavi, A. Z., Li, J. K., MacKinnon, D. P., Marsch, L. A., & Poldrack, R. A. (2018) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/fvqej · Cited by 20
Self-citation
Cited by
Self-regulation in childhood as a predictor of future outcomes: A meta-analytic review.
Robson, D. A., Allen, M. S., & Howard, S. J. (2020) · Psychological Bulletin · doi:10.1037/bul0000227 · Cited by 759
“It is interesting to note that while correlations between selfregulation measures tend to be low (Duckworth & Kern, 2011; Enkavi et al., 2019) they tend to predict outcomes to a similar extent.”
BackgroundUncovering the structure of self-regulation through data-driven ontology discovery
Eisenberg, I. W., Bissett, P. G., Zeynep Enkavi, A., Li, J., MacKinnon, D. P., Marsch, L. A., & Poldrack, R. A. (2019) · Nature Communications · doi:10.1038/s41467-019-10301-1 · Cited by 474
Self-citationMeasuring Adaptive Control in Conflict Tasks
Braem, S., Bugg, J. M., Schmidt, J. R., Crump, M. J. C., Weissman, D. H., Notebaert, W., & Egner, T. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2019.07.002 · Cited by 358
Ekhtiari, H., Tavakoli, H., Addolorato, G., Baeken, C., Bonci, A., Campanella, S., Castelo-Branco, L., Challet-Bouju, G., Clark, V. P., Claus, E., et al. (2019) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2019.06.007 · Cited by 297
Predictors and consequences of intellectual humility
Porter, T., Elnakouri, A., Meyers, E. A., Shibayama, T., Jayawickreme, E., & Grossmann, I. (2022) · Nature Reviews Psychology · doi:10.1038/s44159-022-00081-9 · Cited by 248
Metadata
- Authors
- Enkavi, A. Z., Eisenberg, I. W., Bissett, P. G., Mazza, G. L., MacKinnon, D. P., Marsch, L. A., & Poldrack, R. A.
- Affiliations
- Stanford University (US)
- Mayo Clinic (US)
- Arizona State University (US)
- Dartmouth College (US)
- ORCID
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2019-03-06
- Volume
- 116
- Issue
- 12
- Pages
- 5472-5477
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC6431228
- OpenAlex
- W2920590181
- Cited by
- 489
- References
- 53
- Citations by year
- 2026: 48 · 2025: 60 · 2024: 65 · 2023: 63 · 2022: 81 · 2021: 64 · 2020: 73 · 2019: 34
- FWCI
- 13.16
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- HHS | National Institutes of Health (UH2DA041713)
- HHS | NIH | National Institute on Drug Abuse (P30DA029926)
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Psychology › Applied Psychology › Behavioral Health and Interventions
- Social Sciences › Psychology › Social Psychology › Grit, Self-Efficacy, and Motivation
- Keywords
- self-regulation, test-retest reliability, individual differences, behavioral tasks, self-report questionnaire, measurement reliability, between-subject variability, reaction time (psychology), model parameters, trait measurement
- MeSH terms
- Self-Control, Humans, Individuality, Psychological Tests, Reproducibility of Results, Models, Statistical, Self Report
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.