The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Reaction time in differential and developmental research: A review and commentary on the problems and alternatives.
Open accessAbstract
Reaction time is believed to be a good indicator of the speed and efficiency of mental processes and is a ubiquitous variable in the behavioral sciences. Despite this popularity, there are numerous issues associated with using reaction time (RT), specifically in differential and developmental research. Here, we identify and focus on two main problems-unreliability and sensitivity to speed-accuracy interactions. The use of difference scores is a primary factor that leads to many RT measures having demonstrably low reliability, and RT measures in general often do not properly account for speed-accuracy interactions. Both factors jeopardize the validity and interpretability of results based on RT. Here, we evaluate conceptually and empirically how these issues affect individual differences research. Although the empirical evidence we provide are primarily within the domains of attention control and task switching, we highlight examples from various other areas of psychological inquiry. We also discuss many of the statistical and methodological alternatives available to researchers conducting correlational studies. Ultimately, we encourage researchers comparing individuals of differing cognitive and developmental levels to strongly consider using these alternatives in lieu of RT, specifically RT difference scores. (PsycINFO Database Record (c) 2019 APA, all rights reserved).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“One complication is that the testing effect is a difference score between a condition that receives retrieval practice and one that does not, and difference scores are inherently poor in reliability (Draheim et al. 2019).”
From Practicing Retrieval Facilitates Learning (2021) · cites this paper as Draheim et al. 2019
In the citing paper
“However, treating RT and accuracy as distinct outcomes does not appropriately account for the speed-accuracy trade-off that exists between the two parameters24,25.”
From Delayed goal-directed processing underlies inhibitory control challenges in adult ADHD (2026) · cites this paper as [24]
Citations
Cites
Studies of interference in serial verbal reactions
Stroop, J. R. (1935) · Journal of Experimental Psychology · doi:10.1037/h0054651 · Cited by 18,622
Effects of noise letters upon the identification of a target letter in a nonsearch task
Eriksen, B. A., & Eriksen, C. W. (1974) · Perception & Psychophysics · doi:10.3758/bf03203267 · Cited by 7,230
The processing-speed theory of adult age differences in cognition
Salthouse, T. A. (1996) · Psychological Review · doi:10.1037/0033-295x.103.3.403 · Cited by 5,902
Testing the efficiency and independence of attentional networks
Fan, J., McCandliss, B. D., Sommer, T., Raz, A., & Posner, M. I. (2002) · Journal of Cognitive Neuroscience · doi:10.1162/089892902317361886 · Cited by 3,717
The Sources of Four Commonly Reported Cutoff Criteria: What Did They Really Say?
Lance, C. E., Butts, M. M., & Michels, L. C. (2006) · Organizational Research Methods · doi:10.1177/1094428105284919 · Cited by 2,053
Control and interference in task switching—A review
Kiesel, A., Steinhauser, M., Wendt, M., Falkenstein, M., Jost, K., Philipp, A. M., & Koch, I. (2010) · Psychological Bulletin · doi:10.1037/a0019842 · Cited by 1,485
The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences
Hedge, C., Powell, G., & Sumner, P. (2018) · Behavior Research Methods · doi:10.3758/s13428-017-0935-1 · Cited by 1,457
Long-term reduction in implicit race bias: A prejudice habit-breaking intervention
Devine, P. G., Forscher, P. S., Austin, A. J., & Cox, W. T. L. (2012) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2012.06.003 · Cited by 1,259
Executive Attention, Working Memory Capacity, and a Two-Factor Theory of Cognitive Control
Engle, R. W., & Kane, M. J. (2003) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(03)44005-x · Cited by 1,104
Self-citationWorking Memory and Executive Attention: A Revisit
Engle, R. W. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691617720478 · Cited by 339
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Back to Basics: The Importance of Conceptual Clarification in Psychological Science
Bringmann, L. F., Elmer, T., & Eronen, M. I. (2022) · Current Directions in Psychological Science · doi:10.1177/09637214221096485 · Cited by 221
How does bilingualism modify cognitive function? Attention to the mechanism
Bialystok, E., & Craik, F. I. M. (2022) · Psychonomic Bulletin & Review · doi:10.3758/s13423-022-02057-5 · Cited by 269
Best research practices for using the Implicit Association Test
Greenwald, A. G., Brendl, M., Cai, H., Cvencek, D., Dovidio, J. F., Friese, M., Hahn, A., Hehman, E., Hofmann, W., Hughes, S., et al. (2022) · Behavior Research Methods · doi:10.3758/s13428-021-01624-3 · Cited by 209
A toolbox approach to improving the measurement of attention control.Preprint
Draheim, C., Tsukahara, J., Martin, J., Mashburn, C. A., & Engle, R. (2019) · Journal of experimental psychology. General · doi:10.31234/osf.io/q985d · Cited by 192
“In addition, data and ideas from the present study were disseminated in various conference presentations (Draheim, Martin, Tsukahara, Mashburn, & Engle, 2018; Draheim, Mashburn, & Engle, 2018; Draheim et al., 2019; Engle, 2017).”
MethodSelf-citationAttention Control: A Cornerstone of Higher-Order Cognition
Burgoyne, A. P., & Engle, R. W. (2020) · Current Directions in Psychological Science · doi:10.1177/0963721420969371 · Cited by 186
Self-citationPracticing Retrieval Facilitates Learning
McDermott, K. B. (2021) · Annual review of psychology · doi:10.1146/annurev-psych-010419-051019 · Cited by 113
“One complication is that the testing effect is a difference score between a condition that receives retrieval practice and one that does not, and difference scores are inherently poor in reliability (Draheim et al. 2019).”
BackgroundNature and measurement of attention control.
Burgoyne, A. P., Tsukahara, J. S., Mashburn, C. A., Pak, R., & Engle, R. W. (2023) · Journal of Experimental Psychology General · doi:10.1037/xge0001408 · Cited by 140
“…the use of difference scores in individual differences research for decades because of their unreliability at the level of the participant, and subsequently poor validity (Ackerman & Hambrick, 2020; Cronbach & Furby, 1970; Draheim et al., 2016, 2019; Friedman & Miyake, 2004; Hedge et al., 2018).”
Self-citationDuration and extent of bilingual experience modulate neurocognitive outcomes
DeLuca, V., Rothman, J., Bialystok, E., & Pliatsikas, C. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.116222 · Cited by 141
Individual differences in computational psychiatry: A review of current challenges
Karvelis, P., Paulus, M. P., & Diaconescu, A. O. (2023) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2023.105137 · Cited by 122
“…jointly accounting for choice and reaction time data), it can extract valuable information from trail-level variability, especially if it relates to learning (e.g., with trial-by-trial modelling approaches), and in many cases it circumvents the need to use difference scores (Draheim et al., 2019).”
Improving the Reliability of Cognitive Task Measures: A Narrative ReviewPreprint
Zorowitz, S., & Niv, Y. (2022) · Biological Psychiatry: Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2023.02.004 · Cited by 82
Why Most Studies of Individual Differences With Inhibition Tasks Are Bound To FailPreprint
Rouder, J. N., Kumar, A., & Haaf, J. M. (2019) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/3cjr5 · Cited by 89
“In practice, because there is no subtraction, measures are often highly reliable and lead to higher correlations [Draheim et al. (2019);Hedge:etal:2018].”
BackgroundInfluentialOn the origins and evolution of the Attention Network Tests
Almeida, R. D. S., Faria, A. J. P., & Klein, R. M. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2021.02.028 · Cited by 85
Gidu, D., Badau, D., Stoica, M., Aron, A., Focan, G., Monea, D., Stoica, A., & Calotă, N. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19042028 · Cited by 61
“When speaking about the benefits of PT for neuromuscular control, we refer to improvements in muscle reflex activity [6–11], reaction time [12]), rate of force development and electromiography activity [13].”
BackgroundIndividual Differences in Inhibitory Control: A latent Variable Analysis
Gärtner, A., & Strobel, A. (2021) · Journal of Cognition · doi:10.5334/joc.150 · Cited by 92
“For example, Draheim et al. (2019) extensively discussed the problems and alternatives of using standard reaction time difference scores in differential and developmental research in their recent review.”
BackgroundWhy many studies of individual differences with inhibition tasks may not localize correlations
Rouder, J., Kumar, A., & Haaf, J. M. (2023) · Psychonomic Bulletin & Review · doi:10.3758/s13423-023-02293-3 · Cited by 54
“Draheim et al. (2019) argue that commonly used response time difference scores are susceptible to low reliability and other artifacts such as speed-accuracy tradeoffs.”
BackgroundThe common factor of executive functions measures nothing but speed of information uptake
Löffler, C., Frischkorn, G. T., Hagemann, D., Sadus, K., & Schubert, A. (2024) · Psychological Research · doi:10.1007/s00426-023-01924-7 · Cited by 69
“Second, some researchers caution against using difference scores because they tend to show low reliabilities (Ackerman & Hambrick, 2020; Draheim et al., 2019, 2023; Hedge et al., 2018; Miller & Ulrich, 2013; von Bastian et al., 2020; Weigard et al., 2021).”
Weigard, A., & Sripada, C. (2021) · Biological Psychiatry Global Open Science · doi:10.1016/j.bpsgos.2021.02.001 · Cited by 43
Weigard, A., & Sripada, C. (2020) · Biological Psychiatry Global Open Science · doi:10.31234/osf.io/q9hge · Cited by 45
“Such measures consistently demonstrate poor test-retest reliability (24–27); that is, the rank order of subjects fails to be preserved across testing occasions, limiting the usefulness of these metrics for individual differences research (28,29).”
BackgroundCarlson, J. M., & Fang, L. (2020) · Motivation and Emotion · doi:10.1007/s11031-020-09834-6 · Cited by 45
“Additionally, it should be noted that poor reliability for individual differences does not necessarily mean that the experimental effects associated with differences between conditions are unreliable (Draheim et al. 2019; Goodhew and Edwards 2019; Hedge et al. 2018).”
BackgroundRutkowski, S., Adamczyk, M., Pastuła, A., Gos, E., Luque-Moreno, C., & Rutkowska, A. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18031297 · Cited by 51
“problems arise with the implementation of response time, especially in differential and developmental research [6].”
BackgroundPaying attention to climate change: Positive images of climate change solutions capture attention
Carlson, J. M., Kaull, H., Steinhauer, M., Zigarac, A., & Cammarata, J. (2020) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2020.101477 · Cited by 50
Nash Unsworth, A., Miller, A. L., & Robison, M. K. (2020) · Journal of Experimental Psychology General · doi:10.1037/xge0001000 · Cited by 64
Attention control: The missing link between sensory discrimination and intelligence
Tsukahara, J. S., Harrison, T. L., Draheim, C., Martin, J. D., & Engle, R. W. (2020) · Attention Perception & Psychophysics · doi:10.3758/s13414-020-02044-9 · Cited by 52
Self-citationLuna, F. G., Roca, J., Martín‐Arévalo, E., & Lupiáñez, J. (2020) · Behavior Research Methods · doi:10.3758/s13428-020-01483-4 · Cited by 44
Attentional bias in alcohol drinkers: A systematic review of its link with consumption variables
Bollen, Z., Field, M., Billaux, P., & Maurage, P. (2022) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2022.104703 · Cited by 37
Burgoyne, A. P., Mashburn, C. A., Tsukahara, J. S., & Engle, R. W. (2021) · PsyArXiv (OSF Preprints) · doi:10.1016/j.intell.2022.101629 · Cited by 46
Self-citationTHE CONTRIBUTIONS OF IMPLICIT-STATISTICAL LEARNING APTITUDE TO IMPLICIT SECOND-LANGUAGE KNOWLEDGE
Godfroid, A., & Kim, K. M. (2021) · Studies in Second Language Acquisition · doi:10.1017/s0272263121000085 · Cited by 51
“Another issue concerns the low reliability of aptitude and knowledge measures obtained from reaction time data (Draheim et al., 2019; Rouder & Haaf, 2019), which may obscure any aptitude–knowledge relationships.”
BackgroundMethodIndividual differences in artificial and natural language statistical learning
Isbilen, E. S., McCauley, S. M., & Christiansen, M. H. (2022) · Cognition · doi:10.1016/j.cognition.2022.105123 · Cited by 36
Martin, J. D., Shipstead, Z., Harrison, T. L., Redick, T. S., Bunting, M. F., & Engle, R. W. (2019) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/xlm0000705 · Cited by 28
Self-citationReplication and extension of the toolbox approach to measuring attention control
Draheim, C., Tsukahara, J. S., & Engle, R. W. (2023) · Behavior Research Methods · doi:10.3758/s13428-023-02140-2 · Cited by 37
Self-citationDomingue, B. W., Kanopka, K., Stenhaug, B., Sulik, M. J., Beverly, T., Brinkhuis, M. J. S., Circi, R., Faul, J. D., Liao, D., McCandliss, B. D., et al. (2022) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986221099906 · Cited by 32
“Our results also suggest, dovetailing with others, that accuracy is integral for understanding individual differences (i.e., RT alone may be insufficient; Draheim et al., 2019; Draheim et al., 2020).”
ResultNo relation of need for cognition to basic executive functions.
Gärtner, A., Grass, J., Wolff, M., Goschke, T., Strobel, A., & Strobel, A. (2021) · Journal of Personality · doi:10.1111/jopy.12639 · Cited by 26
Liesefeld, H. R., & Janczyk, M. (2022) · Behavior Research Methods · doi:10.3758/s13428-022-01843-2 · Cited by 31
“By contrast, BIS is by no means restricted to within-participants designs, but we and others consider many use cases even going beyond experimental psychology (e.g., Bakun Emesh et al., 2021; Draheim et al., 2019; Liesefeld & Janczyk, 2019; Liu et al., 2019; Mueller et al., 2019; Palmqvist et al., 2020; Stojan et al.,…”
MethodCalibration of cognitive tests to address the reliability paradox for decision-conflict tasks
Kucina, T., Wells, L., Lewis, I., de Salas, K., Kohl, A. T., Palmer, M. A., Sauer, J., Matzke, D., Aidman, E., & Heathcote, A. (2023) · Nature Communications · doi:10.1038/s41467-023-37777-2 · Cited by 28
“Such considerations have prompted calls to abandon measures based on con fl ict-task RT difference scores in differential research 27 , and instead center on either the development of alternative measurement approaches 28 , or shift focus to differences in processing speed and strategy 29 .”
Using precision approaches to improve brain-behavior prediction
Lee, H. J., Dworetsky, A., Labora, N., & Gratton, C. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.09.007 · Cited by 24
Unpacking the overlap between Autism and ADHD in adults: A multi-method approach
Waldren, L. H., Leung, F. Y. N., Hargitai, L. D., Burgoyne, A. P., Liceralde, V. R. T., Livingston, L. A., & Shah, P. (2024) · Cortex · doi:10.1016/j.cortex.2023.12.016 · Cited by 39
“This is thought to stem from the widespread use of response time difference scores, which minimise between-subject variability to facilitate between-condition comparisons (Draheim et al., 2019; Hedge et al., 2018; Rouder & Haaf, 2019).”
Verdejo‐García, A., Tiego, J., Kakoschke, N., Moskovsky, N., Voigt, K., Anderson, A. C., Koutoulogenis, J., Lubman, D. I., & Bellgrove, M. A. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01127-3 · Cited by 37
Individual differences in updating are not related to reasoning ability and working memory capacity.Preprint
Frischkorn, G. T., von Bastian, C. V., Souza, A., & Oberauer, K. (2020) · Journal of experimental psychology. General · doi:10.31234/osf.io/92uhb · Cited by 27
Individual differences in updating are not related to reasoning ability and working memory capacity.
Frischkorn, G. T., von Bastian, C. C., Souza, A. S., & Oberauer, K. (2022) · Journal of Experimental Psychology: General · doi:10.1037/xge0001141 · Cited by 25
Snijder, J.-P., Tang, R., Bugg, J. M., Conway, A. R. A., & Braver, T. S. (2023) · Behavior Research Methods · doi:10.3758/s13428-023-02111-7 · Cited by 30
“For example, Draheim et al. (2019, 2020) have argued that the use of reaction time difference scores “is the primary cause of null and conflicting results” when examining individual differences in attentional control.”
Resting-state BOLD signal variability is associated with individual differences in metacontrol
Zhang, C., Beste, C., Prochazkova, L., Wang, K., Speer, S. P. H., Smidts, A., Boksem, M. A. S., & Hommel, B. (2022) · Scientific Reports · doi:10.1038/s41598-022-21703-5 · Cited by 24
A review of attentional bias modification trainings for depression
Guo, L., Cai, X., Yang, Q., Cui, Q., Huang, L., Jing, X., & Wang, Y. (2022) · CNS Neuroscience & Therapeutics · doi:10.1111/cns.14022 · Cited by 29
Bădău, D., Bădău, A., Joksimović, M., Mănescu, C. O., Mănescu, D. C., Dinciu, C. C., Mărgărit, I. R., Tudor, V., Mujea, A. M., Neofit, A., et al. (2023) · Symmetry · doi:10.3390/sym16010028 · Cited by 23
Robison, M. K., Ralph, K. J., Gondoli, D. M., Torres, A. S., Campbell, S., Brewer, G. A., & Gibson, B. S. (2023) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/s13415-023-01096-2 · Cited by 29
“The Stroop and flanker effects had low reliability estimates, which is now a well-documented issue with these tasks (Draheim et al., 2019; Enkavi et al., 2019; Feldman and Freitas, 2016; Hedge et al., 2018; Rey-Mermet et al., 2019; Rouder and Haaf, 2019; Whitehead et al., 2019, 2020).”
BackgroundHimi, S. A., Volberg, G., Bühner, M., & Hilbert, S. (2022) · Psychological Research · doi:10.1007/s00426-022-01700-z · Cited by 32
Schubert, A., Hagemann, D., Löffler, C., Rummel, J., & Arnau, S. (2020) · Journal of Experimental Psychology General · doi:10.1037/xge0000865 · Cited by 27
Casagrande, M., Marotta, A., Martella, D., Volpari, E., Agostini, F., Favieri, F., Forte, G., Rea, M., Ferri, R., Giordano, V., et al. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01668-5 · Cited by 22
“The size of reliability indices was interpreted following Draheim et al. (2019): below .70 as problematic, between .70 and .79 as borderline, and above .80 as acceptable.”
MethodSchubert, A., Ferreira, M. B., Mata, A., & Riemenschneider, B. (2020) · Cognition · doi:10.31234/osf.io/e2tq4 · Cited by 4
“…where individuals with higher abilities may strategically adjust their decision criteria differently than individuals with lower abilities to adapt to task demands (Draheim, Hicks, & Engle, 2016; Draheim, Mashburn, Martin, & Engle, 2019; Schmiedek, Oberauer, Wilhelm, Sü ß , & Wittmann, 2007).”
BackgroundNash Unsworth, A., Miller, A. L., & Strayer, D. L. (2024) · Psychonomic Bulletin & Review · doi:10.3758/s13423-024-02516-1 · Cited by 26
“…out of the model, there isn’t sufficient shared variance to find a coherent AC factor (Rey-Mermet et al., 2018; Rey-Mermet et al., 2019; see also Draheim et al., 2019; Draheim et al., 2021; Paap et al., 2020; Rey-Mermet et al., 2020; Schubert & Rey-Mermet, 2019; von Bastian et al., 2020).”
InfluentialBialystok, E., & Shorbagi, S. H. (2021) · Journal of Cognition and Development · doi:10.1080/15248372.2021.1901711 · Cited by 22
No relation of Need for Cognition to basic executive functions.
Gärtner, A., Grass, J., Wolff, M., Goschke, T., Strobel, A., & Strobel, A. (2019) · Journal of Personality · doi:10.31234/osf.io/ts97j · Cited by 20
“More recent findings suggest the reliance on accuracy- based measures (Draheim et al., 2019) or accounting for trial- bytrial variability (Rouder & Haaf, 2019) as more promising approaches.”
BackgroundRoebers, C. M. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.923615 · Cited by 25
“Moreover, measurement problems associated with reaction times are ubiquitous in behavioral science, not limited to developmental research, but definitely serious (Draheim et al., 2019).”
Aranda‐Martín, B., Ballesteros, M. Á., & Lupiáñez, J. (2022) · British Journal of Psychology · doi:10.1111/bjop.12552 · Cited by 21
Retest Reliability of Integrated Speed–Accuracy Measures
Emesh, T. B., Garbi, D., Kaplan, A., Zelicha, H., Meir, A. Y., Tsaban, G., Rinott, E., & Meiran, N. (2021) · Assessment · doi:10.1177/1073191120985609 · Cited by 24
“Although there is some evidence of residual scores being more reliable than the traditional difference scores (for review, see Draheim et al., 2019) it is still unclear under which circumstances this reliability advantage is found (Caruso, 2004; Gollwitzer et al., 2014).”
BackgroundGrundy, J. G., Pavlenko, E., & Bialystok, E. (2020) · Journal of Neurolinguistics · doi:10.1016/j.jneuroling.2020.100933 · Cited by 24
Measuring the acuity of the approximate number system in young children
Krajcsi, A., Chesney, D. L., Cipora, K., Coolen, I. E. J. I., Gilmore, C. K., Inglis, M., Libertus, M. E., Nuerk, H., Simms, V., & Reynvoet, B. (2024) · Developmental Review · doi:10.1016/j.dr.2024.101131 · Cited by 23
O’Shea, B. A., Glenn, J. J., Millner, A. J., Teachman, B. A., & Nock, M. K. (2020) · Suicide and Life-Threatening Behavior · doi:10.1111/sltb.12652 · Cited by 22
“This recommendation is based on recent evidence that RT differences scores are particularly suited to testing between-group differences/context effects (e.g., Draheim et al., 2019; Payne, Vuletich, & Lundberg, 2017).”
MethodPrecise individual measures of inhibitory control
Lee, H. J., Smith, D. M., Hauenstein, C. E., Dworetsky, A., Kraus, B. T., Dorn, M., Nee, D. E., & Gratton, C. (2025) · Nature Human Behaviour · doi:10.1038/s41562-025-02198-2 · Cited by 24
“Study designs for examining within-participant effects of experimental manipulations have come to prominence because of their ability to measure robust group-level effects, including the congruency effect 21,31 .”
InfluentialSame, Same but Different? A Multi-Method Review of the Processes Underlying Executive Control
Toba, M. N., Malkinson, T. S., Howells, H., Mackie, M., & Spagna, A. (2023) · Neuropsychology Review · doi:10.1007/s11065-023-09577-4 · Cited by 26
“…orthogonal constructs, which are able to avoid the measurement errors recently emphasized in different tasks (see for instance Hedge et al., 2018; Draheim et al., 2019; Rey-Mermet et al., 2019; Rouder & Haaf, 2019, for the stimulus response compatibility tasks) and possible sampling errors (see…”
Antoniou, K. (2023) · Psychonomic Bulletin & Review · doi:10.3758/s13423-023-02245-x · Cited by 20
Process-oriented intelligence research: A review from the cognitive perspective
Frischkorn, G. T., Wilhelm, O., & Oberauer, K. (2022) · Intelligence · doi:10.1016/j.intell.2022.101681 · Cited by 25
Spontaneous mind-wandering tendencies linked to cognitive flexibility in young adults.
Wong, Y.-S., Willoughby, A. R., & Machado, L. (2021) · Consciousness and Cognition · doi:10.31234/osf.io/h5zns · Cited by 17
Raising the bar: improving methodological rigour in cognitive alcohol research
Pennington, C. R., Jones, A., Bartlett, J. E., Copeland, A., & Shaw, D. J. (2021) · Addiction · doi:10.1111/add.15563 · Cited by 24
“Thirdly, researchers should explore different ways of analysis which might overcome the limitations inherent in the use of raw RTS and difference scores (see [49] for suggested alternatives).”
BackgroundOpen-source open-access reaction time test (OORTT): an easy tool to assess reaction times
Rigoli, M., Facchin, A., Cardile, D., Beschin, N., & Luzzatti, C. (2020) · Neurological Sciences · doi:10.1007/s10072-020-04839-y · Cited by 17
Rouder, J. N., & Mehrvarz, M. (2024) · Current Directions in Psychological Science · doi:10.1177/09637214231220923 · Cited by 20
“These include avoiding difference scores (Draheim et al., 2019), using so-called gamified tasks (Deveau et al., 2015; Kucina et al., 2022), and diffusion modeling (Haines et al., 2020; Lerche et al., 2020; von Krause et al., 2020; Weigard et al., 2021).”
MethodGarre-Frutos, F., Vadillo, M. A., González, F., & Lupiáñez, J. (2024) · Behavior Research Methods · doi:10.3758/s13428-023-02329-5 · Cited by 20
“These two studies suggest that measures based on RTs may not be suitable for studying individual differences (see Draheim et al., 2019 for a discussion).”
BackgroundA new way of classifying developmental prosopagnosia: Balanced Integration Score
Lowes, J., Hancock, P., & Bobak, A. K. (2024) · Cortex · doi:10.1016/j.cortex.2023.12.011 · Cited by 20
Linking computational models of two core tasks of cognitive control.
Robinson, M. M., & Mark Steyvers, B. (2022) · Psychological Review · doi:10.1037/rev0000395 · Cited by 18
“…of difference scores across individuals because much evidence suggests that peoplevary in howthey strategically approach the task (see Draheim et al., 2019; Hedge et al., 2021; Hedge, Powell, T h i s do c u m e n t i s c opy r i gh t e d by t h e A m e r i ca n P s y c ho l og i ca l…”
Rey-Mermet, A., Singh, K. A., Gignac, G. E., Brydges, C. R., & Ecker, U. K. H. (2020) · PLoS ONE · doi:10.1371/journal.pone.0243053 · Cited by 24
“As pointed out recently by Draheim and colleagues [69, 70], one possible reason for the difficulty in establishing a coherent factor of inhibition might be the use of RTs—and in particular, RT difference scores—because these scores have been associated with low reliability and speed-accuracy…”
Chan, C. G. H., Yow, W. Q., & Oei, A. C. (2020) · The Journals of Gerontology Series B · doi:10.1093/geronb/gbaa058 · Cited by 23
Lee, S. H., & Pitt, M. A. (2024) · Behavior Research Methods · doi:10.3758/s13428-023-02327-7 · Cited by 17
“Further, that this consistency is best for the incongruent RT data across distances suggests that they are a viable alternative for measuring performance across distances (Draheim et al., 2019).”
BackgroundWeigard, A. S., McCurry, K. L., Shapiro, Z. R., Martz, M. E., Angstadt, M., Heitzeg, M. M., Dinov, I. D., & Sripada, C. (2023) · Translational Psychiatry · doi:10.1038/s41398-023-02502-6 · Cited by 19
“Predicting the difference between two highly correlated measures of the same trait is inherently challenging because much of the systematic variance in the individual measures is reflected in their correlation, and their difference will, therefore, necessarily contain a large portion of error variance relative to syst…”
BackgroundLi, Y.-F., Gao, T., Luo, L.-P., & He, S. (2024) · Frontiers in Psychology · doi:10.3389/fpsyg.2024.1457449 · Cited by 12
Working memory capacity preferentially enhances implementation of proactive control.
Lin, Y.-L., Brough, R., Tay, A., Jackson, J., & Braver, T. (2022) · Journal of Experimental Psychology. Learning, Memory and Cognition · doi:10.31234/osf.io/wvpbn · Cited by 14
“Second, the equally common practice of using subtraction-based difference scores can increase measurement error, constrain between-subjects variance, and attenuate reliability (Caruso, 2004; Cronbach & Furby, 1970; Draheim et al., 2019; Hedge et al., 2018).”
BackgroundMason, L., Zimiga, B. M., Anders-Jefferson, R. T., & Paap, K. R. (2020) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-020-04741-8 · Cited by 21
Executive functioning profiles and mathematical and reading achievement in Grades 2, 6, and 10
Carriedo, N., Rodríguez‐Villagra, O. A., Pérez, L., & Iglesias-Sarmiento, V. (2024) · Journal of School Psychology · doi:10.1016/j.jsp.2024.101353 · Cited by 18
The Relevance of Attention Control, Not Working Memory, in Human Factors
Pak, R., McLaughlin, A. C., & Engle, R. W. (2023) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1177/00187208231159727 · Cited by 15
“…measures (i.e., the limitations of using tasks designed for use in an experimental approach designed to minimize individual differences, the use of reaction-time difference scores, low reliability, and speed-accuracy tradeoffs) see Draheim et al. (2019, 2021) and Redick and Engle (2006).”
MethodSelf-citationWilloughby, M. T., Camerota, M., King, K. M., Nduku, T., & Piper, B. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.861441 · Cited by 12
“Although some have criticized a reliance on changes in RT within a task to index IC (Hedge et al., 2018; Draheim et al., 2019), this approach to scoring tasks is common.”
Older Adults Show Intraindividual Variation in the Use of Predictive Processing
Jongman, S. R., Copeland, A., Xu, Y., Payne, B., & Federmeier, K. D. (2022) · Experimental Aging Research · doi:10.1080/0361073x.2022.2137358 · Cited by 13
Machulska, A., Kleinke, K., & Klucken, T. (2022) · Behavior Research Methods · doi:10.3758/s13428-022-01804-9 · Cited by 13
Attentional control may be modifiable with Mindfulness-Based Cognitive Therapy to Prevent Suicide
Chesin, M. S., Keilp, J. G., Kline, A., Stanley, B., Myers, C. E., Latorre, M., Hill, L. M. S., Miller, R. B., King, A. R., Boschulte, D. R., et al. (2021) · Behaviour Research and Therapy · doi:10.1016/j.brat.2021.103988 · Cited by 18
Park, Y., & Lee, J.-H. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.614213 · Cited by 15
Xu, M., Qian, L., Wang, S., Cai, H., Sun, Y., Thakor, N. V., Qi, X., & Sun, Y. (2023) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2023.1193292 · Cited by 11
“Reaction time is considered a good indicator of the speed and efficiency of mental processes (Draheim et al., 2019), whereas local efficiency measures the functional segregation, the capacity of locally processed information (Latora and Marchiori, 2001).”
BackgroundChaku, N., Yan, R., Kelly, D. P., Zhang, Z., Lopez‐Duran, N. L., Weigard, A. S., & Beltz, A. M. (2023) · Research on Child and Adolescent Psychopathology · doi:10.1007/s10802-023-01071-y · Cited by 17
“Since 1975, subtraction methods (e.g., comparing reaction time on incongruent and congruent trials) have gained popularity, but they have significant limitations for accu - rately detecting individual differences (Draheim et al., 2019; Weigard et al., 2021), and thus, intraindividual variation.”
BackgroundMartin, J., Mashburn, C. A., & Engle, R. (2020) · Journal of Applied Research in Memory and Cognition · doi:10.1016/j.jarmac.2020.04.002 · Cited by 12
“Instead, Draheim et al. (2019) advocated that those of the Armed Service Vocational Aptitude Battery with Measures of (2020), https://doi.org/10.1016/j.jarmac.2020.04.002 he experimental and individual differences approach.”
BackgroundInfluentialSelf-citationAttentional control data collection: A resource for efficient data reuse
Haaf, J. M., Hoffstadt, M., & Lesche, S. (2025) · Behavior Research Methods · doi:10.3758/s13428-025-02717-z · Cited by 7
Lacko, D., Prošek, ., Čeněk, J., Helísková, M., Ugwitz, P., Svoboda, V., Počaji, P., Vais, M., Halířová, H., Juřík, V., et al. (2023) · PLoS ONE · doi:10.1371/journal.pone.0287057 · Cited by 18
Gauthier, I., & Fiestan, G. (2022) · Food Quality and Preference · doi:10.1016/j.foodqual.2022.104702 · Cited by 11
Hertz‐Palmor, N., Rozenblit, D., Lavi, S., Zeltser, J., Kviatek, Y., & Lazarov, A. (2023) · Psychological Medicine · doi:10.1017/s0033291723002519 · Cited by 9
“Second, RT-based tasks exhibit poor psychometric properties, including low internal consistency and test – retest reliability (Brown et al., 2014; Draheim, Mashburn, Martin, & Engle, 2019; Rodebaugh et al., 2016; Schmukle, 2005; Staugaard, 2009; Waechter & Stolz, 2015), which are vital for trusting…”
de Bruijn, A. G. M., Meijer, A., Königs, M., Oosterlaan, J., Smith, J., & Hartman, E. (2023) · Psychology of sport and exercise · doi:10.1016/j.psychsport.2023.102390 · Cited by 14
Martinez, D., & Singleton, J. L. (2019) · Acta Psychologica · doi:10.1016/j.actpsy.2019.102892 · Cited by 12
Clarifying the reliability paradox: poor measurement reliability attenuates group differences
Karvelis, P., & Diaconescu, A. O. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1592658 · Cited by 13
Grobe, S. E., Könen, T., David, C., Grüneisen, L., Dörrenbächer‐Ulrich, L., Perels, F., & Karbach, J. (2024) · Journal of Experimental Child Psychology · doi:10.1016/j.jecp.2024.106014 · Cited by 13
Mashburn, C. A., Burgoyne, A. P., Tsukahara, J. S., Pak, R., Coyne, J. T., Sibley, C. M., Foroughi, C. K., & Engle, R. W. (2024) · Intelligence · doi:10.1016/j.intell.2024.101835 · Cited by 12
“Thus, the attention control factor used here was able to capture more of the reliable variance among the observed measures than is often observed in studies of attentional abilities (see, e. g., Draheim, Mashburn, Martin, & Engle, 2019).”
Self-citationFreichel, R., Mrkonja, L., de Jong, P. J., Cousijn, J., Franken, I. H. A., Ruiter, T. A., Pelley, M. E. L., Albertella, L., Watson, P., Veer, I. M., et al. (2023) · Journal of Experimental Psychopathology · doi:10.1177/20438087231204166 · Cited by 9
“Prior research suggests that low test-retest reliability may be related to factors such as low between-subject variability (Hedge et al., 2018) or the use of reaction time difference scores as measures of attentional control (Draheim et al., 2019).”
BackgroundGonthier, C. (2023) · Journal of Intelligence · doi:10.3390/jintelligence11060120 · Cited by 14
DeGutis, J., Li, X., Yosef, B., & Mishra, M. V. (2022) · Cognitive Neuropsychology · doi:10.1080/02643294.2022.2114824 · Cited by 9
“%) may suffer from differential speed-accuracy tradeoffs across participants, limiting the usefulness of RT measures (see Draheim et al., 2019).”
BackgroundWaschbusch, D. A., Babinski, D. E., Fosco, W. D., Haas, S. M., Waxmonsky, J. G., Garon, N., Nichols, S. L., King, S., Santor, D. A., & Andrade, B. F. (2022) · Developmental Neuropsychology · doi:10.1080/87565641.2022.2032713 · Cited by 11
Rey-Mermet, A., Singmann, H., & Oberauer, K. (2025) · Psychonomic Bulletin & Review · doi:10.3758/s13423-025-02696-4 · Cited by 5
“A second reason for the difficulty of establishing attentional control at the latent-variable level is that most atten-tional-control measures rely exclusively on response times, so that individual differences in speed–accuracy trade-offs are disregarded (Draheim et al., 2019).”
InfluentialO'Shea, B. A., & Wiers, R. W. H. J. (2020) · Social Cognition · doi:10.1521/soco.2020.38.supp.s187 · Cited by 15
“Due to restrictive variance and ceiling effects, accu-racy-based calculations are often less reliable (e.g., Draheim, Mashburn, Martin, & Engle, 2019; cf. Gladwin et al., 2015).”
Background
Metadata
- Authors
- Draheim, C., Mashburn, C. A., Martin, J. D., & Engle, R. W.
- Affiliations
- Georgia Institute of Technology (US)
- Venue
- Psychological Bulletin
- Published
- 2019-05
- Volume
- 145
- Issue
- 5
- Pages
- 508-535
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2923932768
- Cited by
- 322
- References
- 0
- Citations by year
- 2026: 46 · 2025: 42 · 2024: 59 · 2023: 55 · 2022: 48 · 2021: 35 · 2020: 31 · 2019: 5
- FWCI
- 17.30
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Psychological Bulletin (publishedVersion, other-oa)
- PubMed (publishedVersion)
- Funders
- Office of Naval Research (N00014-12-1-0406; N00014-12-1-1011)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Cognitive Abilities and Testing
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- reaction time, individual differences, developmental research, speed-accuracy trade-off, reliability statistics, difference scores, attentional control, task switching, alternative measures, cognitive development
- MeSH terms
- Humans, Reaction Time, Reproducibility of Results, Behavioral Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
