Reaction time in differential and developmental research: A review and commentary on the problems and alternatives.

Draheim, Christopher; Mashburn, Cody A.; Martin, Jessie D.; Engle, Randall W.

doi:10.1037/bul0000192

Cited by 322Neural and Behavioral Psychology StudiesCognitive Abilities and TestingPsychometric Methodologies and Testing

Reaction time in differential and developmental research: A review and commentary on the problems and alternatives.

Open access

Draheim, Christopher, Mashburn, Cody A., Martin, Jessie D., Engle, Randall W.·2019·Psychological Bulletin

DOI 10.1037/bul0000192

Open this paper in CoChat — free account

Abstract

Reaction time is believed to be a good indicator of the speed and efficiency of mental processes and is a ubiquitous variable in the behavioral sciences. Despite this popularity, there are numerous issues associated with using reaction time (RT), specifically in differential and developmental research. Here, we identify and focus on two main problems-unreliability and sensitivity to speed-accuracy interactions. The use of difference scores is a primary factor that leads to many RT measures having demonstrably low reliability, and RT measures in general often do not properly account for speed-accuracy interactions. Both factors jeopardize the validity and interpretability of results based on RT. Here, we evaluate conceptually and empirically how these issues affect individual differences research. Although the empirical evidence we provide are primarily within the domains of attention control and task switching, we highlight examples from various other areas of psychological inquiry. We also discuss many of the statistical and methodological alternatives available to researchers conducting correlational studies. Ultimately, we encourage researchers comparing individuals of differing cognitive and developmental levels to strongly consider using these alternatives in lieu of RT, specifically RT difference scores. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Integrity signals

Registry

Crossref · 10.1037/bul0000192

Cites

0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Difference scores are inherently poor in reliability.
“One complication is that the testing effect is a difference score between a condition that receives retrieval practice and one that does not, and difference scores are inherently poor in reliability (Draheim et al. 2019).”
From Practicing Retrieval Facilitates Learning (2021) · cites this paper as Draheim et al. 2019
Treating RT and accuracy as distinct outcomes does not appropriately account for the speed-accuracy trade-off that exists between the two parameters.
“However, treating RT and accuracy as distinct outcomes does not appropriately account for the speed-accuracy trade-off that exists between the two parameters24,25.”
From Delayed goal-directed processing underlies inhibitory control challenges in adult ADHD (2026) · cites this paper as [24]

Studies of interference in serial verbal reactions
Stroop, J. R. (1935) · Journal of Experimental Psychology · doi:10.1037/h0054651 · Cited by 18,622 · see citing works
The processing-speed theory of adult age differences in cognition
Salthouse, T. A. (1996) · Psychological Review · doi:10.1037/0033-295x.103.3.403 · Cited by 5,568 · see citing works
Testing the efficiency and independence of attentional networks
Fan, J., McCandliss, B. D., Sommer, T., Raz, A., & Posner, M. I. (2002) · Journal of Cognitive Neuroscience · doi:10.1162/089892902317361886 · Cited by 3,717 · see citing works
The Sources of Four Commonly Reported Cutoff Criteria: What Did They Really Say?
Lance, C. E., Butts, M. M., & Michels, L. C. (2006) · Organizational Research Methods · doi:10.1177/1094428105284919 · Cited by 2,053 · see citing works
The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences
Hedge, C., Powell, G., & Sumner, P. (2018) · Behavior Research Methods · doi:10.3758/s13428-017-0935-1 · Cited by 1,457 · see citing works
Long-term reduction in implicit race bias: A prejudice habit-breaking intervention
Devine, P. G., Forscher, P. S., Austin, A. J., & Cox, W. T. L. (2012) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2012.06.003 · Cited by 1,259 · see citing works
Executive Attention, Working Memory Capacity, and a Two-Factor Theory of Cognitive Control
Engle, R. W., & Kane, M. J. (2003) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(03)44005-x · Cited by 1,104 · see citing works
Self-citation
Working Memory and Executive Attention: A Revisit
Engle, R. W. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691617720478 · Cited by 339 · see citing works
Self-citation

Flagged first, then by citations

Sorted by citations

Back to Basics: The Importance of Conceptual Clarification in Psychological Science
Bringmann, L. F., Elmer, T., & Eronen, M. I. (2022) · Current Directions in Psychological Science · doi:10.1177/09637214221096485 · Cited by 221 · see citing works
How does bilingualism modify cognitive function? Attention to the mechanism
Bialystok, E., & Craik, F. I. M. (2022) · Psychonomic Bulletin & Review · doi:10.3758/s13423-022-02057-5 · Cited by 238 · see citing works
Best research practices for using the Implicit Association Test
Greenwald, A. G., Brendl, M., Cai, H., Cvencek, D., Dovidio, J. F., Friese, M., Hahn, A., Hehman, E., Hofmann, W., Hughes, S., et al. (2022) · Behavior Research Methods · doi:10.3758/s13428-021-01624-3 · Cited by 209 · see citing works
A toolbox approach to improving the measurement of attention control.Preprint
Draheim, C., Tsukahara, J., Martin, J., Mashburn, C. A., & Engle, R. (2019) · Journal of experimental psychology. General · doi:10.31234/osf.io/q985d · Cited by 192 · see citing works
“In addition, data and ideas from the present study were disseminated in various conference presentations (Draheim, Martin, Tsukahara, Mashburn, & Engle, 2018; Draheim, Mashburn, & Engle, 2018; Draheim et al., 2019; Engle, 2017).”
MethodSelf-citation
Attention Control: A Cornerstone of Higher-Order Cognition
Burgoyne, A. P., & Engle, R. W. (2020) · Current Directions in Psychological Science · doi:10.1177/0963721420969371 · Cited by 186 · see citing works
Self-citation
Practicing Retrieval Facilitates Learning
McDermott, K. B. (2021) · Annual review of psychology · doi:10.1146/annurev-psych-010419-051019 · Cited by 113 · see citing works
“One complication is that the testing effect is a difference score between a condition that receives retrieval practice and one that does not, and difference scores are inherently poor in reliability (Draheim et al. 2019).”
Background
Nature and measurement of attention control.
Burgoyne, A. P., Tsukahara, J. S., Mashburn, C. A., Pak, R., & Engle, R. W. (2023) · Journal of Experimental Psychology General · doi:10.1037/xge0001408 · Cited by 126 · see citing works
“…the use of difference scores in individual differences research for decades because of their unreliability at the level of the participant, and subsequently poor validity (Ackerman & Hambrick, 2020; Cronbach & Furby, 1970; Draheim et al., 2016, 2019; Friedman & Miyake, 2004; Hedge et al., 2018).”
Self-citation
Duration and extent of bilingual experience modulate neurocognitive outcomes
Deluca, V., Rothman, J., Bialystok, E., & Pliatsikas, C. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2019.116222 · Cited by 100 · see citing works
Individual differences in computational psychiatry: A review of current challenges
Karvelis, P., Paulus, M. P., & Diaconescu, A. O. (2023) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2023.105137 · Cited by 113 · see citing works
“…jointly accounting for choice and reaction time data), it can extract valuable information from trail-level variability, especially if it relates to learning (e.g., with trial-by-trial modelling approaches), and in many cases it circumvents the need to use difference scores (Draheim et al., 2019).”
Improving the Reliability of Cognitive Task Measures: A Narrative ReviewPreprint
Zorowitz, S., & Niv, Y. (2022) · Biological Psychiatry: Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2023.02.004 · Cited by 82 · see citing works
Why Most Studies of Individual Differences With Inhibition Tasks Are Bound To FailPreprint
Rouder, J., Kumar, A., & Haaf, J. M. (2019) · doi:10.31234/osf.io/3cjr5 · Cited by 77 · see citing works
“In practice, because there is no subtraction, measures are often highly reliable and lead to higher correlations [Draheim et al. (2019);Hedge:etal:2018].”
BackgroundInfluential
On the origins and evolution of the Attention Network Tests
Almeida, R. D. S., Faria, A. J. P., & Klein, R. M. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2021.02.028 · Cited by 81 · see citing works
The Effects of Proprioceptive Training on Balance, Strength, Agility and Dribbling in Adolescent Male Soccer Players
Gidu, D., Badau, D., Stoica, M., Aron, A., Focan, G., Monea, D., Stoica, A., & Calotă, N. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19042028 · Cited by 61 · see citing works
“When speaking about the benefits of PT for neuromuscular control, we refer to improvements in muscle reflex activity [6–11], reaction time [12]), rate of force development and electromiography activity [13].”
Background
Individual Differences in Inhibitory Control: A latent Variable Analysis
Gärtner, A., & Strobel, A. (2021) · Journal of Cognition · doi:10.5334/joc.150 · Cited by 75 · see citing works
“For example, Draheim et al. (2019) extensively discussed the problems and alternatives of using standard reaction time difference scores in differential and developmental research in their recent review.”
Background
Why many studies of individual differences with inhibition tasks may not localize correlations
Rouder, J., Kumar, A., & Haaf, J. M. (2023) · Psychonomic Bulletin & Review · doi:10.3758/s13423-023-02293-3 · Cited by 54 · see citing works
“Draheim et al. (2019) argue that commonly used response time difference scores are susceptible to low reliability and other artifacts such as speed-accuracy tradeoffs.”
Background
The common factor of executive functions measures nothing but speed of information uptake
Löffler, C., Frischkorn, G. T., Hagemann, D., Sadus, K., & Schubert, A. (2024) · Psychological Research · doi:10.1007/s00426-023-01924-7 · Cited by 60 · see citing works
“Second, some researchers caution against using difference scores because they tend to show low reliabilities (Ackerman & Hambrick, 2020; Draheim et al., 2019, 2023; Hedge et al., 2018; Miller & Ulrich, 2013; von Bastian et al., 2020; Weigard et al., 2021).”
Task-general efficiency of evidence accumulation as a computationally defined neurocognitive trait: implications for clinical neuroscience
Weigard, A., & Sripada, C. (2021) · Biological Psychiatry Global Open Science · doi:10.1016/j.bpsgos.2021.02.001 · Cited by 43 · see citing works
Task-general efficiency of evidence accumulation as a computationally-defined neurocognitive trait: Implications for clinical neurosciencePreprint
Weigard, A., & Sripada, C. (2020) · Biological Psychiatry Global Open Science · doi:10.31234/osf.io/q9hge · Cited by 45 · see citing works
“Such measures consistently demonstrate poor test-retest reliability (24–27); that is, the rank order of subjects fails to be preserved across testing occasions, limiting the usefulness of these metrics for individual differences research (28,29).”
Background
The stability and reliability of attentional bias measures in the dot-probe task: Evidence from both traditional mean bias scores and trial-level bias scores
Carlson, J. M., & Fang, L. (2020) · Motivation and Emotion · doi:10.1007/s11031-020-09834-6 · Cited by 45 · see citing works
“Additionally, it should be noted that poor reliability for individual differences does not necessarily mean that the experimental effects associated with differences between conditions are unreliable (Draheim et al. 2019; Goodhew and Edwards 2019; Hedge et al. 2018).”
Background
Training Using a Commercial Immersive Virtual Reality System on Hand–Eye Coordination and Reaction Time in Young Musicians: A Pilot Study
Rutkowski, S., Adamczyk, M., Pastuła, A., Gos, E., Luque-Moreno, C., & Rutkowska, A. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18031297 · Cited by 51 · see citing works
“problems arise with the implementation of response time, especially in differential and developmental research [6].”
Background
Paying attention to climate change: Positive images of climate change solutions capture attention
Carlson, J. M., Kaull, H., Steinhauer, M., Zigarac, A., & Cammarata, J. (2020) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2020.101477 · Cited by 50 · see citing works
Are individual differences in attention control related to working memory capacity? A latent variable mega-analysis.
Nash Unsworth, A., Miller, A. L., & Robison, M. K. (2020) · Journal of Experimental Psychology General · doi:10.1037/xge0001000 · Cited by 64 · see citing works
Attention control: The missing link between sensory discrimination and intelligence
Tsukahara, J. S., Harrison, T. L., Draheim, C., Martin, J. D., & Engle, R. W. (2020) · Attention Perception & Psychophysics · doi:10.3758/s13414-020-02044-9 · Cited by 52 · see citing works
Self-citation
Measuring attention and vigilance in the laboratory vs. online: The split-half reliability of the ANTI-Vea
Luna, F. G., Roca, J., Martín‐Arévalo, E., & Lupiáñez, J. (2020) · Behavior Research Methods · doi:10.3758/s13428-020-01483-4 · Cited by 39 · see citing works
Attentional bias in alcohol drinkers: A systematic review of its link with consumption variables
Bollen, Z., Field, M., Billaux, P., & Maurage, P. (2022) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2022.104703 · Cited by 37 · see citing works
Attention control and process overlap theory: Searching for cognitive processes underpinning the positive manifoldPreprint
Burgoyne, A. P., Mashburn, C. A., Tsukahara, J., & Engle, R. (2022) · Intelligence · doi:10.1016/j.intell.2022.101629 · Cited by 37 · see citing works
Self-citation
THE CONTRIBUTIONS OF IMPLICIT-STATISTICAL LEARNING APTITUDE TO IMPLICIT SECOND-LANGUAGE KNOWLEDGE
Godfroid, A., & Kim, K. (2021) · Studies in Second Language Acquisition · doi:10.1017/s0272263121000085 · Cited by 35 · see citing works
“Another issue concerns the low reliability of aptitude and knowledge measures obtained from reaction time data (Draheim et al., 2019; Rouder & Haaf, 2019), which may obscure any aptitude–knowledge relationships.”
BackgroundMethod
Individual differences in artificial and natural language statistical learning
Isbilen, E. S., McCauley, S. M., & Christiansen, M. H. (2022) · Cognition · doi:10.1016/j.cognition.2022.105123 · Cited by 36 · see citing works
The role of maintenance and disengagement in predicting reading comprehension and vocabulary learning.
Martin, J. D., Shipstead, Z., Harrison, T. L., Redick, T. S., Bunting, M. F., & Engle, R. W. (2019) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/xlm0000705 · Cited by 28 · see citing works
Self-citation
Replication and extension of the toolbox approach to measuring attention control
Draheim, C., Tsukahara, J. S., & Engle, R. W. (2023) · Behavior Research Methods · doi:10.3758/s13428-023-02140-2 · Cited by 37 · see citing works
Self-citation
Speed–Accuracy Trade-Off? Not So Fast: Marginal Changes in Speed Have Inconsistent Relationships With Accuracy in Real-World Settings
Domingue, B. W., Kanopka, K., Stenhaug, B., Sulik, M. J., Beverly, T., Brinkhuis, M. J. S., Circi, R., Faul, J. D., Liao, D., McCandliss, B. D., et al. (2022) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986221099906 · Cited by 32 · see citing works
“Our results also suggest, dovetailing with others, that accuracy is integral for understanding individual differences (i.e., RT alone may be insufficient; Draheim et al., 2019; Draheim et al., 2020).”
Result
No relation of need for cognition to basic executive functions.
Gärtner, A., Grass, J., Wolff, M., Goschke, T., Strobel, A., & Strobel, A. (2021) · Journal of Personality · doi:10.1111/jopy.12639 · Cited by 26 · see citing works
Same same but different: Subtle but consequential differences between two measures to linearly integrate speed and accuracy (LISAS vs. BIS)
Liesefeld, H. R., & Janczyk, M. (2022) · Behavior Research Methods · doi:10.3758/s13428-022-01843-2 · Cited by 31 · see citing works
“By contrast, BIS is by no means restricted to within-participants designs, but we and others consider many use cases even going beyond experimental psychology (e.g., Bakun Emesh et al., 2021; Draheim et al., 2019; Liesefeld & Janczyk, 2019; Liu et al., 2019; Mueller et al., 2019; Palmqvist et al., 2020; Stojan et al.,…”
Method
Calibration of cognitive tests to address the reliability paradox for decision-conflict tasks
Kucina, T., Wells, L., Lewis, I., de Salas, K., Kohl, A. T., Palmer, M. A., Sauer, J., Matzke, D., Aidman, E., & Heathcote, A. (2023) · Nature Communications · doi:10.1038/s41467-023-37777-2 · Cited by 28 · see citing works
“Such considerations have prompted calls to abandon measures based on con ﬂ ict-task RT difference scores in differential research 27 , and instead center on either the development of alternative measurement approaches 28 , or shift focus to differences in processing speed and strategy 29 .”
Using precision approaches to improve brain-behavior prediction
Lee, H. J., Dworetsky, A., Labora, N., & Gratton, C. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.09.007 · Cited by 20 · see citing works
Unpacking the overlap between Autism and ADHD in adults: A multi-method approach
Waldren, L. H., Leung, F. Y. N., Hargitai, L. D., Burgoyne, A. P., Liceralde, V. R. T., Livingston, L. A., & Shah, P. (2024) · Cortex · doi:10.1016/j.cortex.2023.12.016 · Cited by 34 · see citing works
“This is thought to stem from the widespread use of response time difference scores, which minimise between-subject variability to facilitate between-condition comparisons (Draheim et al., 2019; Hedge et al., 2018; Rouder & Haaf, 2019).”
A unified online test battery for cognitive impulsivity reveals relationships with real-world impulsive behaviours
Verdejo‐García, A., Tiego, J., Kakoschke, N., Moskovsky, N., Voigt, K., Anderson, A. C., Koutoulogenis, J., Lubman, D. I., & Bellgrove, M. A. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01127-3 · Cited by 34 · see citing works
Individual differences in updating are not related to reasoning ability and working memory capacity.Preprint
Frischkorn, G. T., von Bastian, C. V., Souza, A., & Oberauer, K. (2020) · Journal of experimental psychology. General · doi:10.31234/osf.io/92uhb · Cited by 27 · see citing works
Individual differences in updating are not related to reasoning ability and working memory capacity.
Frischkorn, G. T., von Bastian, C. C., Souza, A. S., & Oberauer, K. (2022) · Journal of Experimental Psychology: General · doi:10.1037/xge0001141 · Cited by 25 · see citing works
On the psychometric evaluation of cognitive control tasks: An Investigation with the Dual Mechanisms of Cognitive Control (DMCC) battery
Snijder, J.-P., Tang, R., Bugg, J. M., Conway, A. R. A., & Braver, T. S. (2023) · Behavior Research Methods · doi:10.3758/s13428-023-02111-7 · Cited by 28 · see citing works
“For example, Draheim et al. (2019, 2020) have argued that the use of reaction time difference scores “is the primary cause of null and conflicting results” when examining individual differences in attentional control.”
Resting-state BOLD signal variability is associated with individual differences in metacontrol
Zhang, C., Beste, C., Prochazkova, L., Wang, K., Speer, S. P. H., Smidts, A., Boksem, M. A. S., & Hommel, B. (2022) · Scientific Reports · doi:10.1038/s41598-022-21703-5 · Cited by 24 · see citing works
A review of attentional bias modification trainings for depression
Guo, L., Cai, X., Yang, Q., Cui, Q., Huang, L., Jing, X., & Wang, Y. (2022) · CNS Neuroscience & Therapeutics · doi:10.1111/cns.14022 · Cited by 29 · see citing works
Identifying the Level of Symmetrization of Reaction Time According to Manual Lateralization between Team Sports Athletes, Individual Sports Athletes, and Non-Athletes
Bădău, D., Bădău, A., Joksimović, M., Mănescu, C. O., Mănescu, D. C., Dinciu, C. C., Mărgărit, I. R., Tudor, V., Mujea, A. M., Neofit, A., et al. (2023) · Symmetry · doi:10.3390/sym16010028 · Cited by 23 · see citing works
Testing locus coeruleus-norepinephrine accounts of working memory, attention control, and fluid intelligence
Robison, M. K., Ralph, K. J., Gondoli, D. M., Torres, A. S., Campbell, S., Brewer, G. A., & Gibson, B. S. (2023) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/s13415-023-01096-2 · Cited by 28 · see citing works
“The Stroop and flanker effects had low reliability estimates, which is now a well-documented issue with these tasks (Draheim et al., 2019; Enkavi et al., 2019; Feldman and Freitas, 2016; Hedge et al., 2018; Rey-Mermet et al., 2019; Rouder and Haaf, 2019; Whitehead et al., 2019, 2020).”
Background
Individual differences in everyday multitasking behavior and its relation to cognition and personality
Himi, S. A., Volberg, G., Bühner, M., & Hilbert, S. (2022) · Psychological Research · doi:10.1007/s00426-022-01700-z · Cited by 31 · see citing works
A chronometric model of the relationship between frontal midline theta functional connectivity and human intelligence.
Schubert, A., Hagemann, D., Löffler, C., Rummel, J., & Arnau, S. (2020) · Journal of Experimental Psychology General · doi:10.1037/xge0000865 · Cited by 27 · see citing works
Assessing the three attentional networks in children from three to six years: A child-friendly version of the Attentional Network Test for Interaction
Casagrande, M., Marotta, A., Martella, D., Volpari, E., Agostini, F., Favieri, F., Forte, G., Rea, M., Ferri, R., Giordano, V., et al. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01668-5 · Cited by 22 · see citing works
“The size of reliability indices was interpreted following Draheim et al. (2019): below .70 as problematic, between .70 and .79 as borderline, and above .80 as acceptable.”
Method
A diffusion model analysis of belief bias: Different cognitive mechanisms explain how cognitive abilities and thinking styles contribute to conflict resolution in reasoningPreprint
Schubert, A., Ferreira, M. B., Mata, A., & Riemenschneider, B. (2020) · Cognition · doi:10.31234/osf.io/e2tq4 · Cited by 4 · see citing works
“…where individuals with higher abilities may strategically adjust their decision criteria differently than individuals with lower abilities to adapt to task demands (Draheim, Hicks, & Engle, 2016; Draheim, Mashburn, Martin, & Engle, 2019; Schmiedek, Oberauer, Wilhelm, Sü ß , & Wittmann, 2007).”
Background
Individual differences in attention control: A meta-analysis and re-analysis of latent variable studies
Nash Unsworth, A., Miller, A. L., & Strayer, D. L. (2024) · Psychonomic Bulletin & Review · doi:10.3758/s13423-024-02516-1 · Cited by 22 · see citing works
“…out of the model, there isn’t sufficient shared variance to find a coherent AC factor (Rey-Mermet et al., 2018; Rey-Mermet et al., 2019; see also Draheim et al., 2019; Draheim et al., 2021; Paap et al., 2020; Rey-Mermet et al., 2020; Schubert & Rey-Mermet, 2019; von Bastian et al., 2020).”
Influential
Subtle Increments in Socioeconomic Status and Bilingualism Jointly Affect Children’s Verbal and Nonverbal Performance
Bialystok, E., & Shorbagi, S. H. (2021) · Journal of Cognition and Development · doi:10.1080/15248372.2021.1901711 · Cited by 22 · see citing works
No relation of Need for Cognition to basic executive functions.
Gärtner, A., Grass, J., Wolff, M., Goschke, T., Strobel, A., & Strobel, A. (2019) · Journal of Personality · doi:10.31234/osf.io/ts97j · Cited by 20 · see citing works
“More recent findings suggest the reliance on accuracy- based measures (Draheim et al., 2019) or accounting for trial- bytrial variability (Rouder & Haaf, 2019) as more promising approaches.”
Background
Six- to eight-year-olds’ performance in the Heart and Flower task: Emerging proactive cognitive control
Roebers, C. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.923615 · Cited by 19 · see citing works
“Moreover, measurement problems associated with reaction times are ubiquitous in behavioral science, not limited to developmental research, but definitely serious (Draheim et al., 2019).”
What gaze adds to arrows: Changes in attentional response to gaze versus arrows in childhood and adolescence
Aranda‐Martín, B., Ballesteros, M. Á., & Lupiáñez, J. (2022) · British Journal of Psychology · doi:10.1111/bjop.12552 · Cited by 21 · see citing works
Retest Reliability of Integrated Speed–Accuracy Measures
Emesh, T. B., Garbi, D., Kaplan, A., Zelicha, H., Meir, A. Y., Tsaban, G., Rinott, E., & Meiran, N. (2021) · Assessment (Odessa, Fla.) · doi:10.1177/1073191120985609 · Cited by 19 · see citing works
“Although there is some evidence of residual scores being more reliable than the traditional difference scores (for review, see Draheim et al., 2019) it is still unclear under which circumstances this reliability advantage is found (Caruso, 2004; Gollwitzer et al., 2014).”
Background
Bilingualism modifies disengagement of attention networks across the scalp: A multivariate ERP investigation of the IOR paradigm
Grundy, J. G., Pavlenko, E., & Bialystok, E. (2020) · Journal of Neurolinguistics · doi:10.1016/j.jneuroling.2020.100933 · Cited by 24 · see citing works
Measuring the acuity of the approximate number system in young children
Krajcsi, A., Chesney, D., Cipora, K., Coolen, I., Gilmore, C., Inglis, M., Libertus, M. E., Nuerk, H.-C., Simms, V., & Reynvoet, B. (2024) · Developmental Review · doi:10.1016/j.dr.2024.101131 · Cited by 18 · see citing works
Decomposing implicit associations about life and death improves our understanding of suicidal behavior
O’Shea, B. A., Glenn, J. J., Millner, A. J., Teachman, B. A., & Nock, M. K. (2020) · Suicide and Life-Threatening Behavior · doi:10.1111/sltb.12652 · Cited by 18 · see citing works
“This recommendation is based on recent evidence that RT differences scores are particularly suited to testing between-group differences/context effects (e.g., Draheim et al., 2019; Payne, Vuletich, & Lundberg, 2017).”
Method
Precise Individual Measures of Inhibitory Control
Lee, H. J., Smith, D. M., Hauenstein, C. E., Dworetsky, A., Kraus, B., Dorn, M., Nee, D. E., & Gratton, C. (2025) · Nature Human Behaviour · doi:10.1038/s41562-025-02198-2 · Cited by 17 · see citing works
“Study designs for examining within-participant effects of experimental manipulations have come to prominence because of their ability to measure robust group-level effects, including the congruency effect 21,31 .”
Influential
Same, Same but Different? A Multi-Method Review of the Processes Underlying Executive Control
Toba, M. N., Malkinson, T. S., Howells, H., Mackie, M., & Spagna, A. (2023) · Neuropsychology Review · doi:10.1007/s11065-023-09577-4 · Cited by 24 · see citing works
“…orthogonal constructs, which are able to avoid the measurement errors recently emphasized in different tasks (see for instance Hedge et al., 2018; Draheim et al., 2019; Rey-Mermet et al., 2019; Rouder & Haaf, 2019, for the stimulus response compatibility tasks) and possible sampling errors (see…”
The ups and downs of bilingualism: A review of the literature on executive control using event-related potentials
Antoniou, K. (2023) · Psychonomic Bulletin & Review · doi:10.3758/s13423-023-02245-x · Cited by 17 · see citing works
Process-oriented intelligence research: A review from the cognitive perspective
Frischkorn, G. T., Wilhelm, O., & Oberauer, K. (2022) · Intelligence · doi:10.1016/j.intell.2022.101681 · Cited by 23 · see citing works
Spontaneous mind-wandering tendencies linked to cognitive flexibility in young adults.
Wong, Y.-S., Willoughby, A. R., & Machado, L. (2021) · Consciousness and Cognition · doi:10.31234/osf.io/h5zns · Cited by 17 · see citing works
Raising the Bar: Improving Methodological Rigour in Cognitive Alcohol Research.
Pennington, C., Jones, A., Bartlett, J., Copeland, A., & Shaw, D. (2021) · Addiction · doi:10.1111/add.15563 · Cited by 17 · see citing works
“Thirdly, researchers should explore different ways of analysis which might overcome the limitations inherent in the use of raw RTS and difference scores (see [49] for suggested alternatives).”
Background
Open-source open-access reaction time test (OORTT): an easy tool to assess reaction times
Rigoli, M., Facchin, A., Cardile, D., Beschin, N., & Luzzatti, C. (2020) · Neurological Sciences · doi:10.1007/s10072-020-04839-y · Cited by 17 · see citing works
Hierarchical-Model Insights for Planning and Interpreting Individual-Difference Studies of Cognitive Abilities
Rouder, J., & Mehrvarz, M. (2024) · Current Directions in Psychological Science · doi:10.1177/09637214231220923 · Cited by 16 · see citing works
“These include avoiding difference scores (Draheim et al., 2019), using so-called gamified tasks (Deveau et al., 2015; Kucina et al., 2022), and diffusion modeling (Haines et al., 2020; Lerche et al., 2020; von Krause et al., 2020; Weigard et al., 2021).”
Method
On the reliability of value-modulated attentional capture: An online replication and multiverse analysis
Garre-Frutos, F., Vadillo, M. A., González, F., & Lupiáñez, J. (2024) · Behavior Research Methods · doi:10.3758/s13428-023-02329-5 · Cited by 17 · see citing works
“These two studies suggest that measures based on RTs may not be suitable for studying individual differences (see Draheim et al., 2019 for a discussion).”
Background
A new way of classifying developmental prosopagnosia: Balanced Integration Score
Lowes, J., Hancock, P., & Bobak, A. K. (2024) · Cortex · doi:10.1016/j.cortex.2023.12.011 · Cited by 20 · see citing works
Linking computational models of two core tasks of cognitive control.
Robinson, M. M., & Mark Steyvers, B. (2022) · Psychological Review · doi:10.1037/rev0000395 · Cited by 18 · see citing works
“…of difference scores across individuals because much evidence suggests that peoplevary in howthey strategically approach the task (see Draheim et al., 2019; Hedge et al., 2021; Hedge, Powell, T h i s do c u m e n t i s c opy r i gh t e d by t h e A m e r i ca n P s y c ho l og i ca l…”
Interference control in working memory: Evidence for discriminant validity between removal and inhibition tasks
Rey-Mermet, A., Singh, K. A., Gignac, G., Brydges, C. R., & Ecker, U. K. H. (2020) · PLoS ONE · doi:10.1371/journal.pone.0243053 · Cited by 16 · see citing works
“As pointed out recently by Draheim and colleagues [69, 70], one possible reason for the difficulty in establishing a coherent factor of inhibition might be the use of RTs—and in particular, RT difference scores—because these scores have been associated with low reliability and speed-accuracy…”
Active Bilingualism in Aging: Balanced Bilingualism Usage and Less Frequent Language Switching Relates to Better Conflict Monitoring and Goal Maintenance Ability.
Chan, C. G. H., Yow, W., & Oei, A. (2020) · The journals of gerontology. Series B, Psychological sciences and social sciences · doi:10.1093/geronb/gbaa058 · Cited by 16 · see citing works
Implementation of an online spacing flanker task and evaluation of its test–retest reliability using measures of inhibitory control and the distribution of spatial attention
Lee, S.-H., & Pitt, M. A. (2024) · Behavior Research Methods · doi:10.3758/s13428-023-02327-7 · Cited by 15 · see citing works
“Further, that this consistency is best for the incongruent RT data across distances suggests that they are a viable alternative for measuring performance across distances (Draheim et al., 2019).”
Background
Generalizable prediction of childhood ADHD symptoms from neurocognitive testing and youth characteristics
Weigard, A., McCurry, K. L., Shapiro, Z., Martz, M. E., Angstadt, M., Heitzeg, M., Dinov, I., & Sripada, C. (2023) · Translational Psychiatry · doi:10.1038/s41398-023-02502-6 · Cited by 15 · see citing works
“Predicting the difference between two highly correlated measures of the same trait is inherently challenging because much of the systematic variance in the individual measures is reflected in their correlation, and their difference will, therefore, necessarily contain a large portion of error variance relative to syst…”
Background
Comparative effects of open-skill and closed-skill sports on executive function in university students: a 16-week quasi-experimental study
Li, Y.-F., Gao, T., Luo, L.-P., & He, S. (2024) · Frontiers in Psychology · doi:10.3389/fpsyg.2024.1457449 · Cited by 12 · see citing works
Working memory capacity preferentially enhances implementation of proactive control.
Lin, Y.-L., Brough, R., Tay, A., Jackson, J., & Braver, T. (2022) · Journal of Experimental Psychology. Learning, Memory and Cognition · doi:10.31234/osf.io/wvpbn · Cited by 14
“Second, the equally common practice of using subtraction-based difference scores can increase measurement error, constrain between-subjects variance, and attenuate reliability (Caruso, 2004; Cronbach & Furby, 1970; Draheim et al., 2019; Hedge et al., 2018).”
Background
Autism Traits Predict Self-reported Executive Functioning Deficits in Everyday Life and an Aversion to Exercise
Mason, L., Zimiga, B. M., Anders-Jefferson, R. T., & Paap, K. R. (2020) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-020-04741-8 · Cited by 17 · see citing works
Executive functioning profiles and mathematical and reading achievement in Grades 2, 6, and 10
Carriedo, N., Rodríguez‐Villagra, O. A., Pérez, L., & Iglesias-Sarmiento, V. (2024) · Journal of School Psychology · doi:10.1016/j.jsp.2024.101353 · Cited by 13 · see citing works
The Relevance of Attention Control, Not Working Memory, in Human Factors
Pak, R., McLaughlin, A., & Engle, R. (2023) · Hum. Factors · doi:10.1177/00187208231159727 · Cited by 13 · see citing works
“…measures (i.e., the limitations of using tasks designed for use in an experimental approach designed to minimize individual differences, the use of reaction-time difference scores, low reliability, and speed-accuracy tradeoffs) see Draheim et al. (2019, 2021) and Redick and Engle (2006).”
MethodSelf-citation
Leveraging item-level accuracy and reaction time to address ceiling effects in the measurement of inhibitory control in preschool-aged children
Willoughby, M. T., Camerota, M., King, K., Nduku, T., & Piper, B. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.861441 · Cited by 13 · see citing works
“Although some have criticized a reliance on changes in RT within a task to index IC (Hedge et al., 2018; Draheim et al., 2019), this approach to scoring tasks is common.”
Older Adults Show Intraindividual Variation in the Use of Predictive Processing
Jongman, S. R., Copeland, A., Xu, Y., Payne, B., & Federmeier, K. D. (2022) · Experimental Aging Research · doi:10.1080/0361073x.2022.2137358 · Cited by 13 · see citing works
Same same, but different: A psychometric examination of three frequently used experimental tasks for cognitive bias assessment in a sample of healthy young adults
Machulska, A., Kleinke, K., & Klucken, T. (2022) · Behavior Research Methods · doi:10.3758/s13428-022-01804-9 · Cited by 13 · see citing works
Attentional Control May Be Modifiable with Mindfulness-Based Cognitive Therapy to Prevent Suicide
Chesin, M. S., Keilp, J., Kline, A., Stanley, B., Myers, C., Latorre, M., Hill, L. M. S., Miller, R. B., King, A. R., Boschulte, D. R., et al. (2021) · Behaviour Research and Therapy · doi:10.1016/j.brat.2021.103988 · Cited by 13 · see citing works
The Deficit of Early Selective Attention in Adults With Sluggish Cognitive Tempo: In Comparison With Those With Attention-Deficit/Hyperactivity Disorder
Park, Y., & Lee, J.-H. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.614213 · Cited by 15 · see citing works
Brain network analysis reveals convergent and divergent aberrations between mild stroke patients with cortical and subcortical infarcts during cognitive task performing
Xu, M., Qian, L., Wang, S., Cai, H., Sun, Y., Thakor, N. V., Qi, X., & Sun, Y. (2023) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2023.1193292 · Cited by 10 · see citing works
“Reaction time is considered a good indicator of the speed and efficiency of mental processes (Draheim et al., 2019), whereas local efficiency measures the functional segregation, the capacity of locally processed information (Latora and Marchiori, 2001).”
Background
100 days of Adolescence: Elucidating Externalizing Behaviors Through the Daily Assessment of Inhibitory Control
Chaku, N., Yan, R., Kelly, D. P., Zhang, Z., Lopez‐Duran, N. L., Weigard, A. S., & Beltz, A. M. (2023) · Research on Child and Adolescent Psychopathology · doi:10.1007/s10802-023-01071-y · Cited by 14 · see citing works
“Since 1975, subtraction methods (e.g., comparing reaction time on incongruent and congruent trials) have gained popularity, but they have significant limitations for accu - rately detecting individual differences (Draheim et al., 2019; Weigard et al., 2021), and thus, intraindividual variation.”
Background
Improving the Validity of the Armed Service Vocational Aptitude Battery with Measures of Attention Control
Martin, J., Mashburn, C. A., & Engle, R. (2020) · Journal of Applied Research in Memory and Cognition · doi:10.1016/j.jarmac.2020.04.002 · Cited by 12 · see citing works
“Instead, Draheim et al. (2019) advocated that those of the Armed Service Vocational Aptitude Battery with Measures of (2020), https://doi.org/10.1016/j.jarmac.2020.04.002 he experimental and individual differences approach.”
BackgroundInfluentialSelf-citation
Attentional control data collection: A resource for efficient data reuse
Haaf, J. M., Hoffstadt, M., & Lesche, S. (2025) · Behavior Research Methods · doi:10.3758/s13428-025-02717-z · Cited by 7 · see citing works
Analytic and holistic cognitive style as a set of independent manifests: Evidence from a validation study of six measurement instruments
Lacko, D., Prošek, ‪., Čeněk, J., Helísková, M., Ugwitz, P., Svoboda, V., Počaji, P., Vais, M., Halířová, H., Juřík, V., et al. (2023) · PLoS ONE · doi:10.1371/journal.pone.0287057 · Cited by 18 · see citing works
Food neophobia predicts visual ability in the recognition of prepared food, beyond domain-general factors
Gauthier, I., & Fiestan, G. (2022) · Food Quality and Preference · doi:10.1016/j.foodqual.2022.104702 · Cited by 11 · see citing works
Aberrant reward learning, but not negative reinforcement learning, is related to depressive symptoms: an attentional perspective
Hertz‐Palmor, N., Rozenblit, D., Lavi, S., Zeltser, J., Kviatek, Y., & Lazarov, A. (2023) · Psychological Medicine · doi:10.1017/s0033291723002519 · Cited by 9 · see citing works
“Second, RT-based tasks exhibit poor psychometric properties, including low internal consistency and test – retest reliability (Brown et al., 2014; Draheim, Mashburn, Martin, & Engle, 2019; Rodebaugh et al., 2016; Schmukle, 2005; Staugaard, 2009; Waechter & Stolz, 2015), which are vital for trusting…”
The mediating role of neurocognitive functions in the relation between physical competencies and academic achievement of primary school children
de Bruijn, A. G. M., Meijer, A., Königs, M., Oosterlaan, J., Smith, J., & Hartman, E. (2023) · Psychology of sport and exercise · doi:10.1016/j.psychsport.2023.102390 · Cited by 10 · see citing works
Individual differences in lexical learning across two language modalities: Sign learning, word learning, and their relationship in hearing non-signing adults
Martinez, D., & Singleton, J. L. (2019) · Acta Psychologica · doi:10.1016/j.actpsy.2019.102892 · Cited by 12 · see citing works
Clarifying the reliability paradox: poor measurement reliability attenuates group differences
Karvelis, P., & Diaconescu, A. O. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1592658 · Cited by 10 · see citing works
The factorial structure of executive functions in preschool and elementary school children and relations with intelligence
Grobe, S. E., Könen, T., David, C., Grüneisen, L., Dörrenbächer‐Ulrich, L., Perels, F., & Karbach, J. (2024) · Journal of Experimental Child Psychology · doi:10.1016/j.jecp.2024.106014 · Cited by 11 · see citing works
Knowledge, attention, and psychomotor ability: A latent variable approach to understanding individual differences in simulated work performance
Mashburn, C. A., Burgoyne, A. P., Tsukahara, J. S., Pak, R., Coyne, J. T., Sibley, C. M., Foroughi, C. K., & Engle, R. W. (2024) · Intelligence · doi:10.1016/j.intell.2024.101835 · Cited by 11 · see citing works
“Thus, the attention control factor used here was able to capture more of the reliable variance among the observed measures than is often observed in studies of attentional abilities (see, e. g., Draheim, Mashburn, Martin, & Engle, 2019).”
Self-citation
Value-modulated attentional capture in reward and punishment contexts, attentional control, and their relationship with psychopathology
Freichel, R., Mrkonja, L., de Jong, P. J., Cousijn, J., Franken, I. H. A., Ruiter, T. A., Pelley, M. E. L., Albertella, L., Watson, P., Veer, I. M., et al. (2023) · Journal of Experimental Psychopathology · doi:10.1177/20438087231204166 · Cited by 8 · see citing works
“Prior research suggests that low test-retest reliability may be related to factors such as low between-subject variability (Hedge et al., 2018) or the use of reaction time difference scores as measures of attentional control (Draheim et al., 2019).”
Background
Should Intelligence Tests Be Speeded or Unspeeded? A Brief Review of the Effects of Time Pressure on Response Processes and an Experimental Study with Raven’s Matrices
Gonthier, C. (2023) · Journal of Intelligence · doi:10.3390/jintelligence11060120 · Cited by 12 · see citing works
Not So Fast! Response Times in the Computerized Benton Face Recognition Test May Not Reflect Face Recognition Ability
DeGutis, J., Li, X., Yosef, B., & Mishra, M. V. (2022) · Cognitive Neuropsychology · doi:10.1080/02643294.2022.2114824 · Cited by 9 · see citing works
“%) may suffer from differential speed-accuracy tradeoffs across participants, limiting the usefulness of RT measures (see Draheim et al., 2019).”
Background
Inhibitory Control, Conduct Problems, and Callous Unemotional Traits in Children with ADHD and Typically Developing Children
Waschbusch, D., Babinski, D., Fosco, W. D., Haas, S. M., Waxmonsky, J. G., Garon, N., Nichols, S., King, S., Santor, D., & Andrade, B. F. (2022) · Developmental Neuropsychology · doi:10.1080/87565641.2022.2032713 · Cited by 9 · see citing works
Neither measurement error nor speed–accuracy trade-offs explain the difficulty of establishing attentional control as a psychometric construct: Evidence from a latent-variable analysis using diffusion modeling
Rey-Mermet, A., Singmann, H., & Oberauer, K. (2021) · Psychonomic Bulletin & Review · doi:10.3758/s13423-025-02696-4 · Cited by 9 · see citing works
“A second reason for the difficulty of establishing attentional control at the latent-variable level is that most atten-tional-control measures rely exclusively on response times, so that individual differences in speed–accuracy trade-offs are disregarded (Draheim et al., 2019).”
Influential
Moving Beyond the Relative Assessment of Implicit Biases: Navigating the Complexities of Absolute Measurement
O'Shea, B. A., & Wiers, R. (2020) · Social Cognition · doi:10.1521/soco.2020.38.supp.s187 · Cited by 9 · see citing works
“Due to restrictive variance and ceiling effects, accu-racy-based calculations are often less reliable (e.g., Draheim, Mashburn, Martin, & Engle, 2019; cf. Gladwin et al., 2015).”
Background

Authors: Draheim, C., Mashburn, C. A., Martin, J. D., & Engle, R. W.
Affiliations: Georgia Institute of Technology (US)
ORCID: Draheim, Christopher
Mashburn, Cody A.
Martin, Jessie D.
Engle, Randall W.
Venue: Psychological Bulletin
Publisher: American Psychological Association (APA)
Published: 2019-05
Volume: 145
Issue: 5
Pages: 508-535
ISSN: 0033-2909 (ISSN-L)
1939-1455
DOI: 10.1037/bul0000192
OpenAlex: W2923932768
Cited by: 322
References: 0
Citations by year: 2026: 46 · 2025: 42 · 2024: 59 · 2023: 55 · 2022: 48 · 2021: 35 · 2020: 31 · 2019: 5
FWCI: 17.30
Type: journal article
Language: en
Open access: hybrid
doi.org
Licence: other-oa
www.apa.org/pubs/journals/resources/open-access.aspx (am, 324 d)
Locations: Psychological Bulletin (publishedVersion, other-oa)
PubMed (publishedVersion)
Funders: Office of Naval Research (N00014-12-1-0406; N00014-12-1-1011)
Topics: Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
Social Sciences › Psychology › Experimental and Cognitive Psychology › Cognitive Abilities and Testing
Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
Keywords: reaction time, individual differences, developmental research, speed-accuracy trade-off, reliability statistics, difference scores, attentional control, task switching, alternative measures, cognitive development
MeSH terms: Humans, Reaction Time, Reproducibility of Results, Behavioral Research

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review