The DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
Unreliability as a threat to understanding psychopathology: The cautionary tale of attentional bias.
Open accessAbstract
The use of unreliable measures constitutes a threat to our understanding of psychopathology, because advancement of science using both behavioral and biologically oriented measures can only be certain if such measurements are reliable. Two pillars of the National Institute of Mental Health's portfolio-the Research Domain Criteria (RDoC) initiative for psychopathology and the target engagement initiative in clinical trials-cannot succeed without measures that possess the high reliability necessary for tests involving mediation and selection based on individual differences. We focus on the historical lack of reliability of attentional bias measures as an illustration of how reliability can pose a threat to our understanding. Our own data replicate previous findings of poor reliability for traditionally used scores, which suggests a serious problem with the ability to test theories regarding attentional bias. This lack of reliability may also suggest problems with the assumption (in both theory and the formula for the scores) that attentional bias is consistent and stable across time. In contrast, measures accounting for attention as a dynamic process in time show good reliability in our data. The field is sorely in need of research reporting findings and reliability for attentional bias scores using multiple methods, including those focusing on dynamic processes over time. We urge researchers to test and report reliability of all measures, considering findings of low reliability not just as a nuisance but as an opportunity to modify and improve upon the underlying theory. Full assessment of reliability of measures will maximize the possibility that RDoC (and psychological science more generally) will succeed. (PsycINFO Database Record
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Although in a separate study, ketamine failed to illustrate a similar impact on attentional bias towards negative cues 128 , the use of a reaction time measure (the dot-probe task) with notoriously poor psychometric properties 136-138 may have impeded the ability to sensitively detect changes over time.”
From Neuroplasticity in cognitive and psychological mechanisms of depression: an integrative model (2020) · cites this paper as [138]
In the citing paper
“Our ability to identify individually relevant brain and behavioural changes depends on a high level of measurement accuracy, combined with appropriate statistical modelling, neither of which is widely used in RCTs.38”
From Precision clinical trials: a framework for getting to precision medicine for neurobehavioural disorders (2021) · cites this paper as [38]
In the citing paper
“Our ability to identify individually relevant brain and behavioural changes depends on a high level of measurement accuracy, combined with appropriate statistical modelling, neither of which is widely used in RCTs.38”
From Precision clinical trials: a framework for getting to precision medicine for neurobehavioural disorders (2021) · cites this paper as [38]
In the citing paper
“One research challenge is the limited reliability of AB indices derived from manual response times, which may reflect measurement problems and the variability of anxiety-related ABs, as discussed earlier [19,56-58,98].”
From Anxiety and Threat-Related Attention: Cognitive-Motivational Framework and Treatment (2018) · cites this paper as [56]
In the citing paper
“One research challenge is the limited reliability of AB indices derived from manual response times, which may reflect measurement problems and the variability of anxiety-related ABs, as discussed earlier [19,56-58,98].”
From Anxiety and Threat-Related Attention: Cognitive-Motivational Framework and Treatment (2018) · cites this paper as [56]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical power analyses using G∗Power 3.1: tests for correlation and regression analyses
Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009) · Behavior Research Methods · doi:10.3758/brm.41.4.1149 · Cited by 37,615
ggplot2: Elegant Graphics for Data Analysis
Wickham, H. (2009) · doi:10.1007/978-0-387-98141-3 · Cited by 14,447
Threat-related attentional bias in anxious and nonanxious individuals: a meta-analytic study.
Bar-Haim, Y., Lamy, D., Pergamin, L., Bakermans-Kranenburg, M. J., & van IJzendoorn, M. H. (2007) · Psychological Bulletin · doi:10.1037/0033-2909.133.1.1 · Cited by 3,721
Cole, D. A., & Maxwell, S. E. (2003) · Journal of Abnormal Psychology · doi:10.1037/0021-843x.112.4.558 · Cited by 3,605
How to improve R&D productivity: the pharmaceutical industry's grand challenge
Paul, S. M., Mytelka, D. S., Dunwiddie, C. T., Persinger, C. C., Munos, B. H., Lindborg, S. R., & Schacht, A. L. (2010) · Nature Reviews Drug Discovery · doi:10.1038/nrd3078 · Cited by 3,566
Structured clinical interview for DSM-IV axis I disorders : SCID-I : clinical version : scoresheet
First, M. B., Spitzer, R. L., Gibbon, M., & Williams, J. B. W. (1997) · American Psychiatric Press eBooks · Cited by 3,101
Balota, D. A., Yap, M. J.-M., Hutchison, K. A., Cortese, M. J., Kessler, B., Loftis, B., Neely, J. H., Nelson, D. L., Simpson, G. B., & Rebecca Treiman, A. (2007) · Behavior Research Methods · doi:10.3758/bf03193014 · Cited by 2,758
Attentional bias in emotional disorders.
MacLeod, C., Mathews, A., & Tata, P. R. (1986) · Journal of Abnormal Psychology · doi:10.1037//0021-843x.95.1.15 · Cited by 2,553
Producing high-dimensional semantic spaces from lexical co-occurrence
Lund, K., & Burgess, C. (1996) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03204766 · Cited by 1,875
Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004) · Psychological Review · doi:10.1037/0033-295x.111.4.1061 · Cited by 1,733
Cognitive Approaches to Emotion and Emotional Disorders
Mathews, A., & MacLeod, C. (1994) · Annual Review of Psychology · doi:10.1146/annurev.ps.45.020194.000325 · Cited by 1,023
Selective processing of threat cues in anxiety states
Mathews, A., & MacLeod, C. (1985) · Behaviour Research and Therapy · doi:10.1016/0005-7967(85)90104-4 · Cited by 1,001
(2004) · Cited by 964
Anxiety and the Allocation of Attention to Threat
MacLeod, C., & Mathews, A. (1988) · The Quarterly Journal of Experimental Psychology Section A · doi:10.1080/14640748808402292 · Cited by 817
The Attack of the Psychometricians
Borsboom, D. (2006) · Psychometrika · doi:10.1007/s11336-006-1447-6 · Cited by 599
Wendland, J. R., Martin, B. J., Kruse, M. R., Lesch, K., & Murphy, D. L. (2006) · Molecular Psychiatry · doi:10.1038/sj.mp.4001789 · Cited by 561
Attention modification program in individuals with generalized anxiety disorder
Amir, N., Beard, C., Burns, M., & Bomyea, J. (2009) · Journal of Abnormal Psychology · doi:10.1037/a0012589 · Cited by 539
Unreliability of the dot probe task
Schmukle, S. C. (2005) · European Journal of Personality · doi:10.1002/per.554 · Cited by 538
Attention training in individuals with generalized social phobia: A randomized controlled trial.
Amir, N., Beard, C., Taylor, C. T., Klumpp, H., Elias, J. A., Burns, M. N., & Chen, X. (2009) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0016685 · Cited by 486
A review of current evidence for the causal impact of attentional bias on fear and anxiety
Van Bockstaele, B., Verschuere, B., Tibboel, H., De Houwer, J., Crombez, G., & Koster, E. H. W. (2014) · Psychological Bulletin · doi:10.1037/a0034834 · Cited by 485
Borsboom, D. (2005) · Cambridge University Press eBooks · doi:10.1017/cbo9780511490026 · Cited by 481
Biased processing of emotional information in girls at risk for depression
Joormann, J., Talbot, L., & Gotlib, I. H. (2007) · Journal of Abnormal Psychology · doi:10.1037/0021-843x.116.1.135 · Cited by 420
Constructing constructs for psychopathology: The NIMH research domain criteria.
Cuthbert, B. N., & Kozak, M. J. (2013) · Journal of Abnormal Psychology · doi:10.1037/a0034028 · Cited by 364
Measuring attentional bias to threat: reliability of dot probe and eye movement indices
Waechter, S., Nelson, A. L., Wright, C., Hyatt, A., & Oakman, J. (2014) · Cognitive Therapy and Research · doi:10.1007/s10608-013-9588-2 · Cited by 324
The attentional bias modification approach to anxiety intervention
MacLeod, C., & Clarke, P. J. F. (2015) · Clinical Psychological Science · doi:10.1177/2167702614560749 · Cited by 315
Empirical recommendations for improving the stability of the dot-probe task in clinical research.
Price, R. B., Kuckertz, J. M., Siegle, G. J., Ladouceur, C. D., Silk, J. S., Ryan, N. D., Dahl, R. E., & Amir, N. (2015) · Psychological Assessment · doi:10.1037/pas0000036 · Cited by 312
Automaticity and the anxiety disorders.
Mcnally, R. J. (1995) · Behaviour Research and Therapy · doi:10.1016/0005-7967(95)00015-p · Cited by 281
Kappenman, E. S., Farrens, J. L., Luck, S. J., & Hajcak, G. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.01368 · Cited by 267
Determining target engagement in living systems
Simon, G. M., Niphakis, M. J., & Cravatt, B. F. (2013) · Nature Chemical Biology · doi:10.1038/nchembio.1211 · Cited by 245
Test–retest and between‐site reliability in a multicenter fMRI study
Friedman, L., Stern, H. S., Brown, G. G., Mathalon, D. H., Turner, J. A., Glover, G. H., Gollub, R. L., Lauriello, J., Lim, K. O., Cannon, T. D., et al. (2007) · Human Brain Mapping · doi:10.1002/hbm.20440 · Cited by 245
Temporal dynamics of attentional bias
Zvielli, A., Bernstein, A., & Koster, E. H. W. (2015) · Clinical Psychological Science · doi:10.1177/2167702614551572 · Cited by 237
Self-citationEnock, P. M., Hofmann, S. G., & McNally, R. J. (2014) · Cognitive Therapy and Research · doi:10.1007/s10608-014-9606-z · Cited by 201
Life-threatening danger and suppression of attention bias to threat
Bar-Haim, Y., Holoshitz, Y., Eldar, S., Frenkel, T. I., Muller, D., Charney, D. S., Pine, D. S., Fox, N. A., & Wald, I. (2010) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2009.09070956 · Cited by 193
Clarke, P. J., Notebaert, L., & MacLeod, C. (2014) · BMC Psychiatry · doi:10.1186/1471-244x-14-8 · Cited by 188
Reliability of two versions of the dot-probe task using photographic faces
Staugaard, S. R. (2009) · Cited by 187
Pérez‐Edgar, K. E., Bar‐Haim, Y., McDermott, J. M., Gorodetsky, E. K., Hodgkinson, C. A., Goldman, D. A., Ernst, M., Pine, D. S., & Fox, N. A. (2009) · Biological Psychology · doi:10.1016/j.biopsycho.2009.08.009 · Cited by 175
Pergamin‐Hight, L., Bakermans‐Kranenburg, M. J., van IJzendoorn, M. H., & Bar‐Haim, Y. (2011) · Biological Psychiatry · doi:10.1016/j.biopsych.2011.10.030 · Cited by 144
Attentional retraining: A randomized clinical trial for pathological worry
Hazen, R. A., Vasey, M. W., & Schmidt, N. B. (2008) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2008.07.004 · Cited by 140
Britton, J. C., Bar-Haim, Y., Clementi, M. A., Sankin, L. S., Chen, G., Shechner, T., Norcross, M. A., Spiro, C. N., Lindstrom, K. M., & Pine, D. S. (2013) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2012.11.001 · Cited by 101
Waechter, S., & Stolz, J. A. (2015) · Cognitive Therapy and Research · doi:10.1007/s10608-015-9670-z · Cited by 87
Dear, B. F., Sharpe, L., Nicholas, M. K., & Refshauge, K. M. (2011) · Journal of Pain · doi:10.1016/j.jpain.2011.07.003 · Cited by 80
The effect of attention modification with explicit vs. minimal instructions on worry
Krebs, G., Hirsch, C. R., & Mathews, A. (2010) · Behaviour Research and Therapy · doi:10.1016/j.brat.2009.10.009 · Cited by 79
Hoyle, R. H., & Robinson, J. C. (2008) · Sage eBooks · doi:10.4135/9781412976190.n10 · Cited by 69
Barch, D. M., & Yarkoni, T. (2013) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/s13415-013-0201-7 · Cited by 67
Hirsch, C. R., MacLeod, C., Mathews, A., Sandher, O., Siyani, A., & Hayes, S. (2011) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2010.09.013 · Cited by 65
Emil Kraepelin: Icon and Reality
Engstrom, E. J., & Kendler, K. S. (2015) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2015.15050665 · Cited by 65
Moving From Clinical Trials to Precision Medicine
Pencina, M. J., & Peterson, E. D. (2016) · JAMA · doi:10.1001/jama.2016.4839 · Cited by 65
Schäfer, J., Bernstein, A., Zvielli, A., Höfler, M., Wïttchen, H., & Schönfeld, S. (2016) · Depression and Anxiety · doi:10.1002/da.22526 · Cited by 58
Self-citation(De)coupling of our eyes and our mind’s eye: A dynamic process perspective on attentional bias.
Amir, I., Zvielli, A., & Bernstein, A. (2016) · Emotion · doi:10.1037/emo0000172 · Cited by 51
Self-citationBarch, D. M., & Mathalon, D. H. (2011) · Biological Psychiatry · doi:10.1016/j.biopsych.2011.01.004 · Cited by 41
Attentional Bias Dynamics and Posttraumatic Stress in Survivors of Violent Conflict and Atrocities
Yuval, K., Zvielli, A., & Bernstein, A. (2016) · Clinical Psychological Science · doi:10.1177/2167702616649349 · Cited by 32
Self-citationTo honor Kraepelin...: From symptoms to pathology in the diagnosis of mental illness.
Kihlstrom, J. F. (2004) · American Psychological Association eBooks · doi:10.1037/10456-011 · Cited by 30
Calamaras, M. R., Tone, E. B., & Anderson, P. L. (2012) · Journal of Clinical Psychology · doi:10.1002/jclp.21875 · Cited by 27
Research works printed but not linked: 2
- Harvey AG Watkins E Mansell W 2004 Cognitive behavioural processes across psychological disorders New York Oxford University Press
- Nunnally JD Bernstein IH 1994 Psychometric theory 3 New York McGraw-Hill
Cited by
Sorted by citations
Neuroplasticity in cognitive and psychological mechanisms of depression: an integrative model
Price, R. B., & Duman, R. (2020) · Molecular Psychiatry · doi:10.1038/s41380-019-0615-x · Cited by 739
Parsons, S. D., Kruijt, A., & Fox, E. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919879695 · Cited by 450
Attentional bias for threat: Crisis or opportunity?
McNally, R. J. (2018) · Clinical Psychology Review · doi:10.1016/j.cpr.2018.05.005 · Cited by 272
Anxiety and Threat-Related Attention: Cognitive-Motivational Framework and Treatment
Mogg, K., & Bradley, B. P. (2018) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2018.01.001 · Cited by 210
Fleeting reliability in the dot-probe task
Chapman, A., Devue, C., & Grimshaw, G. M. (2019) · Psychological Research · doi:10.1007/s00426-017-0947-6 · Cited by 172
Metadata
- Authors
- Rodebaugh, T. L., Scullin, R. B., Langer, J. K., Dixon, D. J., Huppert, J. D., Bernstein, A., Zvielli, A., & Lenze, E. J.
- Affiliations
- Hebrew University of Jerusalem (IL)
- University of Haifa (IL)
- ORCID
- Venue
- Journal of Abnormal Psychology
- Published
- 2016-08
- Volume
- 125
- Issue
- 6
- Pages
- 840-851
- ISSN
- 0021-843X (ISSN-L)
- 1939-1846
- PMC
- PMC4980228
- OpenAlex
- W2465300579
- Cited by
- 312
- References
- 53
- Citations by year
- 2026: 21 · 2025: 24 · 2024: 28 · 2023: 39 · 2022: 41 · 2021: 40 · 2020: 37 · 2019: 24 · 2018: 33 · 2017: 22 · 2016: 2
- FWCI
- 38.33
- Type
- journal article
- Language
- en
- Update notices
- correction (2016-09-01)
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Journal of Abnormal Psychology (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institutes of Health (UL1TR000448)
- National Institute of Mental Health (R21MH090308, R01MH070547)
- National Center for Research Resources (UL1RR024992)
- National Center for Advancing Translational Sciences
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Anxiety, Depression, Psychometrics, Treatment, Cognitive Processes
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Mental Health Research Topics
- Keywords
- attentional bias, measurement reliability, psychopathology, Research Domain Criteria, target engagement, replication, individual differences, reliability reporting
- MeSH terms
- Attentional Bias, Mental Disorders, Humans, Individuality, Models, Psychological, National Institute of Mental Health (U.S.), Research Design, United States, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
