It’s Time to Broaden the Replicability Conversation: Thoughts for and From Clinical Psychological Science
The DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index.
Abstract
Psychology is in the early stages of examining a crisis of replicability stemming from several high-profile failures to replicate studies in experimental psychology. This important conversation has largely been focused on social psychology, with some active participation from cognitive psychology. Nevertheless, several other major domains of psychological science—including clinical science—have remained insulated from this discussion. The goals of this article are to (a) examine why clinical psychology and allied fields, such as counseling and school psychology, have not been central participants in the replicability conversation; (b) review concerns and recommendations that are less (or more) applicable to or appropriate for research in clinical psychology and allied fields; and (c) generate take-home messages for scholars and consumers of the literature in clinical psychology and allied fields, as well as reviewers, editors, and colleagues from other areas of psychological science.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Replication of research findings for clinical interventions is important (9), as a pre-condition for evidence-based practice is the replicability of effects (10).”
Citations
Cites
Diagnostic and Statistical Manual of Mental Disorders
American Psychiatric Association (2013) · doi:10.1176/appi.books.9780890425596 · Cited by 116,369
Diagnostic and Statistical Manual of Mental Disorders, 5th ed.
Carter, M. J. (2014) · Therapeutic Recreation Journal · Cited by 15,169
Construct validity in psychological tests.
Cronbach, L. J., & Meehl, P. E. (1955) · Psychological Bulletin · doi:10.1037/h0040957 · Cited by 11,195 · cited 2×
“Mental disorders, along with their putative indicators, are necessarily constructs; they are hypothetical attributes that cannot be perfectly (“operationally”) defined by any set of observable referents (Cronbach & Meehl, 1955).”
BackgroundMethodWhy Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 1×
“One potential benefit of these cultural differences is that it may decrease the likelihood of publication bias stemming from the “file drawer effect” (Rosenthal, 1979).”
BackgroundEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 1×
“…is perhaps surprising given that clinical psychologists frequently collaborate with scholars in domains of psychiatry, neuroimaging, and molecular genetics in which replication dif fi culties have been observed and widely discussed (e.g., Button et al., 2013; Ioannidis & Trikalinos, 2005).”
BackgroundInsel, T., Cuthbert, B., Garvey, M., Heinssen, R., Pine, D. S., Quinn, K., Sanislow, C., & Wang, P. (2010) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2010.09091379 · Cited by 7,067 · cited 1×
“…by dimensional quantification approaches has been noted by others (Markon, Chmielewski, & Miller, 2011), and the newly launched RDoC initiative (Insel et al., 2010) provides a renewed focus on dimensions of psychopathology rather than on categories created by the imposition of largely…”
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
No adjustments are needed for multiple comparisons.
Rothman, K. J. (1990) · Epidemiology · doi:10.1097/00001648-199001000-00010 · Cited by 5,965
What's wrong with Bonferroni adjustments
Perneger, T. V. (1998) · BMJ · doi:10.1136/bmj.316.7139.1236 · Cited by 5,900
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 2×
“The replicability crisis has been discussed at length in both peer-reviewed (e.g., Ioannidis, 2005; Spellman, 2015) and non-peer-reviewed (e.g., Hilgard, 2016) outlets in recent years; hence, our review of this topic is necessarily brief and limited to key concepts that provide a context for…”
BackgroundClinical Psychology: Science and Practice
(2021) · Clinical Psychology Science and Practice · doi:10.1111/(issn)1468-2850 · Cited by 3,672
On the practice of dichotomization of quantitative variables.
MacCallum, R. C., Zhang, S., Preacher, K. J., & Rucker, D. D. (2002) · Psychological Methods · doi:10.1037/1082-989x.7.1.19 · Cited by 3,538
Kotov, R., Krueger, R. F., Watson, D., Achenbach, T. M., Althoff, R. R., Bagby, R. M., Brown, T. A., Carpenter, W. T., Caspi, A., Clark, L. A., et al. (2017) · Journal of Abnormal Psychology · doi:10.1037/abn0000258 · Cited by 3,335 · cited 1×
“One example is the Hierarchical Taxonomy of Psychopathology (HiTOP) consortium, consisting of researchers from clinical psychology, psychiatry, epidemiology, genetics, and beyond, joining together to address overarching concerns regarding the nosology of mental disorders (Kotov et al., 2016).”
BackgroundSelf-citationThe New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200
Effect size guidelines for individual differences researchers
Gignac, G. E., & Szodorai, E. T. (2016) · Personality and Individual Differences · doi:10.1016/j.paid.2016.06.069 · Cited by 2,860
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708 · cited 10×
“Goal 3: Independent replication As noted earlier, a significant contributor to the replicability crisis has been a string of widely publicized failures to replicate findings from prominent experimental psychology studies (e.g., Donnellan et al., 2015; Klein et al., 2014; Open Science Collaboration, 2015).”
BackgroundInfluentialMeasuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
Meta-analysis of psychotherapy outcome studies.
Smith, M. L., & Glass, G. V. (1977) · American Psychologist · doi:10.1037/0003-066x.32.9.752 · Cited by 2,154 · cited 1×
“…means new—having emerged in part from mid-20th-century clinical science research (e.g., Cohen, 1962), including now-classic work on the efficacy of psychotherapy (Smith & Glass, 1977)— but they remain greatly underutilized (Marszalek, Barber, Kohlhart, & Cooper, 2011; Sedlmeier & Gigerenzer, 1989).”
BackgroundHARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
The statistical power of abnormal-social psychological research: A review.
Cohen, J. (1962) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0045186 · Cited by 1,824
Statistical significance in psychological research.
Lykken, D. T. (1968) · Psychological Bulletin · doi:10.1037/h0026141 · Cited by 1,350
Pashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321
Investigating Variation in Replicability: A “Many Labs” Replication Project
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., et al. (2014) · Social Psychology · doi:10.1027/1864-9335/a000178 · Cited by 1,208
A Multilab Preregistered Replication of the Ego-Depletion Effect
Hagger, M. S., Chatzisarantis, N. L. D., Alberts, H., Anggono, C. O., Batailler, C., Birt, A. R., Brand, R., Brandt, M. J., Brewer, G., Bruyneel, S., et al. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616652873 · Cited by 1,079
Is psychology suffering from a replication crisis? What does “failure to replicate” really mean?
Maxwell, S. E., Lau, M. Y., & Howard, G. S. (2015) · American Psychologist · doi:10.1037/a0039400 · Cited by 1,005
Tellegen, A., & Waller, N. G. (2008) · The SAGE Handbook of Personality Theory and Assessment: Volume 2 — Personality Measurement and Testing · doi:10.4135/9781849200479.n13 · Cited by 905 · cited 1×
“This iterative process between theory and data is a fundamental component of a growing science (Cronbach & Meehl, 1955; Tellegen & Waller, 2008) that needs to be accommodated through a more flexible preregistration approach that permits hypotheses and analytic plans to be modified over time— with…”
MethodDennes, W. R., & Reichenbach, H. (1939) · The Philosophical Review · doi:10.2307/2180454 · Cited by 871
Recommendations for Increasing Replicability in Psychology
Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R., Nosek, B. A., et al. (2013) · European Journal of Personality · doi:10.1002/per.1919 · Cited by 823
The rules of the game called psychological science
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459060 · Cited by 800 · cited 5×
“, with new studies building on findings from p studies) has been linked to a higher rather than lower prevalence of QRPs (Bakker et al., 2012).”
BackgroundResultInfluentialDo studies of statistical power have an effect on the power of studies?
Sedlmeier, P., & Gigerenzer, G. (1989) · Psychological Bulletin · doi:10.1037/0033-2909.105.2.309 · Cited by 748
Goldfried, M. R. (1985) · International Handbook of Behavior Modification and Therapy · doi:10.1007/978-1-4615-7278-7_3 · Cited by 687
Markon, K. E., Chmielewski, M., & Miller, C. J. (2011) · Psychological Bulletin · doi:10.1037/a0023678 · Cited by 656
The Value of Direct Replication
Simons, D. J. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613514755 · Cited by 630
The alleged crisis and the illusion of exact replication
Stroebe, W., & Strack, F. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613514450 · Cited by 597
The SAGE Handbook of Personality Theory and Assessment
Austin, E. J., Parker, J. D. A., Petrides, K. V., & Saklofske, D. H. (2008) · Cited by 554
The ironic effect of significant results on the credibility of multiple-study articles.
Schimmack, U. (2012) · Psychological Methods · doi:10.1037/a0029487 · Cited by 455
Fraley, R. C., & Vazire, S. (2014) · PLoS ONE · doi:10.1371/journal.pone.0109019 · Cited by 409
Contextual sensitivity in scientific reproducibility
Van Bavel, J. J., Mende-Siedlecki, P., Brady, W. J., & Reinero, D. A. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1521897113 · Cited by 398
Unpublished results hide the decline effect
Schooler, J. (2011) · Nature · doi:10.1038/470437a · Cited by 333
Ioannidis, J. P. A., & Trikalinos, T. A. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.10.019 · Cited by 325
Addington, J., Cadenhead, K. S., Cannon, T. D., Cornblatt, B., McGlashan, T. H., Perkins, D. O., Seidman, L. J., Tsuang, M., Walker, E. F., Woods, S. W., et al. (2007) · Schizophrenia Bulletin · doi:10.1093/schbul/sbl075 · Cited by 298
Kotov, R., Krueger, R. F., Watson, D., et al. (2017) · doi:10.31234/osf.io/zaadn · Cited by 280
Self-citationMarginally Significant Effects as Evidence for Hypotheses: Changing Attitudes Over Four Decades
Pritschet, L., Powell, D., & Horne, Z. (2016) · Psychological Science · doi:10.1177/0956797616645672 · Cited by 268
Empirically Supported Treatment: Recommendations for a New Model
Tolin, D. F., McKay, D., Forman, E. M., Klonsky, E. D., & Thombs, B. D. (2015) · Clinical Psychology: Science and Practice · doi:10.1111/cpsp.12122 · Cited by 219
A Short (Personal) Future History of Revolution 2.0
Spellman, B. A. (2015) · Perspectives on Psychological Science · doi:10.1177/1745691615609918 · Cited by 213 · cited 5×
“Another increasingly common recommendation for improving the replicability of psychological science findings, which we endorse, is to (a) preregister hypotheses, design, and analyses and (b) make collected data publicly available, or open access (e.g., Gelman & Loken, 2013; Jonas & Cesario, 2015; Lindsay et al., in pr…”
BackgroundInfluentialPsychopathology and Attirition in the Epidemiologic Catchment Area Surveys
Eaton, W. W., Anthony, J. C., Tepper, S., & Dryman, A. (1992) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a116399 · Cited by 210
Finkel, E. J., Eastwick, P. W., & Reis, H. T. (2015) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000007 · Cited by 202
Reversals of Established Medical Practices: Evidence to Abandon Ship
Prasad, V., Cifu, A., & Ioannidis, J. P. A. (2012) · JAMA · doi:10.1001/jama.2011.1960 · Cited by 187
Psychology’s Replication Crisis and the Grant Culture: Righting the Ship
Lilienfeld, S. O. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691616687745 · Cited by 187
Self-citationSample size in psychological research over the past 30 years
Marszalek, J. M., Barber, C., Kohlhart, J., & Cooper, B. H. (2011) · Perceptual and Motor Skills · doi:10.2466/03.11.pms.112.2.331-348 · Cited by 167
The Tree of Knowledge System and the Theoretical Unification of Psychology
Henriques, G. (2003) · Review of General Psychology · doi:10.1037/1089-2680.7.2.150 · Cited by 166
An asocial psychology and a misdirected clinical psychology.
Sarason, S. B. (1981) · American Psychologist · doi:10.1037//0003-066x.36.8.827 · Cited by 129
RDoC: Translating promise into progress
Patrick, C. J., & Hajcak, G. (2016) · Psychophysiology · doi:10.1111/psyp.12612 · Cited by 115
Self-citationFirst, M. B. (2005) · Psychopathology · doi:10.1159/000086093 · Cited by 114
Psychological Science Under Scrutiny: Recent Challenges and Proposed Solutions
(2017) · Wiley eBooks · doi:10.1002/9781119095910 · Cited by 110
An asocial psychology and a misdirected clinical psychology.
Sarason, S. B. (1981) · American Psychologist · doi:10.1037/0003-066x.36.8.827 · Cited by 108
Unity in psychology: Possibility or pipedream?
(2005) · American Psychological Association eBooks · doi:10.1037/10847-000 · Cited by 101
Donnellan, M. B., Lucas, R. E., & Cesario, J. (2015) · Emotion · doi:10.1037/a0036079 · Cited by 51
Biotech giant publishes failures to confirm high-profile science
Baker, M. (2016) · Nature · doi:10.1038/nature.2016.19269 · Cited by 47 · cited 1×
“Large-scale failed replication efforts are not unique to psychological science, a point made especially salient in recent replication failures of biomedical research by large pharmaceutical (e.g., Bayer) and biotechnology (e.g., Amgen) firms (Baker, 2016; Owens, 2011).”
BackgroundWilliams, B. R., Strauss, E. H., Hultsch, D. F., Hunter, M. A., & Tannock, R. (2007) · Journal of Clinical and Experimental Neuropsychology · doi:10.1080/13803390600678020 · Cited by 34
Clinical Case Studies Are Important in the Science and Practice of Psychotherapy
Lazarus, A. A. (2012) · doi:10.4324/9780203942789-14 · Cited by 10
Research works printed but not linked: 3
- Davison G. C., 2007, The great ideas of clinical science: 17 principles that every mental health professional should understand, 149
- Jonas K. J., 2015, Comprehensive Results in Social Psychology, 1, 1–7
- Lindsay S., APS Observer
Cited by
All for One and One for All: Mental Disorders in One Dimension
Caspi, A., & Moffitt, T. E. (2018) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2018.17121383 · Cited by 918
“This evidence has involved bothdirect and conceptual replications (27).”
BackgroundSupervised Machine Learning: A Brief Primer
Jiang, T., Gradus, J. L., & Rosellini, A. J. (2020) · Behavior Therapy · doi:10.1016/j.beth.2020.05.002 · Cited by 847
“This challenge has considerable overlap with current issues surrounding replication and reproducibility in psychological science (Tackett et al., 2017).”
BackgroundPsychology, Science, and Knowledge Construction: Broadening Perspectives from the Replication Crisis
Shrout, P. E., & Rodgers, J. L. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011845 · Cited by 725
“…power from the lower bound of a confidence interval on the effect size, as Maxwell et al. (2015) demonstrated distribution of the unknown effect size to obtain an optimal predictive power estimate (Anderson & Maxwell 2017, McShane & B¨ockenholt 2014, Perugini et al. 2014, Taylor & Muller 1996).”
BackgroundSelf-citationZwaan, R. A., Etz, A., Lucas, R. E., & Donnellan, M. B. (2018) · Behavioral and Brain Sciences · doi:10.1017/s0140525x17001972 · Cited by 444
“, cognitive and social). Consequently, we know much less about the replicability of findings in other domains (e.g., clinical and developmental) let alone how to operationalize replicability in them (Tackett et al., 2017a; Tackett, Brandes, & Reardon, 2018). In these other domains, where data collection is slow an…”
BackgroundFried, E. I., Eidhof, M. B., Palic, S., Costantini, G., Huisman-van Dijk, H. M., Bockting, C. L. H., Engelhard, I., Armour, C., Nielsen, A. B. S., & Karstoft, K.-I. (2018) · Clinical Psychological Science · doi:10.1177/2167702617745092 · Cited by 454
“…Hastie, & Tibshirani, 2008), it is presently unclear whether the findings from these network models replicate across different data sets, a question especially relevant considering the recent attention to replicability in psychology (Open Science Collaboration, 2015; Tackett et al., 2017).”
Background
Metadata
- Authors
- Tackett, J. L., Lilienfeld, S. O., Patrick, C. J., Johnson, S. L., Krueger, R. F., Miller, J. D., Oltmanns, T. F., & Shrout, P. E.
- Affiliations
- Northwestern University (US)
- Emory University (US)
- Florida State University (US)
- University of California, Berkeley (US)
- University of Minnesota (US)
- University of Georgia (US)
- Washington University in St. Louis (US)
- New York University (US)
- Corresponding author
- Tackett, Jennifer L.
- ORCID
- Venue
- Perspectives on Psychological Science
- Publisher
- SAGE Publications
- Published
- 2017-09
- Volume
- 12
- Issue
- 5
- Pages
- 742-756
- ISSN
- 1745-6916 (ISSN-L)
- 1745-6924
- OpenAlex
- W2762387880
- Semantic Scholar
- dcfbdb90a93f74470e096e8125017d2c87da6441
- Cited by
- 251
- References
- 61
- Citations by year
- 2026: 9 · 2025: 12 · 2024: 43 · 2023: 23 · 2022: 23 · 2021: 25 · 2020: 28 · 2019: 44 · 2018: 35 · 2017: 9
- FWCI
- 40.72
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Perspectives on Psychological Science (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › General Psychology › Academic and Historical Perspectives in Psychology
- Keywords
- replication crisis, clinical psychology, counseling psychology, school psychology, research reproducibility, replication failure, research integrity
- MeSH terms
- Communication, Humans, Psychology, Psychology, Experimental, Psychology, Social, Research, Science, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.