It’s Time to Broaden the Replicability Conversation: Thoughts for and From Clinical Psychological Science
The DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index.
Abstract
Psychology is in the early stages of examining a crisis of replicability stemming from several high-profile failures to replicate studies in experimental psychology. This important conversation has largely been focused on social psychology, with some active participation from cognitive psychology. Nevertheless, several other major domains of psychological science—including clinical science—have remained insulated from this discussion. The goals of this article are to (a) examine why clinical psychology and allied fields, such as counseling and school psychology, have not been central participants in the replicability conversation; (b) review concerns and recommendations that are less (or more) applicable to or appropriate for research in clinical psychology and allied fields; and (c) generate take-home messages for scholars and consumers of the literature in clinical psychology and allied fields, as well as reviewers, editors, and colleagues from other areas of psychological science.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Replication of research findings for clinical interventions is important (9), as a pre-condition for evidence-based practice is the replicability of effects (10).”
Citations
Cites
Diagnostic and Statistical Manual of Mental Disorders
American Psychiatric Association (2013) · doi:10.1176/appi.books.9780890425596 · Cited by 116,369
Diagnostic and Statistical Manual of Mental Disorders, 5th ed.
Carter, M. J. (2014) · Therapeutic Recreation Journal · Cited by 15,169
Construct validity in psychological tests.
Cronbach, L. J., & Meehl, P. E. (1955) · Psychological Bulletin · doi:10.1037/h0040957 · Cited by 11,195 · cited 2×
“Mental disorders, along with their putative indicators, are necessarily constructs; they are hypothetical attributes that cannot be perfectly (“operationally”) defined by any set of observable referents (Cronbach & Meehl, 1955).”
BackgroundMethodWhy Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 1×
“One potential benefit of these cultural differences is that it may decrease the likelihood of publication bias stemming from the “file drawer effect” (Rosenthal, 1979).”
BackgroundEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 1×
“…is perhaps surprising given that clinical psychologists frequently collaborate with scholars in domains of psychiatry, neuroimaging, and molecular genetics in which replication dif fi culties have been observed and widely discussed (e.g., Button et al., 2013; Ioannidis & Trikalinos, 2005).”
BackgroundInsel, T., Cuthbert, B., Garvey, M., Heinssen, R., Pine, D. S., Quinn, K., Sanislow, C., & Wang, P. (2010) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2010.09091379 · Cited by 7,067
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
No adjustments are needed for multiple comparisons.
Rothman, K. J. (1990) · Epidemiology · doi:10.1097/00001648-199001000-00010 · Cited by 5,965
Research works printed but not linked: 3
- Davison G. C., 2007, The great ideas of clinical science: 17 principles that every mental health professional should understand, 149
- Jonas K. J., 2015, Comprehensive Results in Social Psychology, 1, 1–7
- Lindsay S., APS Observer
Cited by
All for One and One for All: Mental Disorders in One Dimension
Caspi, A., & Moffitt, T. E. (2018) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2018.17121383 · Cited by 918
“This evidence has involved bothdirect and conceptual replications (27).”
BackgroundSupervised Machine Learning: A Brief Primer
Jiang, T., Gradus, J. L., & Rosellini, A. J. (2020) · Behavior Therapy · doi:10.1016/j.beth.2020.05.002 · Cited by 847
“This challenge has considerable overlap with current issues surrounding replication and reproducibility in psychological science (Tackett et al., 2017).”
BackgroundPsychology, Science, and Knowledge Construction: Broadening Perspectives from the Replication Crisis
Shrout, P. E., & Rodgers, J. L. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011845 · Cited by 725
“…power from the lower bound of a confidence interval on the effect size, as Maxwell et al. (2015) demonstrated distribution of the unknown effect size to obtain an optimal predictive power estimate (Anderson & Maxwell 2017, McShane & B¨ockenholt 2014, Perugini et al. 2014, Taylor & Muller 1996).”
BackgroundSelf-citationZwaan, R. A., Etz, A., Lucas, R. E., & Donnellan, M. B. (2018) · Behavioral and Brain Sciences · doi:10.1017/s0140525x17001972 · Cited by 444
“, cognitive and social). Consequently, we know much less about the replicability of findings in other domains (e.g., clinical and developmental) let alone how to operationalize replicability in them (Tackett et al., 2017a; Tackett, Brandes, & Reardon, 2018). In these other domains, where data collection is slow an…”
BackgroundFried, E. I., Eidhof, M. B., Palic, S., Costantini, G., Huisman-van Dijk, H. M., Bockting, C. L. H., Engelhard, I., Armour, C., Nielsen, A. B. S., & Karstoft, K.-I. (2018) · Clinical Psychological Science · doi:10.1177/2167702617745092 · Cited by 454
“…Hastie, & Tibshirani, 2008), it is presently unclear whether the findings from these network models replicate across different data sets, a question especially relevant considering the recent attention to replicability in psychology (Open Science Collaboration, 2015; Tackett et al., 2017).”
Background
Metadata
- Authors
- Tackett, J. L., Lilienfeld, S. O., Patrick, C. J., Johnson, S. L., Krueger, R. F., Miller, J. D., Oltmanns, T. F., & Shrout, P. E.
- Affiliations
- Northwestern University (US)
- Emory University (US)
- Florida State University (US)
- University of California, Berkeley (US)
- University of Minnesota (US)
- University of Georgia (US)
- Washington University in St. Louis (US)
- New York University (US)
- Corresponding author
- Tackett, Jennifer L.
- ORCID
- Venue
- Perspectives on Psychological Science
- Publisher
- SAGE Publications
- Published
- 2017-09
- Volume
- 12
- Issue
- 5
- Pages
- 742-756
- ISSN
- 1745-6916 (ISSN-L)
- 1745-6924
- OpenAlex
- W2762387880
- Semantic Scholar
- dcfbdb90a93f74470e096e8125017d2c87da6441
- Cited by
- 251
- References
- 61
- Citations by year
- 2026: 9 · 2025: 12 · 2024: 43 · 2023: 23 · 2022: 23 · 2021: 25 · 2020: 28 · 2019: 44 · 2018: 35 · 2017: 9
- FWCI
- 40.72
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Perspectives on Psychological Science (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › General Psychology › Academic and Historical Perspectives in Psychology
- Keywords
- replication crisis, clinical psychology, counseling psychology, school psychology, research reproducibility, replication failure, research integrity
- MeSH terms
- Communication, Humans, Psychology, Psychology, Experimental, Psychology, Social, Research, Science, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: It’s Time to Broaden the Replicability Conversation: Thoughts for and From Clinical Psychological Science. https://app.cochat.ai/papers/10.1177/1745691617690042
- The paper
- Tackett, J. L., Lilienfeld, S. O., Patrick, C. J., Johnson, S. L., Krueger, R. F., Miller, J. D., Oltmanns, T. F., & Shrout, P. E. (2017). It’s Time to Broaden the Replicability Conversation: Thoughts for and From Clinical Psychological Science. Perspectives on Psychological Science. https://doi.org/10.1177/1745691617690042
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.