Biases in research: risk factors for non-replicability in psychotherapy and pharmacotherapy research
Open accessThe DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Abstract
Replicability of findings is an essential prerequisite of research. For both basic and clinical research, however, low replicability of findings has recently been reported. Replicability may be affected by research biases not sufficiently controlled for by the existing research standards. Several biases such as researcher allegiance or selective reporting are well-known for affecting results. For psychotherapy and pharmacotherapy research, specific additional biases may affect outcome (e.g. therapist allegiance, therapist effects or impairments in treatment implementation). For meta-analyses further specific biases are relevant. In psychotherapy and pharmacotherapy research these biases have not yet been systematically discussed in the context of replicability. Using a list of 13 biases as a starting point, we discuss each bias's impact on replicability. We illustrate each bias by selective findings of recent research, showing that (1) several biases are not yet sufficiently controlled for by the presently applied research standards, (2) these biases have a pernicious effect on replicability of findings. For the sake of research credibility, it is critical to avoid these biases in future research. To control for biases and to improve replicability, we propose to systematically implement several measures in psychotherapy and pharmacotherapy research, such as adversarial collaboration (inviting academic rivals to collaborate), reviewing study design prior to knowing the results, triple-blind data analysis (including subjects, investigators and data managers/statisticians), data analysis by other research teams (crowdsourcing), and, last not least, updating reporting standards such as CONSORT or the Template for Intervention Description and Replication (TIDieR).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
PRISMA-P Group, Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., & Stewart, L. A. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-1 · Cited by 26,223
Hoffmann, T. C., Glasziou, P. P., Boutron, I., Milne, R., Perera, R., Moher, D., Altman, D. G., Barbour, V., Macdonald, H., Johnston, M., et al. (2014) · BMJ · doi:10.1136/bmj.g1687 · Cited by 10,548
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Selective publication of antidepressant trials and its influence on apparent efficacy
Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., & Rosenthal, R. (2008) · New England Journal of Medicine · doi:10.1056/nejmsa065779 · Cited by 2,435
Publication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368
Elkin, I. (1989) · Archives of General Psychiatry · doi:10.1001/archpsyc.1989.01810110013002 · Cited by 2,300
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845 · cited 1×
“There is evidence of selective reporting of only favorable results in many areas of research (Chan et al. 2004; Ioannidis, 2005b).”
BackgroundThings I have learned (so far).
Cohen, J. (1990) · American Psychologist · doi:10.1037/0003-066x.45.12.1304 · Cited by 1,781
The Great Psychotherapy Debate: The Evidence for What Makes Psychotherapy Work
Wampold, B. E. (2015) · doi:10.4324/9780203582015 · Cited by 1,773
Contradicted and Initially Stronger Effects in Highly Cited Clinical Research
Ioannidis, J. P. A. (2005) · JAMA · doi:10.1001/jama.294.2.218 · Cited by 1,465
Cited by
Leichsenring, F., Steinert, C., Rabung, S., & Ioannidis, J. P. A. (2022) · World Psychiatry · doi:10.1002/wps.20941 · Cited by 485
Self-citationGrowth mindsets and psychological distress: A meta-analysis
Burnette, J. L., Knouse, L. E., Vavra, D. T., O'Boyle, E., & Brooks, M. A. (2020) · Clinical Psychology Review · doi:10.1016/j.cpr.2020.101816 · Cited by 271
Reid, J. E., Laws, K. R., Drummond, L., Vismara, M., Grancini, B., Mpavaenda, D., & Fineberg, N. A. (2021) · Comprehensive Psychiatry · doi:10.1016/j.comppsych.2021.152223 · Cited by 266
“In their discussion of control choice, Leichsenring & Steinert (2017) [75] remarked correctly that “When examining efficacy, a treatment may be compared with different comparators, that is,with an established treatment, treatment as usual, a placebo, or awaiting list, with decreasing strictness of the empirical test.”
BackgroundSteinert, C., Munder, T., Rabung, S., Hoyer, J., & Leichsenring, F. (2017) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2017.17010057 · Cited by 244
Cochrane’s risk of bias tool in the context of psychotherapy outcome research
Munder, T., & Barth, J. (2018) · Psychotherapy Research · doi:10.1080/10503307.2017.1411628 · Cited by 174
Metadata
- Authors
- Leichsenring, F., Abbass, A., Hilsenroth, M. J., Leweke, F., Luyten, P., Keefe, J. R., Midgley, N., Rabung, S., Salzer, S., & Steinert, C.
- Venue
- Psychological Medicine
- Publisher
- Cambridge University Press (CUP)
- Published
- 2016-12-13
- Volume
- 47
- Issue
- 6
- Pages
- 1000-1011
- ISSN
- 0033-2917 (ISSN-L)
- 1469-8978
- OpenAlex
- W2564414954
- Cited by
- 78
- References
- 41
- FWCI
- 9.78
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Biases in research: risk factors for non-replicability in psychotherapy and pharmacotherapy research. https://app.cochat.ai/papers/10.1017/s003329171600324x
- The paper
- Leichsenring, F., Abbass, A., Hilsenroth, M. J., Leweke, F., Luyten, P., Keefe, J. R., Midgley, N., Rabung, S., Salzer, S., & Steinert, C. (2017). Biases in research: risk factors for non-replicability in psychotherapy and pharmacotherapy research. Psychological Medicine. https://doi.org/10.1017/s003329171600324x
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.