Researcher allegiance in research on psychosocial interventions: meta-research study protocol and pilot study
Open accessAbstract
Introduction: One potential source of bias in randomised clinical trials of psychological interventions is researcher allegiance (RA). The operationalisation of RA differs strongly across studies, and there is not a generally accepted method of operationalising or measuring it. Furthermore, it remains unclear as to how RA affects the outcomes of trials and if it results in better outcomes for a preferred intervention. The aim of this project is to develop and validate a scale that accurately identifies RA, contribute to the understanding of the impact that RA has in a research setting and to make recommendations for addressing RA in practice. Methods and analysis: A scale will first be developed and validated to measure RA in psychotherapy trials. The scale will be validated by surveying authors of psychotherapy trials to assess their opinions, beliefs and preferences of psychotherapy interventions. Furthermore, the scale will be validated for use outside the field of psychotherapy. The validated checklist will then be used to examine two potential mechanisms of how RA may affect outcomes of interventions: publication bias (by assessing grants) and risk of bias (RoB). Finally, recommendations will be developed, and a feasibility study will be conducted at a national mental health agency in The Netherlands. Main analyses comprise inter-rater reliability of checklist items, correlations to examine the relationship between checklist items and author survey (convergent validity) as well as checklist items and trial outcomes and multivariate meta-regression techniques to assess potential mechanisms of how allegiance affects trial outcomes (publication bias and RoB). Ethics and dissemination: This study has been reviewed and approved by the Scientific and Ethical Review Board (VCWE) at the Vrije Universiteit Amsterdam. Study result and advancements will also be published on the Open Science Framework. Furthermore, main findings will be disseminated through articles in international peer-reviewed open access journals. Results and recommendations will be communicated to the Cochrane Collaboration, the Campbell Collaboration and other funding agencies.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
It is well demonstrated that once participants have recovered with antidepressant treatment, continued pharmacotherapy reduces the risk of relapse or recurrence by half, resulting in NNTs of approximately 4 to prevent a relapse within a year among people at high risk of relapse.
“It is well demonstrated that once participants have recovered with antidepressant treatment, continued pharmacotherapy reduces the risk of relapse or recurrence by half, resulting in NNTs of approximately 4 to prevent a relapse within a year among people at high risk of relapse (for example, those who have experienced multiple episodes). 551”
From Time for united action on depression: a Lancet–World Psychiatric Association Commission (2022) · cites this paper as [551]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Interrater reliability: The kappa statistic
McHugh, M. L. (2012) · Biochemia Medica · doi:10.11613/bm.2012.031 · Cited by 19,877 · see citing works
Distribution Theory for Glass's Estimator of Effect Size and Related Estimators
Hedges, L. V. (1981) · Journal of Educational Statistics · doi:10.2307/1164588 · Cited by 5,695 · see citing works
(2008) · Cited by 3,790 · see citing works
Comparative Studies of Psychotherapies: Is It True That "Everyone Has Won and All Must Have Prizes"?
Luborsky, L. (1975) · Archives of General Psychiatry · doi:10.1001/archpsyc.1975.01760260059004 · Cited by 1,485 · see citing works
Moher, D., Schulz, K. F., Altman, D. G., & for the CONSORT Group* (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-134-8-200104170-00011 · Cited by 1,426 · see citing works
Weisz, J. R., Weiss, B., Han, S. S., Granger, D. A., & Morton, T. (1995) · Psychological Bulletin · doi:10.1037/0033-2909.117.3.450 · Cited by 884 · see citing works
Cristea, I. A., Gentili, C., Cotet, C. D., Palomba, D., Barbui, C., & Cuijpers, P. (2017) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2016.4287 · Cited by 655 · see citing works
Self-citationIs cognitive–behavioral therapy more effective than other therapies?
Tolin, D. F. (2010) · Clinical Psychology Review · doi:10.1016/j.cpr.2010.05.003 · Cited by 645 · see citing works
The researcher's own therapy allegiances: A "wild card" in comparisons of treatment efficacy.
Luborsky, L., Diguer, L., Seligman, D. A., Rosenthal, R., Krause, E. D., Johnson, S., Halperin, G., Bishop, M., Berman, J. S., & Schweizer, E. (1999) · Clinical Psychology: Science and Practice · doi:10.1093/clipsy.6.1.95 · Cited by 572 · see citing works
Cuijpers, P., Cristea, I. A., Karyotaki, E., Reijnders, M., & Huibers, M. J. H. (2016) · World Psychiatry · doi:10.1002/wps.20346 · Cited by 572 · see citing works
Self-citationThe efficacy of non-directive supportive therapy for adult depression: A meta-analysis
Cuijpers, P., Driessen, E., Hollon, S. D., van Oppen, P., Barth, J., & Andersson, G. (2012) · Clinical Psychology Review · doi:10.1016/j.cpr.2012.01.003 · Cited by 369 · see citing works
Self-citationThe researcher's own therapy allegiances: a ''wild card'' in comparisons of treatment efficacy
Luborsky, L. (1999) · Clinical Psychology Science and Practice · doi:10.1093/clipsy/6.1.95 · Cited by 308 · see citing works
Researcher allegiance in psychotherapy outcome research: An overview of reviews
Munder, T., Brütsch, O., Leonhart, R., Gerger, H., & Barth, J. (2013) · Clinical Psychology Review · doi:10.1016/j.cpr.2013.02.002 · Cited by 260 · see citing works
Psychological treatment of depression: A meta-analytic database of randomized studies
Cuijpers, P. J., van Straten, A., Warmerdam, L., & Andersson, G. (2008) · BMC Psychiatry · doi:10.1186/1471-244x-8-36 · Cited by 241 · see citing works
Self-citationResearcher allegiance and meta-analysis: The case of cognitive therapy for depression.
Gaffan, E. A., Tsaousis, I., & Kemp-Wheeler, S. M. (1995) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.63.6.966 · Cited by 194 · see citing works
Allegiance in psychotherapy outcome research: Separating association from bias.
Leykin, Y., & DeRubeis, R. J. (2009) · Clinical Psychology: Science and Practice · doi:10.1111/j.1468-2850.2009.01143.x · Cited by 170 · see citing works
Wampold, B. E., Budge, S. L., Laska, K. M., Del Re, A. C., Baardseth, T. P., Flűckiger, C., Minami, T., Kivlighan, D. M., & Gunn, W. (2011) · Clinical Psychology Review · doi:10.1016/j.cpr.2011.07.012 · Cited by 134 · see citing works
Direct comparisons of treatment modalities for youth disorders: a meta-analysis
Miller, S., Wampold, B., & Varhely, K. (2008) · Psychotherapy Research · doi:10.1080/10503300701472131 · Cited by 126 · see citing works
How to prove that your therapy is effective, even when it is not: a guideline
Cuijpers, P., & Cristea, I. A. (2016) · Epidemiology and Psychiatric Sciences · doi:10.1017/s2045796015000864 · Cited by 122 · see citing works
Self-citationCognitive therapy versus systematic desensitization: Is one treatment superior?
Berman, J. S., Miller, R. C., & Massman, P. J. (1985) · Psychological Bulletin · doi:10.1037/0033-2909.97.3.451 · Cited by 99 · see citing works
Testing the allegiance bias hypothesis: a meta-analysis
Munder, T., Gerger, H., Trelle, S., & Barth, J. (2011) · Psychotherapy Research · doi:10.1080/10503307.2011.602752 · Cited by 91 · see citing works
Psychotherapy Versus Second-Generation Antidepressants in the Treatment of Depression
Spielmans, G. I., Berman, M. I., & Usitalo, A. N. (2011) · The Journal of Nervous and Mental Disease · doi:10.1097/nmd.0b013e31820caefb · Cited by 91 · see citing works
Munder, T., Flückiger, C., Gerger, H., Wampold, B. E., & Barth, J. (2012) · Journal of Counseling Psychology · doi:10.1037/a0029571 · Cited by 79 · see citing works
Dragioti, E., Dimoliatis, I., & Evangelou, E. (2015) · BMJ Open · doi:10.1136/bmjopen-2014-007206 · Cited by 44 · see citing works
Investigator allegiance and the evaluation of psychotherapy outcome research
Berman, J. S., & Reich, C. M. (2010) · European Journal of Psychotherapy & Counselling · doi:10.1080/13642531003637775 · Cited by 23 · see citing works
Ioannidis, J. P. A. (2016) · Epidemiology and Psychiatric Sciences · doi:10.1017/s2045796015000888 · Cited by 22 · see citing works
Allegiance effects in psychotherapy research: A constructivist approach
Botella, L., & Beriain, D. (2010) · European Journal of Psychotherapy & Counselling · doi:10.1080/13642531003637759 · Cited by 6 · see citing works
Countering Cognitive Bias: Tips for Recognizing the Impact of Potential Bias on Research
Lomangino, K. (2015) · Journal of the Academy of Nutrition and Dietetics · doi:10.1016/j.jand.2015.07.014 · Cited by 5 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Time for united action on depression: a Lancet–World Psychiatric Association Commission
Herrman, H., Patel, V., Kieling, C., Berk, M., Buchweitz, C., Cuijpers, P., Furukawa, T. A., Kessler, R. C., Kohrt, B. A., Maj, M., et al. (2022) · The Lancet · doi:10.1016/s0140-6736(21)02141-3 · Cited by 1,104 · see citing works
Self-citationSchliemann, D., Paramasivam, D., Dahlui, M., Cardwell, C. R., Somasundaram, S., Tamin, N. S. B. I., Donnelly, C., Su, T. T., & Donnelly, M. (2020) · BMC Cancer · doi:10.1186/s12885-020-06742-3 · Cited by 49 · see citing works
Reith‐Hall, E., & Montgomery, P. (2023) · Campbell Systematic Reviews · doi:10.1002/cl2.1309 · Cited by 73 · see citing works
Increase value and reduce waste in research on psychological therapies
Cristea, I. A., & Naudet, F. (2019) · Behaviour Research and Therapy · doi:10.1016/j.brat.2019.103479 · Cited by 22 · see citing works
Self-citationMontgomery, P., & Weisman, C. B. (2020) · Children and Youth Services Review · doi:10.1016/j.childyouth.2020.105642 · Cited by 11 · see citing works
Zimmerman, J. A., & Marcus, D. K. (2024) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.2974 · Cited by 10 · see citing works
Carvalho, S., Coelho, C. G., & Leite, J. (2025) · Brain Sciences · doi:10.3390/brainsci15050444 · Cited by 8 · see citing works
Maddox, C.-S. S., & Berle, D. (2024) · Clinical Psychologist · doi:10.1080/13284207.2024.2347643 · Cited by 1 · see citing works
Schünemann, O., Jansen, A., Willutzki, U., & Heinrichs, N. (2025) · Clinical Psychology in Europe · doi:10.32872/cpe.9709 · Cited by 1 · see citing works
Lotty, M., White, E. B., & DunnGalvin, A. (2022) · Child Care in Practice · doi:10.1080/13575279.2022.2121683
Braus, N., Flückiger, C., Gräfenkämper, R., Frankman, C., Wichmann, J., & Hunger‐Schoppe, C. (2026) · Psychotherapy Research · doi:10.1080/10503307.2026.2654029
- Authors
- Yoder, W. R., Karyotaki, E., Cristea, I.-A., van Duin, D., & Cuijpers, P.
- Affiliations
- Vrije Universiteit Amsterdam (NL)
- Babeș-Bolyai University (RO)
- Corresponding author
- Yoder, Whitney Rose
- Venue
- BMJ Open
- Publisher
- BMJ
- Published
- 2019-02
- Volume
- 9
- Issue
- 2
- Pages
- e024622
- ISSN
- 2044-6055 (ISSN-L)
- PMC
- PMC6377509
- OpenAlex
- W2914578659
- Cited by
- 11
- References
- 27
- Citations by year
- 2026: 1 · 2025: 2 · 2024: 2 · 2023: 1 · 2022: 2 · 2020: 2 · 2019: 1
- FWCI
- 0.84
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by-nc
- Locations
- BMJ Open (publishedVersion, cc-by-nc)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- HighWire Press Open Archive (acceptedVersion)
- VU Research Portal (submittedVersion, other-oa)
- Padua Research Archive (University of Padova) (submittedVersion, cc-by-nc)
- Data Archiving and Networked Services (DANS) (submittedVersion, cc-by-nc)
- Funders
- ZonMw
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Clinical Psychology › Psychotherapy Techniques and Applications
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- psychosocial interventions, scale development, scale validation, publication bias, risk of bias, inter-rater reliability, mental health services
- MeSH terms
- Humans, Netherlands, Pilot Projects, Psychotherapy, Regression Analysis, Research Design, Research Personnel, Meta-Analysis as Topic, Multivariate Analysis, Randomized Controlled Trials as Topic, Outcome Assessment, Health Care, Publication Bias, Checklist
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
