Therapist variation within randomised trials of psychotherapy: implications for precision, internal and external validity
The DOI matches its Crossref record; 94 of the 94 references it cites were found in a registry or an index.
Abstract
Nesting of patients within therapists in psychotherapy trials creates an additional level within the design. The multilevel nature of this design has implications for the precision, internal and external validity of estimates of the treatment effect. Prior to or during a trial, psychotherapies are allocated to therapists and therapists are assigned to patients such that the therapist becomes part of the causal pathway from the intervention to the patient. It is therefore important to consider not only the relationship between interventions and patients but also relationships between interventions and therapists and between therapists and patients. Research designs comparing the effects of therapeutic approaches, therapist characteristics and packages of the two can be unified by viewing therapists as an important source of variability within psychotherapy outcome studies. Methodological considerations arising from therapist variation will be discussed, drawing together and building upon the associated psychotherapy and statistical literatures. Parallels will also be made with related designs and methods of analysis.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“To account for therapist clustering, we used a design effect of 1.87 for the intervention group, assuming an average of 30 carers per therapist and an intracluster correlation of 0.03.33”
In the citing paper
“Another limitation is related to potential therapist effects, which has implications for evaluating the treatment outcome (Thompson et al. 2012;Walwyn and Roberts 2010).”
From Trial-Based Cognitive Therapy: Efficacy of a New CBT Approach for Treating Social Anxiety Disorder with Comorbid Depression. (2018) · cites this paper as Walwyn and Roberts 2010
Citations
Cites
WinBUGS - A Bayesian modelling framework: Concepts, structure, and extensibility
Lunn, D. J., Thomas, A., Best, N., & Spiegelhalter, D. (2000) · Statistics and Computing · doi:10.1023/a:1008929526011 · Cited by 5,368
Experimental Design: Procedures for the Behavioral Sciences.
Addelman, S., & Kirk, R. E. (1969) · Journal of the American Statistical Association · doi:10.2307/2283497 · Cited by 5,159
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373
Statistical Principles in Experimental Design
Green, P. E., Winer, B. J., Brown, D. R., & Michels, K. M. (1992) · Journal of Marketing Research · doi:10.2307/3172747 · Cited by 4,333
Elkin, I. (1989) · Archives of General Psychiatry · doi:10.1001/archpsyc.1989.01810110013002 · Cited by 2,300
Boutron, I., Moher, D., Altman, D. G., Schulz, K. F., Ravaud, P., & for the CONSORT Group* (2008) · Annals of Internal Medicine · doi:10.7326/0003-4819-148-4-200802190-00008 · Cited by 2,292
The Great Psychotherapy Debate: Models Methods and Findings
Horvath, A. O. (2002) · Psychotherapy Research · doi:10.1093/ptr/12.1.108 · Cited by 1,705
Goulias, K. (2002) · New Directions in Civil Engineering · doi:10.1201/9781420042283.ch9 · Cited by 1,270
Meta‐Analysis Of Controlled Clinical Trials
Whitehead, A. (2002) · doi:10.1002/0470854200 · Cited by 827
Schnurr, P. P., Friedman, M. J., Engel, C. C., Foa, E. B., Shea, M. T., Chow, B. K., Resick, P. A., Thurston, V., Orsillo, S. M., Haug, R., et al. (2007) · JAMA · doi:10.1001/jama.297.8.820 · Cited by 813
Research works printed but not linked: 14
- Luborsky L., 1952, Menninger Quarterly, 6, 1
- Parloff M., Waskow I., Wolfe B. Research on therapist variables in relation to process and outcome . In Garfield SL, Bergin AE, eds. Handbook of psychotherapy and behavior change. Wiley, New York; 1978 : 233-82.
- Beutler LE, Malik M., Alimohamed S. et al. Therapist variables. In Lambert MJ ed. Bergin and Garfield’s handbook of psychotherapy and behavior change. Wiley & Sons, New York; 2004: 227-306.
- Ricks DF, 1974, Supershrink: Methods of a therapist judged successful on the basis of adult outcome of adolescent patients
- Orlinsky D. , Howard K. Gender and psychotherapeutic outcome. In Brodsky AM, Hare-Muslin RT, eds. Women and psychotherapy. Guilford, New York; 1980: 3-34.
- 10.1037/0022-006X.59.1.20 · DOI
- MRC Health Services and Public Health Board., 2000, A Framework for Development and Evaluation of RCTs for Complex Interventions to Improve Health
- 10.1037/1082-989X.5.4.425 · doi:10.1037/1082-989x.5.4.425 · Paper page
- Elashof, J.D., 2002, nQuery Advisor Version 5.0 User’s Guide
- Rasbash J., 2004, A user’s guide to MLwiN version 2.0
- White PD, 2007, BMC Neurology, 7
- Higgins JPT, 2006, Cochrane handbook for systematic reviews of interventions 4.2.6 [Updated September 2006]
- Brien CJ, 2006, Series B, 68, 571
- 10.1037/0022-006X.54.1.95 · doi:10.1037/0022-006x.54.1.95 · Paper page
Cited by
Goodyer, I. M., Reynolds, S., Barrett, B., Byford, S., Dubicka, B., Hill, J., Holland, F., Kelvin, R., Midgley, N., Roberts, C., et al. (2017) · The Lancet Psychiatry · doi:10.1016/s2215-0366(16)30378-9 · Cited by 259
“25 For the cost-eff ectiveness analysis, we calculated incremental cost-eff ectiveness ratios (the diff erence in mean cost divided by the diff erence in mean eff ect) and explored uncertainty with cost-eff ectiveness acceptability curves, which show the probability that each of the treatments is the…”
InfluentialSelf-citationLivingston, G., Barber, J., Rapaport, P., Knapp, M., Griffin, M., King, D., Livingston, D., Mummery, C., Walker, Z., Hoe, J., et al. (2013) · BMJ · doi:10.1136/bmj.f6276 · Cited by 233
Crawford, M. J., Killaspy, H., Barnes, T. R. E., Barrett, B., Byford, S., Clayton, K., Dinsmore, J., Floyd, S., Hoadley, A., Johnson, T., et al. (2012) · BMJ · doi:10.1136/bmj.e846 · Cited by 230
Livingston, G., Barber, J., Rapaport, P., Knapp, M., Griffin, M., Romeo, R., King, D., Livingston, D., Lewis-Holmes, E., Mummery, C., et al. (2014) · Health Technology Assessment · doi:10.3310/hta18610 · Cited by 192
Fjorback, L. O., Arendt, M., Ørnbøl, E., Walach, H., Rehfeld, E., Schröder, A., & Fink, P. (2013) · Journal of Psychosomatic Research · doi:10.1016/j.jpsychores.2012.09.006 · Cited by 144
“We further adjusted for a number of variables where the choice of adjustment variables was made prior to the analysis and based on previous research [51].”
Method
Metadata
- Authors
- Walwyn, R., & Roberts, C.
- Affiliations
- King's College London (GB)
- University of Manchester (GB)
- Corresponding author
- Roberts, Chris
- Venue
- Statistical Methods in Medical Research
- Publisher
- SAGE Publications
- Published
- 2009-07-16
- Volume
- 19
- Issue
- 3
- Pages
- 291-315
- ISSN
- 0962-2802 (ISSN-L)
- 1477-0334
- OpenAlex
- W2164241727
- Cited by
- 93
- References
- 90
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 6 · 2023: 6 · 2022: 9 · 2021: 2 · 2020: 7 · 2019: 4 · 2018: 7 · 2017: 4 · 2016: 9 · 2015: 6 · 2014: 9 · 2013: 2 · 2012: 10
- FWCI
- 7.06
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Statistical Methods in Medical Research (publishedVersion)
- PubMed (publishedVersion)
- Research Portal (King's College London) (submittedVersion)
- Topics
- Social Sciences › Psychology › Clinical Psychology › Psychotherapy Techniques and Applications
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- multi-level design, nested data, precision, internal validity, generalizability, treatment effect estimation, therapist characteristics, causal pathways, randomized controlled trials
- MeSH terms
- Mental Disorders, Psychotherapy, Reproducibility of Results, Models, Statistical, Randomized Controlled Trials as Topic, Treatment Outcome, Biostatistics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Therapist variation within randomised trials of psychotherapy: implications for precision, internal and external validity. https://app.cochat.ai/papers/10.1177/0962280209105017
- The paper
- Walwyn, R., & Roberts, C. (2010). Therapist variation within randomised trials of psychotherapy: implications for precision, internal and external validity. Statistical Methods in Medical Research. https://doi.org/10.1177/0962280209105017
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.