Randomized controlled versus naturalistic studies: A new research agenda
Abstract
The present article addresses the question of what kind of evidence is required to demonstrate that a method of psychotherapy works. Referring to recent conceptualizations of the logical structure of scientific theories, that is, the structuralistic view of theories, the author shows that randomized controlled studies (RCTs) and naturalistic studies (effectiveness studies) refer to different domains of intended applications (laboratory vs. field). This view has several important implications: (1) RCTs and naturalistic studies do not differ concerning their internal and external validity; (2) naturalistic studies do not necessarily provide lower-level evidence than RCTs; (3) evidence from RCTs cannot be transferred to psychotherapeutic practice in the field; (4) naturalistic studies are required to demonstrate that a form of therapy works in the field; (5) The proposed catalogues for levels of evidence focus on RCTs; thus, they cannot be applied to the question if a therapy works in the field; (6) It is necessary to define separate criteria for levels of evidence of naturalistic studies; and (7) a new research agenda for naturalistic studies can be derived, which is analogous to that of efficacy studies. In this article, a proposal is made to define levels of evidence of naturalistic studies. A gold standard for naturalistic studies is proposed.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Compared to RCTs, effectiveness studies tend to include more patients with comorbid disorders.
“For instance, compared to RCTs, effectiveness studies tend to include more patients with comorbid disorders (Leichsenring, 2004), and therefore, serve a more heterogeneous patient population (Knott et al., 2015; Waller et al., 2014).”
From Effectiveness of web-based guided self-help cognitive behavioral therapy-enhanced for binge-eating disorder: An implementation study (2024) · cites this paper as Leichsenring, 2004
RCTs generally include less heterogeneous populations and adhere strictly to treatment protocols compared with real-world clinical settings.
“RCTs generally include less heterogenous populations, and treatment protocols are strictly adhered to compared to in real-world clinical settings (Leichsenring, 2004).”
From Effectiveness of web-based guided self-help cognitive behavioral therapy-enhanced for binge-eating disorder: An implementation study (2024) · cites this paper as Leichsenring, 2004
Cites
Defining empirically supported therapies.
Chambless, D. L., & Hollon, S. D. (1998) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.66.1.7 · Cited by 3,067 · see citing works
Empirically supported psychological interventions: Controversies and evidence
Chambless, D. L., & Ollendick, T. H. (2001) · Annual Review of Psychology · doi:10.1146/annurev.psych.52.1.685 · Cited by 2,097 · see citing works
The effectiveness of psychotherapy: the Consumer Reports study
Seligman, M. E. P. (1995) · American Psychologist · doi:10.1037/0003-066x.50.12.965 · Cited by 1,608 · see citing works
A component analysis of cognitive-behavioral treatment for depression.
Jacobson, N. S., Dobson, K. S., Truax, P. A., Addis, M. E., Koerner, K., Gollan, J. K., Gortner, E., & Prince, S. E. (1996) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.64.2.295 · Cited by 1,413 · see citing works
Users' Guides to the Medical Literature: IX. A Method for Grading Health Care Recommendations
Guyatt, G. H. (1995) · JAMA · doi:10.1001/jama.1995.03530220066035 · Cited by 846 · see citing works
Wells, K. B. (1999) · American Journal of Psychiatry · doi:10.1176/ajp.156.1.5 · Cited by 390 · see citing works
Are results of randomized controlled trials useful to psychotherapists?
Persons, J. B., & Silberschatz, G. (1998) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.66.1.126 · Cited by 233 · see citing works
Predictors of Response to Cognitive-Behavioral Group Therapy for Social Phobia
Chambless, D. L., Tran, G. Q., & Glass, C. R. (1997) · Journal of Anxiety Disorders · doi:10.1016/s0887-6185(97)00008-x · Cited by 228 · see citing works
Comorbid anxiety and mood disorders among persons with social anxiety disorder
Erwin, B. A., Heimberg, R. G., Juster, H., & Mindlin, M. (2002) · Behaviour Research and Therapy · doi:10.1016/s0005-7967(00)00114-5 · Cited by 153 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
National Survey of Psychotherapy Training in Psychiatry, Psychology, and Social Work
Weissman, M. M., Verdeli, H., Gameroff, M. J., Bledsoe, S. E., Betts, K., Mufson, L., Fitterling, H., & Wickramaratne, P. (2006) · Archives of General Psychiatry · doi:10.1001/archpsyc.63.8.925 · Cited by 422 · see citing works
The Effectiveness of Long-Term Psychoanalytic Therapy: A Systematic Review of Empirical Studies
de Maat, S., de Jonghe, F., Schoevers, R., & Dekker, J. (2009) · Harvard Review of Psychiatry · doi:10.1080/10673220902742476 · Cited by 190 · see citing works
Patients' feedback after computer-assisted diagnostic interviews for mental disorders
Hoyer, J., Ruhl, U., Scholz, D., & Wittchen, H.-U. (2006) · Psychotherapy Research · doi:10.1080/10503300500485540 · Cited by 26 · see citing works
Stålsett, G., Gude, T., Rønnestad, M. H., & Monsen, J. T. (2012) · Psychotherapy Research · doi:10.1080/10503307.2012.692214 · Cited by 26 · see citing works
The effectiveness of long-term psychotherapy: Methodological research issues
Maat, S. D., Dekker, J., Schoevers, R., & Jonghe, F. D. (2007) · Psychotherapy Research · doi:10.1080/10503300600607605 · Cited by 21 · see citing works
Melisse, B., Dekker, J., van den Berg, E., de Jonge, M., van Furth, E. F., Peen, J., & de Beurs, E. (2022) · The Cognitive Behaviour Therapist · doi:10.1017/s1754470x22000174 · Cited by 16 · see citing works
Melisse, B., van den Berg, E., & de Beurs, E. (2024) · International Journal of Eating Disorders · doi:10.1002/eat.24079 · Cited by 11 · see citing works
- Authors
- Leichsenring, F.
- Affiliations
- University of Göttingen (DE)
- Corresponding author
- Leichsenring, Falk
- ORCID
- Leichsenring, Falk
- Venue
- Bulletin of the Menninger Clinic
- Publisher
- Guilford Publications
- Published
- 2004-06
- Volume
- 68
- Issue
- 2
- Pages
- 137-151
- ISSN
- 0025-9284 (ISSN-L)
- OpenAlex
- W2118764238
- Cited by
- 183
- References
- 9
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 4 · 2023: 4 · 2022: 4 · 2021: 13 · 2020: 11 · 2019: 6 · 2018: 6 · 2017: 7 · 2016: 10 · 2015: 6 · 2014: 15 · 2013: 12 · 2012: 5
- FWCI
- 9.06
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Bulletin of the Menninger Clinic (publishedVersion)
- PubMed (publishedVersion)
- GRO (submittedVersion)
- Topics
- Social Sciences › Psychology › Clinical Psychology › Psychotherapy Techniques and Applications
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Keywords
- randomized controlled trials, naturalistic study, effectiveness research, level of evidence, structuralism, scientific theories, internal validity, generalizability, research agenda, gold standard, efficacy versus effectiveness
- MeSH terms
- Humans, Psychoanalytic Therapy, Randomized Controlled Trials as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
