Enhancing treatment outcome of patients at risk of treatment failure: Meta-analytic and mega-analytic review of a psychotherapy quality assurance system.
The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
Abstract
OBJECTIVE: Outcome research has documented worsening among a minority of the patient population (5% to 10%). In this study, we conducted a meta-analytic and mega-analytic review of a psychotherapy quality assurance system intended to enhance outcomes in patients at risk of treatment failure. METHOD: Original data from six major studies conducted at a large university counseling center and a hospital outpatient setting (N = 6,151, mean age = 23.3 years, female = 63.2%, Caucasian = 85%) were reanalyzed to examine the effects of progress feedback on patient outcome. In this quality assurance system, the Outcome Questionnaire-45 was routinely administered to patients to monitor their therapeutic progress and was utilized as part of an early alert system to identify patients at risk of treatment failure. Patient progress feedback based on this alert system was provided to clinicians so that they could intervene before treatment failure occurred. Meta-analytic and mega-analytic approaches were applied in intent-to-treat and efficacy analyses of the effects of feedback interventions. RESULTS: Three forms of feedback interventions-integral elements of this quality assurance system-were effective in enhancing treatment outcome, especially for signal alarm patients. Two of the three feedback interventions were also effective in preventing treatment failure (clinical support tools and the provision of patient progress feedback to therapists). CONCLUSIONS: The current state of evidence appears to support the efficacy and effectiveness of feedback interventions in enhancing treatment outcome.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“therapist supervision and feedback can improve patient outcomes, 52,53”
From Canadian Network for Mood and Anxiety Treatments (CANMAT) 2016 Clinical Guidelines for the Management of Adults with Major Depressive Disorder (2016) · cites this paper as [52]
In the citing paper
“Using MBC results in more clinically effective,17 cost-effective,18 and efficient treatment.19”
From Using Machine Learning to Match Clients and Therapy Providers: Evaluating Clinical Quality and Cost of Care (2025) · cites this paper as [19]
Citations
Cites
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
Jacobson, N. S., & Truax, P. (1991) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.59.1.12 · Cited by 7,770
Distribution Theory for Glass's Estimator of Effect Size and Related Estimators
Hedges, L. V. (1981) · Journal of Educational Statistics · doi:10.2307/1164588 · Cited by 5,695 · cited 1×
“Following recommendations by Overton (1998) and Hedges and Vevea (1998), we used a random-effects model, given that the research on providing feedback for the purpose of enhancing treatment outcome is relatively new in psychotherapy outcome literature, that studies included in this meta-analysis contained slight varia…”
MethodBergin and Garfield's Handbook of Psychotherapy and Behavior Change
Lambert, M. J. (2003) · Cited by 3,383
Self-citationFixed-and random-effects models in meta-analysis.
Hedges, L. V., & Vevea, J. L. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.4.486 · Cited by 2,977
Evidence-based practice in psychology.
American Psychological Association (2006) · American Psychologist · doi:10.1037/0003-066x.61.4.271 · Cited by 2,478
A Fail-Safe N for Effect Size in Meta-Analysis
Orwin, R. G. (1983) · Journal of Educational Statistics · doi:10.2307/1164923 · Cited by 2,064
The psychotherapy dose-response effect and its implications for treatment delivery services.
Hansen, N. B., Lambert, M. J., & Forman, E. M. (2002) · Clinical Psychology: Science and Practice · doi:10.1093/clipsy.9.3.329 · Cited by 783 · cited 1×
“Similarly, an argument can be made against causal statements based on the data of studies not directly compared in the same randomized trials, such as in the case of comparing the CST Fb group and the pooled TAU group in the studies of Harmon et al. (2007) and Slade et al.”
BackgroundSelf-citationÆgisdóttir, S., White, M. J., Spengler, P. M., Maugherman, A. S., Anderson, L. A., Cook, R. S., Nichols, C. N., Lampropoulos, G. K., Walker, B. S., Cohen, G., et al. (2006) · The Counseling Psychologist · doi:10.1177/0011000005285875 · Cited by 737 · cited 2×
“As has been repeatedly demonstrated in clinical research, any prediction relying on statistical or actuarial methods tends to fare better than clinical judgment alone (Ægisdóttir et al., 2006; Grove, 2005).”
Background
Cited by
Items 601–660 of 660
The Ubiquity of Discrepant Results in Research
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.003.0002
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.002.0004
How Discrepant Results Reveal Blueprints for Discovery
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.003.0004
Constructing Measurement Batteries
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.003.0009
The Ubiquity of Discrepant Results in the Clinic
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.003.0017
Distinguishing Domain-Relevant Information From Bias and Noise
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.003.0010
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.003.0022
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.002.0008
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.003.0021 · Cited by 1
Insights About Data Sources, Validation Testing, and Theory
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.003.0013
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.002.0006
The Power of Discrepant Results: The Dark Matter in Mental Health Research
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.003.0024
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.003.0007
The Errors of Yesterday and the Dark Matter of Today
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.003.0001
Should Discrepant Results Promote a Crisis of Confidence?
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.003.0003
The Needs-to-Goals Gap Framework
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.003.0018
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.003.0020
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.002.0011
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.002.0003
How the Operations Triad System Informs Assessing Clients
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.003.0019
Structurally Different Informants, Triangulation, and Falsifiability
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.003.0005
Research, Education, and Training
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.003.0016
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.003.0015
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.002.0009
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.003.0014
The CONTEXT Validation Paradigm
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.003.0008
The Operations Triad System and Connections to the History of Science
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.003.0011
Study Design and Measurement Validation Battery
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.003.0012
Effectiveness of group analytic and psychodynamic psychotherapy in routine secondary-care practice
Story, G. W., Dalton, F., Selway, A. G., Caparrotta, L., O’Reilly, J., Colam, A., Larney, A., Ritchie, S., & Hillen, T. (2024) · Group Analysis · doi:10.1177/05333164241257239
“…indicate that building such feed-back loops into everyday clinical practice ‘improves’ responses to psychotherapy (e.g. Hawkins et al., 2004; Slade et al., 2008; Brattland et al., 2018, De Jong et al., 2014; Carlier et al., 2012; for meta-analyses see Shimokawa et al., 2010; Fortney et al., 2017).”
A Scoping Review on Client-Focused Feedback System in Psychotherapy: Routine Outcome Monitoring
ÖZTÜRK, İ., & Yorulmaz, O. (2024) · Psikiyatride Guncel Yaklasimlar - Current Approaches in Psychiatry · doi:10.18863/pgy.1414793
“It has been argued that while there is evidence for the effectiveness of ROM (e.g. Shimokawa et al. 2010, Delgadillo et al. 2018), this evidence is mostly from randomized controlled trials and meta-analyses conducted by the same researchers (Luborsky et al. 1999).”
BackgroundInfluentialGorban, C., Chong, M. K., Poulsen, A., Turner, A., LaMonica, H. M., McKenna, S., Scott, E. M., Hickie, I. B., & Iorfino, F. (2024) · doi:10.2196/preprints.70154
Validation of an Outcome Tracking System for Use in Psychology Training Clinics
Kilmer, E. D. (2020) · University of North Texas Digital Library (University of North Texas) · doi:10.12794/metadc1707394
Bugatti, M. (2020) · Scholars Archive - University at Albany (University at Albany, State University of New York) · doi:10.54014/8m9q-17ee
Introduction to Feedback-Informed Psychological Therapy
Rubel, J. A., & Lutz, W. (2025) · CBT: science into practice · doi:10.1007/978-3-031-90428-8_1
Clinical Support Tools for Treatment Adaptation in Non-improving Cases
Delgadillo, J., de Jong, K., & Barkham, M. (2025) · CBT: science into practice · doi:10.1007/978-3-031-90428-8_6 · Cited by 1
Outcome and Therapeutic Relationship Feedback in Group Therapy
Paxton, T., Griner, D., Beecher, M. E., & Burlingame, G. M. (2025) · CBT: science into practice · doi:10.1007/978-3-031-90428-8_11
Routine Outcome Monitoring With Feedback
Drisko, J. W. (2025) · Families in Society The Journal of Contemporary Social Services · doi:10.1177/10443894251345187
Status of Technologies: Digital Tools
Baldetti, M. (2024) · Emerald Group Publishing Limited eBooks · doi:10.1108/979-8-88730-594-320251004
Further Validation of the Couple Relationship Scale
Starken, P., Stratton, P., Kieran, A., Sherbersky, H., Shepherd, M., Galbraith, M., Whittinger, N., Pomini, V., Teh, Y. Y., Nearchou, F., et al. (2025) · Journal of Marital and Family Therapy · doi:10.1111/jmft.70100 · Cited by 1
“Incorporating such feedback into clinical practice has been shown to enhance therapeutic outcomes, strengthen client engagement, and improve the therapeutic alliance (Reese et al. 2010; Shimokawa et al. 2010).”
Connors, E. H., Lyon, A. R., Garcia, K. A., Sichel, C. E., Hoover, S., Weist, M. D., & Tebes, J. K. (2022) · Figshare · doi:10.6084/m9.figshare.20115798
Martinez, R. G., Weiner, B. J., Meza, R. D., Dorsey, S., Palazzo, L. G., Matson, A., Bain, C. E., Mettert, K. D., Pullmann, M. D., & Lewis, C. C. (2023) · Figshare · doi:10.6084/m9.figshare.24655904
Williams, N. J., Ehrhart, M. G., Aarons, G. A., Esp, S. M., Sklar, M., Carandang, K. M., Vega, N. R., Brookman-Frazee, L. I., & Marcus, S. C. (2024) · Figshare · doi:10.6084/m9.figshare.25506625
Beck, A. K., Hides, L. M., Stirling, R., Larance, B., Campbell, G., Baker, A., Hudson, S., Marsden, J., Pocuca, N., Connor, J. P., et al. (2025) · International Journal of Drug Policy · doi:10.1016/j.drugpo.2025.105071 · Cited by 1
Let's Talk About Us: Insights Into the Impact of Routine Outcome Monitoring on Therapeutic Dyads
Demir, S., Wrase, P., Brakemeier, E., & Kaiser, T. (2025) · Counselling and Psychotherapy Research · doi:10.1002/capr.70068 · Cited by 3
Johnson, L. N., Wagner, K., Anderson, S. R., & Hunt, Q. A. (2026) · Journal of Marital and Family Therapy · doi:10.1111/jmft.70114
When clients with increasing symptoms report improvement in psychotherapy: a qualitative analysis
Ghelfi, E. A., Top, D. N., Fox, S. T., Goates-Jones, M. K., Ogles, B. M., Bailey, R. J., Brown, B., Erekson, D. M., Bledsoe, A., Dixon, R. S., et al. (2026) · Counselling Psychology Quarterly · doi:10.1080/09515070.2026.2643866
Practice-Based Evidence Approaches Extend the Benefits of Routine Outcome Monitoring
Page, A. C., Hooke, G. R., Cameron, A. G., & Ross, R. J. (2026) · Oxford University Press eBooks · doi:10.1093/9780197748336.003.0012
Moltu, C., Brattland, H., Nordberg, S. S., & McAleavey, A. A. (2026) · Oxford University Press eBooks · doi:10.1093/9780197748336.003.0007
Parenting Interventions for Child Mental Health: The Science and Practice of Measurement-Based Care
Turnell, A. I., Hawes, D. J., Sawrikar, V., Dadds, M. R., & Tully, L. A. (2026) · Clinical Child and Family Psychology Review · doi:10.1007/s10567-026-00561-x · Cited by 1
“…health outcomes in adult populations at risk of non-response and intervention failure (i.e., “not-on-track (NOT)” populations; Lambert et al., 2018; Shimokawa et al., 2010; Rognstad et al., 2023), with the strongest effect sizes ( d = 0.36–0.49) demonstrated in MBC that includes clinical…”
InfluentialCompare, A., Tona, A. L., Masella, C., Olive, M. V., Brugnera, A., Ostwald, V., Carcione, A., Pievani, L., Poletti, B., Tasca, G. A., et al. (2026) · Trials · doi:10.1186/s13063-026-09693-5
“Overall, prior research has shown that the feedback with clinical support tools improved the rates of clinically significant change in Not on Track cases [4, 19, 20].”
InfluentialSorvisto, L., Charura, D., Gabriel, L., Hobman, T., Duncan, C., & Sanmartino, D. (2026) · Counselling and Psychotherapy Research · doi:10.1002/capr.70134
Truitt, J. R., Donohue, B. C., Warhola, Z., Webb, M. J., Jahnke, M., Craig, C. D., Condray, R., & Allen, D. N. (2026) · Clinical Case Studies · doi:10.1177/15346501261457444
“…and subscale scores can be interpreted using the reliable change index (RCI) that allows for objective determination of improvement (decrease) or worsening (increase) of symptoms over time and is represented by the difference between a client’s baseline and follow-up score (Shimokawa et al., 2010).”
Persons, J. B., Zieve, G. G., & Kreit, G. P. (2026) · Behavior Therapy · doi:10.1016/j.beth.2026.07.003
Forand, N. R., Nettiksimmons, J., Hardy, C., Anton, M. T., & Green, B. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1816893
Bioethical guidelines of sexual-attraction change in psychotherapy
Nicolosi, J. (2026) · The New Bioethics · doi:10.1080/20502877.2026.2715213
Telman, L. G. E., Marić, M., Bögels, S. M., & van Steensel, F. J. A. (2026) · Behavioural and Cognitive Psychotherapy · doi:10.1017/s1352465826101416
“Moreover, several studies among adults demonstrated that FIT led to more clinically significant improvements and to the same results in less sessions compared with no FIT (Janse et al ., 2017; Shimokawa et al ., 2010).”
Are Therapists Using Outcome Measures and Does It Matter? A Naturalistic Usage Study
Klundt, J. S. (2014) · ScholarsArchive (Brigham Young University)
Metadata
- Authors
- Shimokawa, K., Lambert, M. J., & Smart, D. W.
- Affiliations
- Brigham Young University (US)
- ORCID
- Lambert, Michael J.
- Venue
- Journal of Consulting and Clinical Psychology
- Published
- 2010
- Volume
- 78
- Issue
- 3
- Pages
- 298-311
- ISSN
- 0022-006X (ISSN-L)
- 1939-2117
- OpenAlex
- W2143726303
- Semantic Scholar
- 94c5e3fb3707d960610a79e604c2b070280c0014
- Cited by
- 657
- References
- 38
- Citations by year
- 2026: 18 · 2025: 23 · 2024: 64 · 2023: 24 · 2022: 37 · 2021: 57 · 2020: 47 · 2019: 47 · 2018: 59 · 2017: 54 · 2016: 45 · 2015: 61 · 2014: 44 · 2013: 31 · 2012: 30
- FWCI
- 26.64
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Consulting and Clinical Psychology (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Brigham Young University
- Topics
- Social Sciences › Psychology › Clinical Psychology › Psychotherapy Techniques and Applications
- Social Sciences › Psychology › Clinical Psychology › Counseling Practices and Supervision
- Social Sciences › Psychology › Clinical Psychology › Counseling, Therapy, and Family Dynamics
- Keywords
- treatment failure, early alert systems, progress feedback, patient outcomes, Outcome Questionnaire-45, clinical support tools, therapist feedback, counseling centers
- MeSH terms
- Adult, Mental Disorders, Feedback, Female, Humans, Male, Psychology, Clinical, Psychotherapy, Quality Assurance, Health Care, Surveys and Questionnaires, Referral and Consultation, Risk, Outcome Assessment, Health Care, Treatment Failure, Disease Progression, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.