Enhancing treatment outcome of patients at risk of treatment failure: Meta-analytic and mega-analytic review of a psychotherapy quality assurance system.
The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
Abstract
OBJECTIVE: Outcome research has documented worsening among a minority of the patient population (5% to 10%). In this study, we conducted a meta-analytic and mega-analytic review of a psychotherapy quality assurance system intended to enhance outcomes in patients at risk of treatment failure. METHOD: Original data from six major studies conducted at a large university counseling center and a hospital outpatient setting (N = 6,151, mean age = 23.3 years, female = 63.2%, Caucasian = 85%) were reanalyzed to examine the effects of progress feedback on patient outcome. In this quality assurance system, the Outcome Questionnaire-45 was routinely administered to patients to monitor their therapeutic progress and was utilized as part of an early alert system to identify patients at risk of treatment failure. Patient progress feedback based on this alert system was provided to clinicians so that they could intervene before treatment failure occurred. Meta-analytic and mega-analytic approaches were applied in intent-to-treat and efficacy analyses of the effects of feedback interventions. RESULTS: Three forms of feedback interventions-integral elements of this quality assurance system-were effective in enhancing treatment outcome, especially for signal alarm patients. Two of the three feedback interventions were also effective in preventing treatment failure (clinical support tools and the provision of patient progress feedback to therapists). CONCLUSIONS: The current state of evidence appears to support the efficacy and effectiveness of feedback interventions in enhancing treatment outcome.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“therapist supervision and feedback can improve patient outcomes, 52,53”
From Canadian Network for Mood and Anxiety Treatments (CANMAT) 2016 Clinical Guidelines for the Management of Adults with Major Depressive Disorder (2016) · cites this paper as [52]
In the citing paper
“Using MBC results in more clinically effective,17 cost-effective,18 and efficient treatment.19”
From Using Machine Learning to Match Clients and Therapy Providers: Evaluating Clinical Quality and Cost of Care (2025) · cites this paper as [19]
Citations
Cites
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
Jacobson, N. S., & Truax, P. (1991) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.59.1.12 · Cited by 7,770
Distribution Theory for Glass's Estimator of Effect Size and Related Estimators
Hedges, L. V. (1981) · Journal of Educational Statistics · doi:10.2307/1164588 · Cited by 5,695 · cited 1×
“Following recommendations by Overton (1998) and Hedges and Vevea (1998), we used a random-effects model, given that the research on providing feedback for the purpose of enhancing treatment outcome is relatively new in psychotherapy outcome literature, that studies included in this meta-analysis contained slight varia…”
MethodBergin and Garfield's Handbook of Psychotherapy and Behavior Change
Lambert, M. J. (2003) · Cited by 3,383
Self-citationFixed-and random-effects models in meta-analysis.
Hedges, L. V., & Vevea, J. L. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.4.486 · Cited by 2,977
Evidence-based practice in psychology.
American Psychological Association (2006) · American Psychologist · doi:10.1037/0003-066x.61.4.271 · Cited by 2,478
A Fail-Safe N for Effect Size in Meta-Analysis
Orwin, R. G. (1983) · Journal of Educational Statistics · doi:10.2307/1164923 · Cited by 2,064
The psychotherapy dose-response effect and its implications for treatment delivery services.
Hansen, N. B., Lambert, M. J., & Forman, E. M. (2002) · Clinical Psychology: Science and Practice · doi:10.1093/clipsy.9.3.329 · Cited by 783 · cited 1×
“Similarly, an argument can be made against causal statements based on the data of studies not directly compared in the same randomized trials, such as in the case of comparing the CST Fb group and the pooled TAU group in the studies of Harmon et al. (2007) and Slade et al.”
BackgroundSelf-citationÆgisdóttir, S., White, M. J., Spengler, P. M., Maugherman, A. S., Anderson, L. A., Cook, R. S., Nichols, C. N., Lampropoulos, G. K., Walker, B. S., Cohen, G., et al. (2006) · The Counseling Psychologist · doi:10.1177/0011000005285875 · Cited by 737 · cited 2×
“As has been repeatedly demonstrated in clinical research, any prediction relying on statistical or actuarial methods tends to fare better than clinical judgment alone (Ægisdóttir et al., 2006; Grove, 2005).”
Background
Cited by
Implementing Measurement-Based Care in Behavioral Health: A Review
Lewis, C. C., Boyd, M., Puspitasari, A., Navarro, E., Howard, J., Kassab, H., Hoffman, M., Scott, K., Lyon, A., Douglas, S., et al. (2019) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2018.3329 · Cited by 605
“Measurement-based care can be separated into 4 core components: (1) a routinely administered symptom, outcome, or process measure (ie, PRO), ideally before each clinical encounter; (2) practitioner review of data; (3) patient review of data; and (4) collaborative reevaluation of the treatment plan informed by data.”
Parikh, S. V., Quilty, L. C., Ravitz, P., Rosenbluth, M., Pavlova, B., Grigoriadis, S., Velyvis, V., Kennedy, S. H., Lam, R. W., MacQueen, G. M., et al. (2016) · The Canadian Journal of Psychiatry · doi:10.1177/0706743716659418 · Cited by 562
A Tipping Point for Measurement-Based Care
Fortney, J. C., Unützer, J., Wrenn, G., Pyne, J. M., Smith, G. R., Schoenbaum, M., & Harbin, H. T. (2017) · Psychiatric Services · doi:10.1176/appi.ps.201500439 · Cited by 558
Lindahl, J. R., Fisher, N. E., Cooper, D. J., Rosen, R. K., & Britton, W. B. (2017) · PLOS ONE · doi:10.1371/journal.pone.0176239 · Cited by 476
Implementing routine outcome monitoring in clinical practice: Benefits, challenges, and solutions
Boswell, J. F., Kraus, D. R., Miller, S. D., & Lambert, M. J. (2015) · Psychotherapy Research · doi:10.1080/10503307.2013.817696 · Cited by 494
“The meta-analysis (Shimokawa et al., 2010) involved both intent-to-treat (ITT) and efficacy analyses on the effects of various feedback interventions in relation to TAU (treatment without feedback) on clients who were predicted to have a negative outcome.”
MethodSelf-citationLambert, M. J., & Shimokawa, K. (2011) · Psychotherapy · doi:10.1037/a0022238 · Cited by 452
“%) than studies based on the OQ system (11% to 33%; Shimokawa et al., 2010).”
BackgroundMethodInfluentialSelf-citationExpanding the lens of evidence-based practice in psychotherapy: A common factors perspective.
Laska, K. M., Gurman, A. S., & Wampold, B. E. (2014) · Psychotherapy · doi:10.1037/a0034332 · Cited by 443
Collecting and delivering progress feedback: A meta-analysis of routine outcome monitoring.
Lambert, M. J., Whipple, J. L., & Kleinstäuber, M. (2018) · Psychotherapy · doi:10.1037/pst0000167 · Cited by 392
Self-citationLam, R. W., Kennedy, S. H., Adams, C., Bahji, A., Beaulieu, S., Bhat, V., Blier, P., Blumberger, D. M., Brietzke, E., Chakrabarty, T., et al. (2024) · The Canadian Journal of Psychiatry · doi:10.1177/07067437241245384 · Cited by 369
de Jong, K., Conijn, J. M., Gallagher, R. A. V., Reshetnikova, A. S., Heij, M., & Lutz, M. C. (2021) · Clinical Psychology Review · doi:10.1016/j.cpr.2021.102002 · Cited by 417
Annual Research Review: Building a science of personalized intervention for youth mental health
Ng, M. Y., & Weisz, J. R. (2016) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.12470 · Cited by 291
Outcome in psychotherapy: The past and important advances.
Lambert, M. J. (2013) · Psychotherapy · doi:10.1037/a0030682 · Cited by 286
Self-citation(2021) · Research Design in Clinical Psychology · doi:10.1017/9781108993647.010 · Cited by 276
Expertise in psychotherapy: An elusive goal?
Tracey, T. J. G., Wampold, B. E., Lichtenberg, J. W., & Goodyear, R. K. (2014) · American Psychologist · doi:10.1037/a0035099 · Cited by 265
de Jong, K., van Sluis, P., Nugter, M. A., Heiser, W. J., & Spinhoven, P. (2012) · Psychotherapy Research · doi:10.1080/10503307.2012.673023 · Cited by 247
“The effect sizes of the various feedback conditions compared to the control conditions ranged from .16 to .70 in the full sample (Shimokawa et al., 2010).”
BackgroundWeisz, J. R., Ng, M. Y., & Bearman, S. K. (2014) · Clinical Psychological Science · doi:10.1177/2167702613501307 · Cited by 246
Research Design in Clinical Psychology
Kazdin, A. E. (2021) · doi:10.1017/9781108993647 · Cited by 240
Kendrick, T., El-Gohary, M., Stuart, B., Gilbody, S., Churchill, R., Aiken, L., Bhattacharya, A., Gimson, A., Brütt, A. L., de Jong, K., et al. (2016) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011119.pub2 · Cited by 237
Evidence-Based Therapy Relationships
Norcross, J. C., & Lambert, M. J. (2011) · Psychotherapy Relationships That Work · doi:10.1093/acprof:oso/9780199737208.003.0001 · Cited by 220
Self-citationEvidence-Based Youth Psychotherapy in the Mental Health Ecosystem
Weisz, J. R., Ugueto, A. M., Cheron, D. M., & Herren, J. (2013) · Journal of Clinical Child & Adolescent Psychology · doi:10.1080/15374416.2013.764824 · Cited by 228
Single-case experimental designs. Evaluating interventions in research and clinical practice
Kazdin, A. E. (2019) · Behaviour Research and Therapy · doi:10.1016/j.brat.2018.11.015 · Cited by 217
Walz, L. C., Nauta, M. H., & aan het Rot, M. (2014) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2014.09.022 · Cited by 195
Lutz, W., Rubel, J. A., Schwartz, B., Schilling, V., & Deisenhofer, A.-K. (2019) · Behaviour Research and Therapy · doi:10.1016/j.brat.2019.103438 · Cited by 196
Routine Outcome Monitoring (ROM) and Feedback: Research Review and Recommendations
Barkham, M., De Jong, K., Delgadillo, J., & Lutz, W. (2023) · Psychotherapy Research · doi:10.1080/10503307.2023.2181114 · Cited by 169
“…al., 2015; Mackrill & Sørensen, 2020) and at least 11 meta-analyses (Bergman et al., 2018; De Jong et al., 2021; Kendrick et al., 2016; Knaup et al., 2009; Lambert et al., 2003, 2018; Østergård et al., 2020; Pejtersen et al., 2020; Rognstad et al., 2022; Shimokawa et al., 2010; Tam & Ronan, 2017).”
InfluentialAdvancing psychotherapy and evidence‐based psychological interventions
Emmelkamp, P. M. G., David, D., Beckers, T., Muris, P., Cuijpers, P., Lutz, W., Andersson, G., Araya, R., Banos Rivera, R. M., Barkham, M., et al. (2014) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.1411 · Cited by 164
“These kind of “real time” feedback systems lead to more treatment sessions and better final outcome for patients at risk for negative outcome and shorter treatments for early responders (e.g. Shimokawa et al., 2010).”
BackgroundDelgadillo, J., de Jong, K., Lucock, M., Lutz, W., Rubel, J., Gilbody, S., Ali, S., Aguirre, E., Appleton, M., Nevin, J., et al. (2018) · The Lancet Psychiatry · doi:10.1016/s2215-0366(18)30162-7 · Cited by 152
Krägeloh, C. U., Czuba, K. J., Billington, D. R., Kersten, P., & Siegert, R. J. (2015) · Psychiatric Services · doi:10.1176/appi.ps.201400141 · Cited by 143
Improving Community-Based Mental Health Care for Children: Translating Knowledge into Action
Garland, A. F., Haine-Schlagel, R., Brookman-Frazee, L., Baker-Ericzen, M., Trask, E., & Fawley-King, K. (2013) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-012-0450-8 · Cited by 142
“Use of outcome monitoring systems has demonstrated a positive impact on treatment effectiveness and efficiency for adult and child/family services (Lambert and Shimokawa 2011; Shimokawa et al. 2010; Timimi et al. 2012).”
Weisz, J. R., Krumholz, L. S., Santucci, L., Thomassin, K., & Ng, M. Y. (2015) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-032814-112820 · Cited by 139
“Emerging evidence indicates that such monitoring and feedback systems may enhance the effectiveness of psychotherapy even when ESTs are not being used (Bickman et al. 2011, Shimokawa et al. 2010).”
BackgroundSimon, W., Lambert, M. J., Harris, M. W., Busath, G., & Vazquez, A. (2012) · Psychotherapy Research · doi:10.1080/10503307.2012.698918 · Cited by 137
“Shimokawa et al. (2010) have reported that feedback interventions (coming from our BYU group) that rely on progress feedback (with alarm-signals) and the use of Clinical Support Tools by clinicians reduce failures rates and increase positive outcomes. A majority of studies (five of six) were conducted in the same univ…”
BackgroundInfluentialSelf-citationDelgadillo, J., Moreea, O., & Lutz, W. (2016) · Behaviour Research and Therapy · doi:10.1016/j.brat.2016.02.003 · Cited by 157
The responsiveness problem in psychotherapy: A review of proposed solutions.
Kramer, U., & Stiles, W. B. (2015) · Clinical Psychology: Science and Practice · doi:10.1111/cpsp.12107 · Cited by 134
Progress feedback and the OQ-system: The past and the future.
Lambert, M. J. (2015) · Psychotherapy · doi:10.1037/pst0000027 · Cited by 133
“Because the Shimokawa et al. (2010) mega/meta-analysis was published six additional studies have been published.”
BackgroundInfluentialSelf-citationMeasurement-Based and Data-Informed Psychological Therapy
Lutz, W., Schwartz, B., & Delgadillo, J. (2022) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-071720-014821 · Cited by 127
Maximizing Psychotherapy Outcome beyond Evidence-Based Medicine
Lambert, M. J. (2017) · Psychotherapy and Psychosomatics · doi:10.1159/000455170 · Cited by 123
Self-citationGoldberg, S. B., Babins-Wagner, R., Rousmaniere, T., Berzins, S., Hoyt, W. T., Whipple, J. L., Miller, S. D., & Wampold, B. E. (2016) · Psychotherapy · doi:10.1037/pst0000060 · Cited by 126
Iorfino, F., Cross, S. P., Davenport, T., Carpenter, J. S., Scott, E., Shiran, S., & Hickie, I. B. (2019) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2019.00595 · Cited by 125
Early changes, attrition, and dose–response in low intensity psychological interventions
Delgadillo, J., McMillan, D., Lucock, M., Leach, C., Ali, S., & Gilbody, S. (2014) · British Journal of Clinical Psychology · doi:10.1111/bjc.12031 · Cited by 122
The Partners for Change Outcome Management System (PCOMS): The Heart and Soul of Change Project.
Duncan, B. L. (2012) · Canadian Psychology / Psychologie canadienne · doi:10.1037/a0027762 · Cited by 117
“In a meta-analytic review of the Outcome Questionnaire 45.2 (OQ) system, Shimokawa, Lambert, and Smart (2010) reanalyzed the combined dataset ( N (cid:2) 6,151) from all six feedback studies (Harmon et al., 2007; Hawkins, Lambert, Vermeersch, Slade, & Tuttle, 2004; Lambert et al., 2001; Lambert et…”
InfluentialInterpersonal competencies: Responsiveness, technique, and training in psychotherapy.
Hatcher, R. L. (2015) · American Psychologist · doi:10.1037/a0039803 · Cited by 135
Patient-focused and feedback research in psychotherapy: Where are we and where do we want to go?
Lutz, W., De Jong, K., & Rubel, J. (2015) · Psychotherapy Research · doi:10.1080/10503307.2015.1079661 · Cited by 112
“…the debate about giving feedback directly to patients compared to providing feedback just to therapists resulted in mixed results and it is still unclear whether giving formal written feedback to patients results in a further enhancement of outcome (De Jong et al., 2014; Shimokawa et al., 2010).”
MethodResultInfluentialEvidence‐based couple therapy: current status and future directions
Snyder, D. K., & Halford, W. K. (2012) · Journal of Family Therapy · doi:10.1111/j.1467-6427.2012.00599.x · Cited by 134
Forsell, E., Jernelöv, S., Blom, K., Kraepelien, M., Svanborg, C., Andersson, G., Lindefors, N., & Kaldo, V. (2019) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2018.18060699 · Cited by 100
De Jong, K., Timman, R., Hakkaart-Van Roijen, L., Vermeulen, P., Kooiman, K., Passchier, J., & Busschbach, J. V. (2014) · Psychotherapy Research · doi:10.1080/10503307.2013.871079 · Cited by 100
“Their studies typically demonstrate that feedback is most effective for NOT cases (Lambert et al., 2003; Shimokawa et al., 2010).”
BackgroundInfluentialTrajectories of depression and anxiety symptom change during psychological therapy
Saunders, R., Buckman, J. E. J., Cape, J., Fearon, P., Leibowitz, J., & Pilling, S. (2019) · Journal of Affective Disorders · doi:10.1016/j.jad.2019.02.043 · Cited by 99
The effect of alliance-focused training on a cognitive-behavioral therapy for personality disorders.
Muran, J. C., Safran, J. D., Eubanks, C. F., & Gorman, B. S. (2018) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000284 · Cited by 113
Tam, H. E., & Ronan, K. (2017) · Clinical Psychology Review · doi:10.1016/j.cpr.2017.04.005 · Cited by 96
Webb, C. A., Cohen, Z. D., Beard, C., Forgeard, M., Peckham, A. D., & Björgvinsson, T. (2020) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000451 · Cited by 96
“First, prior research in outpatient samples (Carlier et al., 2012; Knaup et al., 2009; Shimokawa et al., 2010; Kendrick et al., 2016), including a recently published large multi-site psychotherapy trial for depression and anxiety (Delgadillo et al., 2018), indicates that providing clinicians with…”
BackgroundResultFeedback and therapist effects in the context of treatment outcome and treatment length
Lutz, W., Rubel, J., Schiefele, A.-K., Zimmermann, D., Böhnke, J. R., & Wittmann, W. W. (2015) · Psychotherapy Research · doi:10.1080/10503307.2015.1053553 · Cited by 95
“However, early feedback studies were conducted within settings in which relatively short treatments were provided to only moderately impaired patients (e.g., college counseling centers; Newnham & Page, 2010; Lutz et al., 2014; Poston & Hanson, 2010; Shimokawa et al., 2010).”
BackgroundBrattland, H., Koksvik, J. M., Burkeland, O., Gråwe, R. W., Klöckner, C., Linaker, O. M., Ryum, T., Wampold, B., Lara-Cabrera, M. L., & Iversen, V. C. (2018) · Journal of Counseling Psychology · doi:10.1037/cou0000286 · Cited by 94
Supervision As Pedagogy: Attending to Its Essential Instructional and Learning Processes
Goodyear, R. K. (2014) · The Clinical Supervisor · doi:10.1080/07325223.2014.918914 · Cited by 119
Evidence Base Update for Brief, Free, and Accessible Youth Mental Health Measures
Becker‐Haimes, E. M., Tabachnick, A. R., Last, B. S., Stewart, R. E., Hasan-Granier, A., & Beidas, R. S. (2019) · Journal of Clinical Child & Adolescent Psychology · doi:10.1080/15374416.2019.1689824 · Cited by 106
Enrique, A., Palacios, J. E., Ryan, H., & Richards, D. (2019) · Journal of Medical Internet Research · doi:10.2196/12775 · Cited by 81
“One of these strategies could be outcome and engagement monitoring, inbuilt in the feedback system, so that participants could be flagged if they were deviating from the expected results of the intervention, and causes for this could be explored and addressed [40].”
A meta-analysis of confidence and judgment accuracy in clinical decision making.
Miller, D. J., Spengler, E. S., & Spengler, P. M. (2015) · Journal of Counseling Psychology · doi:10.1037/cou0000105 · Cited by 92
“…assessments, such as the use of mechanical prediction techniques (for a comprehensive review, see Spengler, 2013), or the use of objective measures of change in psychotherapy (for a mega-analysis, see Shimokawa et al., 2010), but these approaches are not widely adopted by practitioners.”
MethodPsychological Assessment Training in Clinical Psychology Doctoral Programs
Mihura, J. L., Roy, M., & Graceffo, R. A. (2016) · Journal of Personality Assessment · doi:10.1080/00223891.2016.1201978 · Cited by 97
Schuman, D. L., Slone, N. C., Reese, R. J., & Duncan, B. (2015) · Psychotherapy Research · doi:10.1080/10503307.2014.900875 · Cited by 86
“In a meta-analytic review of the OQ system, Shimokawa, Lambert, and Smart (2010) reanalyzed the combined data-set (N = 6151) from all six of the OQ feedback studies (Harmon et al., 2007; Hawkins, Lambert, Vermeersch, Slade, & Tuttle, 2004; Lambert et al., 2001, 2002; Slade, Lambert, Harmon, Smart,…”
MethodZilcha-Mano, S., & Errázuriz, P. (2015) · Journal of Counseling Psychology · doi:10.1037/cou0000103 · Cited by 86
“The effect of feedback systems appeared to be substantial in reducing deterioration rates and enhancing positive outcomes, especially in patients for whom treatment was predicted to fail (Hansen, Lambert, & Forman, 2002; Lambert, 2013; Shimokawa et al., 2010).”
Backgroundvan Sonsbeek, M. A., Hutschemaekers, G. G., Veerman, J. W., & Tiemens, B. B. (2014) · BMC Psychiatry · doi:10.1186/1471-244x-14-3 · Cited by 86
“By investigating several potentially effective components of feedback from ROM in youth mental health care, we contribute to the identification of important aspects of feedback and to a better understanding of possible mechanisms underlying feedback effects (e.g. what works for whom, when and how).”
BackgroundInfluentialDispelling Myths About Dissociative Identity Disorder Treatment: An Empirically Based Approach
Brand, B. L., Loewenstein, R. J., & Spiegel, D. (2014) · Psychiatry: Interpersonal and Biological Processes · doi:10.1521/psyc.2014.77.2.169 · Cited by 85
Weisz, J. R., Vaughn, R., Evans, S. C., Thomassin, K., Hersh, J., Ng, M. Y., Lau, N., Lee, E. H., Raftery‐Helmer, J. N., & Mair, P. (2019) · Journal of Clinical Child & Adolescent Psychology · doi:10.1080/15374416.2018.1547973 · Cited by 94
Distinct roles of state-like and trait-like patient–therapist alliance in psychotherapy
Zilcha‐Mano, S., & Fisher, H. (2022) · Nature Reviews Psychology · doi:10.1038/s44159-022-00029-z · Cited by 107
Improving the efficiency of psychological treatment using outcome feedback technology
Delgadillo, J., Overend, K., Lucock, M. P., Groom, M., Kirby, N., McMillan, D., Gilbody, S. M., Lutz, W., Rubel, J. A., & de Jong, K. (2017) · Behaviour Research and Therapy · doi:10.1016/j.brat.2017.09.011 · Cited by 89
“Current wisdom in the field suggests that OF is specifically helpful for a subset of signal cases that are classified as NOT during therapy (Carlier et al., 2012; Castonguay, Barkham, Lutz, &McAleavey, 2013; Knaup, Koesters, Schoefer, Becker, & Puschner, 2009; Shimokawa et al., 2010).”
InfluentialEvidence for the Management of Adolescent Depression
Lewandowski, R. E., Acri, M. C., Hoagwood, K. E., Olfson, M., Clarke, G., Gardner, W., Scholle, S. H., Byron, S., Kelleher, K., Pincus, H. A., et al. (2013) · Pediatrics · doi:10.1542/peds.2013-0600 · Cited by 81
Leichsenring, F., & Schauenburg, H. (2014) · Journal of Affective Disorders · doi:10.1016/j.jad.2014.08.007 · Cited by 94
“, 2010), (2) focusing on patients at risk for non-response by giving feedback on patient progress (Lambert et al., 2002; Leichsenring et al., in press; Shimokawa et al., 2010).”
BackgroundMoltu, C., Veseth, M., Stefansen, J., Nøtnes, J. C., Skjølberg, Å., Binder, P., Castonguay, L. G., & Nordberg, S. S. (2016) · Psychotherapy Research · doi:10.1080/10503307.2016.1189619 · Cited by 94
“…by Lambert’s and his colleagues (Harmon et al., 2007; Lambert, 2010; Lambert & Burlingame, 2004; Lambert & Hill, 1994; Lambert & Shimokawa, 2011; Shimokawa et al., 2010) seminal and long-lasting work in establishing and documenting the concept of clinical feedback and its benefits to therapy…”
BackgroundEvidence-Based Practice and Psychological Treatments: The Imperatives of Informed Consent
Blease, C. R., Lilienfeld, S. O., & Kelley, J. M. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01170 · Cited by 79
“…in Psychology | www.frontiersin.org 3 August 2016 | Volume 7 | Article 1170 feedback during therapy is one important means of monitoring progress, thereby helping therapists to enhance patient outcomes (Lambert et al., 2001; Sapyta et al., 2005; Shimokawa et al., 2010; Beidas et al., 2015).”
BackgroundConfessions of a New York rupture researcher: An insider’s guide and critique
Muran, J. C. (2019) · Psychotherapy Research · doi:10.1080/10503307.2017.1413261 · Cited by 79
“(For example, the OQ System developed by Lambert and colleagues [see Shimokawa et al., 2010] includes principles derived from our work in their clinical support tool, and yet it is not clear from their feedback research if and how this tool is actually used by clinicians.”
BackgroundMethodInfluentialBone, C., Simmonds-Buckley, M., Thwaites, R., Sandford, D., Merzhvynska, M., Rubel, J., Deisenhofer, A.-K., Lutz, W., & Delgadillo, J. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00018-2 · Cited by 79
Lambert, M. J. (2011) · Cognitive and Behavioral Practice · doi:10.1016/j.cbpra.2011.02.002 · Cited by 98
Self-citation25 Years of systemic therapies research: Progress and promise
Heatherington, L., Friedlander, M. L., Diamond, G. M., Escudero, V., & Pinsof, W. M. (2014) · Psychotherapy Research · doi:10.1080/10503307.2014.983208 · Cited by 95
Ionita, G., & Fitzpatrick, M. R. (2014) · Canadian Psychology/Psychologie canadienne · doi:10.1037/a0037355 · Cited by 95
The merits of implementing routine outcome monitoring in clinical practice.
Lambert, M. J., & Harmon, K. L. (2018) · Clinical Psychology: Science and Practice · doi:10.1111/cpsp.12268 · Cited by 74
“Using these methods, Shimokawa et al. (2010) found the OQ‐System using Clinical Support Tools reduced deterioration rates in at‐risk cases from the base rate in TAU of 21% to about 6%, while the rate of reliable improvement more than doubled from 21% to 50%.”
MethodSelf-citationHow to Make Practical Use of Therapeutic Alliance Research in Your Clinical Work
Hilsenroth, M. J., Cromer, T. D., & Ackerman, S. J. (2011) · Humana Press eBooks · doi:10.1007/978-1-60761-792-1_22 · Cited by 96
“( [ 21 ] ; also see [ 8, 14 ] ) are particularly relevant to the current review and consistent with an explicit, collaborative discussion of treatment goals, expectations, and structure in the TMA and SI.”
BackgroundMellor‐Clark, J., Cross, S., Macdonald, J. B., & Skjulsvik, T. (2014) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-014-0587-8 · Cited by 89
“…are documented in a growing number of controlled studies (e.g. Kraus et al. 2011; Anker et al. 2009; Lambert and Shimokawa 2011; Lambert 2012), authoritative methodological texts (Lambert 2010), comprehensive clinical guidelines (Duncan 2010), and meta-analytic studies (Shimokawa et al. 2010).”
BackgroundCrits-Christoph, P., Ring-Kurtz, S., Hamilton, J. L., Lambert, M. J., Gallop, R., McClure, B., Kulaga, A., & Rotrosen, J. (2012) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2011.09.003 · Cited by 73
“In addition, Shimokawa, Lambert, and Smart (2010), using meta/mega-analytic techniques, combined data from the six clinical trials and found that negative change rates were four times less likely when progress feedback and clinical support tools were employed.”
Self-citationMacdonald, J. B., & Mellor‐Clark, J. (2014) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.1887 · Cited by 90
Peterson, C. B., Becker, C. B., Treasure, J., Shafran, R., & Bryant-Waugh, R. (2016) · BMC Medicine · doi:10.1186/s12916-016-0615-5 · Cited by 69
“also influence patient values [10, 11].”
Østergård, O. K., Randa, H., & Hougaard, E. (2018) · Psychotherapy Research · doi:10.1080/10503307.2018.1517949 · Cited by 84
Connors, E. H., Douglas, S. R., Jensen‐Doss, A., Landes, S. J., Lewis, C. C., McLeod, B. D., Stanick, C., & Lyon, A. R. (2020) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-020-01063-w · Cited by 74
Client feedback in psychological therapy for children and adolescents with mental health problems
Bergman, H., Kornør, H., Nikolakopoulou, A., Hanssen‐Bauer, K., Soares‐Weiser, K., Tollefsen, T. K., & Bjørndal, A. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011729.pub2 · Cited by 77
Routine outcome monitoring: Coming of age—With the usual developmental challenges.
Wampold, B. E. (2015) · Psychotherapy · doi:10.1037/pst0000037 · Cited by 76
“, the OQ-system and PCOMS) appears to be strong, particularly for preventing deterioration of cases at risk for failure (Lambert & Shimokawa, 2011; Shimokawa et al., 2010).”
BackgroundEvaluating the efficacy of client feedback in group psychotherapy.
Slone, N. C., Reese, R. J., Mathews-Duvall, S., & Kodet, J. (2015) · Group Dynamics: Theory, Research, and Practice · doi:10.1037/gdn0000026 · Cited by 68
“Shimokawa, Lambert, and Smart (2010) conducted a meta-analysis with six of the earliest RCT studies (N 6,151) and found clients in the feedback conditions had approximately 2.6 times higher odds of attaining reliable improvement compared to clients in the TAU conditions.”
InfluentialJohnson, L. N., Miller, R. B., Bradford, A. B., & Anderson, S. R. (2017) · Journal of Marital and Family Therapy · doi:10.1111/jmft.12238 · Cited by 80
Rognstad, K., Wentzel‐Larsen, T., Neumer, S., & Kjøbli, J. (2022) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-022-01236-9 · Cited by 87
“, 2016) or significant small effects (e.g., De Jong et al., 2021; Knaup et al., 2009; Lambert et al., 2001; Shimokawa et al., 2010).”
BackgroundMethodResultInfluentialClinical prediction in group psychotherapy
Chapman, C. L., Burlingame, G. M., Gleave, R., Rees, F., Beecher, M., & Porter, G. S. (2012) · Psychotherapy Research · doi:10.1080/10503307.2012.702512 · Cited by 66
“This research is highlighted since it is the only outcome feedback system that has been the subject of nine large-scale RCTs (Crits-Christoph et al, 2012 ; Harmon et al, 2007 ; Hawkins et al, 2004 ; Shimokawa, Lambert, & Smart, 2010 ; Simon et al, 2012 , submitted; Slade, Lambert, Harmon, Smart, & Bailey 2008).”
Reese, R. J., Gillaspy, J. A., Owen, J. J., Flora, K. L., Cunningham, L. C., Archie, D., & Marsden, T. (2013) · Journal of Clinical Psychology · doi:10.1002/jclp.21946 · Cited by 66
“For instance, Shimokawa et al. (2010) conducted a meta-analysis of six randomized clinical trials (RCT) comparing the OQ System to treatment as usual (TAU; Harmon et al., 2007; Hawkins et al., 2004; Lambert et al., 2001, 2002; Slade, Lambert, Harmon, Smart, & Bailey, 2008; Whipple et al., 2003).”
MethodPinsof, W. M., Zinbarg, R. E., Shimokawa, K., Latta, T. A., Goldsmith, J. Z., Knobloch-Fedders, L. M., Chambers, A. L., & Lebow, J. L. (2015) · Family Process · doi:10.1111/famp.12159 · Cited by 73
Self-citationImproving the efficiency of cognitive-behavioural therapy by using formal client feedback
Janse, P. D., De Jong, K., Van Dijk, M. K., Hutschemaekers, G. J. M., & Verbraak, M. J. P. M. (2017) · Psychotherapy Research · doi:10.1080/10503307.2016.1152408 · Cited by 65
“, 2003), and slightly less so for those “on track” (De Jong et al., 2014; Shimokawa et al., 2010).”
BackgroundEvidence Base for Measurement-Based Care in Child and Adolescent Psychiatry
Parikh, A., Fristad, M. A., Axelson, D., & Krishna, R. (2020) · Child and Adolescent Psychiatric Clinics of North America · doi:10.1016/j.chc.2020.06.001 · Cited by 68
Helping clinicians to use and learn from research-based systems: The OQ-analyst.
Lambert, M. J. (2012) · Psychotherapy · doi:10.1037/a0027110 · Cited by 63
Self-citationPredicting therapist effectiveness from their own practice-based evidence.
Kraus, D. R., Bentley, J. H., Alexander, P. C., Boswell, J. F., Constantino, M. J., Baxter, E. E., & Castonguay, L. G. (2016) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000083 · Cited by 63
Hogue, A., Ozechowski, T. J., Robbins, M. S., & Waldron, H. B. (2013) · Clinical Psychology: Science and Practice · doi:10.1111/cpsp.12023 · Cited by 62
Brattland, H., Koksvik, J. M., Burkeland, O., Klöckner, C. A., Lara-Cabrera, M. L., Miller, S. D., Wampold, B., Ryum, T., & Iversen, V. C. (2019) · Journal of Counseling Psychology · doi:10.1037/cou0000320 · Cited by 60
Simon, W., Lambert, M. J., Busath, G., Vazquez, A., Berkeljon, A., Hyer, K., Granley, M., & Berrett, M. (2013) · Psychotherapy Research · doi:10.1080/10503307.2013.787497 · Cited by 61
“Among other findings, the Shimokawa et al. (2010) meta/mega-analysis of the effects of applying the OQ-Analyst system (applied in the current study) found that deterioration rates could be reduced from the baseline of 20% in NOT cases Correspondence concerning this article should be addressed to…”
ResultSelf-citationThe Gap Between Couple Therapy Research Efficacy and Practice Effectiveness
Halford, W. K., Pepping, C. A., & Petch, J. F. (2015) · Journal of Marital and Family Therapy · doi:10.1111/jmft.12120 · Cited by 69
Progress monitoring measures: A brief guide.
Overington, L., & Ionita, G. (2012) · Canadian Psychology/Psychologie canadienne · doi:10.1037/a0028017 · Cited by 75
Predicting Social Anxiety Treatment Outcome Based on Therapeutic Email Conversations
Hoogendoorn, M., Berger, T., Schulz, A., Stolz, T., & Szolovits, P. (2017) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2016.2601123 · Cited by 59
““Real-time” psychometric feedback over the course of treatment based on self-report measures has shown its effect on therapy outcome, especially for patients at risk for negative outcome (see [36] and [37]).”
BackgroundEnhancing professionalism through self-reflection.
Knapp, S. J., Gottlieb, M. C., & Handelsman, M. M. (2017) · Professional Psychology Research and Practice · doi:10.1037/pro0000135 · Cited by 86
Replicating and extending the good-enough level model of change: considering session frequency
Reese, R. J., Toland, M. D., & Hopkins, N. B. (2011) · Psychotherapy Research · doi:10.1080/10503307.2011.598580 · Cited by 58
Amble, I., Gude, T., Stubdal, S., Andersen, B. J., & Wampold, B. E. (2015) · Psychotherapy Research · doi:10.1080/10503307.2014.928756 · Cited by 58
“At one of the clinics, there were short periods with an unstable Internet function (approximately 10 days in the 2 years in which data were collected), which had as a consequence that the patients were presented the paper/pencil format of the OQ-45, causing increased work load and delays in feedback for the FB conditi…”
BackgroundMethodResultInfluential
Metadata
- Authors
- Shimokawa, K., Lambert, M. J., & Smart, D. W.
- Affiliations
- Brigham Young University (US)
- ORCID
- Lambert, Michael J.
- Venue
- Journal of Consulting and Clinical Psychology
- Published
- 2010
- Volume
- 78
- Issue
- 3
- Pages
- 298-311
- ISSN
- 0022-006X (ISSN-L)
- 1939-2117
- OpenAlex
- W2143726303
- Semantic Scholar
- 94c5e3fb3707d960610a79e604c2b070280c0014
- Cited by
- 657
- References
- 38
- Citations by year
- 2026: 18 · 2025: 23 · 2024: 64 · 2023: 24 · 2022: 37 · 2021: 57 · 2020: 47 · 2019: 47 · 2018: 59 · 2017: 54 · 2016: 45 · 2015: 61 · 2014: 44 · 2013: 31 · 2012: 30
- FWCI
- 26.64
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Consulting and Clinical Psychology (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Brigham Young University
- Topics
- Social Sciences › Psychology › Clinical Psychology › Psychotherapy Techniques and Applications
- Social Sciences › Psychology › Clinical Psychology › Counseling Practices and Supervision
- Social Sciences › Psychology › Clinical Psychology › Counseling, Therapy, and Family Dynamics
- Keywords
- treatment failure, early alert systems, progress feedback, patient outcomes, Outcome Questionnaire-45, clinical support tools, therapist feedback, counseling centers
- MeSH terms
- Adult, Mental Disorders, Feedback, Female, Humans, Male, Psychology, Clinical, Psychotherapy, Quality Assurance, Health Care, Surveys and Questionnaires, Referral and Consultation, Risk, Outcome Assessment, Health Care, Treatment Failure, Disease Progression, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.