Feedback-informed treatment versus usual psychological treatment for depression and anxiety: a multisite, open-label, cluster randomised controlled trial
Open accessAbstract
Background Previous research suggests that the use of outcome feedback technology can enable psychological therapists to identify and resolve obstacles to clinical improvement. We aimed to assess the effectiveness of an outcome feedback quality assurance system applied in stepped care psychological services.Methods This multisite, open-label, cluster randomised controlled trial was done at eight National Health Service (NHS) Trusts in England, involving therapists who were qualified to deliver evidence-based low-intensity or high-intensity psychological interventions. Adult patients (18 years or older) who accessed individual therapy with participating therapists were eligible for inclusion, except patients who accessed group therapies and those who attended less than two individual therapy sessions. Therapists were randomly assigned (1:1) to an outcome feedback intervention group or a treatment-as-usual control group by use of a computer-generated randomisation algorithm. The allocation of patients to therapists was quasi-random, whereby patients on waiting lists were allocated sequentially on the basis of therapist availability. All patients received low-intensity (less than eight sessions) or high-intensity (up to 20 sessions) psychological therapies for the duration of the 1-year study period. An automated computer algorithm alerted therapists in the outcome feedback group to patients who were not on track, and primed them to review these patients in clinical supervision. The primary outcome was symptom severity on validated depression (Patient Health Questionnaire-9 [PHQ-9]) and anxiety (Generalised Anxiety Disorder-7 [GAD-7]) measures after treatment of varying durations, which were compared between groups with multilevel modelling, controlling for cluster (therapist) effects. We used an intention-to-treat approach. This trial was prospectively registered with ISRCTN, number ISRCTN12459454.Findings In total, 79 therapists were recruited to the study between Jan 8, 2016, and July 15, 2016, but two did not participate. Of these participants, 39 (51%) were randomly assigned to the outcome feedback group and 38 (49%) to the control group. Overall, 2233 patients were included in the trial (1176 [53%] were treated by therapists in the outcome feedback group, and 1057 [47%] by therapists in the control group). Patients classified as not on track had less severe symptoms after treatment if they were allocated to the outcome feedback group than those in the control group (PHQ-9 d=0·23, B=-1·03 [95% CI -1·84 to -0·23], p=0·012; GAD-7 d=0·19, B=-0·85 [-1·56 to -0·14], p=0·019).Interpretation Supplementing psychological therapy with low-cost feedback technology can reduce symptom severity in patients at risk of poor response to treatment. This evidence supports the implementation of outcome feedback in stepped care psychological services.Funding English NHS and Department of Health Sciences, University of York, York, UK.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Providing continuous feedback to patients has been shown to be a valuable therapeutic procedure.
“Because providing continuous feedback to patients has been shown to be a valuable therapeutic procedure [29], we also included studies that adopted EMA-based feedback as a therapeutic tool for clinically depressed patients.”
From Current state and future directions of technology-based ecological momentary assessment and intervention for major depressive disorder: a systematic review (2019) · cites this paper as [29]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The PHQ-9: Validity of a brief depression severity measure
Kroenke, K., Spitzer, R. L., & Williams, J. B. W. (2001) · Journal of General Internal Medicine · doi:10.1046/j.1525-1497.2001.016009606.x · Cited by 44,910 · see citing works
Anxiety disorders in primary care: prevalence, impairment, comorbidity, and detection
Kroenke, K., Spitzer, R. L., Williams, J. B. W., Monahan, P. O., & Löwe, B. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-146-5-200703060-00004 · Cited by 4,741 · see citing works
Bergin and Garfield's Handbook of Psychotherapy and Behavior Change
Lambert, M. J. (2003) · Cited by 3,383 · see citing works
The Work and Social Adjustment Scale: a simple measure of impairment in functioning
Mundt, J. C., Marks, I. M., Shear, M. K., & Greist, J. M. (2002) · British Journal of Psychiatry · doi:10.1192/bjp.180.5.461 · Cited by 2,515 · see citing works
Multiple Imputation for Multivariate Missing-Data Problems: A Data Analyst's Perspective
Schafer, J. L., & Olsen, M. K. (1998) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr3304_5 · Cited by 1,518 · see citing works
The psychotherapy dose-response effect and its implications for treatment delivery services.
Hansen, N. B., Lambert, M. J., & Forman, E. M. (2002) · Clinical Psychology: Science and Practice · doi:10.1093/clipsy.9.3.329 · Cited by 783 · see citing works
Clark, D. M. (2011) · International Review of Psychiatry · doi:10.3109/09540261.2011.606803 · Cited by 699 · see citing works
Shimokawa, K., Lambert, M. J., & Smart, D. W. (2010) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0019247 · Cited by 657 · see citing works
Routine outcome monitoring and feedback on physical or mental health status: evidence and theory
Carlier, I. V. E., Meuldijk, D., van Vliet, I. M., van Fenema, E. M., van der Wee, N. J. A., & Zitman, F. G. (2010) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2010.01543.x · Cited by 277 · see citing works
Effect of feedback of treatment outcome in specialist mental healthcare: meta-analysis
Knaup, C., Koesters, M., Schöfer, D., Becker, T. M., & Puschner, B. (2009) · The British Journal of Psychiatry · doi:10.1192/bjp.bp.108.053967 · Cited by 264 · see citing works
de Jong, K., van Sluis, P., Nugter, A., Heiser, W. J., & Spinhoven, P. (2012) · Psychotherapy Research · doi:10.1080/10503307.2012.673023 · Cited by 250 · see citing works
Applied analysis of variance in behavioral science
Edwards, L. K. (1993) · M. Dekker eBooks · Cited by 235 · see citing works
Kendrick, T., El-Gohary, M., Stuart, B. L., Gilbody, S. M., Churchill, R. C., Aiken, L., Bhattacharya, A., Gimson, A., Brütt, A. L., de Jong, K., et al. (2016) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011119.pub2 · Cited by 235 · see citing works
Self-citationCommon mental health disorders : identification and pathways to care
Excellence, C. (2011) · Medical Entomology and Zoology · Cited by 230 · see citing works
Richards, D. A., & Borglin, G. (2011) · Journal of Affective Disorders · doi:10.1016/j.jad.2011.03.024 · Cited by 217 · see citing works
Finch, A. E., Lambert, M. J., & Schaalje, B. G. (2001) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.286 · Cited by 143 · see citing works
Delgadillo, J., Moreea, O., & Lutz, W. (2016) · Behaviour Research and Therapy · doi:10.1016/j.brat.2016.02.003 · Cited by 137 · see citing works
Self-citationLutz, W., Martinovich, Z., & Howard, K. I. (1999) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.67.4.571 · Cited by 124 · see citing works
Self-citationPatient-focused and feedback research in psychotherapy: Where are we and where do we want to go?
Lutz, W., de Jong, K., & Rubel, J. A. (2015) · Psychotherapy Research · doi:10.1080/10503307.2015.1079661 · Cited by 113 · see citing works
Self-citationLutz, W., Martinovich, Z., & Howard, K. I. (1999) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.67.4.571 · Cited by 97 · see citing works
Self-citationImproving the efficiency of psychological treatment using outcome feedback technology
Delgadillo, J., Overend, K., Lucock, M. P., Groom, M., Kirby, N., McMillan, D., Gilbody, S. M., Lutz, W., Rubel, J. A., & de Jong, K. (2017) · Behaviour Research and Therapy · doi:10.1016/j.brat.2017.09.011 · Cited by 81 · see citing works
Self-citationSchiefele, A.-K., Lutz, W., Barkham, M., Rubel, J. A., Boehnke, J. R., Delgadillo, J., Kopta, S. M., Schulte, D., Saxon, D. H., Nielsen, S. L., et al. (2016) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-016-0736-3 · Cited by 79 · see citing works
Self-citationImproving the efficiency of cognitive-behavioural therapy by using formal client feedback
Janse, P. D., de Jong, K., van Dijk, M. K., Hutschemaekers, G. J. M., & Verbraak, M. J. P. M. (2016) · Psychotherapy Research · doi:10.1080/10503307.2016.1152408 · Cited by 65 · see citing works
Amble, I., Gude, T., Stubdal, S., Andersen, B. J., & Wampold, B. E. (2014) · Psychotherapy Research · doi:10.1080/10503307.2014.928756 · Cited by 57 · see citing works
Lutz, W., Martinovich, Z., Howard, K. I., & Leon, S. C. (2002) · Journal of Clinical Psychology · doi:10.1002/jclp.10070 · Cited by 47 · see citing works
Self-citationLucock, M. P., Halstead, J. E., Leach, C., Barkham, M., Tucker, S., Randal, C., Middleton, J., Khan, W., Catlow, H., Waters, E. L., et al. (2015) · Psychotherapy Research · doi:10.1080/10503307.2015.1051163 · Cited by 46 · see citing works
Self-citationde Jong, K., Segaar, J. A., Ingenhoven, T., Busschbach, J. J. V., & Timman, R. (2017) · Journal of Personality Disorders · doi:10.1521/pedi_2017_31_297 · Cited by 42 · see citing works
Lutz, W., Zimmermann, D., Müller, V., Deisenhofer, A., & Rubel, J. A. (2017) · BMC Psychiatry · doi:10.1186/s12888-017-1464-2 · Cited by 38 · see citing works
Self-citationde Jong, K., & de Goede, M. (2015) · Psychotherapy Research · doi:10.1080/10503307.2015.1076198 · Cited by 35 · see citing works
Using outcome feedback in psychological therapy: A guideline for IAPT practitioners
Delgadillo, J., Lucock, M. P., & de Jong, K. (2015) · University of Huddersfield Repository (University of Huddersfield) · Cited by 2 · see citing works
Self-citation
Flagged first, then by citations
1 of the 31 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- Castonguay LG, Barkham M, Lutz W, McAleavey AA. Practice-oriented research: approaches and applications. In Lambert MJ, ed. Bergin and Garfield's handbook of psychotherapy and behavior change, 6th edn. New York, NY: Wiley & Sons, 2013.
- Baldwin SA, Imel ZE. Therapist effects: findings and methods. In Lambert MJ, ed. Bergin and Garfield's handbook of psychotherapy and behavior change, 6th edn. New York, NY: Wiley & Sons, 2013.
Cited by
Sorted by citations
de Jong, K., Conijn, J. M., Gallagher, R. A. V., Reshetnikova, A. S., Heij, M., & Lutz, M. C. (2021) · Clinical Psychology Review · doi:10.1016/j.cpr.2021.102002 · Cited by 384 · see citing works
Lutz, W., Rubel, J. A., Schwartz, B., Schilling, V., & Deisenhofer, A.-K. (2019) · Behaviour Research and Therapy · doi:10.1016/j.brat.2019.103438 · Cited by 196 · see citing works
Self-citationColombo, D., Fernández-Álvarez, J., Patané, A., Semonella, M., Kwiatkowska, M., García-Palacios, A., Cipresso, P., Riva, G., & Botella, C. (2019) · Journal of Clinical Medicine · doi:10.3390/jcm8040465 · Cited by 194 · see citing works
“Because providing continuous feedback to patients has been shown to be a valuable therapeutic procedure [29], we also included studies that adopted EMA-based feedback as a therapeutic tool for clinically depressed patients.”
MethodThieme, A., Hanratty, M., Lyons, M., Palacios, J. E., Marques, R. F., Morrison, C. P., & Doherty, G. J. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3564752 · Cited by 181 · see citing works
Colombo, D., Fernández‐Álvarez, J., Suso‐Ribera, C., Cipresso, P., Valev, H., Leufkens, T., Sas, C., García‐Palacios, A., Riva, G., & Botella, C. (2020) · Emotion · doi:10.1037/emo0000671 · Cited by 154 · see citing works
“EMA could indeed change the way we assess ER in psychopathology by helping us to understand the dynamics of this process in different emotional disorders, as well as the impact of emotion dysregulation feedback to patients, which has been shown to be a valuable practice (Delgadillo et al., 2018).”
BackgroundJuul, S., Gluud, C., Simonsen, S., Frandsen, F. W., Kirsch, I., & Jakobsen, J. C. (2021) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2020-111407 · Cited by 135 · see citing works
Trajectories of depression and anxiety symptom change during psychological therapy
Saunders, R., Buckman, J. E. J., Cape, J., Fearon, P., Leibowitz, J., & Pilling, S. (2019) · Journal of Affective Disorders · doi:10.1016/j.jad.2019.02.043 · Cited by 99 · see citing works
Webb, C. A., Cohen, Z. D., Beard, C., Forgeard, M. J. C., Peckham, A., & Björgvinsson, Þ. (2019) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000451 · Cited by 96 · see citing works
“…monitoring (see Carlier et al., 2012; Knaup, Koesters, Schoefer, Becker, & Puschner, 2009; Shimokawa, Lambert, & Smart, 2010; Kendrick et al., 2016; Delgadillo et al., 2018), (3) highlight which specific baseline characteristics are contributing to an individual’s predicted poor prognosis, which…”
BackgroundInfluentialEnrique, A., Palacios, J. E., Ryan, H., & Richards, D. (2019) · Journal of Medical Internet Research · doi:10.2196/12775 · Cited by 81 · see citing works
“the intervention period, these patients had significantly less severe symptoms after treatment compared with similar patients assigned to therapists who were not receiving this feedback [41].”
Rozental, A., Andersson, G., & Carlbring, P. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00589 · Cited by 74 · see citing works
Bone, C., Simmonds-Buckley, M., Thwaites, R., Sandford, D., Merzhvynska, M., Rubel, J., Deisenhofer, A.-K., Lutz, W., & Delgadillo, J. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00018-2 · Cited by 79 · see citing works
Self-citationHornstein, S., Forman‐Hoffman, V. L., Nazander, A., Ranta, K., & Hilbert, K. (2021) · Digital Health · doi:10.1177/20552076211060659 · Cited by 79 · see citing works
“First studies suggested that monitoring patients, for whom low outcomes are predicted, can help to improve their outcomes.(42) This appears as a promising use case also for a DMH setting, as the monitoring would be easy to implement and not too cost-intensive.”
BackgroundJensen‐Doss, A., Douglas, S. R., Phillips, D. A., Gencdur, O., Zalman, A., & Gomez, N. E. (2020) · Evidence-Based Practice in Child and Adolescent Mental Health · doi:10.1080/23794925.2020.1784062 · Cited by 71 · see citing works
“…a recent trial demonstrated that a MBC feedback condition that included flags that clients were NOT and guidance about how to discuss this feedback in supervision was associated with better treatment outcomes than a MBC conditions without these risk signals and supports (Delgadillo et al., 2018).”
BackgroundWade, T. D., Shafran, R., & Cooper, Z. (2023) · International Journal of Eating Disorders · doi:10.1002/eat.24008 · Cited by 57 · see citing works
The future of psychological treatments: The Marburg Declaration
Rief, W., Asmundson, G. J. G., Bryant, R. A., Clark, D. M., Ehlers, A., Holmes, E. A., McNally, R. J., Neufeld, C. B., Wilhelm, S., Jaroszewski, A. C., et al. (2024) · Clinical Psychology Review · doi:10.1016/j.cpr.2024.102417 · Cited by 54 · see citing works
Nickels, S., Edwards, M. D., Poole, S. F., Winter, D., Gronsbell, J. L., Rozenkrants, B., Miller, D. P., Fleck, M. S., McLean, A., Peterson, B. E., et al. (2021) · JMIR Mental Health · doi:10.2196/27589 · Cited by 50 · see citing works
“The location example (Figure 3) shows one area in which a clinician could gain insights about a patient’s routine that could augment clinical decision-making and strengthen feedback-informed care [25].”
BackgroundUsing Progress Feedback to Enhance Treatment Outcomes: A Narrative Review
de Jong, K., Douglas, S. R., Wolpert, M., Delgadillo, J., Aas, B., Bovendeerd, B., Carlier, I. V. E., Compare, A., Edbrooke-Childs, J. H., Janse, P. D., et al. (2024) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-024-01381-3 · Cited by 60 · see citing works
Self-citationMcAleavey, A. A., de Jong, K., Nissen‐Lie, H. A., Boswell, J. F., Moltu, C., & Lutz, W. (2024) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-024-01351-9 · Cited by 50 · see citing works
Self-citationLatent variable mixture modelling and individual treatment prediction
Saunders, R., Buckman, J. E. J., & Pilling, S. (2019) · Behaviour Research and Therapy · doi:10.1016/j.brat.2019.103505 · Cited by 47 · see citing works
Moving from tradition-based to competence-based psychotherapy
Rief, W. (2021) · Evidence-Based Mental Health · doi:10.1136/ebmental-2020-300219 · Cited by 46 · see citing works
O’Dea, B., Subotic-Kerry, M., KING, C. A., Mackinnon, A. J., Achilles, M. R., Anderson, M., Parker, B., Werner‐Seidler, A., Torok, M. H., Cockayne, N. L., et al. (2021) · The Lancet Regional Health - Western Pacific · doi:10.1016/j.lanwpc.2021.100178 · Cited by 46 · see citing works
Börjesson, S., & Boström, P. (2019) · Psychotherapy Research · doi:10.1080/10503307.2019.1630780 · Cited by 41 · see citing works
“When therapists receive feedback from clients with poor initial treatment response, the percentage of negative outcomes appears to decrease (Delgadillo et al., 2018; Lambert, Harmon, Slade, Whipple, & Hawkins, 2005).”
BackgroundGual-Montolio, P., Martínez-Borba, V., Bretón-López, J. M., Osma, J., & Suso-Ribera, C. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17093170 · Cited by 38 · see citing works
“While the presented results are, overall, encouraging, especially for not-on-track patients, more research is required in this field, especially RCTs comparing technology-supported MBC with traditional MBC without technology.”
BackgroundMethodInfluentialDavey, E., Allen, K. L., Bennett, S. D., Bryant‐Waugh, R., Clarke, T., Cooper, Z., Dixon‐Ward, K., Dudley, J., Eisler, I., Griffiths, J., et al. (2023) · European Eating Disorders Review · doi:10.1002/erv.2981 · Cited by 39 · see citing works
“…extent in practice, it is suggested that (a) such processes are as automated as possible; and (b) clinicians are trained in the importance of PROMs as a technique to improve outcome (e.g., Delgadillo et al., 2018), as well as its importance ethically with regard to clinical decision making.”
BackgroundCost-effectiveness of feedback-informed psychological treatment: Evidence from the IAPT-FIT trial
Delgadillo, J., McMillan, D., Gilbody, S., de Jong, K., Lucock, M., Lutz, W., Rubel, J., Aguirre, E., & Ali, S. (2021) · Behaviour Research and Therapy · doi:10.1016/j.brat.2021.103873 · Cited by 41 · see citing works
Self-citationHovland, R. T., Ytrehus, S., Mellor‐Clark, J., & Moltu, C. (2020) · Journal of Clinical Psychology · doi:10.1002/jclp.22992 · Cited by 37 · see citing works
“…have demonstrated that it has positive effects on therapy outcomes compared to treatment as usual (Brattland et al., 2018; Carlier et al., 2012; Delgadillo et al., 2018; Gondek, Edbrooke‐Childs, Fink, Deighton, & Wolpert, 2016; Knaup, Koesters, Schoefer, Becker, & Puschner, 2009; Lambert,…”
MethodPredicting and preventing dropout in research, assessment and treatment with refugees
Semmlinger, V., & Ehring, T. W. (2021) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.2672 · Cited by 34 · see citing works
Kendrick, T., Dowrick, C. F., Lewis, G., Moore, M. V., Leydon, G., Geraghty, A. W. A., Griffiths, G. O., Zhu, S., Yao, G., May, C. R., et al. (2023) · Health Technology Assessment · doi:10.3310/plrq4216 · Cited by 34 · see citing works
Skelton, M., Carr, E., Buckman, J. E. J., Davies, M. R., Goldsmith, K., Hirsch, C. R., Peel, A. J., Rayner, C., Rimes, K. A., Saunders, R., et al. (2022) · Psychological Medicine · doi:10.1017/s0033291722003403 · Cited by 31 · see citing works
Measuring Alliance and Symptom Severity in Psychotherapy Transcripts Using Bert Topic Modeling
Lalk, C., Steinbrenner, T., Kania, W., Popko, A., Wester, R. A., Schaffrath, J., Eberhardt, S. T., Schwartz, B., Lutz, W., & Rubel, J. A. (2024) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-024-01356-4 · Cited by 29 · see citing works
Self-citationMoshe, I., Terhorst, Y., Paganini, S., Schlicker, S., Pulkki-Råback, L., Baumeister, H., Sander, L. B., & Ebert, D. D. (2022) · Journal of Medical Internet Research · doi:10.2196/38261 · Cited by 28 · see citing works
“…feedback technology that identifies individuals at risk of deterioration during treatment has been shown to improve eventual treatment outcomes [43,44], dropout warning systems could be used to alert the supporting clinician, guide care pathways (eg, in stepped-care models), or personalize the…”
Benito, K., Machan, J. T., Freeman, J. B., Garcia, A. M., Walther, M. R., Frank, H. E., Wellen, B. C. M., Stewart, E., Edmunds, J., Sapyta, J. J., et al. (2020) · Behavior Therapy · doi:10.1016/j.beth.2020.07.004 · Cited by 27 · see citing works
The Future of Enhanced Psychotherapy: Towards Precision Psychotherapy
Zipfel, S., Lutz, W., Schneider, S., Schramm, E., Delgadillo, J., & Giel, K. E. (2024) · Psychotherapy and Psychosomatics · doi:10.1159/000539022 · Cited by 25 · see citing works
Self-citationSchilling, V. N. L. S., Zimmermann, D., Rubel, J. A., Boyle, K. S., & Lutz, W. (2020) · Quality of Life Research · doi:10.1007/s11136-020-02664-6 · Cited by 25 · see citing works
Self-citationEspel‐Huynh, H., Zhang, F., Thomas, J. G., Boswell, J. F., Thompson‐Brenner, H., Juarascio, A. S., & Lowe, M. R. (2021) · International Journal of Eating Disorders · doi:10.1002/eat.23510 · Cited by 24 · see citing works
“If applied in clinical practice and integrated into a decision support tool similar to a “treatment navigator” (Delgadillo et al., 2018; Lutz et al., 2019), this model could yield feedback early enough during a treatment episode to allow clinicians to tailor treatment based upon the patient’s…”
MethodFaija, C., Bee, P., Lovell, K., Lidbetter, N., Gellatly, J. L., Ardern, K., Rushton, K., Brooks, H. L., McMillan, D., Armitage, C. J., et al. (2022) · Psychology and Psychotherapy Theory Research and Practice · doi:10.1111/papt.12400 · Cited by 22 · see citing works
“Empirical and clinical value of session- by- session monitoring using feedback is well documented in the literature (e.g. de Jong et al., 2021; Delgadillo et al., 2018; Lambert et al., 2018; Solstad et al., 2019), including tracking responses at an item level (Cross et al., 2015) and thereby…”
MethodSelf-citationBeyond the Therapist’s Office: Merging Measurement-Based Care and Digital Medicine in the Real World
Chiauzzi, E. J., & Wicks, P. J. (2021) · Digital Biomarkers · doi:10.1159/000517748 · Cited by 21 · see citing works
Prasad, N., Chien, I., Regan, T. W., Enrique, Á., Palacios, J. E., Keegan, D., Munir, U., Tanno, R., Richardson, H., Nori, A., et al. (2023) · PLoS ONE · doi:10.1371/journal.pone.0272685 · Cited by 19 · see citing works
Working Towards the Development and Implementation of Precision Mental Healthcare: An Example
Lutz, W., Schwartz, B., Penedo, J. M. G., Boyle, K. S., & Deisenhofer, A. (2020) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-020-01053-y · Cited by 19 · see citing works
Self-citationPeterson, K., Anderson, J., & Bourne, D. S. (2018) · Europe PMC (PubMed Central) · Cited by 20 · see citing works
Eberhardt, S. T., Vehlen, A., Schaffrath, J., Schwartz, B., Baur, T., Schiller, D., Hallmen, T., André, E., & Lutz, W. (2025) · Scientific Reports · doi:10.1038/s41598-025-14923-y · Cited by 18 · see citing works
Self-citationBear, H. A., Dalzell, K., Edbrooke-Childs, J. H., & Wolpert, M. (2021) · British Journal of Clinical Psychology · doi:10.1111/bjc.12322 · Cited by 18 · see citing works
“…as well as for elevating the efficiency and quality of mental health care is well-documented in the research literature (Bickman, Kelley, Breda, de Andrade, et al., 2011; Connors et al., 2020; Delgadillo et al., 2018; Jensen-Doss et al., 2020; Lambert & Harmon, 2018; Shimokawa et al., 2010).”
Do PROMS improve outcomes in patients with depression in primary care?
Kendrick, T., & Maund, E. (2020) · British medical journal · doi:10.1136/bmj.m3313 · Cited by 15 · see citing works
Kwan, B., Rickwood, D., & Brown, P. M. (2020) · Psychotherapy Research · doi:10.1080/10503307.2020.1829738 · Cited by 18 · see citing works
“The addition of expected change trajectories and risk signals have been shown to enhance the treatment effects already seen in MFS (Delgadillo et al., 2018).”
BackgroundAndrews, L. A., Hayes, A. M., Abel, A., & Kuyken, W. (2020) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000467 · Cited by 16 · see citing works
Mütze, K., Witthöft, M., Lutz, W., & Bräscher, A. (2021) · Psychotherapy Research · doi:10.1080/10503307.2021.1930244 · Cited by 17 · see citing works
“As mentioned, prior experimental research has shown that ROM in combination with feedback to therapists significantly reduces deterioration in psychological treatment (Delgadillo et al., 2018; Lambert et al., 2018).”
MethodSelf-citationHovland, R. T., & Moltu, C. (2019) · International Journal of Mental Health Systems · doi:10.1186/s13033-019-0324-5 · Cited by 16 · see citing works
Weisz, J. R., Fitzpatrick, O. M., Venturo‐Conerly, K. E., Sternberg, A., Steinberg, J. S., & Ng, M. Y. (2023) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.13820 · Cited by 16 · see citing works
Cheron, D. M., Chiu, A. W., Stanick, C., Stern, H. G., Donaldson, A. R., Daleiden, E. L., & Chorpita, B. F. (2019) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-019-00922-5 · Cited by 13 · see citing works
A guide for self-help guides: best practice implementation
Shafran, R., Egan, S. J., de Valle, M. K., Davey, E., Carlbring, P., Creswell, C., & Wade, T. D. (2024) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2024.2369637 · Cited by 12 · see citing works
“This is because of the strong data showing that such progress monitoring improves retention and outcomes in mental health interventions (de Jong et al., 2021; Delgadillo et al., 2018).”
BackgroundHutson, J., Hooke, G. R., & Page, A. C. (2020) · Psychotherapy Research · doi:10.1080/10503307.2019.1710305 · Cited by 12 · see citing works
“A number of studies have provided evidence of the benefits of providing progress feedback to therapists (Delgadillo et al., 2018; Gondek, Edbrooke-Childs, Fink, Deighton, & Wolpert, 2016; Lambert, 2010; Shimokawa, Lambert, & Smart, 2010) and patients (de Jong et al., 2014; Hawkins, Lambert,…”
BackgroundResultInfluentialGates, V., Hsiao, M., Zieve, G. G., Courry, R., & Persons, J. B. (2021) · doi:10.1016/j.brat.2021.103874 · Cited by 12 · see citing works
Cawthorne, T., Käll, A., Bennett, S. D., Baker, E., Andersson, G., & Shafran, R. (2023) · Behavioural and Cognitive Psychotherapy · doi:10.1017/s1352465823000231 · Cited by 11 · see citing works
“Goal-based outcomes have been shown to improve treatment retention, clinical outcomes and client progress (Delgadillo et al ., 2018; Tryon et al ., 2018).”
BackgroundAdam, A., Jain, A., Pletnikova, A., Bagga, R., Vita, A., Richey, L. N., Gould, N. F., Munshaw, S., Misrilall, K., & Peters, M. E. (2020) · JMIR Formative Research · doi:10.2196/17722 · Cited by 11 · see citing works
“In recent years, feedback-informed care (FIC) models have been used to improve patient compliance and psychotherapy outcomes [9,10].”
BackgroundFlygare, O., Chen, L. L., de la Cruz, L. F., Rück, C., Andersson, E., Enander, J., & Mataix‐Cols, D. (2020) · Behavior Therapy · doi:10.1016/j.beth.2020.10.006 · Cited by 11 · see citing works
Praxisorientierte Forschung in der Psychotherapie
Herzog, P., Kaiser, T., & Brakemeier, E. (2022) · Zeitschrift für Klinische Psychologie und Psychotherapie · doi:10.1026/1616-3443/a000665 · Cited by 10 · see citing works
Technology-Based Ecological Momentary Assessment in Clinical Psychology
Colombo, D., Suso‐Ribera, C., Fernández‐Álvarez, J., Cipresso, P., García‐Palacios, A., Riva, G., & Botella, C. (2020) · Elsevier eBooks · doi:10.1016/b978-0-12-818697-8.00003-0 · Cited by 9 · see citing works
Barkham, M., & Broglia, E. (2023) · Counselling and Psychotherapy Research · doi:10.1002/capr.12694 · Cited by 11 · see citing works
Johnson, C., Egan, S. J., Carlbring, P., Shafran, R., & Wade, T. D. (2024) · Internet Interventions · doi:10.1016/j.invent.2024.100795 · Cited by 9 · see citing works
“In addition, a self-scoring version of the CPQ that could chart their progress weekly was supplied to each participant, given the evidence that feedback-informed interventions improve retention and outcomes (Delgadillo et al., 2018).”
Barkham, M., de Jong, K., Delgadillo, J., & Lutz, W. (2023) · doi:10.1093/oso/9780197611012.003.0015 · Cited by 9 · see citing works
Self-citationAltimir, C., Mantilla, C., & Serralta, F. B. (2022) · Studies in Psychology Estudios de Psicología · doi:10.1080/02109395.2022.2139998 · Cited by 8 · see citing works
Hooke, G. R., Savani, P., Stewart, B., Araujo, S., & Page, A. C. (2022) · Journal of Clinical Psychology · doi:10.1002/jclp.23426 · Cited by 8 · see citing works
“ROM benefits clinical practice by identifying patients who are “not on track” and open up a conversation enabling staff and patient to work collaboratively towards a positive outcome (de Jong et al., 2021; Delgadillo et al., 2018).”
BackgroundDepression follow-up monitoring with the PHQ-9: an open cluster-randomised controlled trial
Kendrick, T., Dowrick, C. F., Lewis, G., Moore, M. V., Leydon, G., Geraghty, A. W. A., Griffiths, G. O., Zhu, S., Yao, G., May, C. R., et al. (2024) · British Journal of General Practice · doi:10.3399/bjgp.2023.0539 · Cited by 8 · see citing works
The challenges of making clinical feedback in psychotherapy benefit all users: A qualitative study
Hovland, R. T., & Moltu, C. (2019) · Nordic Psychology · doi:10.1080/19012276.2019.1684348 · Cited by 8 · see citing works
Delli Colli, C., Viglione, A., Poggini, S., Cirulli, F., Chiarotti, F., Giuliani, A., & Branchi, I. (2025) · Translational Psychiatry · doi:10.1038/s41398-025-03246-1 · Cited by 7 · see citing works
Hegland, P. A., Aasprang, A., Kolotkin, R. L., Moltu, C., Tell, G. S., & Andersen, J. R. (2020) · BMJ Open · doi:10.1136/bmjopen-2020-037685 · Cited by 6 · see citing works
Varela, C., Oda-Montecinos, C., Andrés, A., & Saldaña, C. (2021) · Journal of Eating Disorders · doi:10.1186/s40337-021-00432-6 · Cited by 8 · see citing works
“However, traditionally, the intensity of feedback was studied based on its frequency and it was demonstrated that personalized feedback is a relevant variable to consider in webbased interventions [12, 28].”
BackgroundJiang, H., & Yin, X. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.918274 · Cited by 7 · see citing works
“Current research does not specifically study the establishment of user portraits, but for different business scenarios, user portraits are modeled as features of the recommendation system (Jaime et al., 2018).”
BackgroundPredicting Undesired Treatment Outcomes With Machine Learning in Mental Health Care: Multisite Study
van Mens, K., Lokkerbol, J., Wijnen, B. F. M., Janssen, R. T. J. M., de Lange, R. P. J., & Tiemens, B. G. (2023) · JMIR Medical Informatics · doi:10.2196/44322 · Cited by 7 · see citing works
“Inspired by this knowledge, the study aims to use machine learning on ROM data as a feedback device to signal which patients have an elevated risk of a poor response to treatment [16].”
Progress towards clinically informative data-driven decision support tools in psychotherapy
Webb, C. A., & Cohen, Z. D. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00042-x · Cited by 7 · see citing works
Wade, T. D., & Waller, G. (2025) · Behaviour Research and Therapy · doi:10.1016/j.brat.2025.104826 · Cited by 4 · see citing works
Flückiger, C., Barkham, M., & Penedo, J. M. G. (2026) · Psychotherapy Research · doi:10.1080/10503307.2026.2653992 · Cited by 6 · see citing works
Delamain, H., Buckman, J. E. J., O’Driscoll, C., Suh, J. W., Stott, J. C. H., Singh, S., Naqvi, S. A., Leibowitz, J., Pilling, S., & Saunders, R. (2024) · Psychiatry Research · doi:10.1016/j.psychres.2024.115910 · Cited by 6 · see citing works
“This process has been found to be successful with other data-informed treatment predictions in similar settings (Delgadillo et al., 2018).”
ResultThe Effect of Patient Progress Feedback on Psychotherapy Outcome
Schöttke, H., Unrath, M., & Uhlmann, C. (2019) · Verhaltenstherapie · doi:10.1159/000503765 · Cited by 6 · see citing works
“Progress monitoring occurred once per quarter in the present study, whereas other studies conducted weekly assessments [Delgadillo et al., 2018].”
BackgroundMethodResultInfluentialGuided Self-help Teletherapy for Behavioural Difficulties in Children with Epilepsy
Bennett, S. D., Heyman, I., Varadkar, S., Coughtrey, A. E., Walji, F., & Shafran, R. (2021) · Journal of Clinical Psychology in Medical Settings · doi:10.1007/s10880-021-09768-2 · Cited by 6 · see citing works
Bongers, I. L., Buitenweg, D. C., van Kuijk, R. E. F. M., & van Nieuwenhuizen, C. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph191710834 · Cited by 7 · see citing works
Validation of the intersession alliance measure: Individual, couple, and family versions.
Anderson, S. R., Johnson, L. N., Witting, A. B., Miller, R. B., Bradford, A. B., Hunt, Q. A., & Bean, R. A. (2024) · Journal of Marital & Family Therapy · doi:10.1111/jmft.12702 · Cited by 6 · see citing works
Diagnostik in der Psychotherapie
Stieglitz, R., & Spitzer, C. (2018) · Psychotherapeut · doi:10.1007/s00278-018-0308-y · Cited by 6 · see citing works
eXplainable AI for routine outcome monitoring and clinical feedback
Westbye, H. J., Moltu, C., & McAleavey, A. A. (2024) · Counselling and Psychotherapy Research · doi:10.1002/capr.12764 · Cited by 5 · see citing works
Chong, M. K., Hickie, I. B., Ekambareshwar, M., Poulsen, A., McKenna, S., Hutcheon, A., Hockey, S. J., Scott, E. M., & Iorfino, F. (2025) · BMC Health Services Research · doi:10.1186/s12913-025-12889-1 · Cited by 5 · see citing works
Delgadillo, J., Nissen‐Lie, H. A., de Jong, K., Schroder, T. A., & Barkham, M. (2024) · Psychotherapy Research · doi:10.1080/10503307.2024.2310635 · Cited by 6 · see citing works
“In one study, therapists with low job satisfaction and high occupational burnout (specifically an attitude of disengagement with work) were found to be less effective than those who reported lower levels of burnout (Delgadillo et al., 2018a).”
Self-citationLotfaliany, M., Agustini, B., Walker, A. J., Turner, A., Wrobel, A. L., Williams, L. J., Dean, O. M., Miles, S., Rossell, S. L., Berk, M., et al. (2024) · Australian & New Zealand Journal of Psychiatry · doi:10.1177/00048674241253452 · Cited by 5 · see citing works
Rapee, R. M., Kuhnert, R., Bowsher, I., Burns, J. R., Coen, J., Dixon, J. E., Kotselas, P., Lourey, C., McLellan, L. F., Mihalopoulos, C., et al. (2025) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1016/j.jaac.2025.01.032 · Cited by 5 · see citing works
Malins, S., Owen, R., Wright, I., Borrill, H., Limond, J., Gibson, F., Grundy, R. G., Bailey, S., Clifford, S. C., Lowis, S. P., et al. (2021) · BMJ Open · doi:10.1136/bmjopen-2021-051091 · Cited by 5 · see citing works
Senger, K., Rubel, J. A., Kleinstäuber, M., Schröder, A., Köck, K., Lambert, M. J., Lutz, W., & Heider, J. (2021) · Psychotherapy Research · doi:10.1080/10503307.2021.1993376 · Cited by 5 · see citing works
“…as symptom characteristics such as intensity (e.g., Cella et al., 2011; Crawley et al., 2013), duration (e.g., Burger et al., 2016; Crawley et al., 2013) or comorbidity (e.g., Flo & Chalder, 2014; Gilpin et al., 2017; Heider & Zaby, 2010) have been repeatedly investigated with conflicting reports.”
BackgroundSelf-citationMeasurement-Based Care Implementation Challenges and Opportunities in Real-World Practice
Ridout, K. K., Vanderlip, E. R., Ridout, S. J., & Alter, C. L. (2023) · Psychiatric Annals · doi:10.3928/00485713-20230818-02 · Cited by 5 · see citing works
Perspektiven einer evidenzbasierten und personalisierten Psychotherapie
Lutz, W., Clausen, S. A., & Deisenhofer, A. (2019) · Zeitschrift für Klinische Psychologie und Psychotherapie · doi:10.1026/1616-3443/a000518 · Cited by 5 · see citing works
Self-citationMalins, S., Moghaddam, N. G., Morriss, R. K., & Schroder, T. A. (2019) · Psychotherapy Research · doi:10.1080/10503307.2019.1657250 · Cited by 5 · see citing works
“Current evidence from Randomized Controlled Trials (RCTs) and subsequent meta-analyses suggest that ROM can have beneficial effects for both treat-ment efficacy and efficiency (Delgadillo et al., 2018; Lambert, Whipple, & Kleinstäuber, 2018; Shimokawa, Lambert, & Smart, 2010).”
BackgroundLindegaard, T., Persson, L., Thorängen, M., & Rozental, A. (2024) · Psychotherapy Research · doi:10.1080/10503307.2024.2420717 · Cited by 4 · see citing works
“…of negative effects of psychotherapy indicate that around 5–10% of all patients with common mental disorders deteriorate during treatment (Cuijpers et al., 2018; Delgadillo et al., 2018), defined as a change in a negative direction that exceeds measurement error (i.e., the Reliable Change Index).”
James, K., Saxon, D. H., & Barkham, M. (2024) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-024-01403-0 · Cited by 4 · see citing works
Flygare, O., Wallert, J., Chen, L. L., de la Cruz, L. F., Lundström, L., Mataix‐Cols, D., Rück, C., & Andersson, E. (2022) · Behavior Therapy · doi:10.1016/j.beth.2022.06.009 · Cited by 4 · see citing works
Cawthorne, T., Käll, A., Bennett, S. D., Andersson, G., & Shafran, R. (2022) · PLoS ONE · doi:10.1371/journal.pone.0278746 · Cited by 4 · see citing works
“Goal-based outcomes have been shown to improve treatment retention, clinical outcomes and client progress [62, 63].”
BackgroundHuang, S., Wahlquist, A. E., & Dahne, J. (2024) · Substance Use & Misuse · doi:10.1080/10826084.2024.2369155 · Cited by 4 · see citing works
“Another effective approach involves closely monitoring the depressive symptoms of users with lower education levels (e.g., by their primary care providers) throughout treatment (Delgadillo et al., 2018).”
Data-Informed Case Formulation with the Trier Treatment Navigator
Danilo Moggia, N., Schaffrath, J., Bommer, J., Weinmann‐Lutz, B., & Lutz, W. (2022) · Revista de Psicoterapia · doi:10.33898/rdp.v33i123.35971 · Cited by 4 · see citing works
“, CDSS, ROM and feedback systems, CSTs) contributes to better treatment outcomes (Cohen et al., 2021; de Jong et al., 2021; Delgadillo et al., 2018).”
BackgroundSelf-citationBasnet, S., & Chaiton, M. (2023) · JMIR Research Protocols · doi:10.2196/48703 · Cited by 4 · see citing works
“13 for changes in the GAD-7 score (based on the expected effects of the dashboard) [15], an expected dropout rate of no more than 35% [16], and population mean and SD estimates of PHQ-9 and GAD-7 scores [17,18], we estimate having a power of 0.90 and a type I error rate of 0.05 for detecting…”
InfluentialSong, J., Freedman, G., Li, L., & Persons, J. B. (2024) · British Journal of Clinical Psychology · doi:10.1111/bjc.12470 · Cited by 3 · see citing works
Mahon, D., Minami, T., & Brown, J. (2024) · Counselling and Psychotherapy Research · doi:10.1002/capr.12827 · Cited by 3 · see citing works
Using Client and Supervision Feedback to Improve Supervision in Health Care
Mahon, D. (2022) · Counseling and Family Therapy Scholarship Review · doi:10.53309/2576-926x.1044 · Cited by 3 · see citing works
Krause, K. R., Calderón, A., Pino, V. G., Edbrooke-Childs, J. H., Moltrecht, B., & Wolpert, M. (2023) · European Child & Adolescent Psychiatry · doi:10.1007/s00787-023-02140-9 · Cited by 3 · see citing works
Blackwell, S. E. (2024) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-023-01340-4 · Cited by 3 · see citing works
- Authors
- Delgadillo, J., de Jong, K., Lucock, M., Lutz, W., Rubel, J., Gilbody, S., Ali, S., Aguirre, E., Appleton, M., Nevin, J., O'Hayon, H., Patel, U., Sainty, A., Spencer, P., & McMillan, D.
- Affiliations
- University of Sheffield (GB)
- University of Huddersfield (GB)
- Universität Trier (DE)
- University of York (GB)
- National Health Service (GB)
- Pennine Care NHS Foundation Trust (GB)
- Cheshire and Wirral Partnership NHS Foundation Trust (GB)
- Whittington Health NHS Trust (GB)
- Cambridgeshire and Peterborough NHS Foundation Trust (GB)
- Humber Teaching NHS Foundation Trust (GB)
- South West Yorkshire Partnership NHS Foundation Trust (GB)
- Corresponding author
- Delgadillo, Jaime
- ORCID
- Venue
- The Lancet Psychiatry
- Publisher
- Elsevier BV
- Published
- 2018-07
- Volume
- 5
- Issue
- 7
- Pages
- 564-572
- ISSN
- 2215-0366 (ISSN-L)
- OpenAlex
- W2809000673
- Cited by
- 152
- References
- 31
- Citations by year
- 2026: 13 · 2025: 15 · 2024: 28 · 2023: 13 · 2022: 19 · 2021: 21 · 2020: 19 · 2019: 20 · 2018: 3
- FWCI
- 18.76
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- The Lancet Psychiatry (publishedVersion)
- PubMed (publishedVersion)
- Huddersfield Research Portal (University of Huddersfield) (publishedVersion, cc-by-nc-nd)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- Leiden Repository (Leiden University) (submittedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Topics
- Social Sciences › Psychology › Clinical Psychology › Psychotherapy Techniques and Applications
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Anxiety, Depression, Psychometrics, Treatment, Cognitive Processes
- Keywords
- outcome feedback, feedback-informed treatment, depression, anxiety, stepped care, psychotherapy, clinical supervision, low-intensity psychological interventions, symptom severity, treatment progress monitoring, automated alerts, individual psychotherapy, waiting lists, NHS mental health services, intention-to-treat analysis
- MeSH terms
- Adult, Anxiety Disorders, Cost-Benefit Analysis, Depressive Disorder, England, Feedback, Female, Humans, Male, Middle Aged, Psychotherapy, Quality of Life, State Medicine, Treatment Outcome, Evidence-Based Practice
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
