CoChat
Sign Up
Cited by 108Digital Mental Health InterventionsPsychotherapy Techniques and ApplicationsAnxiety, Depression, Psychometrics, Treatment, Cognitive Processes

Quantifying the association between psychotherapy content and clinical outcomes using deep learning

Open access
Ewbank, Michael P., Cummins, Ronan, Tablan, Valentin, Bateup, Sarah, Catarino, Ana, Martin, Alan J., Blackwell, Andrew D.·2020·JAMA Psychiatry
DOI 10.1001/jamapsychiatry.2019.2664PMC PMC6707006
Open this paper in CoChat — free accountOpen PDF

Abstract

Importance: Compared with the treatment of physical conditions, the quality of care of mental health disorders remains poor and the rate of improvement in treatment is slow, a primary reason being the lack of objective and systematic methods for measuring the delivery of psychotherapy. Objective: To use a deep learning model applied to a large-scale clinical data set of cognitive behavioral therapy (CBT) session transcripts to generate a quantifiable measure of treatment delivered and to determine the association between the quantity of each aspect of therapy delivered and clinical outcomes. Design, Setting, and Participants: All data were obtained from patients receiving internet-enabled CBT for the treatment of a mental health disorder between June 2012 and March 2018 in England. Cognitive behavioral therapy was delivered in a secure online therapy room via instant synchronous messaging. The initial sample comprised a total of 17 572 patients (90 934 therapy session transcripts). Patients self-referred or were referred by a primary health care worker directly to the service. Exposures: All patients received National Institute for Heath and Care Excellence-approved disorder-specific CBT treatment protocols delivered by a qualified CBT therapist. Main Outcomes and Measures: Clinical outcomes were measured in terms of reliable improvement in patient symptoms and treatment engagement. Reliable improvement was calculated based on 2 severity measures: Patient Health Questionnaire (PHQ-9) and Generalized Anxiety Disorder 7-item scale (GAD-7), corresponding to depressive and anxiety symptoms respectively, completed by the patient at initial assessment and before every therapy session (see eMethods in the Supplement for details). Results: Treatment sessions from a total of 14 899 patients (10 882 women) aged between 18 and 94 years (median age, 34.8 years) were included in the final analysis. We trained a deep learning model to automatically categorize therapist utterances into 1 or more of 24 feature categories. The trained model was applied to our data set to obtain quantifiable measures of each feature of treatment delivered. A logistic regression revealed that increased quantities of a number of session features, including change methods (cognitive and behavioral techniques used in CBT), were associated with greater odds of reliable improvement in patient symptoms (odds ratio, 1.11; 95% CI, 1.06-1.17) and patient engagement (odds ratio, 1.20, 95% CI, 1.12-1.27). The quantity of nontherapy-related content was associated with reduced odds of symptom improvement (odds ratio, 0.89; 95% CI, 0.85-0.92) and patient engagement (odds ratio, 0.88, 95% CI, 0.84-0.92). Conclusions and Relevance: This work demonstrates an association between clinical outcomes in psychotherapy and the content of therapist utterances. These findings support the principle that CBT change methods help produce improvements in patients' presenting symptoms. The application of deep learning to large clinical data sets can provide valuable insights into psychotherapy, informing the development of new treatments and helping standardize clinical practice.

AI snapshot

Written from the paper’s full text.

TL;DR

A deep learning model categorized therapist utterances from approximately 90,000 hours of internet-enabled CBT and found that greater quantities of CBT change methods were associated with higher odds of reliable improvement and engagement, while nontherapy-related content was associated with lower odds. The study supports key CBT principles and demonstrates large-scale psychotherapy content quantification, but it is observational and cannot establish causality or session quality.

Key findings

  • In 90,934 transcripts from 17,572 patients (reliable improvement rate 63.4%; IAPT engagement rate 87.3%), “change methods” was positively associated with reliable improvement (OR, 1.11; 95% CI, 1.06-1.17) and with first-session IAPT engagement (OR, 1.20; 95% CI, 1.12-1.27).
  • Nontherapy-related “other” content was negatively associated with reliable improvement (OR, 0.89; 95% CI, 0.85-0.92) and engagement (OR, 0.88; 95% CI, 0.84-0.92); “hello” and “goodbye” were also negatively associated with improvement.
  • Other positive associations with reliable improvement included therapeutic praise (OR, 1.21; 95% CI, 1.15-1.27), planning for the future (OR, 1.12; 95% CI, 1.06-1.19), perceptions of change (OR, 1.11; 95% CI, 1.06-1.16), set agenda (OR, 1.08; 95% CI, 1.02-1.14), elicit feedback (OR, 1.06; 95% CI, 1.02-1.11), give feedback (OR, 1.05; 95% CI, 1.00-1.10), and review homework (OR, 1.04; 95% CI, 1.00-1.09). Negative associations included therapeutic empathy (OR, 0.84; 95% CI, 0.81-0.88), risk check (OR, 0.85; 95% CI, 0.81-0.89), and bridge (OR, 0.95; 95% CI, 0.91-0.98).
  • First-session engagement was positively associated with change methods, elicit feedback, set homework, arrange next session, therapeutic thanks, and formulation; it was negatively associated with other, hello, therapeutic empathy, Socratic questioning, bridge, and planning for the future.

Methods

  • Quality improvement study using internet-enabled CBT data from June 2012 to March 2018; outcomes were reliable improvement (PHQ-9/GAD-7) and IAPT engagement (attending ≥2 treatment sessions).
  • A research psychologist annotated 290 therapy session transcripts using 24 feature categories informed by the CBT competences framework and Revised Cognitive Therapy Scale; a deep learning model was trained to classify all therapist utterances, with interrater reliability κ = 0.54.
  • Multivariable logistic regressions examined reliable improvement (n = 13,073) and first-session IAPT engagement (n = 14,899); predictors included mean words per feature plus demographics, starting PHQ-9/GAD-7 scores, medication status, long-term condition, number of sessions, and session duration; continuous predictors were scaled and centered, with significance at P < .05 two-tailed.

Limitations

  • The paper states it cannot determine whether a therapeutic feature was applied appropriately or whether the therapist adhered to the CBT protocol; the model measures association across sessions rather than quality of an individual session.
  • The paper states the model does not assess how treatment was received by patients, results reveal association rather than causation, long-term outcomes are not addressed, and other patient factors are not included; for large data sets, ORs and confidence intervals should be considered more informative than statistical significance alone.
  • Evident from the text: the design is observational, and interrater reliability was moderate (κ = 0.54).

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1001/jamapsychiatry.2019.2664

Cites

33 of 33 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 28 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Increases in therapeutic empathy were negatively associated with reliable improvement (OR, 0.84; 95% CI, 0.81-0.88).

    “increases in nontherapy-related content (“other” [OR, 0.89; 95% CI, 0.85-0.92], “hello” [OR, 0.92; 95% CI, 0.88-0.96], and “goodbye” [OR, 0.95; 95% CI, 0.91-0.99]), along with “therapeutic empathy” (OR, 0.84; 95% CI, 0.81-0.88), “risk check” (OR, 0.85; 95% CI, 0.81-0.89), and “bridge” (OR, 0.95; 95% CI, 0.91-0.98) were negatively associated with improvement.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The mean number of words for each therapy feature is a valid measure of the quantity of that feature delivered in therapy.
  • Findingown data

    Increased quantities of CBT change methods were positively associated with reliable improvement in patient symptoms.

    “Increased quantities of CBT change methods were positively associated with reliable improvement in patient symptoms”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The mean number of words for each therapy feature is a valid measure of the quantity of that feature delivered in therapy.
  • Findingown data

    Increases in the 'hello' feature were negatively associated with reliable improvement (OR, 0.92; 95% CI, 0.88-0.96).

    “increases in nontherapy-related content (“other” [OR, 0.89; 95% CI, 0.85-0.92], “hello” [OR, 0.92; 95% CI, 0.88-0.96], and “goodbye” [OR, 0.95; 95% CI, 0.91-0.99]), along with “therapeutic empathy” (OR, 0.84; 95% CI, 0.81-0.88), “risk check” (OR, 0.85; 95% CI, 0.81-0.89), and “bridge” (OR, 0.95; 95% CI, 0.91-0.98) were negatively associated with improvement.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The mean number of words for each therapy feature is a valid measure of the quantity of that feature delivered in therapy.
  • Findingown data

    Increases in bridge were negatively associated with reliable improvement (OR, 0.95; 95% CI, 0.91-0.98).

    “increases in nontherapy-related content (“other” [OR, 0.89; 95% CI, 0.85-0.92], “hello” [OR, 0.92; 95% CI, 0.88-0.96], and “goodbye” [OR, 0.95; 95% CI, 0.91-0.99]), along with “therapeutic empathy” (OR, 0.84; 95% CI, 0.81-0.88), “risk check” (OR, 0.85; 95% CI, 0.81-0.89), and “bridge” (OR, 0.95; 95% CI, 0.91-0.98) were negatively associated with improvement.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The mean number of words for each therapy feature is a valid measure of the quantity of that feature delivered in therapy.
  • Findingown data

    Increases in risk check were negatively associated with reliable improvement (OR, 0.85; 95% CI, 0.81-0.89).

    “increases in nontherapy-related content (“other” [OR, 0.89; 95% CI, 0.85-0.92], “hello” [OR, 0.92; 95% CI, 0.88-0.96], and “goodbye” [OR, 0.95; 95% CI, 0.91-0.99]), along with “therapeutic empathy” (OR, 0.84; 95% CI, 0.81-0.88), “risk check” (OR, 0.85; 95% CI, 0.81-0.89), and “bridge” (OR, 0.95; 95% CI, 0.91-0.98) were negatively associated with improvement.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The mean number of words for each therapy feature is a valid measure of the quantity of that feature delivered in therapy.
  • May combine two claims. 17
  • Classified as probably not a substantial claim. 28

  • May combine two claims.

    Findingown data

    The initial sample comprised 90,934 session transcripts from 17,572 patients, with a reliable improvement rate of 63.4% and IAPT engagement rate of 87.3%.

    “The initial sample comprised a total of 90 934 session transcripts taken from 17 572 patients, with a reliable improvement rate of 63.4% and IAPT engagement rate of 87.3%.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    Both studies report a positive association between GAD-7 scores and reliable improvement.

    “Both studies report a positive association between GAD-7 scores and reliable improvement.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [32] Catarino A, Bateup S, Tablan V,. Demographic and clinical predictors of response to internet-enabled cognitive-behavioural therapy for depression and anxiety. BJPsych Open. 2018; 4(5): 411- 418. doi: 10.1192/bjo.2018.57 30294451 PMC6171334

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    Factors specific to CBT and factors common to most psychotherapies are associated with increased odds of reliable improvement in patient symptoms.

    “We find that factors specific to CBT, as well as factors common to most psychotherapies, are associated with increased odds of reliable improvement in patient symptoms.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Neither planning for the future nor discussing perceptions of change was significantly associated with outcomes in the first treatment session.

    “Consistent with this, neither feature was significantly associated with outcomes in the first treatment session (eTable 5 in the Supplement).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    Multicollinearity analyses revealed that variance inflation factors were smaller than 2 for all predictor variables, confirming that regression models were not affected by multicollinearity.

    “Multicollinearity analyses revealed that variance inflation factors were smaller than 2 for all predictor variables, confirming that regression models were not affected by the presence of multicollinearity.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given

    A discussion of perceptions of change is only likely to occur following some degree of change.

    “A discussion of perceptions of change is only likely to occur following some degree of change”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    Reliable improvement was calculated based on the Patient Health Questionnaire (PHQ-9) and Generalized Anxiety Disorder 7-item scale (GAD-7).

    “Reliable improvement was calculated based on 2 severity measures: Patient Health Questionnaire (PHQ-9)21 and Generalized Anxiety Disorder 7-item scale (GAD-7),22 corresponding to depressive and anxiety symptoms respectively, completed by the patient at initial assessment and before every therapy session”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [21] Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. 2001; 16(9): 606- 613. doi: 10.1046/j.1525-1497.2001.016009606.x 11556941 PMC1495268 source

      Not checked in the free run.

    • cites [22] Spitzer RL, Kroenke K, Williams JB, Löwe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med. 2006; 166(10): 1092- 1097. doi: 10.1001/archinte.166.10.1092 16717171 source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Rather than the quantity of conversation, it is the therapeutic nature of conversation and/or the dosage of therapy delivered in a session that is associated with improvement in patient symptoms.

    “this suggests that rather than the quantity of conversation, it is the therapeutic nature of conversation and/or the dosage of therapy delivered in a session that is associated with improvement in patient symptoms.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Cases with missing start or end PHQ-9 or GAD-7 scores (n = 1338) were excluded from the analysis.

    “Cases with missing start or end PHQ-9 or GAD-7 scores (n = 1338) were excluded from the analysis.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    The relative effects of different psychotherapeutic interventions appear similar.

    “the relative effects of different psychotherapeutic interventions appear similar.10”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [10] Barth J, Munder T, Gerger H,. Comparative efficacy of seven psychotherapeutic interventions for patients with depression: a network meta-analysis. PLoS Med. 2013; 10(5): e1001454. doi: 10.1371/journal.pmed.1001454 23723742 PMC3665892 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    A total of 24 feature categories were defined, informed by the CBT competences framework and the Revised Cognitive Therapy Scale.

    “We defined a total of 24 feature categories (Box), informed by the CBT competences framework18 and the Revised Cognitive Therapy Scale.23”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [18] Roth AD, Pilling S. Using an evidence-based methodology to identify the competences required to deliver effective cognitive and behavioural therapy for depression and anxiety disorders.. Behav Cogn Psychother. 2008; 36(February): 129- 147. doi: 10.1017/S1352465808004141 source

      Not checked in the free run.

    • cites [23] Blackburn I, James IA, Milne DL,. The revised cognitive therapy scale (CTS-R): psychometric properties. Behav Cogn Psychother. 2001; 29: 431- 446. doi: 10.1017/S1352465801004040 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The final analysis for IAPT engagement included 14,899 patients.

    “making a total of 14 899 patients, at caseness.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductcitation

    Clinical outcomes were defined according to IAPT guidelines and measured as reliable improvement and IAPT engagement.

    “Clinical outcomes were defined according to IAPT guidelines19 and were measured in terms of reliable improvement and IAPT engagement and included as binary measures (ie, 0 or 1).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [19] Clark DM. Implementing NICE guidelines for the psychological treatment of depression and anxiety disorders: the IAPT experience. Int Rev Psychiatry. 2011; 23(4): 318- 327. doi: 10.3109/09540261.2011.606803 22026487 PMC3212920 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    The results of such interventions are difficult to transfer to real-world psychotherapy and require sample sizes larger than typically used.

    “the results of such interventions are difficult to transfer to real-world psychotherapy13 and require sample sizes larger than typically used.14”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [13] Doss BD, Atkins DC. Investigating treatment mediators when simple random assignment to a control group is not possible. Clin Psychol Sci Pract. 2006; 13: 321- 336. doi: 10.1111/j.1468-2850.2006.00045.x source

      Not checked in the free run.

    • cites [14] Bell EC, Marcus DK, Goodlad JK. Are the parts as good as the whole? a meta-analysis of component treatment studies. J Consult Clin Psychol. 2013; 81(4): 722- 736. doi: 10.1037/a0033004 23688145 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    An extended period focused on risk is likely to cause a deviation in the structure of the session and a subsequent reduction in the dosage of active therapy delivered.

    “An extended period focused on risk is also likely to cause a deviation in the structure of the session and a subsequent reduction in the dosage of active therapy delivered.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    The final analysis for reliable improvement included 13,073 patients.

    “leaving a total of 13 073 patients (at a clinical caseness threshold and engaged in treatment) in the analysis.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Premisecitation

    To determine the most effective components of CBT and whether CBT works via the mechanisms proposed by the approach, quantifiable measures of treatment delivered need to be obtained in a natural clinical context and be gathered from a sufficiently large enough sample to draw meaningful conclusions.

    “To determine the most effective components of CBT and whether CBT works via the mechanisms proposed by the approach,15 quantifiable measures of treatment delivered need to be obtained in a natural clinical context and be gathered from a sufficiently large enough sample to draw meaningful conclusions.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [15] Lorenzo-Luaces L, German RE, DeRubeis RJ. It’s complicated: the relation between cognitive change procedures, cognitive change, and symptom change in cognitive therapy for depression. Clin Psychol Rev. 2015; 41: 3- 15. doi: 10.1016/j.cpr.2014.12.003 25595660 PMC10080251 source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    The positive association between CBT change method-related content and outcomes supports the key principles underlying CBT and provides validation for CBT as a treatment.

    “This finding supports the key principles underlying CBT and provides validation for CBT as a treatment (ie, modifying cognitive and behavioral factors produces improvements in patient symptoms).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The final treatment session was excluded because outcome measures are taken prior to the commencement of each treatment session.

    “The final treatment session was excluded because outcome measures are taken prior to the commencement of each treatment session.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Patients provide written informed consent allowing the service to use their anonymized data for audit purposes and to support research, including academic publications.

    “When registering to use the Ieso service, patients provide written informed consent as part of a privacy policy agreement, allowing the service to use their anonymized data for audit purposes and to support research, including academic publications.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    Cases with a mean of fewer than 50 patient words were excluded (n = 16), and sessions with a total of fewer than 50 patient words were excluded (n = 121).

    “Cases with a mean of fewer than 50 patient words were excluded (n = 16)”
    • Also stated in: § Statistical Analysis · plain
      “Sessions with a total of fewer than 50 patient words were excluded (n = 121)”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation

    Whether treatments work through specific factors (eg, CBT change methods) or factors common to most psychotherapies (eg, therapeutic alliance) remains a core issue in the field.

    “whether treatments work through specific factors (eg, CBT change methods) or factors common to most psychotherapies (eg, therapeutic alliance) remains a core issue in the field.11,12”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [11] Blagys MD, Hilsenroth MJ. Distinctive activities of cognitive-behavioral therapy: a review of the comparative psychotherapy process literature. Clin Psychol Rev. 2002; 22(5): 671- 706. doi: 10.1016/S0272-7358(01)00117-9 12113201 source

      Not checked in the free run.

    • cites [12] Mulder R, Murray G, Rucklidge J. Common versus specific factors in psychotherapy: opening the black box. Lancet Psychiatry. 2017; 4(12): 953- 962. doi: 10.1016/S2215-0366(17)30100-1 28689019 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Internet-enabled CBT has been shown to be clinically effective for the treatment of depression.

    “Internet-enabled CBT has been shown to be clinically effective for the treatment of depression16 and is currently deployed within IAPT.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [16] Kessler D, Lewis G, Kaur S,. Therapist-delivered internet psychotherapy for depression in primary care: a randomised controlled trial. Lancet. 2009; 374(9690): 628- 634. doi: 10.1016/S0140-6736(09)61257-5 19700005 source

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    Giving and eliciting feedback helps both the therapist and patient develop a greater understanding of key issues and possibly strengthens the therapeutic alliance.

    “The results also support the principle that giving and eliciting feedback helps both the therapist and patient develop a greater understanding of key issues and possibly strengthens the therapeutic alliance.27”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [27] Beck JS. Cognitive Behavior Therapy: Basics and Beyond. 2nd ed. New York, NY: Guilford Press; 2011.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Premisecitation

    Studies commonly use observational coding methods to investigate the association between treatment delivered and outcomes.

    “Studies commonly use observational coding methods (eg, ratings/transcription of recorded therapeutic conversations) to investigate the association between treatment delivered and outcomes.5”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [5] Perepletchikova F. On the topic of treatment integrity. Clin Psychol (New York). 2011; 18(2): 148- 153. doi: 10.1111/j.1468-2850.2011.01246.x 21769167 PMC3137485

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationcitation

    Such monitoring could help arrest therapist drift.

    “Such monitoring could help arrest therapist drift, ie, the failure to deliver treatments a therapist has been trained to deliver, which may be one of the biggest factors contributing to poor delivery of treatment.33”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [33] Waller G. Evidence-based treatment and therapist drift. Behav Res Ther. 2009; 47(2): 119- 127. doi: 10.1016/j.brat.2008.10.018 19036354 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    Some aspects of therapy (eg, change methods) are typically initiated by the therapist and appear likely to play a causal role.

    “although some aspects of therapy (eg, change methods) are typically initiated by the therapist and appear likely to play a causal role.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    Other patient factors not included are likely to play a role in determining outcomes.

    “In addition, other patient factors not included are likely to play a role in determining outcomes.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Homework in CBT is used to help patients practice skills learned in therapy and generalize these skills to the real world.

    “Homework in CBT is used to help patients practice skills learned in therapy and generalize these skills to the real world.25”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [25] Alford BA, Beck AT. Cognitive therapy of delusional beliefs. Behav Res Ther. 1994; 32(3): 369- 380. doi: 10.1016/0005-7967(94)90134-1 8192636 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    This approach represents an important step in developing a data-driven understanding of mental health treatment and in improving the efficacy of psychotherapy.

    “we believe this approach represents an important step in developing a data-driven understanding of mental health treatment and in improving the efficacy of psychotherapy.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Previous work indicates therapeutic alliance may be reflective of a change in symptoms.

    “although previous work indicates therapeutic alliance may be reflective of a change in symptoms.31”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [31] Strunk DR, Brotman MA, DeRubeis RJ. The process of change in cognitive therapy for depression: predictors of early inter-session symptom gains. Behav Res Ther. 2010; 48(7): 599- 606. doi: 10.1016/j.brat.2010.03.011 20362978 PMC2878902

      Not checked in the free run.

  • May combine two claims.

    Study design and conductnone given

    It is not possible to determine whether a therapeutic feature is applied in an appropriate manner or whether a therapist adheres to the CBT protocol.

    “A limitation of our approach is that it is not possible to determine whether a therapeutic feature is applied in an appropriate manner or whether a therapist adheres to the CBT protocol.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    This work represents a first step toward a practicable approach for quality controlled behavioral health care.

    “We believe that this work represents a first step toward a practicable approach for quality controlled behavioral health care.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The model does not assess how the treatment was received by patients.

    “In addition, the model does not assess how the treatment was received by patients.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Common factors, such as therapeutic alliance, are thought to play a role in all psychotherapeutic treatments and show a moderate association with outcomes.

    “Common factors, such as therapeutic alliance, are thought to play a role in all psychotherapeutic treatments29 and show a moderate association with outcomes.30”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [29] Wampold BE. How important are the common factors in psychotherapy? an update. World Psychiatry. 2015; 14(3): 270- 277. doi: 10.1002/wps.20238 26407772 PMC4592639 source

      Not checked in the free run.

    • cites [30] Martin DJ, Garske JP, Davis MK. Relation of the therapeutic alliance with outcome and other variables: a meta-analytic review. J Consult Clin Psychol. 2000; 68(3): 438- 450. doi: 10.1037/0022-006X.68.3.438 10883561 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductown data

    The logistic regression model for IAPT engagement was adjusted for symptom severity, patient sex, age, medication status, presence of long-term condition, and session duration.

    “Adjusted for symptom severity, patient sex, age, medication status, presence of long-term condition, and session duration.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    Increased praise may be reflective of a patient responding well to treatment.

    “Similarly, increased praise may be reflective of a patient responding well to treatment.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    The logistic regression model for reliable improvement was adjusted for total number of sessions, symptom severity, patient sex, age, medication status, presence of long-term condition, and session duration.

    “Adjusted for total number of sessions, symptom severity, patient sex, age, medication status, presence of long-term condition, and session duration.”
    Quote in the paperSubstantial claim
38 of 38

  • The paper relies on this without support.

    The mean number of words for each therapy feature is a valid measure of the quantity of that feature delivered in therapy.

    “Using the output of the model, the mean number of words for each feature, averaged across all sessions, was calculated for each case.”

  • The paper relies on this without support.

    The exclusion of cases with fewer than 50 patient words does not introduce bias into the association estimates.

    “Cases with a mean of fewer than 50 patient words were excluded (n = 16)”
    “Sessions with a total of fewer than 50 patient words were excluded (n = 121)”

  • The paper relies on this without support.

    The sample of patients receiving IECBT is representative of the broader population of patients receiving CBT for common mental health disorders.

    “The findings support the key principles underlying CBT as a treatment”

  • The paper relies on this without support.

    The 'other' feature category represents nontherapy-related content.

    “increases in nontherapy-related content (“other” [OR, 0.89; 95% CI, 0.85-0.92]”

  • The paper relies on this without support.

    The mean number of words per feature in therapist utterances is a valid measure of the quantity of therapeutic content delivered.

    “mean number of words per feature across treatment”

  • The paper relies on this without support.

    The deep learning model accurately categorizes therapist utterances into the feature categories.

    “automatically categorize therapist utterances”

  • The paper relies on this without support.

    Reliable improvement in PHQ-9/GAD-7 scores is a valid indicator of clinical improvement.

    “reliable improvement in patient symptoms”

  • The paper relies on this without support.

    IAPT engagement is a valid indicator of patient engagement.

    “IAPT engagement”

  • The paper relies on this without support.

    The association between therapy content and outcomes is not confounded by unmeasured variables.

    “Adjusted for total number of sessions, symptom severity, patient sex, age, medication status, presence of long-term condition, and session duration.”

  • The paper relies on this without support.

    The first treatment session is representative of the therapy content.

    “first treatment session”

  • The paper relies on this without support.

    The 90,000 hours of IECBT are representative of CBT practice.

    “approximately 90 000 hours of IECBT”

  • The paper relies on this without support.

    A typical IAPT therapist may accrue approximately 30,000 therapy hours throughout a career.

    “approximately 30 000 therapy hours”
12 of 12

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Studies that ranked high for selective reporting bias tended to report only user satisfaction data and treatment adherence, despite their methods considering collection and assessment of psychiatric scale data and other effectiveness outcomes.

    “The studies that ranked high for selective reporting bias tended to report only user satisfaction data and treatment adherence, despite their methods considering the collection and assessment of psychiatric scale data and other effectiveness outcomes (29, 76, 79, 85–88, 92).”

    From Examining the role of AI technology in online mental healthcare: opportunities, challenges, and implications, a mixed-methods review (2024) · cites this paper as [86]

    Split into 2 atoms — 1 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • The relevant studies highlighted the importance of analyzing the large datasets produced by online mental healthcare interventions and the efficiency and accuracy of AI algorithms, which in some cases were comparable to or better than human assessors.

    “As such, the relevant studies highlighted the importance of analyzing the large datasets produced by online mental healthcare interventions and the efficiency and accuracy of AI algorithms which in some cases were comparable to or better than human assessors (6, 19, 28, 29, 31, 33, 83, 85, 86, 92–94).”

    From Examining the role of AI technology in online mental healthcare: opportunities, challenges, and implications, a mixed-methods review (2024) · cites this paper as [86]

    Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • The studies reporting on the use of ML algorithms found significant success in outcome prediction and feature detection to support online therapy delivery.

    “Additionally, the studies reporting on the use of ML algorithms found significant success in outcome prediction and feature detection to support online therapy delivery (6, 19, 28, 29, 31, 33, 83, 85, 86, 92–94).”

    From Examining the role of AI technology in online mental healthcare: opportunities, challenges, and implications, a mixed-methods review (2024) · cites this paper as [86]

    The Methods (Therapy Feature Categories; page unavailable) describe a deep learning model that classified therapist utterances in internet-enabled CBT, and the Results (Factors Associated With Reliable Improvement Across Treatment; page unavailable) link several detected features with improvement. The paper analyzes associations with outcomes; it does not report that the model successfully predicted patient outcomes. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • CBT is relatively simple to operationalize.

    “This makes CBT relatively simple to operationalize (86).”

    From Waiting for a digital therapist: three challenges on the path to psychotherapy delivered by artificial intelligence (2023) · cites this paper as [86]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Results give a different confidence limit for long-term conditions

    Minor

    The Results give an upper 95% confidence limit of 0.88 for a long-term medical condition, while Table 1 gives 0.80.

  • Results give a different odds ratio for "other" content

    Minor

    The Results report an odds ratio of 0.89, while Table 1 gives 0.88 for the same feature.

2 of 2

Cites

Flagged first, then by citations

1 of the 34 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Related

Authors
Ewbank, M. P., Cummins, R., Tablan, V., Bateup, S., Catarino, A., Martin, A. J., & Blackwell, A. D.
Affiliations
ieso Digital Health (UK) Ltd. (United Kingdom) (GB)
Corresponding author
Ewbank, Michael P.
ORCID
Venue
JAMA Psychiatry
Publisher
American Medical Association (AMA)
Published
2020-01-01
Volume
77
Issue
1
Pages
35
ISSN
2168-622X (ISSN-L)
DOI
10.1001/jamapsychiatry.2019.2664
PMC
PMC6707006
OpenAlex
W2969401291
Semantic Scholar
2c28755adfb8ef67337a98b07dd4c80f96e82e83
Cited by
108
References
34
Citations by year
2026: 16 · 2025: 14 · 2024: 15 · 2023: 12 · 2022: 25 · 2021: 12 · 2020: 14
FWCI
14.41
Type
journal article
Language
en
Open access
Licence
cc-by-nc-nd
Locations
  • JAMA Psychiatry (publishedVersion, cc-by-nc-nd)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
Topics
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
  • Social Sciences › Psychology › Clinical Psychology › Psychotherapy Techniques and Applications
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Anxiety, Depression, Psychometrics, Treatment, Cognitive Processes
Keywords
cognitive behavioral therapy, session transcripts, online therapy, treatment engagement, symptom improvement, depression, anxiety, intervention fidelity, deep learning, PHQ-9, GAD-7, mental illness, primary care referral, patient-reported outcomes
MeSH terms
Deep Learning, Adolescent, Adult, Aged, Mental Disorders, Female, Humans, Language, Male, Middle Aged, Psychiatric Status Rating Scales, Surveys and Questionnaires, Cognitive Behavioral Therapy, Treatment Outcome, Young Adult

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review