Using Psychological Artificial Intelligence (Tess) to Relieve Symptoms of Depression and Anxiety: Randomized Controlled Trial (Preprint)

Studies that ranked high for selective reporting bias tended to report only user satisfaction data and treatment adherence, despite their methods considering collection and assessment of psychiatric scale data and other effectiveness outcomes.

“The studies that ranked high for selective reporting bias tended to report only user satisfaction data and treatment adherence, despite their methods considering the collection and assessment of psychiatric scale data and other effectiveness outcomes (29, 76, 79, 85–88, 92).”

12 of the included studies presented data on treatment dropout; online or remote psychotherapy interventions reported a dropout rate between 0% and 61%, psychoeducation between 4% and 60%, and no intervention between 5% and 34%.

“12 of the included studies (41, 73–83) presented data on treatment dropout, online or remote psychotherapy interventions reported a dropout rate between 0% and 61%, psychoeducation a dropout rate between 4% and 60% and no intervention a dropout rate between 5% and 34%.”

A minority of users, around 30% or less, thought that the tool did not understand them or found their interactions unhelpful or bothersome.

“They also reported that a minority of users, around 30% or less, thought that the tool did not understand them or found their interactions unhelpful or bothersome (73, 74, 76–78, 80–82, 84, 90).”

Previous systematic reviews and meta-analyses have shown significant reductions in symptoms for depression and other mental disorders while using mental health apps, some of which included CBT or gamification elements.

“Additionally, previous systematic reviews and meta-analyses have shown significant reductions in symptoms for depression and other mental disorders while using mental health apps, some of which included cognitive behavioral therapy (CBT) or gamification elements [26,36,37,40,41].”

When used enough, chatbots can have positive mental health outcomes.

“When used enough, there can be positive mental health outcomes16–19.”

Eight studies reported ethnicity; in most studies between 31% and 91% of participants identified as White.

“Eight studies reported on the ethnicity of the participants (30, 32, 41, 74, 76, 77, 80, 83), and in most of the studies between 31% and 91% of the participants identified as White.”

In terms of allegiance bias, most included articles developed and studied their own AI tools instead of AI tools developed by other researchers.

“In terms of allegiance bias, because of the scarcity and novelty of AI-augmented interventions and other AI algorithms, most of the included articles developed and studied their own AI tools, instead of AI tools developed by other researchers (29, 33, 73, 74, 76, 79, 80, 82–87, 90).”

The NMA for main effects considered n=6 RCTs.

“The NMA considered n=6 RCTs (41, 73, 74, 76–78).”

The NMA for anxiety considered n=4 RCTs.

“The NMA considered n=4 RCTs (73, 74, 76, 78).”

The meta-analysis assessing impact of AI-augmented interventions on mental health symptoms considered n=10 studies presenting pre- and post-intervention outcomes.

“The meta-analysis assessing the impact of AI-augmented interventions on mental health symptoms considered n=10 studies (41, 73–78, 80, 84, 90) presenting pre- and post-intervention outcomes.”

The meta-analysis for depression considered n=7 studies presenting pre- and post-intervention depression score outcomes.

“The meta-analysis assessing the impact of AI-augmented interventions for the management of depression symptoms considered n=7 studies (74, 76–78, 80, 84, 90) presenting pre- and post-intervention depression score outcomes.”

The NMA for depression considered n=4 RCTs.

“The NMA considered n=4 RCTs (74, 76–78).”

The meta-analysis for anxiety considered n=4 studies presenting pre- and post-intervention anxiety score outcomes.

“The meta-analysis assessing the impact of AI-augmented interventions for the management of anxiety symptoms considered n=4 studies (73, 74, 76, 78)presenting pre- and post-intervention anxiety score outcomes.”

Aside from AI-self-guided eCBT (n=6) and psychoeducation (n=4), other interventions and comparators were reported in three or fewer studies.

“Aside from AI-self-guided eCBT [n=6 (73, 74, 77, 78, 84, 90)] and psychoeducation [n=4 (74, 76–78)], other interventions and comparators were reported in three or fewer studies”

The NMA for dropout considered n=6 RCTs.

“The NMA considered n=6 RCTs (41, 73, 74, 76–78).”

The therapeutic AI agents or chatbots identified by our systematic search (Wysa, Woebot, TEO, Tess, XiaoE, and Sexpert) were effective at delivering self-guided therapy content by communicating intelligently with their users, promoting a significant reduction in mental health symptom severity with high user satisfaction.

“For instance, the therapeutic AI agents or chatbots identified by our systematic search (i.e., Wysa, Woebot, TEO, Tess, XiaoE, and Sexpert) were effective at delivering self-guided therapy content by communicating intelligently with their users, promoting a significant reduction in mental health symptom severity with high user satisfaction (73, 74, 76–78, 82, 84, 90).”

A growing number of studies report effectiveness of chatbots in improving symptoms of anxiety.

“thereby improving symptoms of anxiety [10,23-25]”

The GAD-7 is a 7-item self-report scale assessing anxiety symptoms over the past 2 weeks, with total scores categorized as none (0-4), mild (5-9), moderate (10-14), and severe (≥15).

“The 7-item Generalized Anxiety Disorder scale (GAD-7) [34] is a 7-item self-report scale based on a 4-point Likert scale, ranging from 0 (not at all) to 3 (nearly every day).”

Overall, our quantitative and qualitative findings are aligned with recent chatbot evaluation studies and will inform our design decisions for future developments.

“Overall, our quantitative and qualitative findings are aligned with recent chatbot evaluation studies [10,23-25] and will inform our design decisions for future developments.”

A growing number of studies report effectiveness of chatbots in reducing perceived stress.

“as well as their effectiveness in reducing perceived stress [16-25]”

The effectiveness of health care conversational agents was assessed in 8 studies.

“The effectiveness of health care conversational agents was assessed in 8 studies [47,52,57,61,70,75,81,84].”

Eight studies noted the effectiveness of conversational agents for mental health applications.

“Eight studies noted the effectiveness of conversational agents for mental health applications [57,61,64,67,75,80,81,84].”

The conversational agent Tess by Fulmer et al initiated a statistically significant improvement in depression and anxiety compared with the control group.

“The conversational agent Tess by Fulmer et al [81] initiated a statistically significant improvement in depression and anxiety compared with the control group.”

Psychological conditions were the most commonly addressed type of condition, the focus of 3 studies.

“In the reviewed articles, psychological conditions were the most commonly addressed type of condition, which was the focus of 3 studies [52-54].”

Four studies found high levels of engagement with the conversational agent or reported the conversational agent to be engaging.

“Four studies found high levels of engagement with the conversational agent or reported the conversational agent to be engaging [50,52,53,58].”

In 7 studies, user experience was assessed and was generally positive regarding acceptability, understanding, comprehensibility, interaction rates, or content relevance.

“In 7 studies, user experience was assessed. User experience was generally positive regarding the acceptability, understanding of the conversational agents, comprehensibility of the systems’ responses, interaction rates, or content relevance [51,53,54,56-59].”

The remaining 4 studies used a written format of interaction modality, with both input and output in written form.

“The remaining 4 studies use a written format of interaction modality, which means that both input and output were in a written form [51-53,57].”

Two RCTs found that interaction with conversational agents led to decreased symptoms of depression and anxiety compared with control groups.

“Two RCTs reported on health-related outcomes and found that interaction with the conversational agents led to decreased symptoms of depression and anxiety compared with the control groups [52,53].”

In 3 papers, students served as main study participants.

“In 3 papers, students served as main study participants [52,53,59].”

Among the included studies, natural language processing was the most frequently used AI technique, followed by speech recognition, machine learning, natural language understanding, neural networks, and artificial intelligence markup language.

“Natural language processing represented the most used technique [50-53,55,56,59] before speech recognition (including speech-to-text and text-to-speech) [50,54,56,58,59], machine learning [53,54,59], natural language understanding [54,59], neural networks [54,59] and artificial intelligence markup language [56,57], as shown in Table 3.”

Emotion algorithms were used in one included study.

“The following techniques were used in one study each: deep learning [59], natural language generation [54], emotion algorithms [53], and decision trees [52].”

Five included studies relied on external sources for the artificial intelligence system.

“A total of 4 studies developed the artificial intelligence system internally [50,51,57,59], and 5 studies relied on external sources [52-54,56,58].”

Two conversational agents were commercially available, and one was available for free on Android app store.

“Two conversational agents were commercially available [52,53], and 1 was available for free on Android app store [56].”

Four included studies were classified as AI.

“Four studies were classified as AI [53,56-58].”

In Fulmer et al [29], chatbots improved positive and negative affect at the 2-week follow-up.

“While one study concluded that chatbots improved the positive and negative affect at the 2-week follow-up [29], another study did not find any significant influence of chatbots at the 2-week follow-up [28].”

The chatbot in [29] was more advanced than the one in [28]; it depended on artificial intelligence and machine learning to generate responses to users.

“Specifically, the chatbot in the first study [29] was more advanced than the one in the second study [28]; it depended on artificial intelligence and machine learning to generate responses to users, and this makes it more humanlike and lets users feel more socially connected [5].”

Fulmer et al found a statistically significant difference favoring chatbot use over information on positive and negative affect at the 2-week follow-up (P=.03).

“Fulmer et al [29] found a statistically significant difference favoring chatbot use over information on positive and negative affect at the 2-week follow-up (P=.03).”

The first study assessed the effect of the chatbot on positive and negative affect together, while the other study examined positive affect and negative affect separately.

“With regards to the second difference, while the first study assessed the effect of the chatbot on positive and negative affect together [29], the other study examined the effect of the chatbot on positive affect and negative affect separately [28].”

Participants reported being satisfied with the conversational agent interventions offered.

“Generally, from studies that reported user experience outcomes (n=11), participants reported being satisfied with the conversational agent interventions offered [22,23,25,26,29,30,32].”

Five controlled studies reported significant posttreatment improvements in psychological distress in the intervention group compared with a no treatment or information control group.

“Of the 13 studies included, 5 controlled studies reported significant posttreatment improvements in psychological distress in the intervention group compared with a no treatment or information control group [23,25,26,31,34].”

Significant improvements were observed on measures of depression, psychological distress, anxiety, fear of heights, and positive affect.

“Significant improvements were observed on measures of depression [25,26,31], psychological distress [34], anxiety [26], fear of heights [23] and positive affect [26,34].”

Greater engagement with the conversational agent resulted in greater reductions in psychological distress.

“Generally, greater engagement with the conversational agent resulted in greater reductions in psychological distress [22,26,28,29,33].”

The use of conversational agents for treating mental health problems appears to be limited but is growing quickly, with 5 of the included studies published in 2018 alone.

“The use of conversational agents for treating mental health problems appears to be limited but is growing quickly, with 5 of the included studies published in 2018 alone [23,24,26,29,34].”

Three studies reported that participants found the conversational agent interventions available and accessible.

“In addition, 3 studies reported that participants found the conversational agent interventions available and accessible [26,32,33].”

This provides some support for the utility of conversational agents in treating mild-to-moderate psychological distress in adults.

“This provides some support for the utility of conversational agents in treating mild-to-moderate psychological distress in adults [23,25,26,31,34].”

Participants reported that they found the agent empathic, liked the interactivity, the agent's personality, the agent's ability to form a relationship, and the agent's ability to learn from input.

“Participants reported that they found the agent empathic [26], that they liked the interactivity [30], the agent’s personality [22,25], the agent’s ability to form a relationship [28,33], and the agent’s ability to learn from input [26].”

The predominant challenges to intervention with a conversational agent included repetitive content, limitations in the agents ability to understand or respond appropriately, a shallow or superficial relationship, the sound and quality of the agents voice, and specific intervention tools or content.

“The predominant challenges to intervention with a conversational agent included repetitive content [22,25,26,28,29,33], limitations in the agents ability to understand or respond appropriately [22,25,26,29], a shallow or superficial relationship [28,33], the sound and quality of the agents voice [32], and specific intervention tools or content [25,29].”

A large proportion of agents were eclectic interventions comprising a variety of strategies and psychoeducation drawing on a range of therapeutic orientations.

“Finally, a large proportion of agents were eclectic interventions comprising a variety of strategies and psychoeducation drawing on a range of therapeutic orientations [22,26,29,33].”

Using an identical search strategy, the authors updated the review in January 2020 and identified a further 11 studies; most recent studies were aimed at mental health self-management (n=5).

“Using an identical search strategy, we updated this review in January 2020 and identified a further 11 studies. Similar to the previous review, most recent studies were aimed at mental health self-management (n=5) [10-12].”

The depression symptom reduction, converted to Patient Health Questionnaire-9 equivalent, in the experimental group (1.5 points over 4 weeks) was comparable to that reported in the trial of the X2AI tool (0.9 points over 4 weeks).

“Of note, the depression symptom reduction, when converted to the Patient Health Questionnaire - 9 items equivalent, in the experimental group in our trial (1.5 points over 4 weeks) was comparable to that reported in the trial of the X2AI tool (0.9 points over 4 weeks) [51].”

AI users highlighted positive aspects of including AI, such as accessibility and empathy.

“AI users highlighted positive aspects of including AI, such as accessibility and empathy [78,86].”

The remaining controlled intervention studies were classified as "good" quality investigations as they met most of the criteria.

“The remaining controlled intervention studies were classified as “good” quality investigations as they met most of the criteria [78,86].”

The largest proportion of included conversational agents (12/31, 39%) addressed mental health issues.

“The largest proportion of them (12/31, 39%) addressed mental health issues [13,32-42]”

The most frequently raised issue with conversational agents (9 studies) was poor understanding because of limited vocabulary, voice recognition accuracy, or error management of word inputs.

“The most frequently raised issue with conversational agents (9 studies) was poor understanding because of limited vocabulary, voice recognition accuracy, or error management of word inputs [13,32-37,41,52].”

Users reported liking reminders and assistance in forming routines, accountability, facilitated learning, and ease of learning and use.

“Other features of the agents that users reported liking were the reminders and assistance in forming routines [37,48] and that the agents provided accountability [13,34,48], facilitated learning [13,34,37], and were easy to learn and use [8,15].”

Users in 4 studies reported liking that the agent had a personality and/or showed empathy, whereas users in other studies reported disliking the lack of personal connection or had difficulty empathizing with the agent or reported disliking its limited conversation and responses.

“Similarly, users in 4 studies reported liking that the agent had a personality and/or showed empathy [13,32,34,42], whereas users in other studies reported disliking the lack of personal connection or had difficulty in empathizing with the agent [35,37,50] or reported disliking its limited conversation and responses [35,56].”

Feedback from users in 5 studies expressed a preference for interactivity; users in 1 study liked the interactivity of the chatbot, and users in 4 studies desired greater interactivity or relational skills.

“Feedback from users in 5 studies expressed a preference for interactivity, with users in 1 study noting that they liked the interactivity of the chatbot [35,37], and users in the other 4 studies expressed a desire for greater interactivity or relational skills in the conversational agent [14,32,34,53].”

CBT is seldom used due to pervasive and persistent problems with access, cost, and stigma.

“However, CBT is seldom used due to pervasive and persistent problems with access, cost, and stigma [15-18].”

Conversational agents or chatbots that deliver CBT via a text-based, semiautomated algorithm are known to reduce mild-to-moderate depressive symptoms in nonclinical populations.

“Conversational agents or “chatbots” that deliver CBT via a text-based, semiautomated algorithm are known to reduce mild-to-moderate depressive symptoms in nonclinical populations [17,20-23].”