CoChat

A Conversational Artificial Intelligence Agent for a Mental Health Care App: Evaluation Study of Its Participatory Design

Open access

Published version of a preprint

The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index; 8 of the 27 sources its claims cite were checked for support.

Abstract

Background: Mobile apps for mental health are available on the market. Although they seem to be promising for improving the accessibility of mental health care, little is known about their acceptance, design methodology, evaluation, and integration into psychotherapy protocols. This makes it difficult for health care professionals to judge whether these apps may help them and their patients. Objective: Our aim is to describe and evaluate a protocol for the participatory design of mobile apps for mental health. In this study, participants and psychotherapists are engaged in the early phases of the design and development of the app empowered by conversational artificial intelligence (AI). The app supports interventions for stress management training based on cognitive behavioral theory. Methods: A total of 21 participants aged 33-61 years with mild to moderate levels of stress, anxiety, and depression (assessed by administering the Italian versions of the Symptom Checklist-90-Revised, Occupational Stress Indicator, and Perceived Stress Scale) were assigned randomly to 2 groups, A and B. Both groups received stress management training sessions along with cognitive behavioral treatment, but only participants assigned to group A received support through a mobile personal health care agent, designed for mental care and empowered by AI techniques. Psychopathological outcomes were assessed at baseline (T1), after 8 weeks of treatment (T2), and 3 months after treatment (T3). Focus groups with psychotherapists who administered the therapy were held after treatment to collect their impressions and suggestions. Results: Although the intergroup statistical analysis showed that group B participants could rely on better coping strategies, group A participants reported significant improvements in obsessivity and compulsivity and positive distress symptom assessment. The psychotherapists’ acceptance of the protocol was good. In particular, they were in favor of integrating an AI-based mental health app into their practice because they could appreciate the increased engagement of patients in pursuing their therapy goals. Conclusions: The integration into practice of an AI-based mobile app for mental health was shown to be acceptable to both mental health professionals and users. Although it was not possible in this experiment to show that the integration of AI-based conversational technologies into traditional remote psychotherapy significantly decreased the participants’ levels of stress and anxiety, the experimental results showed significant trends of reduction of symptoms in group A and their persistence over time. The mental health professionals involved in the experiment reported interest in, and acceptance of, the proposed technology as a promising tool to be included in a blended model of psychotherapy. ©Morena Danieli, Tommaso Ciulli, Seyed Mahed Mousavi, Giuseppe Riccardi.

Overview

CoChat Summary

TL;DR

This exploratory study evaluated a participatory-design protocol for a conversational AI mobile app (m-PHA/TEO) integrated with CBT-based stress management for work-related stress in 21 Italian workers. The integrated app was acceptable to therapists and users, but the study did not show a clear overall symptom advantage for the app-supported group over CBT-only treatment.

Key findings

  • 21 participants aged 33-61 years were randomized to two groups: 11 received SMT-CBT plus the m-PHA app, and 10 received SMT-CBT only.
  • In between-group analyses, group A (app support) showed significantly different scores at T1 for obsessivity/compulsivity (mean 62.5 vs 51.3; *P*=.006), at T2 for anxiety (mean 59.3 vs 49.6; *P*=.01), and at T3 for perceived stress (PSS mean 18 vs 10.3; *P*=.04).
  • Nonparametric results also found T3 differences favoring group B on coping-related OSI scales: task-oriented (median 6 vs 7; *P*=.004) and involvement (median 6 vs 7.5; *P*=.02).
  • Therapists’ acceptance was reported as good; they said the app increased patient engagement and helped patients complete ABC homework.

Methods

  • Exploratory, randomized two-group study with repeated assessments at baseline (T1), after 8 weeks of treatment (T2), and 3 months after treatment (T3).
  • Participants were recruited in Italy from aging workers with mild to high distress or mild to moderate anxiety/depression; 29 were assessed and 21 were recruited.
  • Main analysis used parametric independent t tests / repeated-measures ANOVA and nonparametric Mann–Whitney, Friedman, and Wilcoxon tests on SCL-90-R, PSS, and OSI outcomes; therapists also took part in two post-treatment focus groups.

Limitations

  • The paper states that the small number of participants did not allow allocation to more than two experimental groups, and that assigning a control group was not planned because the sample was small.
  • The authors note the m-PHA had limited dialog capabilities at the time, aiming mainly to motivate ABC homework rather than sustain extended conversations.
  • Evident from the text: the study was exploratory and coincided with the first COVID-19 lockdown in Italy, which the authors say may have influenced anxiety and stress results.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.2196/30053

Cites

36 of 36 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 The information flow in the proposed approach. The patients interact with the mobile personal health care agent (m-PHA) to share personal recollections of their life events. The therapists supervise the interaction of the m-PHA with the patients and elaborate on the patients’ personal narratives during the therapy session. ABC: antecedents, beliefs, and consequences; m-PHA: mobile personal health care agent.
Figure 1 The information flow in the proposed approach. The patients interact with the mobile personal health care agent (m-PHA) to share personal recollections of their life events. The therapists supervise the interaction of the m-PHA with the patients and elaborate on the patients’ personal narratives during the therapy session. ABC: antecedents, beliefs, and consequences; m-PHA: mobile personal health care agent.CC BY · PubMed Central
Figure 2 The CONSORT (Consolidated Standards of Reporting Trials) diagram shows the flow of the intervention, the enrollment of participants, their allocation to treatment, follow-up, and analysis. CBT: cognitive behavioral therapy; m-PHA: mobile personal health care agent; SMT: stress management training; T3: assessment of psychopathological outcomes 3 months after treatment.
Figure 2 The CONSORT (Consolidated Standards of Reporting Trials) diagram shows the flow of the intervention, the enrollment of participants, their allocation to treatment, follow-up, and analysis. CBT: cognitive behavioral therapy; m-PHA: mobile personal health care agent; SMT: stress management training; T3: assessment of psychopathological outcomes 3 months after treatment.CC BY · PubMed Central

Verification

Claims the paper makes

7 Valid · 1 Not valid · 8 Not checked · 90 Own results and interpretations

54 more claims

Create a free account to see all 106 claims and to verify the 5 references still open.

See all claims — free account
  • Probably not a substantial claim. 27
  • It may state two separate claims as one. 33

Create a free account to see these claims.

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Not confirmedz = -0.65reported p = .66 · recomputed p = .516

    “Z=-0.65, p = .66”

  • Not confirmedz = -1.72reported p = .16 · recomputed p = .085

    “Z=-1.72, p = .16”

  • Not confirmedz = -2.25reported p = .03 · recomputed p = .024

    “Z=-2.25, p = .03”

  • Consistentz = -2.3reported p = .02 · recomputed p = .021

    “Z=-2.30, p = .02”

  • Consistentz = -2.01reported p = .04 · recomputed p = .044

    “Z=-2.01, p = .04”

5 of 34 · the rest aren’t loaded

29 more recomputed tests

Create a free account to see every test the Stats check recomputed.

See all tests — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Self-check results appear here once an independent check has confirmed them.

Citations

Cites

Show all 36 references

Research works printed but not linked: 6

  1. Armontrout, J, 2018, J Am Acad Psychiatry Law, 46, 000
  2. Matheny, K, 2000, J Indiv Psychol, 56, 74
  3. Thompson, C, 2007, Exploring the Work and Non-work Interface (Research in Occupational Stress and Well Being, Vol. 6), 73
  4. Derogatis, L, 1977, SCL-90: Administration Scoring and Procedures Manual-i and Other Instruments of the Psychopathology Rating Scale Series
  5. Cohen, S, 1988, The Social Psychology of Health: The Claremont Symposium on Applied Social Psychology, 31
  6. Cooper, C, 1988, Occupational Stress Indicator: Management Set

Cited by

See all 64

Metadata

Authors
Danieli, M., Ciulli, T., Mousavi, S. M., & Riccardi, G.
Affiliations
University of Trento (IT)
Corresponding author
Danieli, Morena
Venue
JMIR Formative Research
Published
2021-12-01
Volume
5
Issue
12
Pages
e30053
ISSN
2561-326X (ISSN-L)
OpenAlex
W3200694407
Cited by
64
References
35
Citations by year
2026: 15 · 2025: 23 · 2024: 18 · 2023: 5 · 2022: 3
FWCI
5.84
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JMIR Formative Research (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Iris (University of Trento) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
European Commission (826266, HORIZON2020)
Topics
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
  • Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
Keywords
conversational AI, mental health apps, participatory design, stress management training, cognitive behavioral therapy, obsessive-compulsive symptoms, anxiety and depression, remote psychotherapy, mobile health

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A Conversational Artificial Intelligence Agent for a Mental Health Care App: Evaluation Study of Its Participatory Design. https://app.cochat.ai/papers/10.2196/30053
The paper
Danieli, M., Ciulli, T., Mousavi, S. M., & Riccardi, G. (2021). A Conversational Artificial Intelligence Agent for a Mental Health Care App: Evaluation Study of Its Participatory Design. JMIR Formative Research. https://doi.org/10.2196/30053

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.