A Conversational Artificial Intelligence Agent for a Mental Health Care App: Evaluation Study of Its Participatory Design
Open accessPublished version of a preprint
The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index; 8 of the 27 sources its claims cite were checked for support.
Abstract
Background: Mobile apps for mental health are available on the market. Although they seem to be promising for improving the accessibility of mental health care, little is known about their acceptance, design methodology, evaluation, and integration into psychotherapy protocols. This makes it difficult for health care professionals to judge whether these apps may help them and their patients. Objective: Our aim is to describe and evaluate a protocol for the participatory design of mobile apps for mental health. In this study, participants and psychotherapists are engaged in the early phases of the design and development of the app empowered by conversational artificial intelligence (AI). The app supports interventions for stress management training based on cognitive behavioral theory. Methods: A total of 21 participants aged 33-61 years with mild to moderate levels of stress, anxiety, and depression (assessed by administering the Italian versions of the Symptom Checklist-90-Revised, Occupational Stress Indicator, and Perceived Stress Scale) were assigned randomly to 2 groups, A and B. Both groups received stress management training sessions along with cognitive behavioral treatment, but only participants assigned to group A received support through a mobile personal health care agent, designed for mental care and empowered by AI techniques. Psychopathological outcomes were assessed at baseline (T1), after 8 weeks of treatment (T2), and 3 months after treatment (T3). Focus groups with psychotherapists who administered the therapy were held after treatment to collect their impressions and suggestions. Results: Although the intergroup statistical analysis showed that group B participants could rely on better coping strategies, group A participants reported significant improvements in obsessivity and compulsivity and positive distress symptom assessment. The psychotherapists’ acceptance of the protocol was good. In particular, they were in favor of integrating an AI-based mental health app into their practice because they could appreciate the increased engagement of patients in pursuing their therapy goals. Conclusions: The integration into practice of an AI-based mobile app for mental health was shown to be acceptable to both mental health professionals and users. Although it was not possible in this experiment to show that the integration of AI-based conversational technologies into traditional remote psychotherapy significantly decreased the participants’ levels of stress and anxiety, the experimental results showed significant trends of reduction of symptoms in group A and their persistence over time. The mental health professionals involved in the experiment reported interest in, and acceptance of, the proposed technology as a promising tool to be included in a blended model of psychotherapy. ©Morena Danieli, Tommaso Ciulli, Seyed Mahed Mousavi, Giuseppe Riccardi.
Overview
CoChat Summary
TL;DR
This exploratory study evaluated a participatory-design protocol for a conversational AI mobile app (m-PHA/TEO) integrated with CBT-based stress management for work-related stress in 21 Italian workers. The integrated app was acceptable to therapists and users, but the study did not show a clear overall symptom advantage for the app-supported group over CBT-only treatment.
Key findings
- 21 participants aged 33-61 years were randomized to two groups: 11 received SMT-CBT plus the m-PHA app, and 10 received SMT-CBT only.
- In between-group analyses, group A (app support) showed significantly different scores at T1 for obsessivity/compulsivity (mean 62.5 vs 51.3; *P*=.006), at T2 for anxiety (mean 59.3 vs 49.6; *P*=.01), and at T3 for perceived stress (PSS mean 18 vs 10.3; *P*=.04).
- Nonparametric results also found T3 differences favoring group B on coping-related OSI scales: task-oriented (median 6 vs 7; *P*=.004) and involvement (median 6 vs 7.5; *P*=.02).
- Therapists’ acceptance was reported as good; they said the app increased patient engagement and helped patients complete ABC homework.
Methods
- Exploratory, randomized two-group study with repeated assessments at baseline (T1), after 8 weeks of treatment (T2), and 3 months after treatment (T3).
- Participants were recruited in Italy from aging workers with mild to high distress or mild to moderate anxiety/depression; 29 were assessed and 21 were recruited.
- Main analysis used parametric independent t tests / repeated-measures ANOVA and nonparametric Mann–Whitney, Friedman, and Wilcoxon tests on SCL-90-R, PSS, and OSI outcomes; therapists also took part in two post-treatment focus groups.
Limitations
- The paper states that the small number of participants did not allow allocation to more than two experimental groups, and that assigning a control group was not planned because the sample was small.
- The authors note the m-PHA had limited dialog capabilities at the time, aiming mainly to motivate ABC homework rather than sustain extended conversations.
- Evident from the text: the study was exploratory and coincided with the first COVID-19 lockdown in Italy, which the authors say may have influenced anxiety and stress results.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
7 Valid · 1 Not valid · 8 Not checked · 90 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“Changing cognitions and thoughts can help to reduce symptoms [25].”
§ A Conversational Artificial Intelligence Agent for a Mental Health Care App: Evaluation Study of Its Participatory Design > Introduction > Objective
What the sources say
- Supportedcites [25] source
The paper’s Abstract states that CBT aims to reduce symptoms by challenging maladaptive cognitions and that strategies to change those cognitions lead to changes in emotional distress. This supports the citing sentence’s qualified claim that changing thoughts can help reduce symptoms. (LLM, conf=0.90)
plainstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“4 meta-analyses showed how CBT performed better than the other interventions in the treatment of occupational stress [25].”
§ A Conversational Artificial Intelligence Agent for a Mental Health Care App: Evaluation Study of Its Participatory Design > Introduction > Objective
What the sources say
- Partly supportedcites [25] sourceExperimental — accuracy not yet measured
The Results > General Stress section says four meta-analyses examined occupational stress and that most found CBT more effective than other intervention types. It also says one of the four did not compare CBT with other interventions, so the claim overstates what all four showed. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“We chose the focus group technique because in the past this method has been found appropriate for evaluating attitudes of health care personnel, among others [19,40].”
§ A Conversational Artificial Intelligence Agent for a Mental Health Care App: Evaluation Study of Its Participatory Design > Results > Qualitative Evaluation of the Intervention
What the sources say
- Could not verifycites [19] sourceExperimental — accuracy not yet measured
The Results (Study Characteristics) discuss control groups, and the Discussion (Principal Findings) mentions clinicians’ attitudes toward digital interventions. Neither passage addresses whether focus groups are appropriate for evaluating health care personnel’s attitudes. Searches for “focus group” and related terms found no passage making that methodological claim. (LLM, conf=0.75)
- Not checked — no full text of the sourcecites [40] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the OSI coping strategies, only the task-oriented and involvement scales at T3 were significantly different.”
§ A Conversational Artificial Intelligence Agent for a Mental Health Care App: Evaluation Study of Its Participatory Design > Results > Parametric Data Analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In particular, they were in favor of integrating an AI-based mental health app into their practice because they could appreciate the increased engagement of patients in pursuing their therapy goals.”
§ A Conversational Artificial Intelligence Agent for a Mental Health Care App: Evaluation Study of Its Participatory Design > Abstract > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
54 more claims
Create a free account to see all 106 claims and to verify the 5 references still open.
- Probably not a substantial claim. 27
- It may state two separate claims as one. 33
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Not confirmedz = -0.65reported p = .66 · recomputed p = .516
“Z=-0.65, p = .66”
- Not confirmedz = -1.72reported p = .16 · recomputed p = .085
“Z=-1.72, p = .16”
- Not confirmedz = -2.25reported p = .03 · recomputed p = .024
“Z=-2.25, p = .03”
- Consistentz = -2.3reported p = .02 · recomputed p = .021
“Z=-2.30, p = .02”
- Consistentz = -2.01reported p = .04 · recomputed p = .044
“Z=-2.01, p = .04”
29 more recomputed tests
Create a free account to see every test the Stats check recomputed.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Self-check results appear here once an independent check has confirmed them.
Citations
Cites
A Global Measure of Perceived Stress
Cohen, S., Kamarck, T., & Mermelstein, R. (1983) · Journal of Health and Social Behavior · doi:10.2307/2136404 · Cited by 32,807 · cited 2×
“Description of the Questionnaires for Initial and Final Assessment The tests used for the initial assessment (T1) were the Italian versions of the SCL-90-R [28,29], the Perceived Stress Scale (PSS) [30,31], and the Occupational Stress Indicator (OSI) [32,33].”
(1962) · Medical Entomology and Zoology · Cited by 4,856
The Efficacy of Cognitive Behavioral Therapy: A Review of Meta-analyses
Hofmann, S. G., Asnaani, A., Vonk, I. J. J., Sawyer, A. T., & Fang, A. (2012) · Cognitive Therapy and Research · doi:10.1007/s10608-012-9476-1 · Cited by 3,445 · cited 2×
“Changing cognitions and thoughts can help to reduce symptoms [25].”
Analyzing and Interpreting Data From Likert-Type Scales
Sullivan, G. M., & Artino, A. R. (2013) · Journal of Graduate Medical Education · doi:10.4300/jgme-5-4-18 · Cited by 3,044 · cited 1×
“In addition, by following the suggestions made by an anonymous reviewer and by Sullivan and Artino [38], a parametric independent 2-tailed t test analysis of data was performed.”
Cognitive Therapy and the Emotional Disorders
Ince, L. P. (1977) · American Journal of Psychotherapy · doi:10.1176/appi.psychotherapy.1977.31.4.633 · Cited by 2,892
Rossi, R., Socci, V., Talevi, D., Mensi, S., Niolu, C., Pacitti, F., Di Marco, A., Rossi, A., Siracusano, A., & Di Lorenzo, G. (2020) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2020.00790 · Cited by 1,448
Torous, J., Nicholas, J., Larsen, M. E., Firth, J., & Christensen, H. (2018) · Evidence Based Mental Health · doi:10.1136/eb-2018-102891 · Cited by 790 · cited 1×
“However, there is little consensus on the usability and effectiveness of such systems [13].”
An overview of the features of chatbots in mental health: A scoping review
Abd-alrazaq, A. A., Alajlani, M., Alalwan, A. A., Bewick, B. M., Gardner, P., & Househ, M. (2019) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2019.103978 · Cited by 605 · cited 1×
“In the mental health domain, chatbots are often based on scripted or Eliza-style dialogs [6,16].”
Overview of Friedman’s test and post-hoc analysis
Pereira, D. G., Afonso, A., & Medeiros, F. M. (2015) · Communications in Statistics - Simulation and Computation · doi:10.1080/03610918.2014.931971 · Cited by 585 · cited 2×
“Nonparametric Data Analysis A nonparametric analysis of data was performed using the Friedman test (Pereira et al [39]), which allowed us to compare the different results reported in group B at T1, T2, and T3.”
Measuring and improving the quality of mental health care: a global perspective
Kilbourne, A. M., Beck, K., Spaeth‐Rublee, B., Ramanuj, P., O'Brien, R. W., Tomoyasu, N., & Pincus, H. A. (2018) · World Psychiatry · doi:10.1002/wps.20482 · Cited by 457 · cited 1×
“As the requirement for mental health services is widespread [8] and with the current COVID-19 pandemic creating a spike in demand (as stated by the World Health Organization surveys on October 10, 2020 [9,10]), there is a greater awareness of these apps among mental health professionals [11,12].”
Research works printed but not linked: 6
- Armontrout, J, 2018, J Am Acad Psychiatry Law, 46, 000
- Matheny, K, 2000, J Indiv Psychol, 56, 74
- Thompson, C, 2007, Exploring the Work and Non-work Interface (Research in Occupational Stress and Well Being, Vol. 6), 73
- Derogatis, L, 1977, SCL-90: Administration Scoring and Procedures Manual-i and Other Instruments of the Psychopathology Rating Scale Series
- Cohen, S, 1988, The Social Psychology of Health: The Claremont Symposium on Applied Social Psychology, 31
- Cooper, C, 1988, Occupational Stress Indicator: Management Set
Cited by
Ni, Y., & Jia, F. (2025) · Healthcare · doi:10.3390/healthcare13101205 · Cited by 148
“[48] MH support Conversational AI agent A mental health app design involving participatory protocol with therapists and users for stress management.”
InfluentialDanieli, M., Ciulli, T., Mousavi, S. M., Silvestri, G., Barbato, S., Di Natale, L., & Riccardi, G. (2022) · JMIR Mental Health · doi:10.2196/38067 · Cited by 84
“A growing number of studies have indicated that the development of conversational AI systems (also known as chatbots) as applications in the mental health domain can improve access to mental health care support in an easy and inexpensive manner [8,11,12].”
InfluentialSelf-citationJabir, A. I., Lin, X., Martinengo, L., Sharp, G., Theng, Y.-L., & Tudor Car, L. (2024) · Journal of Medical Internet Research · doi:10.2196/48168 · Cited by 66
Polyportis, A., & Pahos, N. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-023-02464-6 · Cited by 64
Review of artificial intelligence‐based question‐answering systems in healthcare
Budler, L. C., Gosak, L., & Stiglic, G. (2023) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1487 · Cited by 52
Metadata
- Authors
- Danieli, M., Ciulli, T., Mousavi, S. M., & Riccardi, G.
- Affiliations
- University of Trento (IT)
- Corresponding author
- Danieli, Morena
- Venue
- JMIR Formative Research
- Publisher
- JMIR Publications Inc.
- Published
- 2021-12-01
- Volume
- 5
- Issue
- 12
- Pages
- e30053
- ISSN
- 2561-326X (ISSN-L)
- PMC
- PMC8686486
- OpenAlex
- W3200694407
- Cited by
- 64
- References
- 35
- Citations by year
- 2026: 15 · 2025: 23 · 2024: 18 · 2023: 5 · 2022: 3
- FWCI
- 5.84
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Formative Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Iris (University of Trento) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- European Commission (826266, HORIZON2020)
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- conversational AI, mental health apps, participatory design, stress management training, cognitive behavioral therapy, obsessive-compulsive symptoms, anxiety and depression, remote psychotherapy, mobile health
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.