Cited by 63Digital Mental Health InterventionsSocial Robot Interaction and HRIEmotion and Mood Recognition

Evaluating Conversational Agents for Mental Health: Scoping Review of Outcomes and Outcome Measurement Instruments

Open access
Jabir, Ahmad Ishqi, Martinengo, Laura, Lin, Xiaowen, Torous, John, Subramaniam, Mythily, Tudor Car, Lorainne·2023·Journal of Medical Internet Research

Published version of a preprint

Abstract

Background: Rapid proliferation of mental health interventions delivered through conversational agents (CAs) calls for high-quality evidence to support their implementation and adoption. Selecting appropriate outcomes, instruments for measuring outcomes, and assessment methods are crucial for ensuring that interventions are evaluated effectively and with a high level of quality. Objective: We aimed to identify the types of outcomes, outcome measurement instruments, and assessment methods used to assess the clinical, user experience, and technical outcomes in studies that evaluated the effectiveness of CA interventions for mental health. Methods: We undertook a scoping review of the relevant literature to review the types of outcomes, outcome measurement instruments, and assessment methods in studies that evaluated the effectiveness of CA interventions for mental health. We performed a comprehensive search of electronic databases, including PubMed, Cochrane Central Register of Controlled Trials, Embase (Ovid), PsychINFO, and Web of Science, as well as Google Scholar and Google. We included experimental studies evaluating CA mental health interventions. The screening and data extraction were performed independently by 2 review authors in parallel. Descriptive and thematic analyses of the findings were performed. Results: We included 32 studies that targeted the promotion of mental well-being (17/32, 53%) and the treatment and monitoring of mental health symptoms (21/32, 66%). The studies reported 203 outcome measurement instruments used to measure clinical outcomes (123/203, 60.6%), user experience outcomes (75/203, 36.9%), technical outcomes (2/203, 1.0%), and other outcomes (3/203, 1.5%). Most of the outcome measurement instruments were used in only 1 study (150/203, 73.9%) and were self-reported questionnaires (170/203, 83.7%), and most were delivered electronically via survey platforms (61/203, 30.0%). No validity evidence was cited for more than half of the outcome measurement instruments (107/203, 52.7%), which were largely created or adapted for the study in which they were used (95/107, 88.8%). Conclusions: The diversity of outcomes and the choice of outcome measurement instruments employed in studies on CAs for mental health point to the need for an established minimum core outcome set and greater use of validated instruments. Future studies should also capitalize on the affordances made available by CAs and smartphones to streamline the evaluation and reduce participants’ input burden inherent to self-reporting.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.2196/44548

Cites

68 of 68 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Relational agents have been shown to foster therapeutic rapport with users, support long-term engagement, and have been deployed across various clinical and nonclinical populations, including cognitively healthy older adults.

    “Relational agents have been shown to foster therapeutic rapport with users [32-34], support long-term engagement [35], and have been deployed across various clinical [35-37] and nonclinical populations [38-40], including (cognitively healthy) older adults [41-43].”

    From Toward a Smartphone-Based and Conversational Agent–Delivered Just-in-Time Adaptive Holistic Lifestyle Intervention for Older Adults Affected by Cognitive Decline: Two-Week Proof-of-Concept Study (2025) · cites this paper as [34]

    The Discussion (Comparison With Prior Work; page unavailable) says early evidence suggests users may develop a therapeutic alliance with conversational agents, but describes that evidence as limited. The Results (Characteristics of the Included Studies; page unavailable) include healthy adults and inpatients or outpatients, and the references identify a study of older adults. The review measures engagement and reports some studies lasting at least eight weeks, but does not establish that relational agents foster long-term engagement or specify that the older adults were cognitively healthy. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. [55]Hodrien A Fernando T A Review of Post-Study and Post- Task Subjective Questionnaires to Guide Assessment of System Usability Journal of Usability Studies 2021 16 3 203 232

Cited by

Sorted by citations

Authors
Jabir, A. I., Martinengo, L., Lin, X., Torous, J., Subramaniam, M., & Tudor Car, L.
Affiliations
  • Nanyang Technological University (SG)
  • Beth Israel Deaconess Medical Center (US)
  • National University of Singapore (SG)
Venue
Journal of Medical Internet Research
Published
2023-04-19
Volume
25
Pages
e44548
ISSN
1438-8871 (ISSN-L)
OpenAlex
W4361994630
Cited by
63
References
68
Citations by year
2026: 24 · 2025: 13 · 2024: 16 · 2023: 10
FWCI
2.14
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Journal of Medical Internet Research (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • DR-NTU (Nanyang Technological University) (submittedVersion, cc-by)
Funders
  • National Research Foundation
  • National Research Foundation Singapore
  • Eidgenössische Technische Hochschule Zürich
  • Lee Kong Chian School of Medicine, Nanyang Technological University
Topics
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
Keywords
chatbots, mental health interventions, outcome measurement instruments, outcome validity, clinical outcomes, self-report questionnaire, smartphone-based assessment
MeSH terms
Communication, Humans, Mental Health, Outcome Assessment, Health Care

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review