CoChat

Conversational agents in healthcare: a systematic review

Open access

The DOI matches its Crossref record; 55 of the 60 references it cites were found in a registry or an index; 14 of the 36 sources its claims cite were checked for support.

Abstract

Objective: Our objective was to review the characteristics, current applications, and evaluation measures of conversational agents with unconstrained natural language input capabilities used for health-related purposes. Methods: We searched PubMed, Embase, CINAHL, PsycInfo, and ACM Digital using a predefined search strategy. Studies were included if they focused on consumers or healthcare professionals; involved a conversational agent using any unconstrained natural language input; and reported evaluation measures resulting from user interaction with the system. Studies were screened by independent reviewers and Cohen's kappa measured inter-coder agreement. Results: The database search retrieved 1513 citations; 17 articles (14 different conversational agents) met the inclusion criteria. Dialogue management strategies were mostly finite-state and frame-based (6 and 7 conversational agents, respectively); agent-based strategies were present in one type of system. Two studies were randomized controlled trials (RCTs), 1 was cross-sectional, and the remaining were quasi-experimental. Half of the conversational agents supported consumers with health tasks such as self-care. The only RCT evaluating the efficacy of a conversational agent found a significant effect in reducing depression symptoms (effect size d = 0.44, p = .04). Patient safety was rarely evaluated in the included studies. Conclusions: The use of conversational agents with unconstrained natural language input capabilities for health-related purposes is an emerging field of research, where the few published studies were mainly quasi-experimental, and rarely evaluated efficacy or safety. Future studies would benefit from more robust experimental designs and standardized reporting. Protocol Registration: The protocol for this systematic review is registered at PROSPERO with the number CRD42017065917.

Overview

CoChat Summary

TL;DR

This systematic review found that conversational agents with unconstrained natural language input were still rarely studied in healthcare: 17 articles covering 14 agents, mostly evaluated with quasi-experimental designs. Evidence of efficacy and safety was limited, with only one RCT showing reduced depression symptoms (effect size d = 0.44, p = .04).

Key findings

  • The search retrieved 1513 citations; 17 articles evaluating 14 different conversational agents met inclusion criteria.
  • Study designs were mainly weak for efficacy claims: 2 RCTs, 1 cross-sectional study, and the rest quasi-experimental. The paper notes that the RCTs had moderate to high risk of bias.
  • Dialogue management was mostly finite-state (6 agents / 8 studies) or frame-based (7 agents / 8 studies); agent-based dialogue management appeared in 1 agent.
  • The only RCT evaluating efficacy found a significant reduction in depression symptoms with effect size d = 0.44, p = .04; patient safety was rarely evaluated.

Methods

  • Systematic review of studies of conversational agents using any unconstrained natural language input for health-related purposes; protocol registered at PROSPERO CRD42017065917 and the review followed PRISMA.
  • Searched PubMed, Embase, CINAHL, PsycInfo, and ACM Digital Library in April 2017, updated in February 2018; included primary studies of consumers, caregivers, or healthcare professionals that tested the full system with human users and reported evaluation measures.
  • Screening was done by independent reviewers with Cohen’s kappa for agreement; because of heterogeneity, the authors conducted a narrative synthesis rather than a meta-analysis.

Limitations

  • The paper states it focused only on agents with unconstrained natural language input, excluding constrained-input systems that are also relevant to healthcare.
  • The paper states that the small number of included studies, heterogeneity of agents, and predominance of quasi-experimental pilot studies limited its ability to conduct a meta-analysis.
  • Evident from the text: reporting was inconsistent, making eligibility assessment and synthesis difficult, and several studies lacked clear funding/conflict-of-interest reporting.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1093/jamia/ocy072

Cites

55 of 55 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1. Flow diagram of included studies in which 17 studies (14 conversational agents) were identified from 1513 articles in the initial database search (April 2017). Search updates were conducted until February 2018, with 3 new papers being identified for full-text screening.
Figure 1. Flow diagram of included studies in which 17 studies (14 conversational agents) were identified from 1513 articles in the initial database search (April 2017). Search updates were conducted until February 2018, with 3 new papers being identified for full-text screening.CC BY-NC · PubMed Central
Figure 2. Characteristics of included conversational agents in terms of task-orientation, dialogue management, and dialogue initiative.
Figure 2. Characteristics of included conversational agents in terms of task-orientation, dialogue management, and dialogue initiative.CC BY-NC · PubMed Central

Verification

Claims the paper makes

10 Valid · 3 Not valid · 13 Not checked · 25 Own results and interpretations

17 more claims · 2 with their cited sources checked

Create a free account to see all 51 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 12
  • It may state two separate claims as one. 20
  • Borderline: one of the tests for a substantial claim is unclear. 1
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

62 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Research works printed but not linked: 2

  1. Radziwill, 2017, arXiv Prepr, 1704
  2. Coiera, 2017, First compute no harm, BMJ Opin

Cited by

See all 1,388

Metadata

Authors
Laranjo, L., Dunn, A. G., Tong, H. L., Kocaballi, A. B., Chen, J., Bashir, R., Surian, D., Gallego, B., Magrabi, F., Lau, A. Y. S., & Coiera, E.
Affiliations
Macquarie University (AU)
Corresponding author
Laranjo, Liliana
Venue
Journal of the American Medical Informatics Association
Published
2018-07-11
Volume
25
Issue
9
Pages
1248-1258
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
OpenAlex
W2843010082
Cited by
1,388
References
60
Citations by year
2026: 291 · 2025: 246 · 2024: 182 · 2023: 182 · 2022: 151 · 2021: 143 · 2020: 124 · 2019: 62 · 2018: 6
FWCI
1.87
Type
journal article
Language
en
Open access
Locations
  • Journal of the American Medical Informatics Association (publishedVersion, cc-by-nc)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • UNSWorks (University of New South Wales, Sydney, Australia) (submittedVersion, cc-by-nc-nd)
Funders
  • National Health and Medical Research Council (APP1134919)
  • Centre for Research Excellence in Digital Health (APP1054146)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
Keywords
chatbots, healthcare chatbots, natural language input, dialogue management, self-care support, depression symptom reduction, patient safety, evaluation metrics
MeSH terms
Artificial Intelligence, Communication, Delivery of Health Care, Natural Language Processing, Speech Recognition Software

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.