Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 89 of the 89 references it cites were found in a registry or an index.

Technical Metrics Used to Evaluate Health Care Chatbots: Scoping Review

Open access

Published version of a preprint

Abstract

Background: Dialog agents (chatbots) have a long history of application in health care, where they have been used for tasks such as supporting patient self-management and providing counseling. Their use is expected to grow with increasing demands on health systems and improving artificial intelligence (AI) capability. Approaches to the evaluation of health care chatbots, however, appear to be diverse and haphazard, resulting in a potential barrier to the advancement of the field. Objective: This study aims to identify the technical (nonclinical) metrics used by previous studies to evaluate health care chatbots. Methods: Studies were identified by searching 7 bibliographic databases (eg, MEDLINE and PsycINFO) in addition to conducting backward and forward reference list checking of the included studies and relevant reviews. The studies were independently selected by two reviewers who then extracted data from the included studies. Extracted data were synthesized narratively by grouping the identified metrics into categories based on the aspect of chatbots that the metrics evaluated. Results: Of the 1498 citations retrieved, 65 studies were included in this review. Chatbots were evaluated using 27 technical metrics, which were related to chatbots as a whole (eg, usability, classifier performance, speed), response generation (eg, comprehensibility, realism, repetitiveness), response understanding (eg, chatbot understanding as assessed by users, word error rate, concept error rate), and esthetics (eg, appearance of the virtual agent, background color, and content). Conclusions: The technical metrics of health chatbot studies were diverse, with survey designs and global usability metrics dominating. The lack of standardization and paucity of objective measures make it difficult to compare the performance of health chatbots and could inhibit advancement of the field. We suggest that researchers more frequently include metrics computed from conversation logs. In addition, we recommend the development of a framework of technical metrics with recommendations for specific circumstances for their inclusion in chatbot studies.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.2196/18301

Cites

89 of 89 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Flowchart of the study selection process.
Figure 1 Flowchart of the study selection process.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 88 references

Research works printed but not linked: 6

  1. Denecke, K, 2020, Studies in Health Technology and Informatics, 1
  2. Ali, M, 2018, arXiv, 1
  3. Hess, G, 2019, Studies in health technology and informatics, 259, 101
  4. Ghosh, S, 2018, Stud Health Technol Inform, 252, 51
  5. Bickmore, T, 2003, Relational agents affecting change through human-computer relationships, 1
  6. Jadeja, M, 2017, arXiv preprint, 1

Cited by

See all 163

Metadata

Authors
Abd-Alrazaq, A., Safi, Z., Alajlani, M., Warren, J., Househ, M., & Denecke, K.
Affiliations
  • Hamad bin Khalifa University (QA)
  • University of Warwick (GB)
  • University of Auckland (NZ)
  • Bern University of Applied Sciences (CH)
Corresponding author
Denecke, Kerstin
Venue
Journal of Medical Internet Research
Published
2020-06-05
Volume
22
Issue
6
Pages
e18301
ISSN
1438-8871 (ISSN-L)
OpenAlex
W3021626742
Cited by
150
References
88
Citations by year
2026: 25 · 2025: 32 · 2024: 22 · 2023: 27 · 2022: 21 · 2021: 18 · 2020: 4
FWCI
1.34
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Journal of Medical Internet Research (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • ARBOR - Bern University of Applied Sciences Repository (submittedVersion, other-oa)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Figshare (submittedVersion, cc-by)
  • Warwick Research Archive Portal (University of Warwick) (submittedVersion)
  • BFH: ARBOR
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
  • Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
Keywords
chatbot evaluation, usability, response generation, word error rate, conversation logs, dialogue agents
MeSH terms
Artificial Intelligence, Communication, Delivery of Health Care, Humans

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.