How to Evaluate Health Applications with Conversational User Interface?
Open accessThe DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
Application of conversational user interfaces (CUI) or chatbots to healthcare is gaining interest fueled by the rising power of artificial intelligence, increasing popularity of mobile health applications and the desire for engagement and usability. While their use is mainly justified by increasing adherence to mobile health applications and facilitating interactions with the system, the question arises: How can such systems be evaluated in a reliable manner? This paper introduces an evaluation framework for health systems whose core interaction principle is a CUI. We derive quality dimensions and attributes by collecting relevant evaluation aspects from applications that have been developed in previous work and from literature on health chatbots. The collected aspects are aggregated into six thematic categories for chatbot quality, including user experience, linguistic, task-oriented and artificial intelligence perspectives, but also healthcare quality and system quality perspectives. The framework is intended to support developers and researchers in the domain of chatbots in healthcare in selecting relevant quality attributes to be assessed before their systems are distributed to patients.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Perceived usefulness, perceived ease of use, and user acceptance of information technology
Davis, F. D. (1989) · MIS Quarterly · doi:10.2307/249008 · Cited by 68,483
Conversational agents in healthcare: a systematic review
Laranjo, L., Dunn, A. G., Tong, H. L., Kocaballi, A. B., Chen, J., Bashir, R., Surian, D., Gallego, B., Magrabi, F., Lau, A. Y. S., et al. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy072 · Cited by 1,388 · cited 1×
“[1].”
Inkster, B., Sarda, S., & Subramanian, V. (2018) · doi:10.2196/preprints.12106 · Cited by 1,063
From Eliza to XiaoIce: Challenges and opportunities with social chatbots
Shum, H.-Y., He, X.-D., & Li, D. (2018) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.1700826 · Cited by 749
Self-Anamnesis with a Conversational User Interface: Concept and Usability Study
Hochreutener, S., Pöpel, A., May, R., & Denecke, K. (2018) · Methods of Information in Medicine · doi:10.1055/s-0038-1675822 · Cited by 47
Self-citation
Cited by
Perceptions and opinions of patients about mental health chatbots: scoping review
Abd-Alrazaq, A. A., Alajlani, M., Ali, N., Denecke, K., Bewick, B. M., & Househ, M. (2021) · Journal of Medical Internet Research · doi:10.2196/17828 · Cited by 301
“The first attempts toward evaluation frameworks for digital health interventions [82] and health chatbots [83,84] have been recently published.”
Self-citationTechnical metrics used to evaluate health care chatbots: scoping review
Abd-Alrazaq, A., Safi, Z., Alajlani, M., Warren, J., Househ, M., & Denecke, K. (2020) · Journal of Medical Internet Research · doi:10.2196/18301 · Cited by 150
Self-citationImplementation of Cognitive Behavioral Therapy in e–Mental Health Apps: Literature Review
Denecke, K., Schmid, N., & Nüssli, S. (2022) · Journal of Medical Internet Research · doi:10.2196/27791 · Cited by 107
Self-citationJabir, A. I., Martinengo, L., Lin, X., Torous, J., Subramaniam, M., & Tudor Car, L. (2023) · Journal of Medical Internet Research · doi:10.2196/44548 · Cited by 63
“This may be done via a Delphi study or via existing guidelines [68] to establish the core outcome set directly related to CA functionalities, such as the definition of meaningful engagement with the CA, or determine user attitudes and perceptions toward the CA.”
Standardizing and Scaffolding Health Care AI-Chatbot Evaluation: Systematic Review
Hua, Y., Xia, W., Bates, D., Hartstein, G. L., Kim, H. T., Li, M., Nelson, B. W., Stromeyer IV, C., King, D., Suh, J., et al. (2025) · JMIR AI · doi:10.2196/69006 · Cited by 30
“Several evaluation frameworks [12-22] have emerged in response to these challenges over the last few years, particularly following the popularity of generative AI.”
Metadata
- Authors
- Kerstin, D., & Jim, W.
- Venue
- Studies in Health Technology and Informatics
- Publisher
- IOS Press
- Published
- 2020
- Volume
- 270
- Pages
- 976-980
- ISSN
- 0926-9630 (ISSN-L)
- OpenAlex
- W3035867101
- Cited by
- 18
- References
- 5
- FWCI
- 14.25
- Type
- book chapter
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: How to Evaluate Health Applications with Conversational User Interface? https://app.cochat.ai/papers/10.3233/shti200307
- The paper
- Kerstin, D., & Jim, W. (2020). How to Evaluate Health Applications with Conversational User Interface? Studies in Health Technology and Informatics. https://doi.org/10.3233/shti200307
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.