Technical Metrics Used to Evaluate Health Care Chatbots: Scoping Review
Open accessPublished version of a preprint
The DOI matches its Crossref record; 89 of the 89 references it cites were found in a registry or an index.
Abstract
Background: Dialog agents (chatbots) have a long history of application in health care, where they have been used for tasks such as supporting patient self-management and providing counseling. Their use is expected to grow with increasing demands on health systems and improving artificial intelligence (AI) capability. Approaches to the evaluation of health care chatbots, however, appear to be diverse and haphazard, resulting in a potential barrier to the advancement of the field. Objective: This study aims to identify the technical (nonclinical) metrics used by previous studies to evaluate health care chatbots. Methods: Studies were identified by searching 7 bibliographic databases (eg, MEDLINE and PsycINFO) in addition to conducting backward and forward reference list checking of the included studies and relevant reviews. The studies were independently selected by two reviewers who then extracted data from the included studies. Extracted data were synthesized narratively by grouping the identified metrics into categories based on the aspect of chatbots that the metrics evaluated. Results: Of the 1498 citations retrieved, 65 studies were included in this review. Chatbots were evaluated using 27 technical metrics, which were related to chatbots as a whole (eg, usability, classifier performance, speed), response generation (eg, comprehensibility, realism, repetitiveness), response understanding (eg, chatbot understanding as assessed by users, word error rate, concept error rate), and esthetics (eg, appearance of the virtual agent, background color, and content). Conclusions: The technical metrics of health chatbot studies were diverse, with survey designs and global usability metrics dominating. The lack of standardization and paucity of objective measures make it difficult to compare the performance of health chatbots and could inhibit advancement of the field. We suggest that researchers more frequently include metrics computed from conversation logs. In addition, we recommend the development of a framework of technical metrics with recommendations for specific circumstances for their inclusion in chatbot studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“However, a recent scoping review of health care chatbots showed that the use of JITAIs in designing and evaluating chatbots in health care in general and promoting physical activity and a healthy diet in particular is sparse, suggesting that future research needs to consider using more of these adaptive approaches [89].”
From Artificial Intelligence Chatbot Behavior Change Model for Designing Artificial Intelligence Chatbots to Promote Physical Activity and a Healthy Diet: Viewpoint (2020) · cites this paper as [89]
Experimental — accuracy not yet measuredIn the citing paper
“From the emergence of the first chatbot, ELIZA, developed by Joseph Weizenbaum (1966), chatbots have been trying to ‘mimic human behaviour in a text-based conversation’ (Shum et al. 2018, p. 10; Abd-Alrazaq et al. 2020).”
From Chatbot breakthrough in the 2020s? An ethical reflection on the trend of automated consultations in health care (2022) · cites this paper as Abd-Alrazaq et al. 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The PHQ-9: Validity of a brief depression severity measure
Kroenke, K., Spitzer, R. L., & Williams, J. B. W. (2001) · Journal of General Internal Medicine · doi:10.1046/j.1525-1497.2001.016009606.x · Cited by 44,910
PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation
Tricco, A. C., Lillie, E., Zarin, W., O'Brien, K. K., Colquhoun, H., Levac, D., Moher, D., Peters, M. D. J., Horsley, T., Weeks, L., et al. (2018) · Annals of Internal Medicine · doi:10.7326/m18-0850 · Cited by 43,565
Practical Statistics for Medical Research
Altman, D. G. (1990) · doi:10.1201/9780429258589 · Cited by 10,668
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697
An Empirical Evaluation of the System Usability Scale
Bangor, A., Kortum, P. T., & Miller, J. T. (2008) · International Journal of Human–Computer Interaction · doi:10.1080/10447310802205776 · Cited by 5,181 · cited 2×
“The system usability scale (SUS [82,83]) is a well-established usability instrument that we observed was used repeatedly, although it was not used in the majority of the studies assessing usability; in many cases, a single survey question was used instead.”
BackgroundMethodELIZA—a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1966) · Communications of the ACM · doi:10.1145/365153.365168 · Cited by 4,452 · cited 2×
“Given that this review was executed and reported in line with PRISMA-ScR guidelines [1], it could be considered a high-quality review.”
BackgroundResultUsability Evaluation In Industry
(1996) · CRC Press eBooks · doi:10.1201/9781498710411 · Cited by 4,354
Fitzpatrick, K. K., Darcy, A., & Vierhile, M. (2017) · JMIR Mental Health · doi:10.2196/mental.7785 · Cited by 2,730
Webb, T. L., Joseph, J., Yardley, L., & Michie, S. (2010) · Journal of Medical Internet Research · doi:10.2196/jmir.1376 · Cited by 2,517 · cited 1×
“At the same time, internet/web-based interventions have shown the ability to promote positive health behavior change [4,5], and the interaction components associated with users sticking with an internet intervention are increasingly well understood and include the inclusion of dialog elements [6].”
BackgroundConversational agents in healthcare: a systematic review
Laranjo, L., Dunn, A. G., Tong, H. L., Kocaballi, A. B., Chen, J., Bashir, R., Surian, D., Gallego, B., Magrabi, F., Lau, A. Y. S., et al. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy072 · Cited by 1,388 · cited 2×
“Laranjo et al [86] reviewed the characteristics, current applications and evaluation measures of health chatbots.”
BackgroundResult
Research works printed but not linked: 6
- Denecke, K, 2020, Studies in Health Technology and Informatics, 1
- Ali, M, 2018, arXiv, 1
- Hess, G, 2019, Studies in health technology and informatics, 259, 101
- Ghosh, S, 2018, Stud Health Technol Inform, 252, 51
- Bickmore, T, 2003, Relational agents affecting change through human-computer relationships, 1
- Jadeja, M, 2017, arXiv preprint, 1
Cited by
Items 101–163 of 163
Lim, S., Yang, Q., Meng, J., & Rains, S. A. (2025) · Behaviour and Information Technology · doi:10.1080/0144929x.2025.2567989 · Cited by 3
Ahamed, S. A., Chowdhury, M. H., Zaman, M., Agarwala, A., Mashud, M. A. A., Tariqul, Muntasir, T. A., Rahman, R., Shahriyar, R., Sarker, F., et al. (2025) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2025.101720 · Cited by 5
Ma, A., Chen, J., & Yang, Z. (2026) · Healthcare · doi:10.3390/healthcare14060820 · Cited by 3
Liu, C., Zhang, B., & Peng, G. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-77015-0_24 · Cited by 3
Enhancing Freelancer Project Matching with a BERT-Powered Deep Learning Indonesian Chatbot
Aziz, A., Khadija, M. A., Nurharjadmo, W., Febrianto, D. C., Firmansyah, M. R., & Pratama, R. A. (2024) · International Conference on Computer, Control, Informatics and its Applications · doi:10.1109/ic3ina64086.2024.10732347 · Cited by 3
Ask NAEP: A Generative AI Assistant for Querying Assessment Information
Zhang, T., Patterson, L., Beiting‐Parrish, M., Webb, B., Abeysinghe, B., Bailey, P., & Sikali, E. (2024) · Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi · doi:10.21031/epod.1548128 · Cited by 3
Unlocking AI Chatbot Potential in Healthcare: Trust-Enhanced DeLone & McLean IS Success Model
Sarhan, M. Y., Alarify, M., & Khojah, M. (2026) · Healthcare · doi:10.3390/healthcare14101324 · Cited by 2
Parents’ information needs and perceptions of chatbots regarding self medicating their children
Wei, X., Chen, X., Yue, L., & Liao, P. (2025) · Scientific Reports · doi:10.1038/s41598-025-15130-5 · Cited by 2
The role of human-like AI in effective human–machine communication
Simfa, E., Sprogis, D. K., & Melbardis, M. (2025) · Discover Artificial Intelligence · doi:10.1007/s44163-025-00559-4 · Cited by 3
Vowels, L. M., Vohra, P., Li, D., Zeinoddin, P. S., Elswick, A., Marcantonio, T. L., Wood, N. D., & Vowels, M. J. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001352 · Cited by 2
Hayakawa, M., Watanabe, O., Shiga, K., Fujishita, M., Yamaki, C., Ogo, Y., Takahashi, T., Ikeguchi, Y., & Takayama, T. (2022) · Patient Education and Counseling · doi:10.1016/j.pec.2022.10.004 · Cited by 2
Harnessing Artificial Intelligence (AI) for Psychological Assessment and Treatment in Older Adults
Şenol-Durak, E. (2024) · Journal of Aging and Long-Term Care · doi:10.51819/jaltc.2024.1595910 · Cited by 5
Chatbots: A Game Changer in mHealthPreprint
Chowdhury, M. N.-U.-R., Haque, A., & Soliman, H. (2023) · doi:10.36227/techrxiv.22548532.v2 · Cited by 2
Scalia, J. L., Laprise, J. K., Thrift, J. R., Farrell, C. L., & Sarasua, S. M. (2026) · JMIR AI · doi:10.2196/76400 · Cited by 1
Leveraging AI Chatbots to Transform University Mental Health Support: A Case Study
Beem, V. R. (2025) · Journal of Computer Science and Technology Studies · doi:10.32996/jcsts.2025.7.5.80 · Cited by 1
Nyongesa, M. J. (2025) · Pan-African journal of education and social sciences · doi:10.56893/pajes2025v06i02.10 · Cited by 2
A Generative AI Framework for Learning Management System (LMS) Chatbot Integration
Ramakrisnan, P., & Hoo, M. S. C. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3808-9_25 · Cited by 1
da Silva, J. C., Félix, B., Meoli, L., Ntinga, X., Melo, R., Joolay, Y., Till, S. C., Densmore, M., & Nunes, F. (2025) · African Human Computer Interaction Conference (AfriCHI) · doi:10.1145/3757232.3757252 · Cited by 1
DBT-wellMind-GameTree: a game-theoretic framework for student mental health support using LLMs
Kaysar, M. N., Nagasawa, F., & Shiramatsu, S. (2026) · BMC Artificial Intelligence · doi:10.1186/s44398-026-00025-w · Cited by 1
Novello, P., Weeks, R., Sedoc, J., Ou, Y., Sangha, P., Miller, E. A., Jamison, A., & Agarwal, S. (2026) · BMJ Digital Health & AI · doi:10.1136/bmjdh-2026-000036 · Cited by 1
Roman, M. K., de Oliveira, L. Z., Veiga, J. E., Bellei, E. A., & Marchi, A. C. B. D. (2026) · Healthcare · doi:10.3390/healthcare14121641 · Cited by 1
Detecting the patient’s need for help with machine learning based on expressions
Lahti, L. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01502-8 · Cited by 1
Critical information quality dimensions of conversational agents for healthcare
Liu, C., Peng, G., Kong, S., Lan, C., & Zhu, H. (2023) · Information Research · doi:10.47989/ir284561 · Cited by 1
A multifaceted overview of AI in different fields: A survey
Tingare, B. A., Nikam, S. V., & Narlawar, S. S. (2025) · AIP conference proceedings · doi:10.1063/5.0254217 · Cited by 1
Engagement of Parents with the Aurora Child Health Chatbot: A Conversation Log Analysis Study
Liebeatrau, D., Densmore, M., & Nunes, F. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2492805 · Cited by 2
Supporting Caregivers in Complex Home Care: Towards Designing a Voice User Interface
Tennant, R. (2021) · UWSpace (University of Waterloo) · Cited by 3
Healthcare Delivery in the Digital Age
Gibbons, M. C., Shaihk, Y., & Ayalasomayajula, F. (2022) · Health informatics · doi:10.1007/978-3-031-07912-2_22
da Luz, M. A. P., Guimarães, R. J., & Moro, C. M. C. (2023) · doi:10.2196/preprints.47313
Chatbots: A Game Changer in mHealthPreprint
Chowdhury, M. N.-U.-R., Haque, A., & Soliman, H. (2023) · doi:10.36227/techrxiv.22548532
Inoue, K., Lala, D., Ochi, K., Kawahara, T., & Skantze, G. (2023) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3610661.3617151
Knowledge-Grounded Dialogue Generation for Medical Conversations: A Survey
Liu, X., Chang, J. M., & Zhang, J. (2023) · International Conference on Information Visualisation (IV) · doi:10.1109/iv60283.2023.00075
Agarwal, S. (2024) · International Journal of Research in Medical Sciences & Technology · doi:10.37648/ijrmst.v17i01.011
Chatbot Systems: Measuring Effectiveness, Challenges, and Future Directions Across Industries
Xhaferra, E., Rista, A., Mukli, L., Prodani, F., & Gjermeni, F. (2025) · Interdisciplinary Journal of Research and Development · doi:10.56345/ijrdv12n2013
Overview of chatbot usage on mental health: A scoping review
Permatasari, R. I., Artha, D. P., Wiratama, B. S., & Wulandari, H. (2024) · Springer Link (Chiba Institute of Technology) · doi:10.1051/bioconf/202413205002/pdf
1177, 24/7? User perceptions of human elements in the chatbot "1177 direkt" in Region Uppsala
Luo, S. (2024) · Uppsala University Publications (Uppsala University)
Informing the Development of a Chatbot for Pharmacist Needs in HIV Care
Laymouna, M. A. (2024) · eScholarship@McGill (McGill) · doi:10.82308/48190 · Cited by 3
Mäkiaho, H. (2023) · Jyväskylä University Digital Archive (University of Jyväskylä)
Luijendijk, R., & Brinkman, W. (2022) · Research Repository (Delft University of Technology) · doi:10.17605/osf.io/783gv
An Energy-Efficient Blockchain Security Framework for Sustainable IoT and Edge Environments
Mohamed, Y. A., Bashir, M., & Khanan, A. (2025) · IEEE International Conference on Advanced Infocomm Technology · doi:10.1109/icait66450.2025.11353325
Developing a Service Quality Index System for AI Health Care Chatbots: Mixed Methods Study
Gu, Y., & Wang, X. (2026) · Journal of Medical Internet Research · doi:10.2196/83051
Design and Implementation of Healix AI Smart Medical Chatbot Using NLP and Deep Neural Networks
Khokher, B., Reddy, N. V. U., Palimar, S. G., Suman, S., Surya, R., & Naik, S. (2025) · Global AI Summit - International Conference on Artificial Intelligence and Emerging Technology (AI Summit) · doi:10.1109/aisummit66170.2025.11411307
Implementation and Evaluation of an Open-Source Chatbot for Patient Information Leaflets
Heiler, L., Kirchsteiger, K., Hanke, S., & Bödenler, M. (2026) · Future Internet · doi:10.3390/fi18030139
Subbulakshmi, M. M., Jayaraman, M., & Jeyaraman, J. (2026) · Expert Review of Medical Devices · doi:10.1080/17434440.2026.2649559
Automated health information question-answering reimagined: Medical Relation-Aware Encoding Network
Pan, T., Chua, A. Y. K., Shen, J., & Xu, M. (2026) · Journal of Information Science · doi:10.1177/01655515261433926
Jabin, M. S. R., Mbwayo, A., Misoi, I. C., Ngaruiya, M., & Macharia, P. M. (2026) · doi:10.2196/preprints.97360
Weinreich, L., von Kontz, L.-M., Witzel, B., Hermansson, O., Hampe, H. L., Volkmer, S., Schulte‐Körne, G., & Moll, K. (2026) · Journal of Medical Internet Research · doi:10.2196/64903
Zhang, C., Wei, W., & Hu, X. (2026) · Education Sciences · doi:10.3390/educsci16060824
A Chatbot for Teaching Secure Programming
Walden, J., Atnafu, L., & Caporusso, N. (2023) · Cybersecurity Pedagogy and Practice Journal · doi:10.62273/gigf6782
Rizwan, M., Alrasheed, K. A., Alotaibi, A., Hadi, A., Ghani, M. U., Almujibah, H., & Elshekh, A. E. A. (2026) · Journal of Engineering Research · doi:10.1016/j.jer.2026.06.011
Nurfadhilah, E., Aini, L. R., Santosa, A., Uliniansyah, M. T., Gunarso, G., Putra, P. K., Pebiana, S., Fajri, R., Hidayati, N. N., Prastowo, R. D., et al. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-7862-6.ch003
Beyond AI Capability – Governance, Trust, and Realistic Scope in Rural Health Chatbots
Krayter, S., Bhattacharjee, S., Fatima, F., Patanker, A., Epping, L., & Müller, C. (2026) · ACM Interactive Health Conference · doi:10.1145/3786579.3804951
The Impact of Chatbots on Organizations
Lopes, I., Guarda, T., Fernandes, A., & Ribeiro, M. I. (2026) · Springer proceedings in business and economics · doi:10.1007/978-3-032-24600-4_31
da Silva, I. S., & Fernandes, M. G. (2026) · Revista de Geopolítica · doi:10.56238/revgeov17n8-023
Chawla, M., Alsadoon, A., Fadhil, A., Ameen, M., Elhag, A. A. M., Althubaity, A., & Osman, A. H. (2026) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2026.01708100
TAMBOURA, A., Diarra, F., Kante, M., Kanté, M., & Konate, I. (2025) · African Journal of Development Informatics · doi:10.68173/ajdi.v0i2.2
Liao, Y., Madeo, A. C., Allen, C. G., Frey, M. K., Espinel, W., Schlechter, C. R., Sharaf, R., Kawamoto, K., Fiol, G. D., & Kaphingst, K. A. (2026) · Journal of Medical Internet Research · doi:10.2196/97773
Macharia, P., Ngaruiya, N., Misoi, I. C., Mbwayo, A., & Jabin, M. S. R. (2026) · Research Square · doi:10.21203/rs.3.rs-10504873/v1
Baek, J. S., Choi, S., Sung, S., Rajaguru, V., & Park, Y. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001720
Divya, S., Nevarez, N., Levitz, N., Shum, K., & Gust, E. (2026) · doi:10.2196/preprints.114147
Cheng, N., Yu, R., & Wang, X. (2026) · Ergonomics · doi:10.1080/00140139.2026.2748363
Macharia, P. M., Ngaruiya, N., Misoi, I. C., Mbwayo, A., & Jabin, M. S. R. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1908306
Metadata
- Authors
- Abd-Alrazaq, A., Safi, Z., Alajlani, M., Warren, J., Househ, M., & Denecke, K.
- Affiliations
- Hamad bin Khalifa University (QA)
- University of Warwick (GB)
- University of Auckland (NZ)
- Bern University of Applied Sciences (CH)
- Corresponding author
- Denecke, Kerstin
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2020-06-05
- Volume
- 22
- Issue
- 6
- Pages
- e18301
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC7305563
- OpenAlex
- W3021626742
- Semantic Scholar
- e4286342557d57067ecf35a4969c621d2484ef3a
- Cited by
- 150
- References
- 88
- Citations by year
- 2026: 25 · 2025: 32 · 2024: 22 · 2023: 27 · 2022: 21 · 2021: 18 · 2020: 4
- FWCI
- 1.34
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- ARBOR - Bern University of Applied Sciences Repository (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Warwick Research Archive Portal (University of Warwick) (submittedVersion)
- BFH: ARBOR
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Keywords
- chatbot evaluation, usability, response generation, word error rate, conversation logs, dialogue agents
- MeSH terms
- Artificial Intelligence, Communication, Delivery of Health Care, Humans
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
