The DOI matches its Crossref record; 89 of the 89 references it cites were found in a registry or an index.
Technical Metrics Used to Evaluate Health Care Chatbots: Scoping Review
Open accessPublished version of a preprint
Abstract
Background: Dialog agents (chatbots) have a long history of application in health care, where they have been used for tasks such as supporting patient self-management and providing counseling. Their use is expected to grow with increasing demands on health systems and improving artificial intelligence (AI) capability. Approaches to the evaluation of health care chatbots, however, appear to be diverse and haphazard, resulting in a potential barrier to the advancement of the field. Objective: This study aims to identify the technical (nonclinical) metrics used by previous studies to evaluate health care chatbots. Methods: Studies were identified by searching 7 bibliographic databases (eg, MEDLINE and PsycINFO) in addition to conducting backward and forward reference list checking of the included studies and relevant reviews. The studies were independently selected by two reviewers who then extracted data from the included studies. Extracted data were synthesized narratively by grouping the identified metrics into categories based on the aspect of chatbots that the metrics evaluated. Results: Of the 1498 citations retrieved, 65 studies were included in this review. Chatbots were evaluated using 27 technical metrics, which were related to chatbots as a whole (eg, usability, classifier performance, speed), response generation (eg, comprehensibility, realism, repetitiveness), response understanding (eg, chatbot understanding as assessed by users, word error rate, concept error rate), and esthetics (eg, appearance of the virtual agent, background color, and content). Conclusions: The technical metrics of health chatbot studies were diverse, with survey designs and global usability metrics dominating. The lack of standardization and paucity of objective measures make it difficult to compare the performance of health chatbots and could inhibit advancement of the field. We suggest that researchers more frequently include metrics computed from conversation logs. In addition, we recommend the development of a framework of technical metrics with recommendations for specific circumstances for their inclusion in chatbot studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“However, a recent scoping review of health care chatbots showed that the use of JITAIs in designing and evaluating chatbots in health care in general and promoting physical activity and a healthy diet in particular is sparse, suggesting that future research needs to consider using more of these adaptive approaches [89].”
From Artificial Intelligence Chatbot Behavior Change Model for Designing Artificial Intelligence Chatbots to Promote Physical Activity and a Healthy Diet: Viewpoint (2020) · cites this paper as [89]
Experimental — accuracy not yet measuredIn the citing paper
“From the emergence of the first chatbot, ELIZA, developed by Joseph Weizenbaum (1966), chatbots have been trying to ‘mimic human behaviour in a text-based conversation’ (Shum et al. 2018, p. 10; Abd-Alrazaq et al. 2020).”
From Chatbot breakthrough in the 2020s? An ethical reflection on the trend of automated consultations in health care (2022) · cites this paper as Abd-Alrazaq et al. 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The PHQ-9: Validity of a brief depression severity measure
Kroenke, K., Spitzer, R. L., & Williams, J. B. W. (2001) · Journal of General Internal Medicine · doi:10.1046/j.1525-1497.2001.016009606.x · Cited by 44,910
PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation
Tricco, A. C., Lillie, E., Zarin, W., O'Brien, K. K., Colquhoun, H., Levac, D., Moher, D., Peters, M. D. J., Horsley, T., Weeks, L., et al. (2018) · Annals of Internal Medicine · doi:10.7326/m18-0850 · Cited by 43,565
Practical Statistics for Medical Research
Altman, D. G. (1990) · doi:10.1201/9780429258589 · Cited by 10,668
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697
An Empirical Evaluation of the System Usability Scale
Bangor, A., Kortum, P. T., & Miller, J. T. (2008) · International Journal of Human–Computer Interaction · doi:10.1080/10447310802205776 · Cited by 5,181 · cited 2×
“The system usability scale (SUS [82,83]) is a well-established usability instrument that we observed was used repeatedly, although it was not used in the majority of the studies assessing usability; in many cases, a single survey question was used instead.”
BackgroundMethodELIZA—a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1966) · Communications of the ACM · doi:10.1145/365153.365168 · Cited by 4,452
Usability Evaluation In Industry
(1996) · CRC Press eBooks · doi:10.1201/9781498710411 · Cited by 4,354
Fitzpatrick, K. K., Darcy, A., & Vierhile, M. (2017) · JMIR Mental Health · doi:10.2196/mental.7785 · Cited by 2,730
Webb, T. L., Joseph, J., Yardley, L., & Michie, S. (2010) · Journal of Medical Internet Research · doi:10.2196/jmir.1376 · Cited by 2,517 · cited 1×
“At the same time, internet/web-based interventions have shown the ability to promote positive health behavior change [4,5], and the interaction components associated with users sticking with an internet intervention are increasingly well understood and include the inclusion of dialog elements [6].”
BackgroundConversational agents in healthcare: a systematic review
Laranjo, L., Dunn, A. G., Tong, H. L., Kocaballi, A. B., Chen, J., Bashir, R., Surian, D., Gallego, B., Magrabi, F., Lau, A. Y. S., et al. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy072 · Cited by 1,388 · cited 2×
“Laranjo et al [86] reviewed the characteristics, current applications and evaluation measures of health chatbots.”
BackgroundResult
Research works printed but not linked: 6
- Denecke, K, 2020, Studies in Health Technology and Informatics, 1
- Ali, M, 2018, arXiv, 1
- Hess, G, 2019, Studies in health technology and informatics, 259, 101
- Ghosh, S, 2018, Stud Health Technol Inform, 252, 51
- Bickmore, T, 2003, Relational agents affecting change through human-computer relationships, 1
- Jadeja, M, 2017, arXiv preprint, 1
Cited by
A review of opportunities and challenges of chatbots in education
Hwang, G.-J., & Chang, C.-Y. (2023) · Interactive Learning Environments · doi:10.1080/10494820.2021.1952615 · Cited by 530
Zhang, J., Oh, Y. J., Lange, P., Yu, Z., & Fukuoka, Y. (2020) · Journal of Medical Internet Research · doi:10.2196/22845 · Cited by 363
Roles, Users, Benefits, and Limitations of Chatbots in Health Care: Rapid Review
Laymouna, M., Ma, Y., Lessard, D., Schuster, T., Engler, K., & Lebouché, B. (2024) · Journal of Medical Internet Research · doi:10.2196/56930 · Cited by 305
Roles, Users, Benefits, and Limitations of Chatbots in Health Care: Rapid Review (Preprint)
Laymouna, M., Ma, Y., Lessard, D., Schuster, T., Engler, K., & Lebouché, B. (2024) · doi:10.2196/preprints.56930 · Cited by 302
Perceptions and opinions of patients about mental health chatbots: scoping review
Abd-Alrazaq, A. A., Alajlani, M., Ali, N., Denecke, K., Bewick, B. M., & Househ, M. (2021) · Journal of Medical Internet Research · doi:10.2196/17828 · Cited by 301
“The first attempts toward evaluation frameworks for digital health interventions [82] and health chatbots [83,84] have been recently published.”
Self-citationOh, Y. J., Zhang, J., Fang, M.-L., & Fukuoka, Y. (2021) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-021-01224-6 · Cited by 278
A critical review of state‐of‐the‐art chatbot designs and applications
Luo, B., Lau, R. Y. K., Li, C., & Si, Y. (2022) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1434 · Cited by 224
He, Y., Yang, L., Zhu, X., Wu, B., Zhang, S., Qian, C., & Tian, T. (2022) · Journal of Medical Internet Research · doi:10.2196/40719 · Cited by 178
Artificial Intelligence for Chatbots in Mental Health: Opportunities and Challenges
Denecke, K., Abd-Alrazaq, A., & Househ, M. (2021) · Lecture Notes in Bioengineering · doi:10.1007/978-3-030-67303-1_10 · Cited by 177
Self-citationParviainen, J., & Rantala, J. (2022) · Medicine, Health Care and Philosophy · doi:10.1007/s11019-021-10049-w · Cited by 146
Bérubé, C., Schachner, T., Keller, R., Fleisch, E., v Wangenheim, F., Barata, F., & Kowatsch, T. (2021) · Journal of Medical Internet Research · doi:10.2196/25933 · Cited by 133
“Text-based conversational agents have been extensively studied in the domain of digital health interventions [80-83] and can be considered as a precursor to VCAs [9].”
The evaluation of chatbot as a tool for health literacy education among undergraduate students
Mokmin, N. A. M., & Ibrahim, N. A. (2021) · Education and Information Technologies · doi:10.1007/s10639-021-10542-y · Cited by 128
Nadarzynski, T., Knights, N., Husbands, D., Graham, C. A., Llewellyn, C. D., Buchanan, T., Montgomery, I., & Ridge, D. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000492 · Cited by 117
Izadi, S., & Forouzanfar, M. (2024) · AI · doi:10.3390/ai5020041 · Cited by 138
Health-focused conversational agents in person-centered care: a review of apps
Parmar, P., Ryu, J., Pandya, S., Sedoc, J., & Agarwal, S. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00560-6 · Cited by 113
Chatbots in Health Care: Connecting Patients to Information
Clark, M., & Bailey, S. M. (2024) · Canadian Journal of Health Technologies · doi:10.51731/cjht.2024.818 · Cited by 137
Implementation of Cognitive Behavioral Therapy in e–Mental Health Apps: Literature Review
Denecke, K., Schmid, N., & Nüssli, S. (2022) · Journal of Medical Internet Research · doi:10.2196/27791 · Cited by 107
Self-citationText Messaging-Based Medical Diagnosis Using Natural Language Processing and Fuzzy Logic
Omoregbe, N. A., Ndaman, I. O., Misra, S., Abayomi‐Alli, O., & Damaševičius, R. (2020) · Journal of Healthcare Engineering · doi:10.1155/2020/8839524 · Cited by 112
Talyshinskii, A., Naik, N., Hameed, B. M. Z., Juliebø‐Jones, P., & Somani, B. K. (2023) · Current Urology Reports · doi:10.1007/s11934-023-01184-3 · Cited by 96
Wang, J., Hwang, G., & Chang, C. (2021) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2021.100023 · Cited by 95
Martínez-Miranda, J., Martínez, A., Ramos, R., Aguilar, H., Jiménez, L., Arias, H., Rosales, G., & Valencia, E. (2019) · Journal of Medical Systems · doi:10.1007/s10916-019-1387-1 · Cited by 79
Ta, V. P., Boatfield, C., Wang, X., DeCero, E., Krupica, I. C., Rasof, S. D., Motzer, A., & Pedryc, W. M. (2022) · JMIR Human Factors · doi:10.2196/38876 · Cited by 75
Jabir, A. I., Martinengo, L., Lin, X., Torous, J., Subramaniam, M., & Tudor Car, L. (2023) · Journal of Medical Internet Research · doi:10.2196/44548 · Cited by 63
“Technical outcomes were measures used to evaluate the performance of the CA itself related to its technical interface, system crashes, and dialogue system, such as chatbot response generation [9].”
InfluentialDing, H., Simmich, J., Vaezipour, A., Andrews, N. E., & Russell, T. G. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad222 · Cited by 71
Wang, X., Luo, R., Liu, Y., Chen, P., Tao, Y., & He, Y. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103444 · Cited by 63
The Challenges in Designing a Prevention Chatbot for Eating Disorders: Observational Study
Chan, W. W., Fitzsimmons‐Craft, E. E., Smith, A. C., Firebaugh, M., Fowler, L. A., DePietro, B., Topooco, N., Wilfley, D. E., Taylor, C. B., & Jacobson, N. C. (2022) · JMIR Formative Research · doi:10.2196/28003 · Cited by 72
Embodied Conversational Agents for Chronic Diseases: Scoping Review
Jiang, Z., Huang, X., Wang, Z., Liu, Y., Huang, L., & Luo, X. (2023) · Journal of Medical Internet Research · doi:10.2196/47134 · Cited by 63
Experimentation for Chatbot Usability Evaluation: A Secondary Study
Ren, R., Zapata, M., Castro, J. W., Dieste, Ó., & Acuña, S. T. (2022) · IEEE Access · doi:10.1109/access.2022.3145323 · Cited by 56
Kiuchi, K., Otsu, K., & Hayashi, Y. (2023) · Behaviour and Information Technology · doi:10.1080/0144929x.2023.2286528 · Cited by 62
Noble, J. M., Zamani, A., Gharaat, M., Merrick, D., Maeda, N., Lambe Foster, A., Nikolaidis, I., Goud, R., Stroulia, E., Agyapong, V. I. O., et al. (2022) · JMIR Research Protocols · doi:10.2196/33717 · Cited by 53
Li, B., Gross, O., Crampton, N., Kapoor, M., Tauseef, S., Jain, M., Truong, K. N., & Mariakakis, A. T. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641913 · Cited by 53
Kolenik, T., & Gams, M. (2021) · Electronics · doi:10.3390/electronics10111250 · Cited by 55
Clinical-chatbot AHP evaluation based on “quality in use” of ISO/IEC 25010
Barletta, V. S., Caivano, D., Colizzi, L., Dimauro, G., & Piattini, M. G. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2022.104951 · Cited by 51
Griffin, A. C., Khairat, S., Bailey, S. C., & Chung, A. E. (2023) · JAMIA Open · doi:10.1093/jamiaopen/ooad073 · Cited by 46
ASHABot: An LLM-Powered Chatbot to Support the Informational Needs of Community Health Workers
Ramjee, P., Chhokar, M., Sachdeva, B., Meena, M., Abdullah, H., Vashistha, A., Nagar, R., & Jain, M. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713680 · Cited by 69
CataractBot: An LLM-powered Expert-in-the-Loop Chatbot for Cataract Patients
Ramjee, P., Sachdeva, B., Golechha, S., Kulkarni, S., Fulari, G., Murali, K., & Jain, M. (2025) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3729479 · Cited by 55
Chow, J. C. L., & Li, K. (2025) · JMIR Cancer · doi:10.2196/66633 · Cited by 46
Mauriello, M. L., Tantivasadakarn, N., Mora-Mendoza, M. A., Lincoln, E. T., Hon, G., Nowruzi, P., Simon, D., Hansen, L., Goenawan, N. H., Kim, J., et al. (2021) · JMIR Formative Research · doi:10.2196/25294 · Cited by 45
Issom, D.-Z., Hardy-Dessources, M.-D., Romana, M., Hartvigsen, G., & Lovis, C. (2021) · Frontiers in Digital Health · doi:10.3389/fdgth.2021.600333 · Cited by 39
Chou, Y.-H., Lin, C., Lee, S.-H., Lee, Y.-F., & Cheng, L.-C. (2024) · JMIR Formative Research · doi:10.2196/49462 · Cited by 38
Chagas, B. A., Pagano, A. S., Prates, R. O., Praes, E. C., Ferreguetti, K., Vaz, H., Reis, Z. S. N., Ribeiro, L. B., Ribeiro, A. L. P., Pedroso, T. M., et al. (2023) · JMIR Human Factors · doi:10.2196/43135 · Cited by 38
Sagstad, M. H., Morken, N., Lund, A., Dingsør, L. J., Nilsen, A. B. V., & Sørbye, L. M. (2022) · JMIR Formative Research · doi:10.2196/28091 · Cited by 40
Chau, L. W., Lam, R. W. M., Minas, H., Hayashi, K., Nguyen, V. C., & O’Neil, J. (2025) · JMIR Mental Health · doi:10.2196/68296 · Cited by 39
Sezgın, E., Kocaballi, A. B., Dolcé, M., Skeens, M. A., Militello, L. K., Huang, Y., Stevens, J. H., & Kemper, A. R. (2024) · JMIR Human Factors · doi:10.2196/57114 · Cited by 38
Evaluation Metrics for Health Chatbots: A Delphi Study
Denecke, K., Abd-Alrazaq, A. A., Househ, M. S., & Warren, J. (2021) · Methods of Information in Medicine · doi:10.1055/s-0041-1736664 · Cited by 31
Self-citationSuryanto, T. L. M., Wibawa, A. P., Hariyono, H., & Nafalski, A. (2023) · International Journal of Robotics and Control Systems · doi:10.31763/ijrcs.v3i4.1195 · Cited by 33
Conversational Physical Activity Coaches for Spanish and English Speaking Women: A User Design Study
Figueroa, C. A., Luo, T. C., Jacobo, A., Munoz, A., Manuel, M., Chan, D., Canny, J., & Aguilera, A. (2021) · Frontiers in Digital Health · doi:10.3389/fdgth.2021.747153 · Cited by 29
Chatbots Utility in Healthcare Industry: An Umbrella Review
Afsahi, A. M., SeyedAlinaghi, S., Molla, A., Mirzapour, P., Jahani, S., Razi, A., Mojdeganlou, P., Karimi, E., Mehrtak, M., Dadras, O., et al. (2024) · Frontiers in Health Informatics · doi:10.30699/fhi.v13i0.561 · Cited by 27
Pupong, K., Hunsrisakhun, J., Pithpornchaiyakul, S., & Naorungroj, S. (2025) · JMIR Pediatrics and Parenting · doi:10.2196/62738 · Cited by 28
Giansanti, D. (2023) · Life · doi:10.3390/life13051130 · Cited by 29
Denecke, K., & May, R. (2022) · Journal of Medical Internet Research · doi:10.2196/41583 · Cited by 26
Self-citationA systematic review of AI-based chatbot usages in healthcare services
Jasim, K. M., Malathi, A., Bhardwaj, S., & Aw, E. C. (2025) · Journal of Health Organization and Management · doi:10.1108/jhom-12-2023-0376 · Cited by 29
Liu, D., Zhang, Y., Zhao, B., Ma, S., Shi, C., & Ma, X. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714196 · Cited by 28
User Experience of COVID-19 Chatbots: Scoping Review
White, B. K., Martin, A., & White, J. A. (2022) · Journal of Medical Internet Research · doi:10.2196/35903 · Cited by 22
Analysing Conversation Pathways with a Chatbot Tutor to Enhance Self-Regulation in Higher Education
Martins, L., Fernández-Ferrer, M., & Puertas, E. (2024) · Education Sciences · doi:10.3390/educsci14060590 · Cited by 24
The Art of Talking Machines: A Comprehensive Literature Review of Conversational User Interfaces
Namvarpour, M. (., & Razi, A. (2025) · Proceedings of the 7th ACM Conference on Conversational User Interfaces · doi:10.1145/3719160.3736621 · Cited by 19
Framework for Guiding the Development of High-Quality Conversational Agents in Healthcare
Denecke, K. (2023) · Healthcare · doi:10.3390/healthcare11081061 · Cited by 21
Self-citationChatbots: A Game Changer in mHealth
Chowdhury, M. N.-U.-R., Haque, A., & Soliman, H. (2023) · International Symposium on Computer, Consumer and Control · doi:10.1109/is3c57901.2023.00103 · Cited by 20
Li, B., Tauseef, S., Truong, K. N., & Mariakakis, A. T. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713613 · Cited by 22
Andrews, N. E., Ireland, D., Vijayakumar, P., Burvill, L., Hay, E., Westerman, D., Rose, T., Schlumpf, M., Strong, J., & Claus, A. P. (2023) · JMIR Formative Research · doi:10.2196/47267 · Cited by 19
Viswanathan, V. K., Jain, V. K., Vaish, A., Jeyaraman, M., Iyengar, K. P., & Vaishya, R. (2024) · Apollo Medicine · doi:10.1177/09760016241259851 · Cited by 17
Polireddi, N. S. A., & Kavitha, J. (2023) · Soft Computing · doi:10.1007/s00500-023-08541-x · Cited by 17
RetractedImproving the user experience of a gambling support and education website using a chatbot
Merkouris, S., Loram, G., Abdelrazek, M., Rodda, S. N., Ibrahim, A. I., Bonti, A., & Dowling, N. A. (2022) · Universal Access in the Information Society · doi:10.1007/s10209-022-00932-5 · Cited by 16
Koscelny, S. N., Sadralashrafi, S., & Neyens, D. M. (2025) · Applied Ergonomics · doi:10.1016/j.apergo.2025.104515 · Cited by 16
Chatbots and young people in emerging economies: Factors affecting user satisfaction
López-Sánchez, J. A., Bermeo-Giraldo, M. C., Valencia-Arías, A., García, R. E. B., & Gallegos, A. (2023) · Cogent Social Sciences · doi:10.1080/23311886.2023.2252256 · Cited by 17
Evaluating ChatGPT as an Agent for Providing Genetic EducationPreprint
Walton, N., Graceffo, S., Sutherland, N., Kozel, B. A., Danford, C. J., & McGrath, S. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.10.25.564074 · Cited by 14
Development of a practical training method for a healthcare artificial intelligence (AI) chatbot
Lin, J., Joseph, T., Parga-Belinkie, J. J., Mandel, A., Schumacher, R., Neumann, K., Scalise, L., Gaulton, J., Christ, L., Leitner, K., et al. (2021) · BMJ Innovations · doi:10.1136/bmjinnov-2020-000530 · Cited by 13
Begum, T. U. S., Devi, R., Haridas, D., Bačanin, N., Djuric-Jovicic, M. D., & Nikolić, B. (2025) · Scientific Reports · doi:10.1038/s41598-025-24172-8 · Cited by 15
Decoding the cry for help: AI's emerging role in suicide risk assessment
Esmaeilzadeh, P. (2025) · AI and Ethics · doi:10.1007/s43681-025-00758-w · Cited by 13
Information quality of conversational agents in healthcare
Liu, C., Zowghi, D., Peng, G., & Kong, S. (2023) · Information Development · doi:10.1177/02666669231172434 · Cited by 15
AI Chatbot Technology to Predict Disease: A Systematic Literature Review
Velasco, V., Setiawan, K., Sanjaya, R. R., Anggreainy, M. S., & Kurniawan, A. (2023) · International Conference on Artificial Intelligence and Data Sciences (AiDAS) · doi:10.1109/aidas60501.2023.10284717 · Cited by 13
ChatGPT for Addressing Patient-centered Frequently Asked Questions in Glaucoma Clinical Practice
Wang, H., Masselos, K., Tong, J., Connor, H. R. M., Scully, J., Zhang, S., Rafla, D., Posarelli, M., Tan, J. C. K., Agar, A., et al. (2024) · Ophthalmology Glaucoma · doi:10.1016/j.ogla.2024.10.005 · Cited by 13
Su, B., Jones, R., Chen, K. S., Kostenko, E., Schmid, M., DeMaria, A. L., Villa, A., Swarup, M., Weida, J., & Tuuli, M. G. (2024) · Patient Education and Counseling · doi:10.1016/j.pec.2024.108557 · Cited by 14
Cost-effectiveness of using chatbots in healthcare: a systematic review
Fgaier, M., & Zrubka, Z. (2022) · IEEE International Symposium on Computational Intelligence and Informatics (CINTI) · doi:10.1109/cinti-macro57952.2022.10029478 · Cited by 11
Er-Rays, Y., M’dioud, M., & Lemqeddem, H. A. (2025) · AI and Ethics · doi:10.1007/s43681-025-00672-1 · Cited by 11
The Accuracy of Artificial Intelligence (AI) Chatbots in Telemedicine
Swick, R. K. (2021) · Scholar Commons (University of South Carolina) · Cited by 9
Al-Shamaileh, O., Mubin, O., & Nizamuddin, N. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2532712 · Cited by 8
Review of Time Domain Electronic Medical Record Taxonomies in the Application of Machine Learning
Ali, H., Niazi, I. K., Russell, B. K., Crofts, C. A. P., Madanian, S., & White, D. E. (2023) · Electronics · doi:10.3390/electronics12030554 · Cited by 9
Factors influencing the adoption of chatbots by healthcare users
Bermeo-Giraldo, M. C., Patiño-Toro, O. N., Valencia-Arías, A., & Rodríguez-Correa, P. A. (2023) · Journal of Innovation Management · doi:10.24840/2183-0606_011.003_0004 · Cited by 9
Kaushik, A., Jacob, B., & Velavan, P. (2022) · Knowledge · doi:10.3390/knowledge2010007 · Cited by 8
Hsu, Y., Chou, Y.-R., Chang, C.-J., Chang, Y.-C., Lee, Z. H., Gipiškis, R., Li, R., Shih, C., Peng, J., Huang, H., et al. (2026) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2026.101388 · Cited by 7
AI-Powered Virtual Assistant: To Enhance Saudi Arabia Travel Experience and Support Tourism Growth
Alamoudi, Y., Alasmari, H., Alamoudi, G., & Alghamdi, H. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-66336-9_18 · Cited by 7
Pawar, B., Mahajan, S. R., & Kolhar, S. (2025) · Discover Psychology · doi:10.1007/s44202-025-00513-6 · Cited by 7
Answerability: A custom metric for evaluating chatbot performance
Gupta, P., Rajasekar, A. A., Patel, A., Kulkarni, M., Sunell, A., Kim, K., Ganapathy, K., & Trivedi, A. (2022) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2022.gem-1.27 · Cited by 6
Conversational Agents in Healthcare: A Variability Perspective
May, R., & Denecke, K. (2024) · International Workshop on Variability Modelling of Software-Intensive Systems (VaMoS) · doi:10.1145/3634713.3634717 · Cited by 6
Self-citationGamberini, L., Pluchino, P., Porto, L., Zordan, F., Monterosso, A., Palanivel, K., & Fontanari, A. (2021) · Communications in computer and information science · doi:10.1007/978-3-030-78645-8_39 · Cited by 6
Chatbots: A game changer in mHealthPreprint
Chowdhury, M. N.-U.-R., Haque, A., & Soliman, H. (2023) · doi:10.36227/techrxiv.22548532.v1 · Cited by 6
Onyekwelu, I. B., Shrestha, R., & Sandnes, F. E. (2024) · International Conference on PErvasive Technologies Related to Assistive Environments (PETRA) · doi:10.1145/3652037.3652042 · Cited by 7
HelpBot: A Web-Based Chatbot to Handle Depression Among Adolescents
Jusoh, S., Alfawareh, H. M., Kadir, R. A., & Hosseinzadeh, H. (2024) · International Conference on Computing and Artificial Intelligence (ICCAI) · doi:10.1145/3669754.3669777 · Cited by 5
Pre-operative Planning of High Tibial Osteotomy With ChatGPT: Are We There Yet?
Gengatharan, D., Saggi, S. S., & Razak, H. R. B. A. (2024) · Cureus · doi:10.7759/cureus.54858 · Cited by 4
Experiences of patients with hard-to-heal wounds: insights from a pilot survey
Wahab, N., & Forsyth, A. (2024) · Journal of Wound Care · doi:10.12968/jowc.2024.0109 · Cited by 4
Sundaramoorthy, S., Ratra, V., Shankar, V., Dorairajan, R., Maskati, Q., Fredrick, T. N., Ratra, A., & Ratra, D. (2025) · Ophthalmic Epidemiology · doi:10.1080/09286586.2025.2484772 · Cited by 4
Paediatric rare diseases: Can large language models assist off‐label prescribing?
Flamigni, A., Zamagni, G., Paternuosto, G., & Arbo, A. (2025) · British Journal of Clinical Pharmacology · doi:10.1002/bcp.70168 · Cited by 4
Zidoun, Y., Kaladhara, S., Powell, L., Nour, R., Suwaidi, H. S. A., & Zary, N. (2022) · JMIR Research Protocols · doi:10.2196/38043 · Cited by 4
Overview of chatbot usage on mental health: A scoping review
Permatasari, R. I., Artha, D. P., Wiratama, B. S., & Wulandari, H. (2024) · BIO Web of Conferences · doi:10.1051/bioconf/202413205002 · Cited by 4
A Chatbot for Preliminary Patient Guidance System
Tuncel, F., Mumcu, B., & Tanberk, S. (2021) · Signal Processing and Communications Applications Conference (SIU) · doi:10.1109/siu53274.2021.9478023 · Cited by 4
He, Y., Yang, L., Zhu, X., Wu, B., Zhang, S., Qian, C., & Tian, T. (2022) · doi:10.2196/preprints.40719 · Cited by 3
Liu, I., Chen, W., Ge, Q., Song, D., & Ni, S. (2022) · Proceedings of the Tenth International Symposium of Chinese CHI · doi:10.1145/3565698.3565787 · Cited by 3
Kim, J., Kim, S. J., Ahn, J., & Lee, S. (2025) · Healthcare Informatics Research · doi:10.4258/hir.2025.31.2.136 · Cited by 3
Collins, C., Fackler, J. C., Sacco, M. J., & Jacobs, M. L. (2025) · Frontiers in Pediatrics · doi:10.3389/fped.2025.1646989 · Cited by 3
Metadata
- Authors
- Abd-Alrazaq, A., Safi, Z., Alajlani, M., Warren, J., Househ, M., & Denecke, K.
- Affiliations
- Hamad bin Khalifa University (QA)
- University of Warwick (GB)
- University of Auckland (NZ)
- Bern University of Applied Sciences (CH)
- Corresponding author
- Denecke, Kerstin
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2020-06-05
- Volume
- 22
- Issue
- 6
- Pages
- e18301
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC7305563
- OpenAlex
- W3021626742
- Semantic Scholar
- e4286342557d57067ecf35a4969c621d2484ef3a
- Cited by
- 150
- References
- 88
- Citations by year
- 2026: 25 · 2025: 32 · 2024: 22 · 2023: 27 · 2022: 21 · 2021: 18 · 2020: 4
- FWCI
- 1.34
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- ARBOR - Bern University of Applied Sciences Repository (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Warwick Research Archive Portal (University of Warwick) (submittedVersion)
- BFH: ARBOR
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Keywords
- chatbot evaluation, usability, response generation, word error rate, conversation logs, dialogue agents
- MeSH terms
- Artificial Intelligence, Communication, Delivery of Health Care, Humans
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
