The DOI matches its Crossref record; 89 of the 89 references it cites were found in a registry or an index.
Technical Metrics Used to Evaluate Health Care Chatbots: Scoping Review
Open accessPublished version of a preprint
Abstract
Background: Dialog agents (chatbots) have a long history of application in health care, where they have been used for tasks such as supporting patient self-management and providing counseling. Their use is expected to grow with increasing demands on health systems and improving artificial intelligence (AI) capability. Approaches to the evaluation of health care chatbots, however, appear to be diverse and haphazard, resulting in a potential barrier to the advancement of the field. Objective: This study aims to identify the technical (nonclinical) metrics used by previous studies to evaluate health care chatbots. Methods: Studies were identified by searching 7 bibliographic databases (eg, MEDLINE and PsycINFO) in addition to conducting backward and forward reference list checking of the included studies and relevant reviews. The studies were independently selected by two reviewers who then extracted data from the included studies. Extracted data were synthesized narratively by grouping the identified metrics into categories based on the aspect of chatbots that the metrics evaluated. Results: Of the 1498 citations retrieved, 65 studies were included in this review. Chatbots were evaluated using 27 technical metrics, which were related to chatbots as a whole (eg, usability, classifier performance, speed), response generation (eg, comprehensibility, realism, repetitiveness), response understanding (eg, chatbot understanding as assessed by users, word error rate, concept error rate), and esthetics (eg, appearance of the virtual agent, background color, and content). Conclusions: The technical metrics of health chatbot studies were diverse, with survey designs and global usability metrics dominating. The lack of standardization and paucity of objective measures make it difficult to compare the performance of health chatbots and could inhibit advancement of the field. We suggest that researchers more frequently include metrics computed from conversation logs. In addition, we recommend the development of a framework of technical metrics with recommendations for specific circumstances for their inclusion in chatbot studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“However, a recent scoping review of health care chatbots showed that the use of JITAIs in designing and evaluating chatbots in health care in general and promoting physical activity and a healthy diet in particular is sparse, suggesting that future research needs to consider using more of these adaptive approaches [89].”
From Artificial Intelligence Chatbot Behavior Change Model for Designing Artificial Intelligence Chatbots to Promote Physical Activity and a Healthy Diet: Viewpoint (2020) · cites this paper as [89]
Experimental — accuracy not yet measuredIn the citing paper
“From the emergence of the first chatbot, ELIZA, developed by Joseph Weizenbaum (1966), chatbots have been trying to ‘mimic human behaviour in a text-based conversation’ (Shum et al. 2018, p. 10; Abd-Alrazaq et al. 2020).”
From Chatbot breakthrough in the 2020s? An ethical reflection on the trend of automated consultations in health care (2022) · cites this paper as Abd-Alrazaq et al. 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The PHQ-9: Validity of a brief depression severity measure
Kroenke, K., Spitzer, R. L., & Williams, J. B. W. (2001) · Journal of General Internal Medicine · doi:10.1046/j.1525-1497.2001.016009606.x · Cited by 44,910
PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation
Tricco, A. C., Lillie, E., Zarin, W., O'Brien, K. K., Colquhoun, H., Levac, D., Moher, D., Peters, M. D. J., Horsley, T., Weeks, L., et al. (2018) · Annals of Internal Medicine · doi:10.7326/m18-0850 · Cited by 43,565
Practical Statistics for Medical Research
Altman, D. G. (1990) · doi:10.1201/9780429258589 · Cited by 10,668
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697
An Empirical Evaluation of the System Usability Scale
Bangor, A., Kortum, P. T., & Miller, J. T. (2008) · International Journal of Human–Computer Interaction · doi:10.1080/10447310802205776 · Cited by 5,181 · cited 2×
“The system usability scale (SUS [82,83]) is a well-established usability instrument that we observed was used repeatedly, although it was not used in the majority of the studies assessing usability; in many cases, a single survey question was used instead.”
BackgroundMethodELIZA—a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1966) · Communications of the ACM · doi:10.1145/365153.365168 · Cited by 4,452
Usability Evaluation In Industry
(1996) · CRC Press eBooks · doi:10.1201/9781498710411 · Cited by 4,354
Fitzpatrick, K. K., Darcy, A., & Vierhile, M. (2017) · JMIR Mental Health · doi:10.2196/mental.7785 · Cited by 2,730
Webb, T. L., Joseph, J., Yardley, L., & Michie, S. (2010) · Journal of Medical Internet Research · doi:10.2196/jmir.1376 · Cited by 2,517 · cited 1×
“At the same time, internet/web-based interventions have shown the ability to promote positive health behavior change [4,5], and the interaction components associated with users sticking with an internet intervention are increasingly well understood and include the inclusion of dialog elements [6].”
BackgroundConversational agents in healthcare: a systematic review
Laranjo, L., Dunn, A. G., Tong, H. L., Kocaballi, A. B., Chen, J., Bashir, R., Surian, D., Gallego, B., Magrabi, F., Lau, A. Y. S., et al. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy072 · Cited by 1,388 · cited 2×
“Laranjo et al [86] reviewed the characteristics, current applications and evaluation measures of health chatbots.”
BackgroundResultPersuasive System Design Does Matter: a Systematic Review of Adherence to Web-based Interventions
Kelders, S. M., Kok, R. N., Ossebaard, H. C., & Van Gemert-Pijnen, J. E. (2012) · Journal of Medical Internet Research · doi:10.2196/jmir.2104 · Cited by 1,308 · cited 1×
“At the same time, internet/web-based interventions have shown the ability to promote positive health behavior change [4,5], and the interaction components associated with users sticking with an internet intervention are increasingly well understood and include the inclusion of dialog elements [6].”
BackgroundMobile phone-based interventions for smoking cessation
Whittaker, R., McRobbie, H., Bullen, C., Rodgers, A., & Gu, Y. (2016) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006611.pub4 · Cited by 1,079 · cited 1×
“As mobile phones have become commonplace, a range of SMS text messaging–based interventions have been developed and trialed, with particular success in smoking cessation [3].”
BackgroundInkster, B., Sarda, S., & Subramanian, V. (2018) · doi:10.2196/preprints.12106 · Cited by 1,063
Chatbots and Conversational Agents in Mental Health: A Review of the Psychiatric Landscape
Vaidyam, A. N., Wisniewski, H., Halamka, J. D., Kashavan, M. S., & Torous, J. B. (2019) · The Canadian Journal of Psychiatry · doi:10.1177/0706743719828977 · Cited by 1,048
The design and implementation of Xiaoice, an empathetic social chatbot
Zhou, L., Gao, J., Li, D., & Shum, H.-Y. (2020) · Computational Linguistics · doi:10.1162/coli_a_00368 · Cited by 777
The Design and Implementation of XiaoIce, an Empathetic Social Chatbot
Zhou, L., Gao, J., Li, D., & Shum, H.-Y. (2018) · arXiv · doi:10.48550/arxiv.1812.08989 · Cited by 777
From Eliza to XiaoIce: Challenges and opportunities with social chatbots
Shum, H.-Y., He, X.-D., & Li, D. (2018) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.1700826 · Cited by 749 · cited 1×
“Conversational-turns per session (CPS) has been suggested as a success metric for social chatbots as exemplified by XiaoIce [85].”
MethodFulmer, R., Joerin, A., Gentile, B., Lakerink, L., & Rauws, M. (2018) · doi:10.2196/preprints.9782 · Cited by 715
An overview of the features of chatbots in mental health: A scoping review
Abd-alrazaq, A. A., Alajlani, M., Alalwan, A. A., Bewick, B. M., Gardner, P., & Househ, M. (2019) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2019.103978 · Cited by 605 · cited 1×
“The abovementioned restrictions were applied by previous reviews about chatbots as these features are not part of ordinary chatbots [87-90].”
BackgroundSelf-citationAbd-Alrazaq, A. A., Rababeh, A., Alajlani, M., Bewick, B. M., & Househ, M. (2020) · Journal of Medical Internet Research · doi:10.2196/16021 · Cited by 503
Self-citationNeve, M., Morgan, P. J., Jones, P. R., & Collins, C. E. (2010) · Obesity Reviews · doi:10.1111/j.1467-789x.2009.00646.x · Cited by 459 · cited 1×
“At the same time, internet/web-based interventions have shown the ability to promote positive health behavior change [4,5], and the interaction components associated with users sticking with an internet intervention are increasingly well understood and include the inclusion of dialog elements [6].”
BackgroundShould Machines Express Sympathy and Empathy? Experiments with a Health Advice Chatbot
Liu, B., & Sundar, S. S. (2018) · Cyberpsychology, Behavior, and Social Networking · doi:10.1089/cyber.2018.0110 · Cited by 447
Embodied conversational agents in clinical psychology: a scoping review
Provoost, S., Lau, H. M., Ruwaard, J., & Riper, H. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.6553 · Cited by 433
Virtual Reality Job Interview Training in Adults with Autism Spectrum Disorder
Smith, M. J., Ginger, E. J., Wright, K., Wright, M. A., Taylor, J. L., Humm, L. B., Olsen, D. E., Bell, M. D., & Fleming, M. F. (2014) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-014-2113-y · Cited by 401
Ly, K. H., Ly, A.-M., & Andersson, G. (2017) · Internet Interventions · doi:10.1016/j.invent.2017.10.002 · Cited by 368
Perceptions and opinions of patients about mental health chatbots: scoping review
Abd-Alrazaq, A. A., Alajlani, M., Ali, N., Denecke, K., Bewick, B. M., & Househ, M. (2021) · Journal of Medical Internet Research · doi:10.2196/17828 · Cited by 301
Self-citationTaking the time to care: empowering low health literacy hospital patients with virtual nurse agents
Bickmore, T. W., Pfeifer, L. M., & Jack, B. W. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518891 · Cited by 289
Clinical Implications and Challenges of Artificial Intelligence and Deep Learning
Stead, W. W. (2018) · JAMA · doi:10.1001/jama.2018.11029 · Cited by 250
Bickmore, T. W., Pfeifer, L. M., Byron, D., Forsythe, S., Henault, L. E., Jack, B. W., Silliman, R., & Paasche-Orlow, M. K. (2010) · Journal of Health Communication · doi:10.1080/10810730.2010.499991 · Cited by 249
I Can Help You Change! An Empathic Virtual Agent Delivers Behavior Change Health Interventions
Lisetti, C., Amini, R., Yasavur, U., & Rishe, N. (2013) · ACM Transactions on Management Information Systems · doi:10.1145/2544103 · Cited by 242
Response to a relational agent by hospital patients with depressive symptoms
Bickmore, T. W., Mitchell, S. E., Jack, B. W., Paasche-Orlow, M. K., Pfeifer, L. M., & O’Donnell, J. (2010) · Interacting with Computers · doi:10.1016/j.intcom.2009.12.001 · Cited by 218
Maintaining reality: Relational agents for antipsychotic medication adherence
Bickmore, T. W., Puskar, K., Schlenk, E. A., Pfeifer, L. M., & Sereika, S. M. (2010) · Interacting with Computers · doi:10.1016/j.intcom.2010.02.001 · Cited by 213
Automated interventions for multiple health behaviors using conversational agents
Bickmore, T. W., Schulman, D., & Sidner, C. (2013) · Patient Education and Counseling · doi:10.1016/j.pec.2013.05.011 · Cited by 206
Holmes, S., Moorhead, A., Bond, R., Zheng, H., Coates, V., & Mctear, M. (2019) · Proceedings of the 31st European Conference on Cognitive Ergonomics · doi:10.1145/3335082.3335094 · Cited by 206
Crutzen, R., Peters, G.-J. Y., Portugal, S. D., Fisser, E. M., & Grolleman, J. J. (2011) · Journal of Adolescent Health · doi:10.1016/j.jadohealth.2010.09.002 · Cited by 205
Pocket Skills: A Conversational Mobile Web App To Support Dialectical Behavioral Therapy
Schroeder, J., Wilkes, C., Rowan, K., Toledo, A., Paradiso, A., Czerwinski, M., Mark, G., & Linehan, M. M. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173972 · Cited by 175
Patient assessment of chatbots for the scalable delivery of genetic counseling
Schmidlen, T., Schwartz, M., DiLoreto, K., Kirchner, H. L., & Sturm, A. C. (2019) · Journal of Genetic Counseling · doi:10.1002/jgc4.1169 · Cited by 168
MANDY: Towards a Smart Primary Care Chatbot Application
Ni, L., Lu, C., Liu, N., & Liu, J. (2017) · Communications in computer and information science · doi:10.1007/978-981-10-6989-5_4 · Cited by 161
Pharmabot: A Pediatric Generic Medicine Consultant Chatbot
Comendador, B. E. V., Francisco, B. M. B., Medenilla, J. S., Nacion, S. M. T., & Serac, T. B. E. (2015) · Journal of Automation and Control Engineering · doi:10.12720/joace.3.2.137-140 · Cited by 150
Automated Telephone Conversations to Assess Health Behavior and Deliver Behavioral Interventions
Friedman, R. H. (1998) · Journal of Medical Systems · doi:10.1023/a:1022695119046 · Cited by 147
Acceptance and usability of a relational agent interface by urban older adults
Bickmore, T. W., Caruso, L., & Clough-Gorr, K. (2005) · CHI '05 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/1056808.1056879 · Cited by 142
Gardiner, P. M., McCue, K. D., Negash, L. M., Cheng, T., White, L. F., Yinusa-Nyahkoon, L., Jack, B. W., & Bickmore, T. W. (2017) · Patient Education and Counseling · doi:10.1016/j.pec.2017.04.015 · Cited by 139 · cited 1×
“%) studies evaluated clarity or quality of speech using either interviews [51,62,75] or a single question in a self-administrated questionnaire [27,69,77].”
A design and evaluation framework for digital health interventions
Kowatsch, T., Otto, L., Harperink, S., Cotti, A., & Schlieter, H. (2019) · it - Information Technology · doi:10.1515/itit-2019-0019 · Cited by 130
Philip, P., Micoulaud-Franchi, J.-A., Sagaspe, P., Sevin, E. D., Olive, J., Bioulac, S., & Sauteraud, A. (2017) · Scientific Reports · doi:10.1038/srep42656 · Cited by 128
Detecting Dementia Through Interactive Computer Avatars
Tanaka, H., Adachi, H., Ukita, N., Ikeda, M., Kazui, H., Kudo, T., & Nakamura, S. (2017) · IEEE Journal of Translational Engineering in Health and Medicine · doi:10.1109/jtehm.2017.2752152 · Cited by 127
Burton, C., Szentagotai Tatar, A., McKinstry, B., Matheson, C., Matu, S., Moldovan, R., Macnab, M., Farrow, E., David, D., Pagliari, C., et al. (2016) · Journal of Telemedicine and Telecare · doi:10.1177/1357633x15609793 · Cited by 125
kBot: Knowledge-Enabled Personalized Chatbot for Asthma Self-Management
Kadariya, D., Venkataramanan, R., Yip, H. Y., Kalra, M., Thirunarayanan, K., & Sheth, A. (2019) · 2019 IEEE International Conference on Smart Computing (SMARTCOMP) · doi:10.1109/smartcomp.2019.00043 · Cited by 102
Conlin, P. R., Colburn, J. A., Aron, D. C., Pries, R. M., Tschanz, M. P., & POGACH, L. M. (2017) · Annals of Internal Medicine · doi:10.7326/m17-1362 · Cited by 101
Tielman, M. L., Neerincx, M. A., Bidarra, R., Kybartas, B., & Brinkman, W.-P. (2017) · Journal of Medical Systems · doi:10.1007/s10916-017-0771-y · Cited by 94 · cited 2×
“Comprehensibility of responses was evaluated using a single question in a s e l f - a d m i n i s t r a t e d q u e s t i o n n a i r e [20,23,31,36,39,42,44,52,59,60,63,69] and interviews [34,51].”
MethodAssessing the Usability of a Chatbot for Mental Health Care
Cameron, G., Cameron, D., Megaw, G., Bond, R., Mulvenna, M. D., O’Neill, S. M., Armour, C., & McTear, M. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-17705-8_11 · Cited by 92
Virtual Reality Job Interview Training for Individuals With Psychiatric Disabilities
Smith, M. J., Ginger, E. J., Wright, M., Wright, K., Humm, L. B., Olsen, D. E., Bell, M. D., & Fleming, M. F. (2014) · The Journal of Nervous and Mental Disease · doi:10.1097/nmd.0000000000000187 · Cited by 83
Martínez-Miranda, J., Martínez, A., Ramos, R., Aguilar, H., Jiménez, L., Arias, H., Rosales, G., & Valencia, E. (2019) · Journal of Medical Systems · doi:10.1007/s10916-019-1387-1 · Cited by 79
Oxygenation of frog gastric mucosa in vitro
Kidder, G. W., & Montgomery, C. (1975) · American Journal of Physiology-Legacy Content · doi:10.1152/ajplegacy.1975.229.6.1510 · Cited by 75
Kocaballi, A. B., Quiroz, J. C., Rezazadegan, D., Berkovsky, S., Magrabi, F., Coiera, E., & Laranjo, L. (2020) · Journal of Medical Internet Research · doi:10.2196/15823 · Cited by 73
Virtual reality job interview training for veterans with posttraumatic stress disorder
Smith, M. J., Humm, L. B., Fleming, M. F., Jordan, N., Wright, M. A., Ginger, E. J., Wright, K., Olsen, D. E., & Bell, M. D. (2015) · Journal of Vocational Rehabilitation · doi:10.3233/jvr-150748 · Cited by 71
A Conversational Agent for an Online Mental Health Intervention
Elmasri, D., & Maeder, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-47103-7_24 · Cited by 68 · cited 8×
“%) examined the appropriateness and adequacy of verbal [18,19,27,28,31,34,38,39,51,58,66-69] and nonverbal responses of chatbots [32].”
BackgroundMethodInfluentialDevelopment and validation of a virtual agent to screen tobacco and alcohol use disorders
Auriacombe, M., Moriceau, S., Serre, F., Denis, C., Micoulaud‐Franchi, J., de Sevin, É., Bonhomme, É., Bioulac, S., Fatseas, M., & Philip, P. (2018) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2018.08.025 · Cited by 64
Tielman, M. L., Neerincx, M. A., van Meggelen, M., Franken, I., & Brinkman, W.-P. (2018) · Technology and Health Care · doi:10.3233/thc-170899 · Cited by 63
Swartout, W., Artstein, R., Forbell, E., Foutz, S., Lane, H. C., Lange, B., Morie, J., Noren, D., Rizzo, S., & Traum, D. (2013) · AI Magazine · doi:10.1609/aimag.v34i4.2487 · Cited by 61
Quro: Facilitating User Symptom Check Using a Personalised Chatbot-Oriented Dialogue System
Ghosh, S., Bhatia, S., & Bhatia, A. (2018) · Studies in health technology and informatics · doi:10.3233/978-1-61499-890-7-51 · Cited by 60
Development of a virtual agent based social tutor for children with autism spectrum disorders
Milne, M., Luerssen, M. H., Lewis, T. W., Leibbrandt, R. E., & Powers, D. M. W. (2010) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2010.5596584 · Cited by 58
Dworkin, M. S., Lee, S., Chakraborty, A., Monahan, C., Hightow‐Weidman, L., Garofalo, R., Qato, D. M., Liu, L., & Jimenez, A. D. (2019) · AIDS Education and Prevention · doi:10.1521/aeap.2019.31.1.17 · Cited by 58
The potential of conversational agents to provide a rapid HIV counseling and testing services
van Heerden, A., Ntinga, X., & Vilakazi, K. (2017) · 2017 International Conference on the Frontiers and Advances in Data Science (FADS) · doi:10.1109/fads.2017.8253198 · Cited by 57
Yokotani, K., Takagi, G., & Wakashima, K. (2018) · Computers in Human Behavior · doi:10.1016/j.chb.2018.03.045 · Cited by 56
Coerced Change-talk with Conversational Agents Promotes Confidence in Behavior Change
Ólafsson, S., O’Leary, T. K., & Bickmore, T. (2019) · International Conference on Pervasive Computing Technologies for Healthcare · doi:10.1145/3329189.3329202 · Cited by 55
Automated Social Skills Trainer
Tanaka, H., Sakti, S., Neubig, G., Toda, T., Negoro, H., Iwasaka, H., & Nakamura, S. (2015) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/2678025.2701368 · Cited by 54
Multimodal and mobile conversational Health and Fitness Companions
Turunen, M., Hakulinen, J., Ståhl, O., Gambäck, B., Hansen, P., Gancedo, M. C. R., de la Cámara, R. S., Smith, C., Charlton, D., & Cavazza, M. (2010) · Computer Speech & Language · doi:10.1016/j.csl.2010.04.004 · Cited by 47
Self-Anamnesis with a Conversational User Interface: Concept and Usability Study
Hochreutener, S., Pöpel, A., May, R., & Denecke, K. (2018) · Methods of Information in Medicine · doi:10.1055/s-0038-1675822 · Cited by 47
Self-citationPinto, M. D., Greenblatt, A. M., Hickman, R. L., Rice, H. M., Thomas, T. L., & Clochesy, J. M. (2016) · Perspectives in Psychiatric Care · doi:10.1111/ppc.12112 · Cited by 46
The Atrial Fibrillation Health Literacy Information Technology System: Pilot Assessment
Magnani, J. W., Schlusser, C. L., Kimani, E., Rollman, B. L., Paasche-Orlow, M. K., & Bickmore, T. W. (2017) · JMIR Cardio · doi:10.2196/cardio.8543 · Cited by 46
Abdullah, A. S. M., Gaehde, S. A., & Bickmore, T. (2018) · Journal of Epidemiology and Global Health · doi:10.2991/j.jegh.2018.08.104 · Cited by 46
OlloBot - Towards A Text-Based Arabic Health Conversational Agent: Evaluation and Results
Fadhil, A., AbuRa’ed, A., & Information & Communication Technologies, U. P. F. B. (2019) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-056-4_034 · Cited by 38
Usability and acceptability assessment of an empathic virtual agent to prevent major depression
Bresó, A., Martínez‐Miranda, J., Botella, C., Baños, R. M., & García‐Gómez, J. M. (2016) · Expert Systems · doi:10.1111/exsy.12151 · Cited by 36
The LISSA Virtual Human and ASD Teens: An Overview of Initial Experiments
Razavi, S. Z., Ali, M. R., Smith, T. H., Schubert, L. K., & Hoque, M. E. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-47665-0_55 · Cited by 34
Mobile Conversational Agents for Context-Aware Care Applications
Griol, D., & Callejas, Z. (2016) · Cognitive Computation · doi:10.1007/s12559-015-9352-x · Cited by 33
Let’s talk! speaking virtual counselor offers you a brief intervention
Yasavur, U., Lisetti, C., & Rishe, N. (2014) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-014-0169-9 · Cited by 32
Wargnier, P., Benveniste, S., Jouvelot, P., & Rigaud, A.-S. (2018) · Technology and Disability · doi:10.3233/tad-180189 · Cited by 32 · cited 8×
“%) examined the appropriateness and adequacy of verbal [18,19,27,28,31,34,38,39,51,58,66-69] and nonverbal responses of chatbots [32].”
BackgroundMethodInfluentialAcceptability of Embodied Conversational Agent in a Health Care Context
Micoulaud‐Franchi, J., Sagaspe, P., de Sevin, É., Bioulac, S., Sauteraud, A., & Philip, P. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-47665-0_45 · Cited by 28
Could a virtual human be used to explore excessive daytime sleepiness in patients?
Philip, P., Bioulac, S., Sauteraud, A., Chaufton, C., & Olive, J. (2014) · Presence: Teleoperators and Virtual Environments · doi:10.1162/pres_a_00197 · Cited by 25
Thompson, D. I., Callender, C. S., Gonynor, C., Cullen, K. W., REDONDO, M. J., Butler, A. M., & Anderson, B. J. (2019) · Journal of Medical Internet Research · doi:10.2196/15318 · Cited by 25
Kang, J., & Wei, L. (2018) · International Conference on Human-Agent Interaction (HAI) · doi:10.1145/3284432.3284462 · Cited by 20
How to Evaluate Health Applications with Conversational User Interface?
Denecke, K., & Jim, W. (2020) · Studies in health technology and informatics · doi:10.3233/shti200307 · Cited by 20
Self-citationLook on the Bright Side: A Model of Cognitive Change in Virtual Agents
Martínez-Miranda, J., Bresó, A., & García-Gómez, J. M. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-09767-1_37 · Cited by 18
Improving and Evaluating eMMA's Communication Skills: A Chatbot for Managing Medication
Hess, G., Guillaume, F., & Denecke, K. (2019) · PubMed · doi:10.24451/arbor.9011 · Cited by 13
Self-citationDesign of interactive cancer education technology for Latina farmworkers
Wu, Y., Samant, D., Squibbs, K., Chaet, A., Morshedi, B., & Barnes, L. E. (2014) · Systems and Information Engineering Design Symposium · doi:10.1109/sieds.2014.6829908 · Cited by 12
Daily life support at home through a virtual support partner
Hanke, S., Sandner, E., Kadyrov, S., & Stainer-Hochgatterer, A. (2016) · 2nd IET International Conference on Technologies for Active and Assisted Living (TechAAL 2016) · doi:10.1049/ic.2016.0058 · Cited by 11
DEMİRCİ, H. M. (2018) · OpenMETU (Middle East Technical University) · Cited by 8
Mental Health And Wellness Chatbot
Beiley, M. R. (2019) · UA Campus Repository (The University of Arizona) · Cited by 4
Kocaballi, A. B., Quiroz, J. C., Rezazadegan, D., et al. (2019) · doi:10.2196/preprints.15823 · Cited by 3
Research works printed but not linked: 6
- Denecke, K, 2020, Studies in Health Technology and Informatics, 1
- Ali, M, 2018, arXiv, 1
- Hess, G, 2019, Studies in health technology and informatics, 259, 101
- Ghosh, S, 2018, Stud Health Technol Inform, 252, 51
- Bickmore, T, 2003, Relational agents affecting change through human-computer relationships, 1
- Jadeja, M, 2017, arXiv preprint, 1
Cited by
A review of opportunities and challenges of chatbots in education
Hwang, G.-J., & Chang, C.-Y. (2023) · Interactive Learning Environments · doi:10.1080/10494820.2021.1952615 · Cited by 530
Zhang, J., Oh, Y. J., Lange, P., Yu, Z., & Fukuoka, Y. (2020) · Journal of Medical Internet Research · doi:10.2196/22845 · Cited by 363
Roles, Users, Benefits, and Limitations of Chatbots in Health Care: Rapid Review
Laymouna, M., Ma, Y., Lessard, D., Schuster, T., Engler, K., & Lebouché, B. (2024) · Journal of Medical Internet Research · doi:10.2196/56930 · Cited by 305
Roles, Users, Benefits, and Limitations of Chatbots in Health Care: Rapid Review (Preprint)
Laymouna, M., Ma, Y., Lessard, D., Schuster, T., Engler, K., & Lebouché, B. (2024) · doi:10.2196/preprints.56930 · Cited by 302
Perceptions and opinions of patients about mental health chatbots: scoping review
Abd-Alrazaq, A. A., Alajlani, M., Ali, N., Denecke, K., Bewick, B. M., & Househ, M. (2021) · Journal of Medical Internet Research · doi:10.2196/17828 · Cited by 301
“The first attempts toward evaluation frameworks for digital health interventions [82] and health chatbots [83,84] have been recently published.”
Self-citation
Metadata
- Authors
- Abd-Alrazaq, A., Safi, Z., Alajlani, M., Warren, J., Househ, M., & Denecke, K.
- Affiliations
- Hamad bin Khalifa University (QA)
- University of Warwick (GB)
- University of Auckland (NZ)
- Bern University of Applied Sciences (CH)
- Corresponding author
- Denecke, Kerstin
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2020-06-05
- Volume
- 22
- Issue
- 6
- Pages
- e18301
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC7305563
- OpenAlex
- W3021626742
- Semantic Scholar
- e4286342557d57067ecf35a4969c621d2484ef3a
- Cited by
- 150
- References
- 88
- Citations by year
- 2026: 25 · 2025: 32 · 2024: 22 · 2023: 27 · 2022: 21 · 2021: 18 · 2020: 4
- FWCI
- 1.34
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- ARBOR - Bern University of Applied Sciences Repository (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Warwick Research Archive Portal (University of Warwick) (submittedVersion)
- BFH: ARBOR
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Keywords
- chatbot evaluation, usability, response generation, word error rate, conversation logs, dialogue agents
- MeSH terms
- Artificial Intelligence, Communication, Delivery of Health Care, Humans
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
