Conversational agents in healthcare: a systematic review
Open accessThe DOI matches its Crossref record; 55 of the 60 references it cites were found in a registry or an index; 14 of the 36 sources its claims cite were checked for support.
Abstract
Objective: Our objective was to review the characteristics, current applications, and evaluation measures of conversational agents with unconstrained natural language input capabilities used for health-related purposes. Methods: We searched PubMed, Embase, CINAHL, PsycInfo, and ACM Digital using a predefined search strategy. Studies were included if they focused on consumers or healthcare professionals; involved a conversational agent using any unconstrained natural language input; and reported evaluation measures resulting from user interaction with the system. Studies were screened by independent reviewers and Cohen's kappa measured inter-coder agreement. Results: The database search retrieved 1513 citations; 17 articles (14 different conversational agents) met the inclusion criteria. Dialogue management strategies were mostly finite-state and frame-based (6 and 7 conversational agents, respectively); agent-based strategies were present in one type of system. Two studies were randomized controlled trials (RCTs), 1 was cross-sectional, and the remaining were quasi-experimental. Half of the conversational agents supported consumers with health tasks such as self-care. The only RCT evaluating the efficacy of a conversational agent found a significant effect in reducing depression symptoms (effect size d = 0.44, p = .04). Patient safety was rarely evaluated in the included studies. Conclusions: The use of conversational agents with unconstrained natural language input capabilities for health-related purposes is an emerging field of research, where the few published studies were mainly quasi-experimental, and rarely evaluated efficacy or safety. Future studies would benefit from more robust experimental designs and standardized reporting. Protocol Registration: The protocol for this systematic review is registered at PROSPERO with the number CRD42017065917.
Overview
CoChat Summary
TL;DR
This systematic review found that conversational agents with unconstrained natural language input were still rarely studied in healthcare: 17 articles covering 14 agents, mostly evaluated with quasi-experimental designs. Evidence of efficacy and safety was limited, with only one RCT showing reduced depression symptoms (effect size d = 0.44, p = .04).
Key findings
- The search retrieved 1513 citations; 17 articles evaluating 14 different conversational agents met inclusion criteria.
- Study designs were mainly weak for efficacy claims: 2 RCTs, 1 cross-sectional study, and the rest quasi-experimental. The paper notes that the RCTs had moderate to high risk of bias.
- Dialogue management was mostly finite-state (6 agents / 8 studies) or frame-based (7 agents / 8 studies); agent-based dialogue management appeared in 1 agent.
- The only RCT evaluating efficacy found a significant reduction in depression symptoms with effect size d = 0.44, p = .04; patient safety was rarely evaluated.
Methods
- Systematic review of studies of conversational agents using any unconstrained natural language input for health-related purposes; protocol registered at PROSPERO CRD42017065917 and the review followed PRISMA.
- Searched PubMed, Embase, CINAHL, PsycInfo, and ACM Digital Library in April 2017, updated in February 2018; included primary studies of consumers, caregivers, or healthcare professionals that tested the full system with human users and reported evaluation measures.
- Screening was done by independent reviewers with Cohen’s kappa for agreement; because of heterogeneity, the authors conducted a narrative synthesis rather than a meta-analysis.
Limitations
- The paper states it focused only on agents with unconstrained natural language input, excluding constrained-input systems that are also relevant to healthcare.
- The paper states that the small number of included studies, heterogeneity of agents, and predominance of quasi-experimental pilot studies limited its ability to conduct a meta-analysis.
- Evident from the text: reporting was inconsistent, making eligibility assessment and synthesis difficult, and several studies lacked clear funding/conflict-of-interest reporting.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
10 Valid · 3 Not valid · 13 Not checked · 25 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“A crossover RCT evaluated the diagnostic performance for depression of an embodied conversational agent compared to a psychiatrist, finding a sensitivity of 49% and specificity of 93%.”
§ Conversational agents in healthcare: a systematic review > Results > Evaluation measures
What the sources say
- Supportedcites [22] Philip, 2017, Virtual human as a new diagnostic tool, a proof of concept study in the field of major depressive disorders, Sci Rep source
The Method (Recorded data in the study; page not given) says patients completed both interviews in randomized order, one with the ECA and one with the psychiatrist. The Results (Global performance of ECA diagnosis of MDD; page not given) report overall sensitivity of 49% and specificity of 93% against the psychiatrist’s diagnosis. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“We found that only one study evaluated the efficacy of a conversational agent on health outcomes using adequate study design methods (RCT) and validated questionnaires.”
§ Conversational agents in healthcare: a systematic review > DISCUSSION > Standards for reporting the evaluation of conversational agents in healthcare
What the sources say
- Supportedcites [19] Fitzpatrick, 2017, Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): a randomized controlled trial, JMIR Ment Health source
The Results report that participants were randomized to Woebot or an information-control group, and the Methods > Measures section identifies the PHQ-9 and GAD-7 as validated questionnaires. This confirms that the cited paper is an RCT evaluating a conversational agent with validated health-outcome measures. “Only one study” describes the citing authors’ review finding, not a claim made by this source. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Over the last two decades, a solid body of evidence has shown the potential benefits of using embodied conversational agents for health-related purposes.”
§ Conversational agents in healthcare: a systematic review > Introduction
What the sources say
- Not checked — no full text of the sourcecites [3] Bickmore, 2013, A randomized controlled trial of an automated exercise coach for older adults, J Am Geriatr Soc source
- Partly supportedcites [4] Bickmore, 2013, Automated interventions for multiple health behaviors using conversational agents, Patient Educ Couns sourceExperimental — accuracy not yet measured
The Results and Discussion report benefits from this conversational-agent intervention, particularly increased fruit and vegetable consumption compared with the control group. The Conclusion also describes the intervention as effective at changing health behavior. The paper does not establish the claim’s broader characterization of a solid body of evidence accumulated over two decades. (LLM, conf=0.86)
- Partly supportedcites [5] Watson, 2012, An internet-based virtual coach to promote physical activity adherence in overweight adults: randomized controlled trial, J Med Internet Res sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) describes prior research on embodied computer agents, and the Discussion conclusion reports that a virtual coach helped maintain activity in overweight participants. This supports potential health benefits, but the paper does not establish a solid body of evidence spanning two decades or benefits across health-related purposes generally. (LLM, conf=0.85)
- Not checked — no full text of the sourcecites [6] Edwards, 2013, Use of an interactive computer agent to support breastfeeding, Matern Child Health J source
boostedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The only RCT evaluating the efficacy of a conversational agent found a significant effect in reducing depression symptoms (effect size d = 0.44, p = .04).”
§ Conversational agents in healthcare: a systematic review > Abstract > Results
What the sources say
- Partly supportedcites [19] Fitzpatrick, 2017, Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): a randomized controlled trial, JMIR Ment Health sourceExperimental — accuracy not yet measured
The Results (Preliminary Efficacy) report a significant reduction in depression symptoms for the Woebot group relative to the information control group, with d=0.44 and a Bonferroni-corrected P=.04. The Discussion (Principal Results) calls this, to the authors’ knowledge, the first randomized trial of a nonembodied text-based conversational agent designed for therapeutic use; it does not establish that it was the only RCT evaluating any conversational agent. (LLM, conf=0.86)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Several randomized controlled trials of interventions involving embodied conversational agents have shown significant improvements in physical activity, fruit and vegetable consumption, and accessibility to online health information, among other outcomes.”
§ Conversational agents in healthcare: a systematic review > Introduction
What the sources say
- Not checked — no full text of the sourcecites [3] Bickmore, 2013, A randomized controlled trial of an automated exercise coach for older adults, J Am Geriatr Soc source
- Partly supportedcites [4] Bickmore, 2013, Automated interventions for multiple health behaviors using conversational agents, Patient Educ Couns sourceExperimental — accuracy not yet measured
The Abstract (Methods) describes one randomized trial, not several. The Results (§3.1–3.3) report a significant improvement in fruit and vegetable consumption and faster growth in walking steps for the physical-activity group, but no significant difference in self-reported physical activity. Our copy does not report an outcome concerning accessibility to online health information. (LLM, conf=0.90)
- Partly supportedcites [5] Watson, 2012, An internet-based virtual coach to promote physical activity adherence in overweight adults: randomized controlled trial, J Med Internet Res sourceExperimental — accuracy not yet measured
Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
- Not checked — no full text of the sourcecites [6] Edwards, 2013, Use of an interactive computer agent to support breastfeeding, Matern Child Health J source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
17 more claims · 2 with their cited sources checked
Create a free account to see all 51 claims and every source verdict.
- Probably not a substantial claim. 12
- It may state two separate claims as one. 20
- Borderline: one of the tests for a substantial claim is unclear. 1
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Previously conducted reviews [5,18,19] did not assess the effectiveness and safety of chatbots in mental health.”
From Effectiveness and safety of using chatbots to improve mental health: systematic review and meta-analysis (2020) · cites this paper as [18]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Most studies reviewed in 2018 [14] were limited to task-oriented conversational agents to support patients and clinicians in highly specific scenarios (e.g., information retrieval and predefined clinical interview), restricting user input to predetermined utterances.”
From Conversational Affective Social Robots for Ageing and Dementia Support (2022) · cites this paper as [14]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“On the same note, conversational agents have been proven a valuable tool to deliver digital health interventions [7-9].”
From Voice-Based Conversational Agents for the Prevention and Management of Chronic and Mental Health Conditions: Systematic Literature Review (2021) · cites this paper as [7]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These databases were chosen as they cover relevant aspects in the fields of medicine, technology, and interdisciplinary research and have also been used in other systematic reviews covering similar topics [7,8].”
From Voice-Based Conversational Agents for the Prevention and Management of Chronic and Mental Health Conditions: Systematic Literature Review (2021) · cites this paper as [7]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Existing reviews on conversational agents focus on a certain subtype of agents such as virtual coaches [31-33] or embodied conversational agents (ECAs) [34] or on specific functionalities of these agents such as behavior change [35] or mental health applications [36,37].”
From Conversational Agents in Health Care: Scoping Review and Conceptual Analysis (2020) · cites this paper as [32]
Experimental — accuracy not yet measured
62 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164 · cited 1×
“This systematic review is compliant with the PRISMA statement.(18)”
MethodCochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
Hoffmann, T. C., Glasziou, P. P., Boutron, I., Milne, R., Perera, R., Moher, D., Altman, D. G., Barbour, V., Macdonald, H., Johnston, M., et al. (2014) · BMJ · doi:10.1136/bmj.g1687 · Cited by 10,548 · cited 1×
“49,50 Implications for healthcare, public health, and future research Patient safety was rarely evaluated in the included studies.”
BackgroundEvaluating the quality of medical care
DONABEDIAN, A. (2005) · The Milbank Quarterly · doi:10.1111/j.1468-0009.2005.00397.x · Cited by 6,102 · cited 2×
“…of a conversational system from a health research perspective was considered to involve any health-related results present in the included studies, including process and outcome measures as defined by Donabedian; 16 for example, effectiveness in symptom reduction, diagnostic accuracy, or referrals.”
BackgroundELIZA—a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1966) · Communications of the ACM · doi:10.1145/365153.365168 · Cited by 4,452 · cited 1×
“The first well-established chatbot of this kind—ELIZA—was programmed in 1966 to simulate a text-based conversation with a psychotherapist.(2)”
MethodA model for types and levels of human interaction with automation
Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2000) · IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans · doi:10.1109/3468.844354 · Cited by 4,195 · cited 1×
“10,56,57 Therefore, the use of conversational agents with unconstrained natural language input capabilities and other artificial intelligence applications in healthcare needs to be carefully monitored.”
BackgroundSTARD 2015: an updated list of essential items for reporting diagnostic accuracy studies
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2015) · BMJ · doi:10.1136/bmj.h5527 · Cited by 3,660 · cited 1×
“…telehealth (CONSORT-EHEALTH), 46 the Transparent Reporting of Evaluations with Nonrandomized Designs (TREND) statement, 47 and the Standards for Reporting Diagnostic accuracy studies (STARD), 48 among others from the Enhancing the Quality and Transparency Of health Research (EQUATOR) network.”
MethodDes Jarlais, D. C., Lyles, C., Crepaz, N., & the TREND Group (2004) · American Journal of Public Health · doi:10.2105/ajph.94.3.361 · Cited by 2,817
ELIZA — a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1983) · Communications of the ACM · doi:10.1145/357980.357991 · Cited by 2,738
Fitzpatrick, K. K., Darcy, A., & Vierhile, M. (2017) · JMIR Mental Health · doi:10.2196/mental.7785 · Cited by 2,730 · cited 7×
“The only RCT evaluating the efficacy of a conversational agent found a significant effect in reducing depression symptoms.(19) Two studies comparing diagnostic performance of conversational agents and clinicians found acceptable sensitivity and specificity.”
Influential
Research works printed but not linked: 2
- Radziwill, 2017, arXiv Prepr, 1704
- Coiera, 2017, First compute no harm, BMJ Opin
Cited by
Big data and machine learning algorithms for health-care delivery
Ngiam, K. Y., & Khor, I. W. (2019) · The Lancet Oncology · doi:10.1016/s1470-2045(19)30149-4 · Cited by 1,424
Torous, J., Bucci, S., Bell, I. H., Kessing, L. V., Faurholt‐Jepsen, M., Whelan, P., Carvalho, A. F., Keshavan, M., Linardon, J., & Firth, J. (2021) · World Psychiatry · doi:10.1002/wps.20883 · Cited by 1,199
“However, their evidence base is not well established 85 .”
BackgroundChatbots and Conversational Agents in Mental Health: A Review of the Psychiatric Landscape
Vaidyam, A. N., Wisniewski, H., Halamka, J. D., Kashavan, M. S., & Torous, J. B. (2019) · The Canadian Journal of Psychiatry · doi:10.1177/0706743719828977 · Cited by 1,048
Artificial Intelligence-Based Chatbots for Promoting Health Behavioral Changes: Systematic Review.
Aggarwal, A., Tam, C. C., Wu, D., Li, X., & Qiao, S. (2023) · Journal of medical Internet research · doi:10.2196/40789 · Cited by 683
“Artificial intelligence (AI)–driven chatbots (AI chatbots) are conversational agents that mimic human interaction through written, oral, and visual forms of communication with a user [1,2].”
InfluentialRapp, A., Curti, L., & Boldi, A. (2021) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2021.102630 · Cited by 643
An overview of the features of chatbots in mental health: A scoping review
Abd-alrazaq, A. A., Alajlani, M., Alalwan, A. A., Bewick, B. M., Gardner, P., & Househ, M. (2019) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2019.103978 · Cited by 605
“In addition that such guidelines improve reporting of studies, they enable readers to assess the quality of studies, combine their results, and compare between interventions [29].”
BackgroundMethodInfluentialConversational Agents in Health Care: Scoping Review and Conceptual Analysis
Tudor Car, L., Dhinagaran, D. A., Kyaw, B. M., Kowatsch, T., Joty, S., Theng, Y.-L., & Atun, R. (2020) · Journal of Medical Internet Research · doi:10.2196/17158 · Cited by 553
“A recent systematic review on the effectiveness of ECAs and other conversational agents noted a lack of an established method for evaluating health care conversational agents in health care and a dearth of data on adverse effects [32].”
BackgroundThe Effectiveness of Artificial Intelligence Conversational Agents in Health Care: Systematic Review
Milne-Ives, M., de Cock, C., Lim, E., Shehadeh, M. H., de Pennington, N., Mole, G., Normando, E., & Meinert, E. (2020) · Journal of Medical Internet Research · doi:10.2196/20346 · Cited by 626
“They also found a limited quality of design and evidence in the included studies, with inconsistent reporting of study methods (including methods of selection, attrition, and a lack of validated outcome measures) and conflicts of interest [2].”
InfluentialXu, L., Sanders, L., Li, K., & Chow, J. C. L. (2021) · JMIR Cancer · doi:10.2196/27850 · Cited by 571
“With their ability for complex dialog management and conversational flexibility, integration of chatbot technology into clinical practice may reduce costs, refine workflow efficiencies, and improve patient outcomes [5].”
Chaves, A. P., & Gerosa, M. A. (2021) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2020.1841438 · Cited by 572
“elements (such as quick replies) compared to typing the responses, Lasek and Jessa (2013) observed that users that start the interaction by clicking an option are more likely to continue the conversation if the next exchange also has visual elements as optional affordances.”
ResultArtificial intelligence in healthcare: An essential guide for health leaders
Chen, M., & Decary, M. (2020) · Healthcare Management Forum · doi:10.1177/0840470419873123 · Cited by 538
“However, the reliability of such text-based chatbots is often achieved at the cost of restricting user input to predetermined words and phrases, not allowing users to take the initiative in the dialogue.(6) With the high expectation of investors and the efforts from both big tech companies and start-ups, it is certain…”
BackgroundAbd-Alrazaq, A. A., Rababeh, A., Alajlani, M., Bewick, B. M., & Househ, M. (2020) · Journal of Medical Internet Research · doi:10.2196/16021 · Cited by 503
“Previously conducted reviews [5,18,19] did not assess the effectiveness and safety of chatbots in mental health.”
The Internet of Things: Impact and Implications for Health Care Delivery
Kelly, J. T., Campbell, K. L., Gong, E., & Scuffham, P. (2020) · Journal of Medical Internet Research · doi:10.2196/20135 · Cited by 469
“A summary of IoT devices that may support and improve health service delivery is provided in Multimedia Appendix 1 [14-47].”
BackgroundAI-Based Digital Assistants: Opportunities, Threats, and Research Perspectives
Maedche, A., Legner, C., Benlian, A., Berger, B., Gimpel, H., Hess, T., Hinz, O., Morana, S., & Söllner, M. (2019) · Business & Information Systems Engineering · doi:10.1007/s12599-019-00600-8 · Cited by 432
Fitzpatrick, P. J. (2023) · Frontiers in Digital Health · doi:10.3389/fdgth.2023.1264780 · Cited by 416
“Additionally, conversational agents in healthcare have shown promise in delivering tailored education and facilitating patient engagement (39).”
BackgroundInfluentialD’Alfonso, S. (2020) · Current Opinion in Psychology · doi:10.1016/j.copsyc.2020.04.005 · Cited by 408
“The underlying system can be based on a variety of foundations, ranging from a set of simple rule-based responses and keyword matching to sophisticated NLP techniques [35,36].”
BackgroundDigital transformation in healthcare – architectures of present and future information technologies
Gopal, G., Suter-Crazzolara, C., Toldo, L., & Eberhardt, W. (2019) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2018-0658 · Cited by 369
Zhang, J., Oh, Y. J., Lange, P., Yu, Z., & Fukuoka, Y. (2020) · Journal of Medical Internet Research · doi:10.2196/22845 · Cited by 363
“AI chatbots have been rapidly transforming multiple fields, including business [17], governance [18], education [19], and health care [16,20].”
BackgroundAI-generated characters for supporting personalized learning and well-being
Pataranutaporn, P., Danry, V., Leong, J., Punpongsanon, P., Novy, D., Maes, P., & Sra, M. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00417-9 · Cited by 358
“…Siri, Google Assistant or Amazon Alexa, which respond to questions and instructions; and embodied conversational agents, which involve a virtual character (for example, avatar, virtual agent or photorealistic character) simulating face-to-face conversation with verbal and nonverbal behaviours 42 .”
BackgroundThe Personalization of Conversational Agents in Health Care: Systematic Review
Kocaballi, A. B., Berkovsky, S., Quiroz, J. C., Laranjo, L., Tong, H. L., Rezazadegan, D., Briatore, A., & Coiera, E. (2019) · Journal of Medical Internet Research · doi:10.2196/15360 · Cited by 332
“A recent systematic review in this area examined technical performance, user experience, and health-related outcomes and found that most studies had not employed standardized evaluation methods or had failed to address aspects of patient safety [2].”
BackgroundSelf-citationSystematic Review: Trust-Building Factors and Implications for Conversational Agent Design
Rheu, M., Shin, J. Y., Peng, W., & Huh-Yoo, J. (2021) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2020.1807710 · Cited by 360
Shahsavar, Y., & Choudhury, A. (2023) · JMIR Human Factors · doi:10.2196/47564 · Cited by 341
“In recent years, research has increasingly focused on developing and evaluating AI chatbots for various health care purposes [5,36].”
BackgroundUnderstanding, explaining, and utilizing medical artificial intelligence
Cadario, R., Longoni, C., & Morewedge, C. K. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01146-0 · Cited by 342
Mariani, M. M., Hashemi, N., & Wirtz, J. (2023) · Journal of Business Research · doi:10.1016/j.jbusres.2023.113838 · Cited by 343
Survey of conversational agents in health
Montenegro, J. L. Z., da Costa, C. A., & da Rosa Righi, R. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2019.03.054 · Cited by 322
Chatbots in the fight against the COVID-19 pandemic
Miner, A. S., Laranjo, L., & Kocaballi, A. B. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0280-0 · Cited by 322
Self-citationPerceptions and opinions of patients about mental health chatbots: scoping review
Abd-Alrazaq, A. A., Alajlani, M., Ali, N., Denecke, K., Bewick, B. M., & Househ, M. (2021) · Journal of Medical Internet Research · doi:10.2196/17828 · Cited by 301
“Currently, systems are rather restricted in the number of responses, which might be because of the early development stage of many chatbots, as has already been reported by Laranjo et al [71].”
InfluentialConversational agents in the treatment of mental health problems: mixed-method systematic review
Gaffney, H., Mansell, W., & Tai, S. (2019) · JMIR Mental Health · doi:10.2196/14166 · Cited by 275
“Evidence has begun to accumulate around the potential benefits of conversational agents in diverse fields [5] within health and medical care [6] and specifically in mental health [7-11].”
Roles, Users, Benefits, and Limitations of Chatbots in Health Care: Rapid Review (Preprint)
Laymouna, M., Ma, Y., Lessard, D., Schuster, T., Engler, K., & Lebouché, B. (2024) · doi:10.2196/preprints.56930 · Cited by 302
Alexa, what do we know about conversational commerce? Insights from a systematic literature review.
Lim, W. M., Kumar, S., Verma, S., & Chaturvedi, R. (2022) · Psychology & Marketing · doi:10.1002/mar.21654 · Cited by 293
Roles, Users, Benefits, and Limitations of Chatbots in Health Care: Rapid Review
Laymouna, M., Ma, Y., Lessard, D., Schuster, T., Engler, K., & Lebouché, B. (2024) · Journal of Medical Internet Research · doi:10.2196/56930 · Cited by 305
“…in language processing and a lack of empathic response, can lead to trust issues and increased clinical workload and align with past literature [3-5, 68, 72,73 , Our results indicate that chatbots serve a wide range of populations from various groups in terms of age, gender, ethnicity, and…”
Schachner, T., Keller, R., & v Wangenheim, F. (2020) · Journal of Medical Internet Research · doi:10.2196/20701 · Cited by 267
“Conversational agents or chatbots are computer systems that imitate natural conversation with human users through images and written or spoken language [1].”
BackgroundMethodResultInfluentialHuman-like communication in conversational agents: a literature review and research agenda
Van Pinxteren, M. M. E., Pluymaekers, M., & Lemmink, J. G. A. M. (2020) · Journal of Service Management · doi:10.1108/josm-06-2019-0175 · Cited by 280
“Conversational agents are “ systems that mimic human conversation ” using communication channels such as speech, text but also facial expressions and gestures (Laranjo et al. , 2018, p. 1,248; Radziwill and Benton, 2017).”
BackgroundOh, Y. J., Zhang, J., Fang, M.-L., & Fukuoka, Y. (2021) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-021-01224-6 · Cited by 278
Greer, S., Ramo, D., Chang, Y.-J., Fu, M., Moskowitz, J., & Haritatos, J. (2019) · JMIR mHealth and uHealth · doi:10.2196/15018 · Cited by 265
“A review of 14 chatbots in health care found that their use is still rare relative to other areas [34].”
Prochaska, J. J., Vogel, E. A., Chieng, A., Kendra, M., Baiocchi, M., Pajarito, S., & Robinson, A. (2021) · Journal of Medical Internet Research · doi:10.2196/24850 · Cited by 257
Beatty, C., Malik, T., Meheli, S., & Sinha, C. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.847991 · Cited by 224
“Conversational agents use machine learning and AI methods to simulate human-like behaviors and support (24, 25).”
MethodSocial companionship with artificial intelligence: Recent trends and future avenues
Chaturvedi, R., Verma, S., Das, R., & Dwivedi, Y. K. (2023) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2023.122634 · Cited by 244
Hauser-Ulrich, S., Künzli, H., Meier-Peterhans, D., & Kowatsch, T. (2020) · JMIR mHealth and uHealth · doi:10.2196/15806 · Cited by 210
“Another study focused on conversational agents with unconstrained natural language input and found it to be an emerging field of research, but included studies that rarely evaluated efficacy or safety and were mainly quasiexperimental [111].”
BackgroundA Mental Health Chatbot for Regulating Emotions (SERMO) - Concept and Usability Test
Denecke, K., Vaaheesan, S., & Arulnathan, A. (2021) · IEEE Transactions on Emerging Topics in Computing · doi:10.1109/tetc.2020.2974478 · Cited by 237
Haugeland, I. K. F., Følstad, A., Taylor, C., & Bjørkli, C. A. (2022) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102788 · Cited by 235
A critical review of state‐of‐the‐art chatbot designs and applications
Luo, B., Lau, R. Y. K., Li, C., & Si, Y. (2022) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1434 · Cited by 224
“Laranjo et al. (2018) also noted that the evaluation of patient safety was a rarity.”
BackgroundA Review on Innovation in Healthcare Sector (Telehealth) through Artificial Intelligence
Amjad, A., Kordel, P., & Fernandes, G. (2023) · Sustainability · doi:10.3390/su15086655 · Cited by 218
He, Y., Yang, L., Qian, C., Li, T., Su, Z., Zhang, Q., & Hou, X. (2023) · Journal of Medical Internet Research · doi:10.2196/43862 · Cited by 213
“Conversational agent interventions (CAIs) [12] were the new wave of digital mental health interventions to cope with the insufficient and inadequate mental health services [1,13].”
Bibault, J.-E., Chaix, B., Guillemassé, A., Cousin, S., Escande, A., Perrin, M., Pienkowski, A., Delamon, G., Nectoux, P., & Brouard, B. (2019) · Journal of Medical Internet Research · doi:10.2196/15787 · Cited by 202
Using Health Chatbots for Behavior Change: A Mapping Study
Pereira, J., & Díaz, Ó. (2019) · Journal of Medical Systems · doi:10.1007/s10916-019-1237-1 · Cited by 199
Factors influencing users' adoption and use of conversational agents: A systematic review
Ling, E. C., Tussyadiah, I., Tuomi, A., Stienmetz, J., & Ioannou, A. (2021) · Psychology & Marketing · doi:10.1002/mar.21491 · Cited by 199
Systematic review and meta-analysis of the effectiveness of chatbots on lifestyle behaviours
Singh, B., Olds, T., Brinsley, J., Dumuid, D., Virgara, R., Matricciani, L., Watson, A., Szeto, K., Eglitis, E., Miatke, A., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00856-1 · Cited by 224
Design Features of Embodied Conversational Agents in eHealth: a Literature Review
ter Stal, S., Kramer, L. L., Tabak, M., op den Akker, H., & Hermens, H. (2020) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2020.102409 · Cited by 188
“However, they either focus on conversational agents in general, not on ECAs specifically (Laranjo et al., 2018; Rist et al., 2004), are not up-to-date (Rist et al., 2004), focus on a broader context than health (Scholten et al., 2017) or focus on a subarea of health, such as clinical psychology…”
BackgroundConveRT: Efficient and Accurate Conversational Representations from Transformers
Henderson, M., Casanueva, I., Mrkšić, N., Su, P.-H., Wen, T.-H., & Vulić, I. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.196 · Cited by 176
“…flights (Hemphill et al., 1990; Williams, 2012; El Asri et al., 2017), with further use in tourist information (Budzianowski et al., 2018), language learning (Raux et al., 2003; Chen et al., 2017), entertainment (Fraser et al., 2018), and healthcare (Laranjo et al., 2018; Fadhil and Schiavo, 2019).”
BackgroundKuziemsky, C., Maeder, A. J., John, O., Gogia, S. B., Basu, A., Meher, S., & Ito, M. (2019) · Yearbook of Medical Informatics · doi:10.1055/s-0039-1677897 · Cited by 183
“The value of synchronous and task-oriented computer-generated dialogue has been observed for mental health applications [16] and a broader range of applications has been noted in recent years [17].”
BackgroundTemsah, M.-H., Aljamaan, F., Malki, K. H., Alhasan, K., Altamimi, I., Aljarbou, R., Bazuhair, F., Alsubaihin, A., Abdulmajeed, N., Alshahrani, F. S., et al. (2023) · Healthcare · doi:10.3390/healthcare11131812 · Cited by 179
“Introduction Artificial Intelligence (AI) is the ability of digital computers or computer-controlled robots to perform tasks typically associated with intelligent beings [1,2].”
BackgroundHe, Y., Yang, L., Zhu, X., Wu, B., Zhang, S., Qian, C., & Tian, T. (2022) · Journal of Medical Internet Research · doi:10.2196/40719 · Cited by 178
“In addition, chatbots were designed to focus on interactive capabilities instead of single psychological education for facilitating the process of psychotherapy [19].”
Artificial Intelligence for Chatbots in Mental Health: Opportunities and Challenges
Denecke, K., Abd-Alrazaq, A., & Househ, M. (2021) · Lecture Notes in Bioengineering · doi:10.1007/978-3-030-67303-1_10 · Cited by 177
Kurniawan, M. H., Handiyani, H., Nuraini, T., Hariyati, R. T. S., & Sutrisno, S. (2024) · Annals of Medicine · doi:10.1080/07853890.2024.2302980 · Cited by 177
Speech Technology for Healthcare: Opportunities, Challenges, and State of the Art
Latif, S., Qadir, J., Qayyum, A., Usama, M., & Younis, S. (2021) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2020.3006860 · Cited by 173
AI Chatbots in Digital Mental Health
Balcombe, L. (2023) · Informatics · doi:10.3390/informatics10040082 · Cited by 174
Key Considerations for Incorporating Conversational AI in Psychotherapy
Miner, A. S., Shah, N., Bullock, K. D., Arnow, B. A., Bailenson, J., & Hancock, J. (2019) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2019.00746 · Cited by 166
“Other areas of medicine have seen success with AI, such as lung cancer imaging and building diagnostic or prognostic models (69–73), and conversational AI for health is an emerging field with limited research on efficacy and safety (40, 63, 74).”
BackgroundConversational Agents: Goals, Technologies, Vision and Challenges
Allouch, M., Azaria, A., & Azoulay, R. (2021) · Sensors · doi:10.3390/s21248448 · Cited by 168
“There have been several recent reviews on CAs in this field (see [164–167]).”
BackgroundMaher, C. A., Davis, C. R., Curtis, R. G., Short, C. E., & Murphy, K. J. (2020) · JMIR mHealth and uHealth · doi:10.2196/17558 · Cited by 165
“Single studies (n=3) used virtual assistants to provide patients with education and support for asthma, sexual health, and language impairment [9].”
BackgroundDonaldson, S. I., van Zyl, L. E., & Donaldson, S. I. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.817244 · Cited by 166
Readiness for voice assistants to support healthcare delivery during a health crisis and pandemic
Sezgin, E., Huang, Y., Ramtekkar, U., & Lin, S. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00332-0 · Cited by 155
Digital interventions in mental health: An overview and future perspectives
Löchner, J., Carlbring, P., Schuller, B. W., Torous, J. B., & Sander, L. (2025) · Internet Interventions · doi:10.1016/j.invent.2025.100824 · Cited by 204
“Chatbots with unrestricted natural language input options for health-related purposes are an emerging field of research (Laranjo et al., 2018), showing first promising results regarding the reduction of depressive symptoms and psychological stress (Li et al., 2023).”
Digital health during COVID-19: lessons from operationalising new models of care in ophthalmology
Gunasekeran, D. V., Tham, Y.-C., Ting, D. S. W., Tan, G. S. W., & Wong, T. Y. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30287-9 · Cited by 151
Technical metrics used to evaluate health care chatbots: scoping review
Abd-Alrazaq, A., Safi, Z., Alajlani, M., Warren, J., Househ, M., & Denecke, K. (2020) · Journal of Medical Internet Research · doi:10.2196/18301 · Cited by 150
“Laranjo et al [86] reviewed the characteristics, current applications and evaluation measures of health chatbots.”
BackgroundResultAn intelligent knowledge-based chatbot for customer service
Ngai, E. W. T., Lee, M. C. M., Luo, M., Chan, P. S. L., & Liang, T. (2021) · Electronic Commerce Research and Applications · doi:10.1016/j.elerap.2021.101098 · Cited by 218
García de Blanes Sebastián, M., Sarmiento Guede, J. R., & Antonovica, A. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.993935 · Cited by 147
“These communicate with users, via speech, text, facial recognition, and gestures (Laranjo et al., 2018) to allow user interaction via natural language (Stieglitz et al., 2018).”
BackgroundBérubé, C., Schachner, T., Keller, R., Fleisch, E., v Wangenheim, F., Barata, F., & Kowatsch, T. (2021) · Journal of Medical Internet Research · doi:10.2196/25933 · Cited by 133
“These databases were chosen as they cover relevant aspects in the fields of medicine, technology, and interdisciplinary research and have also been used in other systematic reviews covering similar topics [7,8].”
Baclic, O., Tunis, M., Young, K., Doan, C., & Swerdfeger, H. (2020) · Canada Communicable Disease Report · doi:10.14745/ccdr.v46i06a02 · Cited by 134
The Development and Use of Chatbots in Public Health: Scoping Review
Wilson, L., & Marasoiu, M. (2022) · JMIR Human Factors · doi:10.2196/35882 · Cited by 130
“As an emerging field of research, the future implications of human interactions with AI and chatbot interfaces is unpredictable, and there is a need for standardized reporting, study design [54,55], and evaluation [56].”
The virtual patient as a learning tool: a mixed quantitative qualitative study
Isaza-Restrepo, A., Gómez, M. T., Cifuentes, G., & Argüello, A. (2018) · BMC Medical Education · doi:10.1186/s12909-018-1395-8 · Cited by 129
Designing Personality-Adaptive Conversational Agents for Mental Health Care
Ahmad, R., Siemon, D., Gnewuch, U., & Robra-Bissantz, S. (2022) · Information Systems Frontiers · doi:10.1007/s10796-022-10254-9 · Cited by 130
“This means that a successful conversation is particularly dependent on each patient’s individual dynamic behavior and the therapist’s ability to adapt to the patient’s specific personality in order to form a therapeutic relationship (Laranjo et al., 2018; Luxton, 2014).”
BackgroundTechnical Aspects of Developing Chatbots for Medical Applications: Scoping Review
Safi, Z., Abd-Alrazaq, A., Khalifa, M., & Househ, M. (2020) · Journal of Medical Internet Research · doi:10.2196/19127 · Cited by 128
“Two similar systematic review studies were conducted earlier in 2018 and 2019 [57,58].”
Chatbots to Support Young Adults’ Mental Health: An Exploratory Study of Acceptability
Koulouri, T., Macredie, R. D., & Olakitan, D. (2022) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3485874 · Cited by 127
Vaira, L. A., Lechien, J. R., Abbate, V., Allevi, F., Audino, G., Beltramini, G. A., Bergonzani, M., Bolzoni, A., Committeri, U., Crimi, S., et al. (2024) · Otolaryngology–Head and Neck Surgery · doi:10.1002/ohn.489 · Cited by 122
Health-focused conversational agents in person-centered care: a review of apps
Parmar, P., Ryu, J., Pandya, S., Sedoc, J., & Agarwal, S. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00560-6 · Cited by 113
McGreevey, J. D., Hanson, C. W., & Koppel, R. (2020) · JAMA · doi:10.1001/jama.2020.2724 · Cited by 112
Bin Sawad, A., Narayan, B., Alnefaie, A., Maqbool, A., Mckie, I., Smith, J., Yuksel, B., Puthal, D., Prasad, M., & Kocaballi, A. B. (2022) · Sensors · doi:10.3390/s22072625 · Cited by 113
“This is aligned with the findings of the previous systematic reviews of CAs in healthcare [1,27].”
BackgroundMethodResultInfluentialSelf-citationRobotics and Intelligent Systems Against a Pandemic
Khamis, A. M., Meng, J., Wang, J., Azar, A. T., Prestes, E., Li, H., Hameed, I. A., Takács, Á., Rudas, I. J., & Haidegger, T. P. (2021) · Acta Polytechnica Hungarica · doi:10.12700/aph.18.5.2021.5.3 · Cited by 121
“There is a strong demand of more robust and efficient methods and standards for evaluating these technologies [31, 32].”
Yusuf, H., Money, A., & Daylamani-Zad, D. (2025) · Educational technology research and development · doi:10.1007/s11423-025-10447-4 · Cited by 112
Seitz, L., Bekmeier‐Feuerhahn, S., & Gohil, K. (2022) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102848 · Cited by 142
Wang, H., Gupta, S., Singhal, A., Muttreja, P., Singh, S., Sharma, P., & Piterova, A. (2022) · Journal of Medical Internet Research · doi:10.2196/29969 · Cited by 111
“Not surprisingly, AI chatbot health apps are burgeoning [18-21], and they have unique advantages for addressing sensitive and taboo issues such as SRH [22].”
BackgroundAI Technologies for Social Emotional Learning: Recent Research and Future Directions
Sethi, S. S., & Jain, K. (2024) · Journal of Research in Innovative Teaching & Learning · doi:10.1108/jrit-03-2024-0073 · Cited by 111
Ethical implications of conversational agents in global public health
Luxton, D. D. (2020) · Bulletin of the World Health Organization · doi:10.2471/blt.19.237636 · Cited by 108
“Thus, users may experience less anxiety when discussing private concerns, such as disease exposure risk behaviours, with a conversational agent than they would with another person.(3,4,7) Developers can design conversational agents to have modifiable physical appearance and mannerisms, language and speech dialect, and…”
BackgroundArnold, M. H. (2021) · Journal of Bioethical Inquiry · doi:10.1007/s11673-020-10080-1 · Cited by 105
Ruggiano, N., Brown, E. L., Roberts, L., Framil Suarez, C. V., Luo, Y., Hao, Z., & Hristidis, V. (2021) · Journal of Medical Internet Research · doi:10.2196/25006 · Cited by 100
“A recent systematic review by Laranjo et al [10] concluded that despite the potential benefits of conversational agents in the health care field, limited research has evaluated the efficacy and safety of this technology.”
InfluentialKowatsch, T., Schachner, T., Harperink, S., Barata, F., Dittler, U., Xiao, G., Stanger, C., v Wangenheim, F., Fleisch, E., Oswald, H., et al. (2021) · Journal of Medical Internet Research · doi:10.2196/25060 · Cited by 103
“Although there have been recent review papers discussing the use of conversational agents in health care [63,96-101], the current conversational agent is the first (to the best of our knowledge) that takes over the role of a scalable social actor framed as a scalable assistant of a health care professional that mediat…”
Philip, P., Dupuy, L., Auriacombe, M., Serre, F., de Sevin, E., Sauteraud, A., & Micoulaud-Franchi, J.-A. (2020) · npj Digital Medicine · doi:10.1038/s41746-019-0213-y · Cited by 103
Easton, K., Potter, S., Bec, R., Bennion, M., Christensen, H., Grindell, C., Mirheidari, B., Weich, S., de Witte, L., Wolstenholme, D., et al. (2019) · Journal of Medical Internet Research · doi:10.2196/12996 · Cited by 88
“These have rarely been tested with samples of older adults (where most comorbidities are evident) [37,38].”
Nißen, M., Selimi, D., Janssen, A., Cardona, D. R., Breitner, M. H., Kowatsch, T., & von Wangenheim, F. (2022) · Computers in Human Behavior · doi:10.1016/j.chb.2021.107043 · Cited by 101
“Thus, similar to prior research that has provided classifications and frameworks for various specific foci such as healthcare chatbots (Laranjo et al., 2018), business-to-business chatbots (Janssen, Rodríguez Cardona, & Breitner, 2020), or collaborative team chatbots (Bittner et al., 2019; Seering…”
ResultSmart Smile: Revolutionizing Dentistry With Artificial Intelligence
Dhopte, A., & Bagde, H. S. (2023) · Cureus · doi:10.7759/cureus.41227 · Cited by 202
“Limitations of AI in Different Specialties [22]”
BackgroundUX Research on Conversational Human-AI Interaction: A Literature Review of the ACM Digital Library
Zheng, Q., Tang, Y., Liu, Y., Liu, W., & Huang, Y. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501855 · Cited by 100
“reviews [44, 88, 139, 165] and was developed through several iterations and refnement by the research team, in line with prior HCI work.”
BackgroundInfluentialGrodniewicz, J. P., & Hohol, M. (2023) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2023.1190084 · Cited by 100
“Contemporary mental health chatbots are precisely such simple frame-based systems guiding users through predesigned steps of built-in therapeutic interventions or exercises (27, 34, 138).”
BackgroundA bibliometric analysis of artificial intelligence chatbots in educational contexts
Lin, Y., & Yu, Z. (2024) · Interactive Technology and Smart Education · doi:10.1108/itse-12-2022-0165 · Cited by 99
Health Care Chatbots Are Here to Help
Bates, M. (2019) · IEEE Pulse · doi:10.1109/mpuls.2019.2911816 · Cited by 97
Chaves, A. P., & Gerosa, M. A. (2020) · Figshare · doi:10.6084/m9.figshare.13207221.v1 · Cited by 97
Wutz, M., Hermes, M., Winter, V., & Köberlein-Neu, J. (2023) · Journal of Medical Internet Research · doi:10.2196/46548 · Cited by 99
“One promising technology is conversational agents (CAs), also known as chatbots [3,4].”
The Revival of the Notes Field: Leveraging the Unstructured Content in Electronic Health Records
Assale, M., Dui, L. G., Cina, A., Seveso, A., & Cabitza, F. (2019) · Frontiers in Medicine · doi:10.3389/fmed.2019.00066 · Cited by 94
“This is the reason why we will not cover the application of NLP to realize either text-or voice-based conversational agents (24) and scribe-like transcription systems [e.g., (25)], or we will just hint at applications to build complex phenotipic data by which to train highly-accurate predictive…”
BackgroundSystematic review of context-aware digital behavior change interventions to improve health
Craig, K. J. T., Morgan, L. C., Chen, C.-H., Michie, S., Fusco, N., Snowdon, J. L., Scheufele, E. L., Gagliardi, T., & Sill, S. (2020) · Translational Behavioral Medicine · doi:10.1093/tbm/ibaa099 · Cited by 118
“Users in ECA-based interventions have reportedly favored the computer interviewer to human counselors in some cases [51]; the”
MethodCould artificial intelligence make doctors obsolete?
Goldhahn, J., Rampton, V., & Spinas, G. A. (2018) · BMJ · doi:10.1136/bmj.k4563 · Cited by 92
Metadata
- Authors
- Laranjo, L., Dunn, A. G., Tong, H. L., Kocaballi, A. B., Chen, J., Bashir, R., Surian, D., Gallego, B., Magrabi, F., Lau, A. Y. S., & Coiera, E.
- Affiliations
- Macquarie University (AU)
- Corresponding author
- Laranjo, Liliana
- ORCID
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2018-07-11
- Volume
- 25
- Issue
- 9
- Pages
- 1248-1258
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC6118869
- OpenAlex
- W2843010082
- Cited by
- 1,388
- References
- 60
- Citations by year
- 2026: 291 · 2025: 246 · 2024: 182 · 2023: 182 · 2022: 151 · 2021: 143 · 2020: 124 · 2019: 62 · 2018: 6
- FWCI
- 1.87
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of the American Medical Informatics Association (publishedVersion, cc-by-nc)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- UNSWorks (University of New South Wales, Sydney, Australia) (submittedVersion, cc-by-nc-nd)
- Funders
- National Health and Medical Research Council (APP1134919)
- Centre for Research Excellence in Digital Health (APP1054146)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Keywords
- chatbots, healthcare chatbots, natural language input, dialogue management, self-care support, depression symptom reduction, patient safety, evaluation metrics
- MeSH terms
- Artificial Intelligence, Communication, Delivery of Health Care, Natural Language Processing, Speech Recognition Software
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.