The Effectiveness of Artificial Intelligence Conversational Agents in Health Care: Systematic Review

May combine two claims.

The included studies varied widely in design: 9/31 used cross-sectional designs, 8/31 used randomized controlled trials, 7/31 used qualitative methods, 6/31 used cohort studies, and 1 used a cluster crossover design.

“The study designs also varied widely, with 29% (9/31) using cross-sectional designs, 26% (8/31) using RCTs, 23% (7/31) using qualitative methods, 19% (6/31) using cohort studies, and 1 using a cluster crossover design.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A total of 18 of the 31 included studies included more specific user feedback.

“A total of 18 of the 31 studies included more specific user feedback.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Of the included agents, 45% were chatbots, 19% were embodied conversational agents, and 10% were interactive voice response phone calls, virtual patients, and speech recognition screening systems.

“Of the agents, 45% (14/31) were chatbots (including 2 voice chatbots and 1 chatbot that also used a wizard), 19% (6/31) were ECAs (including 1 virtual doctor), and 10% (3/31) were interactive voice response (IVR) phone calls, virtual patients, and speech recognition screening systems.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Of the 31 included studies, 45% evaluated conversational agents that had some type of audio or speech element.

“Of these studies, 45% (14/31) evaluated conversational agents that had some type of audio or speech element.”

May combine two claims.

Very few included studies discussed cost-effectiveness (5/30, 17%) or safety, privacy, and security (14/30, 47%) outcomes for the agents being evaluated.

“However, very few studies discussed the cost-effectiveness (5/30, 17%, coded as positive or mixed) or safety, privacy, and security (14/30, 47%, coded as positive or mixed) outcomes for the agents being evaluated.”

May combine two claims.

The 31 included studies evaluated a variety of conversational agent types, including 14 chatbots (2 voice chatbots), 6 embodied conversational agents, 1 contextual question-answering agent, and 1 voice recognition triage system.

“A total of 31 studies were selected and included a variety of conversational agents, including 14 chatbots (2 of which were voice chatbots), 6 embodied conversational agents (3 of which were interactive voice response calls, virtual patients, and speech recognition screening systems), 1 contextual question-answering agent, and 1 voice recognition triage system.”

May combine two claims.

Perceived ease of use or usefulness (27/30, 90%), process of service delivery or performance (26/30, 87%), appropriateness (24/30, 80%), and satisfaction (26/31, 84%) were the outcomes with the most support from the included studies.

“Perceived ease of use or usefulness (27/30, 90%), the process of service delivery or performance (26/30, 87%), appropriateness (24/30, 80%), and satisfaction (26/31, 84%) were the outcomes that had the most support from the studies.”

May combine two claims.

Users in 4 studies reported liking that the agent had a personality and/or showed empathy, whereas users in other studies reported disliking the lack of personal connection or had difficulty empathizing with the agent or reported disliking its limited conversation and responses.

“Similarly, users in 4 studies reported liking that the agent had a personality and/or showed empathy [13,32,34,42], whereas users in other studies reported disliking the lack of personal connection or had difficulty in empathizing with the agent [35,37,50] or reported disliking its limited conversation and responses [35,56].”

• cites [13] sourceNot checked yet

  Not checked in the free run.

• cites [32] Cameron, G, 2019, Assessing the Usability of a Chatbot for Mental Health CareNot checked yet

  Not checked in the free run.

• cites [34] sourceNot checked yet

  Not checked in the free run.

• cites [42] sourceNot checked yet

  Not checked in the free run.

• cites [35] sourceNot checked yet

  Not checked in the free run.

• cites [37] sourceNot checked yet

  Not checked in the free run.

• cites [50] sourceNot checked yet

  Not checked in the free run.

• cites [56] Ireland, D, 2016, Stud Health Technol InformNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Users in 2 studies suggested that better integration of the agent with electronic health record systems or health care providers would be useful.

“Additionally, users in 2 studies suggested that better integration of the agent with electronic health record (EHR) systems (for a virtual doctor [42]) or health care providers (for an asthma self-management chatbot [48]) would be useful.”

• cites [42] sourceNot checked yet

  Not checked in the free run.

• cites [48] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Users reported liking reminders and assistance in forming routines, accountability, facilitated learning, and ease of learning and use.

“Other features of the agents that users reported liking were the reminders and assistance in forming routines [37,48] and that the agents provided accountability [13,34,48], facilitated learning [13,34,37], and were easy to learn and use [8,15].”

• cites [37] sourceNot checked yet

  Not checked in the free run.

• cites [48] sourceNot checked yet

  Not checked in the free run.

• cites [13] sourceNot checked yet

  Not checked in the free run.

• cites [34] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Feedback from users in 5 studies expressed a preference for interactivity; users in 1 study liked the interactivity of the chatbot, and users in 4 studies desired greater interactivity or relational skills.

“Feedback from users in 5 studies expressed a preference for interactivity, with users in 1 study noting that they liked the interactivity of the chatbot [35,37], and users in the other 4 studies expressed a desire for greater interactivity or relational skills in the conversational agent [14,32,34,53].”

• cites [35] sourceNot checked yet

  Not checked in the free run.

• cites [37] sourceNot checked yet

  Not checked in the free run.

• cites [14] sourceNot checked yet

  Not checked in the free run.

• cites [32] Cameron, G, 2019, Assessing the Usability of a Chatbot for Mental Health CareNot checked yet

  Not checked in the free run.

• cites [34] sourceNot checked yet

  Not checked in the free run.

• cites [53] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Several studies reported feedback concerned with customization or availability of feature options: 2 studies commented positively, and 3 studies desired more features and more control.

“However, several studies reported feedback concerned with customization or availability of feature options, with 2 studies commenting on it positively (eg, having both voice and touch modes to allow hands-free work and rapid data input on a triage system for nurses) [8,35], and 3 studies desiring more features and more control [33,37,48].”

• cites [8] sourceNot checked yet

  Not checked in the free run.

• cites [35] sourceNot checked yet

  Not checked in the free run.

• cites [33] sourceNot checked yet

  Not checked in the free run.

• cites [37] sourceNot checked yet

  Not checked in the free run.

• cites [48] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Of the 957 citations remaining after initial screening, 293 were selected for full-text review.

“Of these 957 citations, 293 were selected for full-text review.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The review's database search retrieved 9441 studies, of which 2782 were duplicates.

“Overall, 9441 studies were retrieved from the 6 databases, of which 2782 were duplicates.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The most common reasons for exclusion after full-text review were that the conversational agent did not use natural language processing (n=81), the full text was not available (n=71), or there was no conversational agent in the study (n=51).

“with the most common reason being that the conversational agent did not use NLP (n=81), the full text was not available (n=71), or there was no conversational agent in the study (n=51).”

Probably not a substantial claim: may not pass S3 (load-bearing).

There were a wide variety of areas of health care targeted by the conversational agents of the included studies.

“There were a wide variety of areas of health care targeted by the conversational agents of the included studies.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In the 26 studies that reported the device used, 35% used computers, 27% used web-based apps, 23% used mobile phone apps, 15% used telephone calls, and 1 study used a tablet.

“In the 26 studies that reported the device that their conversational agent was used on; 35% (9/26) used computers, 27% (7/26) used web-based apps, 23% (6/26) used mobile phone apps, 15% (4/26) used telephone calls; 1 study used a tablet (the percentages do not add up to 100% because one agent could be used on a computer and also the telephone).”

May combine two claims.

The included studies generally reported positive or mixed evidence for the effectiveness, usability, and satisfactoriness of the conversational agents investigated, but qualitative user perceptions were more mixed.

“The studies generally reported positive or mixed evidence for the effectiveness, usability, and satisfactoriness of the conversational agents investigated, but qualitative user perceptions were more mixed.”

Probably not a substantial claim: may not pass S3 (load-bearing).

After 6 passes, 957 citations remained for abstract screening.

“After 6 passes, 957 citations remained for abstract screening.”

May combine two claims.

Over 80% of users reported that the agents helped them follow their treatment more effectively, and over 80% reported being more prepared for pediatric visits.

“Over 80% of users also reported that the agents helped them follow their treatment more effectively [45] and be more prepared for pediatric visits [9].”

• cites [45] sourceNot checked yet

  Not checked in the free run.

• cites [9] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Very few included studies quantified claims about potential value to the health care system in terms of time or money, and even fewer measured these outcomes with objective measures.

“However, as demonstrated above, very few studies quantified these claims and even fewer measured these outcomes with objective measures.”

May combine two claims.

Among the included study types, RCTs and qualitative studies were generally determined to have the highest quality and lowest risk of bias, while none of the other three study types met more than half the criteria for quality assessment.

“On average, RCTs [9,13,34,37,46,47,49,53] and qualitative studies [41,48,56] evaluated were generally determined to have the highest quality and lowest risk of bias, with none of the other 3 study types meeting more than half the criteria for quality assessment.”

May combine two claims.

Limitations of the agents consistently raised across studies included difficulty understanding them, being repetitive and not sufficiently interactive, and users having difficulty forming personal connections with the agents.

“These included the following: the agents had difficulty understanding them, the agents were repetitive and not sufficiently interactive, and the users had difficulty forming personal connections with the agents.”

May combine two claims.

The other studies performed best on questions about whether the study design was appropriate for the aims and whether the conclusions were justified by the results (6/6 yes for both).

“The other studies performed best on the questions about whether the study design was appropriate for the aims and whether the conclusions were justified by the results (6/6 yes for both).”

• AssumptionRests on an assumption: The AXIS tool, although designed for cross-sectional studies, provides the most systematic evaluation of the various elements of the other studies.

May combine two claims.

Most of the 8 RCTs (5/8) had a high risk of performance bias, predominantly because blinding was not possible given the nature of the intervention.

“Most of the studies (5/8) had a high risk of performance bias, but this was predominantly because blinding was not possible given the nature of the intervention.”

May combine two claims.

Users highlighted positive factors of the agents, particularly their personality and ability to provide empathy and emotional support, that they support learning, are easy to use and access, and help them be accountable.

“Users highlighted many positive factors of the agents, particularly their personality and ability to provide empathy and emotional support, that they support learning, they are easy to use and access, and they help them be accountable, all of which support the generally positive evaluations of usability and satisfaction outcomes.”

May combine two claims.

Usability, agent performance, and satisfaction had the most support across the studies, while cost-effectiveness received hardly any support.

“However, this ranged significantly, with usability, agent performance, and satisfaction having the most support across the studies, and cost-effectiveness receiving hardly any support.”

May combine two claims.

The cross-sectional studies performed best on questions about clarity of aims (5/5 yes), appropriate outcome variables for the aims (5/5 yes), internal consistency (5/5 yes), and adequate description of basic data (4/5 yes).

“Overall, the cross-sectional studies performed best on questions about the clarity of aims (5/5 yes), appropriate outcome variables for the aims (5/5 yes), internal consistency (5/5 yes), and adequate description of basic data (4/5 yes).”

• AssumptionRests on an assumption: The AXIS tool, although designed for cross-sectional studies, provides the most systematic evaluation of the various elements of the other studies.

May combine two claims.

The definitions of effectiveness were highly varied and rarely evaluated with the scrutiny expected for medical devices.

“It should also be noted that the definitions of effectiveness were highly varied and, as evidenced by the methodological limitations identified in the quality assessment, rarely evaluated with the scrutiny expected for medical devices.”

May combine two claims.

Very few studies in this review even discussed the cost analysis of the agent in question, let alone provided substantive evidence about its cost-effectiveness.

“Very few studies in this review even discussed the cost analysis of the agent in questions, let alone provide substantive evidence about its cost-effectiveness.”

May combine two claims.

Only half of the included studies (18/31) reported specific user feedback, despite the fact that 7 of the remaining 13 studies included some measure of usability or user perceptions.

“Only half (18/31) of the studies included in this review reported specific user feedback, despite the fact that 7 of the remaining 13 studies included some measure of usability or user perceptions.”

May combine two claims.

Several of the studies did not sufficiently report their recruitment strategies, so their potential selection bias cannot be accurately evaluated.

“To make matters worse, several of the studies also did not sufficiently report their recruitment strategies, and so their potential selection bias cannot be accurately evaluated.”

May combine two claims.

Almost two-thirds of the studies (19/31) used samples of less than 100 participants or items of analysis, with a median sample size of 48 across all the studies.

“Almost two-thirds of the studies (19/31) used samples of less than 100 participants or items of analysis (eg, voice clips and clinical scenarios) with a median sample size of 48 across all the studies.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Given the small number of studies for each category of agents, comparisons by agent health care scope should be interpreted with caution.

“However, given the small number of studies for each category of agents, these comparisons should be interpreted with caution.”

Classified as probably not a substantial claim. May combine two claims.

Poor understanding and repetitive conversations are key areas of improvement for future research and development because they represent limitations in the usability of the agents in a real-world context.

“Both of these issues are key areas of improvement for future research and development of conversational agents because they represent limitations in the usability of the agents in a real-world context.”

May combine two claims.

Much of the qualitative user feedback was agent dependent, although some preferences and complaints were frequently reported.

“Although there were some preferences and complaints that were frequently reported, much of the feedback was agent dependent.”

Classified as probably not a substantial claim.

The review's analysis was limited to the interpretation of impact as reported by study authors to reflect evaluation outcomes.

“the analysis for this review was limited to the interpretation of impact as reported by study authors to reflect evaluation outcomes.”

Classified as probably not a substantial claim.

The impact of conversational agents on health care depends on people being willing to use them and preferring to use them over alternatives.

“because their impact depends on people being willing to use them and preferring to use them over alternatives.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Data were extracted by 1 reviewer and validated by a second reviewer.

“Data were extracted by 1 reviewer, and key data points from the studies, specified in the protocol and identified on further study of the publications, were recorded in a spreadsheet and validated by a second reviewer.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Studies coded as other design types were assessed using the AXIS tool, which was deemed the most rigorous and appropriate tool because it systematically evaluates introduction, methods, results, and discussion sections and is not limited to RCT-specific questions.

“Studies that were coded as other design types were also assessed using the AXIS tool, which was deemed to be the most rigorous and appropriate tool because it systematically evaluates elements of the introduction, methods, results, and discussion sections, and is not limited to the RCT-specific questions used in the risk-of-bias tool.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The search strategy was finalized and tailored to different databases in consultation with a medical librarian.

“The search strategy was finalized and tailored to different databases in consultation with a medical librarian.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Full-text screening was conducted by 1 reviewer.

“full-text screening was conducted by 1 reviewer.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

All quality assessments were conducted by 2 independent reviewers, with disagreements resolved by consensus.

“All quality assessments were conducted by 2 independent reviewers, with disagreements resolved by consensus.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The search strategy was implemented following the PRISMA-P checklist.

“The population, intervention, comparison, and outcome framework [25] was used to develop the search strategy, which was implemented following the PRISMA-P (Preferred Reporting Items for Systematic Review and Meta-Analyses Protocols) checklist [26].”

• cites [25] sourceNot checked yet

  Not checked in the free run.

• cites [26] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The Cochrane Collaboration risk-of-bias tool was used to evaluate risk of bias in randomized controlled trials, CASP tools were used for cohort and qualitative studies, and the AXIS tool was used for cross-sectional survey studies.

“The Cochrane Collaboration risk-of-bias tool was used to evaluate the risk of bias in randomized controlled trials (RCTs) [28]. The CASP (Critical Appraisal Skills Programme) tools for cohort and qualitative studies were used for the respective studies [29], and the Appraisal tool for Cross-Sectional Studies (AXIS) tool was used to assess the quality of cross-sectional survey studies [30].”

• cites [28] sourceNot checked yet

  Not checked in the free run.

• cites [29] CASP ChecklistsCritical Appraisal Skills Programme: CASP2020-09-11https://casp-uk.net/casp-tools-checklists/Not checked yet

  Not checked in the free run.

• cites [30] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The full methods for this review have been published in detail in a systematic review protocol.

“The full methods for this review have been published in detail in a systematic review protocol [24].”

• cites [24] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Conversational agents are a promising tool for the advancement of patient-centered care and can support users' involvement in the management of their own health.

“Due to this accessibility, conversational agents are also a promising tool for the advancement of patient-centered care and can support users’ involvement in the management of their own health [17,18].”

• cites [17] sourceNot checked yet

  Not checked in the free run.

• cites [18] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The initial database searches of a previous systematic review retrieved only 1531 articles, raising concern that some relevant articles may have been overlooked.

“Additionally, their initial database searches only retrieved 1531 articles, which raises the concern that some relevant articles may have been overlooked [2].”

• cites [2] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

Given the rapid pace of technological development, there is a need to provide an update and expansion to previous systematic reviews of conversational agents in health care.

“Their search was updated in February 2018, but given the rapid pace of technological development, there is a need to provide an update and expansion to these previous systematic reviews.”

May combine two claims.

Although the results reported are promising for the use of conversational agents in health care, limitations in the included studies and in the structure of this review question the validity of this finding.

“Although the results reported are promising for the use of conversational agents in health care, there are a number of limitations in both the studies analyzed and the structure of this review that questions the validity of this finding.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Limiting the focus to include only unconstrained NLP and interaction may have excluded studies of relevant conversational agents that could be satisfactory, useful, and effective in addressing current health care challenges.

“Further limitations of this review are that we limited the focus to include only unconstrained NLP and interaction. This was chosen as a focus because of the advantages NLP offers for simulating human-to-human interaction. However, it may have excluded studies of relevant conversational agents that could be satisfactory, useful, and effective in addressing current health care challenges.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Combining positive and mixed outcomes into one category reduces the granularity of the results.

“However, this combination of positive and mixed outcomes reduces the granularity of the results.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

It is possible that the lack of evaluation of the implications of the agents for health care provision and resources was because of an emphasis on technology development and evaluation, rather than system integration.

“It is possible that the lack of evaluation of the implications of the agents for health care provision and resources was because of an emphasis on technology development and evaluation, rather than system integration.”

Probably not a substantial claim: may not pass S4 (based).

This study extended the literature by expanding its summary to examine a whole system set of evaluation outcomes, including cost-effectiveness, privacy, and security, which have not been systematically examined in previous reviews.

“This study extended the literature by expanding its summary to examine a whole system set of evaluation outcomes, including cost-effectiveness, privacy, and security, which have not been systematically examined in previous reviews.”

Classified as probably not a substantial claim.

This study provides a distinct contribution by conducting a thematic analysis of the qualitative user perceptions of the agents.

“In addition, it provides a distinct contribution by conducting a thematic analysis of the qualitative user perceptions of the agents.”

May combine two claims.

The previous systematic review identified that high-quality evidence of effectiveness and patient safety was limited, which was also observed in this review.

“The previous systematic review identified that high-quality evidence of effectiveness and patient safety was limited, which was also observed in this review.”

• cites [2] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Despite the generally positive usability reported by the studies, there are barriers to the successful use of conversational agents in health care that will need to be addressed before they can achieve the greatest impact.

“This suggests that despite the generally positive usability reported by the studies, there are a number of barriers to the successful use of conversational agents in health care that will need to be addressed before they can achieve the greatest impact.”

May combine two claims.

Although the studies generally reported positive outcomes relating to the agents' usability and effectiveness, the quality of the evidence was not sufficient to provide strong evidence to support these claims.

“Although the studies generally reported positive outcomes relating to the agents’ usability and effectiveness, the quality of the evidence was not sufficient to provide strong evidence to support these claims.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The previous systematic review noted that high overall satisfaction was generally reported by the studies, but that the most common issues with conversational agents related to language understanding or poor dialogue management, which is consistent with our findings.

“Similarly, it noted that high overall satisfaction was generally reported by the studies, but that the most common issues with conversational agents related to language understanding or poor dialogue management, which is consistent with our findings [2].”

• cites [2] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

As the included data indicated a self-reported impact in the studies of effectiveness, the study effectiveness is biased favorably toward the authors' reporting of impact.

“Additionally, as the included data indicated a self-reported impact in the studies of effectiveness, the study effectiveness is biased favorably toward the authors’ reporting of impact.”

May combine two claims.

The risk with self-selection recruitment is that participants who elect to take part in the study are already more positively predisposed to new technologies than those who do not participate, and would tend to evaluate the technology more positively.

“The risk with self-selection recruitment is that participants who elect to take part in the study are already more positively predisposed to new technologies than those who do not participate, and would tend to evaluate the technology more positively.”

May combine two claims.

The previous systematic review [2] also found limited quality of design and evidence in the included studies, with inconsistent reporting of study methods (including methods of selection, attrition, and a lack of validated outcome measures) and conflicts of interest.

“They also found a limited quality of design and evidence in the included studies, with inconsistent reporting of study methods (including methods of selection, attrition, and a lack of validated outcome measures) and conflicts of interest [2].”

• cites [2] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The exclusion of conference abstracts might have caused relevant papers that were classified as abstracts to be missed; however, a previous systematic review that included conference abstracts in their search only had 1 included in their final selection [2].

“The exclusion of conference abstracts might also have caused relevant papers that were classified as abstracts to be missed; however, a previous systematic review that included conference abstracts in their search only had 1 included in their final selection [2].”

• cites [2] sourceNot checked yet

  Not checked in the free run.