Patient and Consumer Safety Risks When Using Conversational Assistants for Medical Information: An Observational Study of Siri, Alexa, and Google Assistant
Open accessPublished version of a preprint
The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Conversational assistants, such as Siri, Alexa, and Google Assistant, are ubiquitous and are beginning to be used as portals for medical services. However, the potential safety issues of using conversational assistants for medical information by patients and consumers are not understood. OBJECTIVE: To determine the prevalence and nature of the harm that could result from patients or consumers using conversational assistants for medical information. METHODS: Participants were given medical problems to pose to Siri, Alexa, or Google Assistant, and asked to determine an action to take based on information from the system. Assignment of tasks and systems were randomized across participants, and participants queried the conversational assistants in their own words, making as many attempts as needed until they either reported an action to take or gave up. Participant-reported actions for each medical task were rated for patient harm using an Agency for Healthcare Research and Quality harm scale. RESULTS: Fifty-four subjects completed the study with a mean age of 42 years (SD 18). Twenty-nine (54%) were female, 31 (57%) Caucasian, and 26 (50%) were college educated. Only 8 (15%) reported using a conversational assistant regularly, while 22 (41%) had never used one, and 24 (44%) had tried one "a few times." Forty-four (82%) used computers regularly. Subjects were only able to complete 168 (43%) of their 394 tasks. Of these, 49 (29%) reported actions that could have resulted in some degree of patient harm, including 27 (16%) that could have resulted in death. CONCLUSIONS: Reliance on conversational assistants for actionable medical information represents a safety risk for patients and consumers. Patients should be cautioned to not use these technologies for answers to medical questions they intend to act on without further consultation from a health care provider.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Nevertheless, the nature of 'in the wild' research [15] and the compounding difficulties of conducting research remotely and in a timely manner, during COVID-19, reassured us that it was correct to gift these devices, both as financial compensation, but also as an reliable means of studying their use during a pandemic.”
From Amazon Echo Show as a Multimodal Human-to-Human Care Support Tool within Self-Isolating Older UK Households (2022) · cites this paper as [15]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Two studies examined the acceptability of conversational agents for health care service delivery [48,87].”
From Conversational Agents in Health Care: Scoping Review and Conceptual Analysis (2020) · cites this paper as [87]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Finally, although chatbots have demonstrated feasibility in behavior change and mental health treatment, they are untested in pandemics and have demonstrated limits in health crisis detection and response4,6–9.”
From Chatbots in the fight against the COVID-19 pandemic (2020) · cites this paper as [9]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With more than a billion voice searches per month, any health-related mistakes, such as misidentifying key symptoms, would be amplified with extensive harmful repercussions4,9.”
From Chatbots in the fight against the COVID-19 pandemic (2020) · cites this paper as [9]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Currently, it is possible to develop apps for consumer voice assistants (eg, skills for Amazon Alexa or actions for Google Assistant). However, these products may be of privacy [76] or safety concerns [77].”
From Voice-Based Conversational Agents for the Prevention and Management of Chronic and Mental Health Conditions: Systematic Literature Review (2021) · cites this paper as [76]
13 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Levinson, S. C. (1983) · doi:10.1017/cbo9780511813313 · Cited by 5,137
ELIZA—a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1966) · Communications of the ACM · doi:10.1145/365153.365168 · Cited by 4,452 · cited 1×
“The earliest such program is the ELIZA system, developed to simulate a Rogerian psychotherapist [18].”
BackgroundFitzpatrick, K. K., Darcy, A., & Vierhile, M. (2017) · JMIR Mental Health · doi:10.2196/mental.7785 · Cited by 2,730 · cited 1×
“The Woebot depression counseling system, evaluated in a randomized clinical trial, does allow free-text input via Facebook Messenger, but the counseling dialogue advances primarily via fully-constrained user input choices [25].”
MethodRapid estimate of adult literacy in medicine: a shortened screening instrument.
Davis, T. C., Long, S. W., Jackson, R. H., Mayeaux, E. J., George, R. B., Murphy, P. W., & Crouch, M. A. (1993) · PubMed · Cited by 1,597
Establishing the computer-patient working alliance in automated health behavior change interventions
Bickmore, T., Gruber, A., & Picard, R. (2005) · Patient Education and Counseling · doi:10.1016/j.pec.2004.09.008 · Cited by 494
Self-citationChang, A. (2005) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzi021 · Cited by 396
Miner, A. S., Milstein, A., Schueller, S., Hegde, R., Mangurian, C., & Linos, E. (2016) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2016.0400 · Cited by 368
A randomized controlled trial of an automated exercise coach for older adults
Bickmore, T. W., Silliman, R. A., Nelson, K., Cheng, D. M., Winter, M., Henault, L., & Paasche‐Orlow, M. K. (2013) · Journal of the American Geriatrics Society · doi:10.1111/jgs.12449 · Cited by 305 · cited 1×
“…to patients with atrial fibrillation [8], human papillomavirus vaccination advice to mothers of girls in the Netherlands [9], exercise and sun avoidance counseling to reduce cancer risk [10], exercise promotion for geriatrics patients [11], and assistance searching for clinical trials [12].”
BackgroundSelf-citationMorris, R. R., Kouddous, K., Kshirsagar, R., & Schueller, S. M. (2018) · Journal of Medical Internet Research · doi:10.2196/10148 · Cited by 273
Health dialog systems for patients and consumers
Bickmore, T., & Giorgino, T. (2006) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2005.12.004 · Cited by 242 · cited 1×
“Bickmore and Giorgino [6] reviewed research and methods in patient-facing natural language dialogue systems in health care.”
BackgroundSelf-citation
Research works printed but not linked: 2
- 2018, US Census Bureau, 2018
- Davis, TC, 1993, Fam Med, 25, 391
Cited by
Taxonomy of Risks posed by Language Models
Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533088 · Cited by 764
“For example, misin-formation on medical dosages may lead a user to cause harm to themselves [21, 130].”
BackgroundConversational Agents in Health Care: Scoping Review and Conceptual Analysis
Tudor Car, L., Dhinagaran, D. A., Kyaw, B. M., Kowatsch, T., Joty, S., Theng, Y.-L., & Atun, R. (2020) · Journal of Medical Internet Research · doi:10.2196/17158 · Cited by 553
“Two studies examined the acceptability of conversational agents for health care service delivery [48,87].”
BackgroundMethodInfluentialEthical and social risks of harm from Language Models
Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.04359 · Cited by 477
The Personalization of Conversational Agents in Health Care: Systematic Review
Kocaballi, A. B., Berkovsky, S., Quiroz, J. C., Laranjo, L., Tong, H. L., Rezazadegan, D., Briatore, A., & Coiera, E. (2019) · Journal of Medical Internet Research · doi:10.2196/15360 · Cited by 332
“Using CAs with unconstrained natural language input can be risky [66].”
BackgroundChatbots in the fight against the COVID-19 pandemic
Miner, A. S., Laranjo, L., & Kocaballi, A. B. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0280-0 · Cited by 322
Conversational agents in the treatment of mental health problems: mixed-method systematic review
Gaffney, H., Mansell, W., & Tai, S. (2019) · JMIR Mental Health · doi:10.2196/14166 · Cited by 275
Schachner, T., Keller, R., & v Wangenheim, F. (2020) · Journal of Medical Internet Research · doi:10.2196/20701 · Cited by 267
“Patients, who are often laypeople when it comes to assessing any technical or medical capabilities of AI-based conversational agents, might follow a chatbot’s advice without additional medical clarification [71].”
BackgroundDigital transformation of mental health services
Bond, R. R., Mulvenna, M. D., Potts, C., O’Neill, S., Ennis, E., & Torous, J. (2023) · npj Mental Health Research · doi:10.1038/s44184-023-00033-y · Cited by 199
At Your Service: Designing Voice Assistant Personalities to Improve Automotive User Interfaces
Braun, M., Mainz, A., Chadowitz, R., Pfleging, B., & Alt, F. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300270 · Cited by 182
Key Considerations for Incorporating Conversational AI in Psychotherapy
Miner, A. S., Shah, N., Bullock, K. D., Arnow, B. A., Bailenson, J., & Hancock, J. (2019) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2019.00746 · Cited by 166
"Alexa is a Toy": Exploring Older Adults' Reasons for Using, Limiting, and Abandoning Echo
Trajkova, M., & Martin-Hammond, A. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376760 · Cited by 154
“While most IVA devices are not approved to access personal health data, Alexa is already being explored to help reduce social isolation in older adults [21], and there is emerging research examining IVA use for health and wellness [5,8,40,41].”
BackgroundBérubé, C., Schachner, T., Keller, R., Fleisch, E., v Wangenheim, F., Barata, F., & Kowatsch, T. (2021) · Journal of Medical Internet Research · doi:10.2196/25933 · Cited by 133
“However, these products may be of privacy [76] or safety concerns [77].”
Differentiate ChatGPT-generated and Human-written Medical Texts
Liao, W., Liu, Z., Dai, H., Xu, S., Wu, Z., Zhang, Y., Huang, X., Zhu, D., Cai, H., Liu, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.11567 · Cited by 119
Modeling adoption of intelligent agents in medical imaging
Calisto, F. M., Nunes, N., & Nascimento, J. C. (2022) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102922 · Cited by 111
Investigating the Accessibility of Voice Assistants With Impaired Users: Mixed Methods Study
Masina, F., Orso, V., Pluchino, P., Dainese, G., Volpato, S., Nelini, C., Mapelli, D., Spagnolli, A., & Gamberini, L. (2020) · Journal of Medical Internet Research · doi:10.2196/18431 · Cited by 105
“Voice-activated technologies are becoming pervasive in our everyday life [1,2].”
BackgroundAn Empirical Study of Older Adult’s Voice Assistant Use for Health Information Seeking
Brewer, R., Pierce, C., Upadhyay, P., & Park, L. (2022) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3484507 · Cited by 89
Kocaballi, A. B., Quiroz, J. C., Rezazadegan, D., Berkovsky, S., Magrabi, F., Coiera, E., & Laranjo, L. (2020) · Journal of Medical Internet Research · doi:10.2196/15823 · Cited by 73
Sezgin, E., Militello, L. K., Huang, Y., & Lin, S. (2020) · Translational Behavioral Medicine · doi:10.1093/tbm/ibz141 · Cited by 83
Voice-Controlled Intelligent Personal Assistants in Health Care: International Delphi Study
(2021) · doi:10.2196/25312 · Cited by 78
Nobles, A. L., Leas, E. C., Caputi, T. L., Zhu, S.-H., Strathdee, S. A., & Ayers, J. W. (2020) · npj Digital Medicine · doi:10.1038/s41746-019-0215-9 · Cited by 67
Palanica, A., Thommandram, A., Lee, A., Li, M., & Fossat, Y. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0133-x · Cited by 63
“Siri elicited the lowest failure rate (23%), but because of this, it had the highest likelihood of potentially causing harm for the tasks tested (30%).(6) A study that explored the use of digital assistants for providing health information and advice on smoking cessation found that Google Assistant generated high qual…”
BackgroundInfluentialTruth machines: synthesizing veracity in AI language models
Munn, L., Magee, L., & Arora, V. (2024) · AI & SOCIETY · doi:10.1007/s00146-023-01756-4 · Cited by 65
“For instance, incorrect advice on medical symptoms and drugs can lead to patient harm or death (Bickmore et al. 2018), with one medical chatbot based on GPT-3 already advising a patient to kill themselves (Quach 2020).”
Arnold, A., Kolody, S., Comeau, A., & Miguel Cruz, A. (2024) · Disability and Rehabilitation: Assistive Technology · doi:10.1080/17483107.2022.2065369 · Cited by 59
Svenningsson, N., & Faraon, M. (2019) · Proceedings of the 2019 2nd Artificial Intelligence and Cloud Computing Conference · doi:10.1145/3375959.3375973 · Cited by 56
User Interaction Patterns and Breakdowns in Conversing with LLM-Powered Voice Assistants
Mahmood, A., Wang, J., Yao, B., Wang, D., & Huang, C.-M. (2025) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2024.103406 · Cited by 51
Griffin, A. C., Khairat, S., Bailey, S. C., & Chung, A. E. (2023) · JAMIA Open · doi:10.1093/jamiaopen/ooad073 · Cited by 46
Wei, J., Tag, B., Trippas, J. R., Dingler, T., & Kostakos, V. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517432 · Cited by 46
Healthcare Voice AI Assistants: Factors Influencing Trust and Intention to Use
Zhan, X., Abdi, N., Seymour, W., & Such, J. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637339 · Cited by 46
EvoPrompting: Language Models for Code-Level Neural Architecture Search
Chen, A., Dohan, D. M., & So, D. R. (2023) · arXiv · doi:10.48550/arxiv.2302.14838 · Cited by 45
Brewer, R. N. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519642 · Cited by 43
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling
Dinan, E., Abercrombie, G., Bergman, A. S., Spruit, S., Hovy, D., Boureau, Y.-L., & Rieser, V. (2021) · arXiv · doi:10.48550/arxiv.2107.03451 · Cited by 44
“Sample conversational assistant interactions resulting in potential harm to the user from (Bickmore et al., 2018).”
BackgroundDesai, S., & Chin, J. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581507 · Cited by 38
“However, there are examples of using other forms of conversational agents, including chatbots and social robots, to provide medical information and interventions to a diverse set of users [10].”
BackgroundInfluentialLanguage Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey
Kumar, S., Balachandran, V., Njoo, L., Anastasopoulos, A., & Tsvetkov, Y. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.241 · Cited by 38
Medical Instructed Real-Time Assistant for Patient with Glaucoma and Diabetic Conditions
Rehman, U. U., Chang, D. J., Jung, Y., Akhtar, U., Razzaq, M. A., & Lee, S. (2020) · Applied Sciences · doi:10.3390/app10072216 · Cited by 37
“The recommendation is predicted based on one symptom and requires approval from the medical specialist [23].”
BackgroundGriffin, A. C., Xing, Z., Mikles, S. P., Bailey, S., Khairat, S., Arguello, J., Wang, Y., & Chung, A. E. (2021) · JAMIA Open · doi:10.1093/jamiaopen/ooab021 · Cited by 31
The Use of Smart Speakers in Care Home Residents: Implementation Study
Edwards, K. J., Jones, R. B., Shenton, D., Page, T., Maramba, I., Warren, A., Fraser, F., Križaj, T., Coombe, T., Cowls, H., et al. (2021) · Journal of Medical Internet Research · doi:10.2196/26767 · Cited by 32
“This extends the use beyond entertainment, and further research is needed to evaluate effective workflows [41].”
BackgroundUddin, J., Feng, C., & Xu, J. (2025) · Journal of Medical Internet Research · doi:10.2196/66032 · Cited by 31
“…[13], health forums and communities [29], health-related webinars and podcasts [30,31], telemedicine [32,33], electronic health records [34], voice assistants [35,36], and augmented and virtual realities [37], all of which support a wide variety of health promotion practices and research [13].”
Bickmore, T., Rubin, A., & Simon, S. (2020) · Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents · doi:10.1145/3383652.3423869 · Cited by 29
Self-citationDesai, S., Lundy, M., & Chin, J. (2023) · Proceedings of the 5th International Conference on Conversational User Interfaces · doi:10.1145/3571884.3597141 · Cited by 25
“[7] found that the health information the participants’ voice assistants shared, if it had been applied in practice, could have resulted in some degree of harm in 29% and potential death in 16% of the”
BackgroundSoubutts, E., Ayobi, A., Eardley, R., McNaney, R., Cater, K., & O'Kane, A. A. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555193 · Cited by 23
"This machine is for the aides": Tailoring Voice Assistant Design to Home Health Care Work
Bartle, V., Albright, L., & Dell, N. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581118 · Cited by 18
Beyeler, M., Légeret, C., Kiwitz, F., & van der Horst, K. (2023) · JMIR Human Factors · doi:10.2196/47913 · Cited by 14
“When patients use an AI-based digital information tool without surveillance by a health care professional, there is a certain risk of misunderstanding or misinterpretation the provided answer [74].”
BackgroundJiang, X.-H., Yuan, X.-H., Zhao, H., & Peng, J.-S. (2025) · Journal of Medical Internet Research · doi:10.2196/76968 · Cited by 10
Should Alexa be a Police Officer, a Doctor, or a Priest?: Towards CUI Relationships Worth Having
Simpson, J., & Crone, C. (2022) · Proceedings of the 4th Conference on Conversational User Interfaces · doi:10.1145/3543829.3544522 · Cited by 6
“However, such a portrayal avoids the often-harsh truth, which is rarely disclosed to users: the CUI can be a nosey neighbour that gossips about you and can even put your health at risk [7, 28].”
BackgroundDesigning Conversational Assistants to Support Older Adults’ Personal Health Record Access
Karimi, P., Ballard, K., Vazirani, P., Jorigay, R. T. N., & Martin-Hammond, A. (2022) · Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-030-99194-4_17 · Cited by 3
Data Capture and Analyses from Conversational Devices in the Homes of the Elderly
Purao, S., & Meng, C. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-34146-6_14 · Cited by 3
Metadata
- Authors
- Bickmore, T. W., Trinh, H., Olafsson, S., O'Leary, T. K., Asadi, R., Rickles, N. M., & Cruz, R.
- Affiliations
- Northeastern University (US)
- University of Connecticut (US)
- Boston Medical Center (US)
- Corresponding author
- Bickmore, Timothy W
- ORCID
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2018-09-04
- Volume
- 20
- Issue
- 9
- Pages
- e11510
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC6231817
- OpenAlex
- W2886572631
- Semantic Scholar
- 42a3e7c85e46b6e6a135e3a1331fd86a2ba6394c
- Cited by
- 308
- References
- 28
- Citations by year
- 2026: 62 · 2025: 36 · 2024: 32 · 2023: 47 · 2022: 35 · 2021: 44 · 2020: 32 · 2019: 20
- FWCI
- 6.26
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Patient Safety and Medication Errors
- Keywords
- chatbots, voice assistants, Siri, Amazon Alexa, Google Assistant, health information seeking, patient safety, consumer health information
- MeSH terms
- Adult, Consumer Behavior, Female, Humans, Male, Social Media, Health Information Exchange
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.