The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Responses of Conversational Agents to Health and Lifestyle Prompts: Investigation of Appropriateness and Presentation Structures
Open accessPublished version of a preprint
Abstract
Background: Conversational agents (CAs) are systems that mimic human conversations using text or spoken language. Their widely used examples include voice-activated systems such as Apple Siri, Google Assistant, Amazon Alexa, and Microsoft Cortana. The use of CAs in health care has been on the rise, but concerns about their potential safety risks often remain understudied. Objective: This study aimed to analyze how commonly available, general-purpose CAs on smartphones and smart speakers respond to health and lifestyle prompts (questions and open-ended statements) by examining their responses in terms of content and structure alike. Methods: We followed a piloted script to present health- and lifestyle-related prompts to 8 CAs. The CAs’ responses were assessed for their appropriateness on the basis of the prompt type: responses to safety-critical prompts were deemed appropriate if they included a referral to a health professional or service, whereas responses to lifestyle prompts were deemed appropriate if they provided relevant information to address the problem prompted. The response structure was also examined according to information sources (Web search–based or precoded), response content style (informative and/or directive), confirmation of prompt recognition, and empathy. Results: The 8 studied CAs provided in total 240 responses to 30 prompts. They collectively responded appropriately to 41% (46/112) of the safety-critical and 39% (37/96) of the lifestyle prompts. The ratio of appropriate responses deteriorated when safety-critical prompts were rephrased or when the agent used a voice-only interface. The appropriate responses included mostly directive content and empathy statements for the safety-critical prompts and a mix of informative and directive content for the lifestyle prompts. Conclusions: Our results suggest that the commonly available, general-purpose CAs on smartphones and smart speakers with unconstrained natural language interfaces are limited in their ability to advise on both the safety-critical health prompts and lifestyle prompts. Our study also identified some response structures the CAs employed to present their appropriate responses. Further investigation is needed to establish guidelines for designing suitable response structures for different prompt types.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Finally, although chatbots have demonstrated feasibility in behavior change and mental health treatment, they are untested in pandemics and have demonstrated limits in health crisis detection and response4,6–9.”
From Chatbots in the fight against the COVID-19 pandemic (2020) · cites this paper as [8]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Chatbots have varied widely in their responses to questions about physical health, suicide, intimate partner violence, substance abuse, and other sensitive conversations4,6–9.”
From Chatbots in the fight against the COVID-19 pandemic (2020) · cites this paper as [8]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Content Analysis: An Introduction to Its Methodology
Krippendorff, K. (2019) · doi:10.4135/9781071878781 · Cited by 15,448
(1993) · Elsevier eBooks · doi:10.1016/c2009-0-21512-1 · Cited by 5,504
Perspectives on socially shared cognition.
(1991) · American Psychological Association eBooks · doi:10.1037/10096-000 · Cited by 3,446
(2013) · Cited by 1,951
Conversational agents in healthcare: a systematic review
Laranjo, L., Dunn, A. G., Tong, H. L., Kocaballi, A. B., Chen, J., Bashir, R., Surian, D., Gallego, B., Magrabi, F., Lau, A. Y. S., et al. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy072 · Cited by 1,388 · cited 2×
“Given their expanding capabilities and widespread availability, CAs are being increasingly used for health purposes, particularly to support patients and health consumers with health-related aspects of their daily lives [5-9].”
BackgroundSelf-citationLuger, E., & Sellen, A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858288 · Cited by 1,176
Chatbots and Conversational Agents in Mental Health: A Review of the Psychiatric Landscape
Vaidyam, A. N., Wisniewski, H., Halamka, J. D., Kashavan, M. S., & Torous, J. B. (2019) · The Canadian Journal of Psychiatry · doi:10.1177/0706743719828977 · Cited by 1,048
Fulmer, R., Joerin, A., Gentile, B., Lakerink, L., & Rauws, M. (2018) · doi:10.2196/preprints.9782 · Cited by 715
McTear, M., Callejas, Z., & Griol, D. (2016) · doi:10.1007/978-3-319-32967-3 · Cited by 594
Handbook of interpersonal communication
Pettegrew, L. S. (1987) · Social Science & Medicine · doi:10.1016/0277-9536(87)90318-2 · Cited by 524
Research works printed but not linked: 5
- Ogden, W, 1997, Handbook of Human-Computer Interaction (Second Edition)
- Walls, R, 2018, Rosen's Emergency Medicine: Concepts Clinical Practice
- 10.2196/mental.9782 · doi:10.2196/mental.9782 · Paper page
- Edu, J, 2019, ArXiv.org, 1
- Kumar, A, 2017, arXiv.org, 1
Cited by
Chatbots in the fight against the COVID-19 pandemic
Miner, A. S., Laranjo, L., & Kocaballi, A. B. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0280-0 · Cited by 322
Self-citationArtificial intelligence technologies and compassion in healthcare: A systematic scoping review
Morrow, E., Zidaru, T., Ross, F., Mason, C., Patel, K. D., Ream, M., & Stockley, R. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.971044 · Cited by 250
A critical review of state‐of‐the‐art chatbot designs and applications
Luo, B., Lau, R. Y. K., Li, C., & Si, Y. (2022) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1434 · Cited by 224
Kurniawan, M. H., Handiyani, H., Nuraini, T., Hariyati, R. T. S., & Sutrisno, S. (2024) · Annals of Medicine · doi:10.1080/07853890.2024.2302980 · Cited by 177
What Does it Mean for a Language Model to Preserve Privacy?
Brown, H., Lee, K., Mireshghallah, F., Shokri, R., & Tramèr, F. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3534642 · Cited by 165
Metadata
- Authors
- Kocaballi, A. B., Quiroz, J. C., Rezazadegan, D., Berkovsky, S., Magrabi, F., Coiera, E., & Laranjo, L.
- Affiliations
- Macquarie University (AU)
- Comprehensive Health Research Centre (PT)
- Corresponding author
- Kocaballi, Ahmet Baki
- ORCID
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2020-02-09
- Volume
- 22
- Issue
- 2
- Pages
- e15823
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC7055771
- OpenAlex
- W2996664720
- Cited by
- 73
- References
- 34
- Citations by year
- 2026: 10 · 2025: 8 · 2024: 9 · 2023: 10 · 2022: 13 · 2021: 16 · 2020: 7
- FWCI
- 2.86
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- UTS ePRESS (University of Technology Sydney) (submittedVersion)
- Swinburne figshare (Swinburne University of Technology) (submittedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Keywords
- chatbots, voice assistants, smart speakers, response structure, empathy, health information provision, natural language interfaces
- MeSH terms
- Communication, Humans, Life Style
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Responses of Conversational Agents to Health and Lifestyle Prompts: Investigation of Appropriateness and Presentation Structures. https://app.cochat.ai/papers/10.2196/15823
- The paper
- Kocaballi, A. B., Quiroz, J. C., Rezazadegan, D., Berkovsky, S., Magrabi, F., Coiera, E., & Laranjo, L. (2020). Responses of Conversational Agents to Health and Lifestyle Prompts: Investigation of Appropriateness and Presentation Structures. Journal of Medical Internet Research. https://doi.org/10.2196/15823
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.