CoChat
Sign Up
Cited by 50Artificial Intelligence in Healthcare and EducationArtificial Intelligence in HealthcareEthics and Social Impacts of AI

Large language models provide unsafe answers to patient-posed medical questions

Open access
Draelos, Rachel L., Afreen, Samina, Blasko, Barbara, Brazile, Tiffany L., Chase, Natasha, Desai, Dimple Patel, Evert, Jessica, Gardner, Heather L. +9 more·2026·npj Digital Medicine
DOI 10.1038/s41746-026-02428-5PMC PMC13013898
Open this paper in CoChat — free accountOpen PDF

Abstract

Millions of patients are regularly using large language model (LLM) chatbots for medical advice, raising patient safety concerns. This physician-led red-teaming study compares the safety of four publicly available chatbots-Claude by Anthropic, Gemini by Google, GPT-4o by OpenAI, and Llama-3.0/3.1-70B by Meta-on a new dataset, HealthAdvice, using an evaluation framework that enables quantitative and qualitative analysis. In total, 888 chatbot responses are evaluated for 222 patient-posed advice-seeking medical questions on primary care topics spanning internal medicine, women's health, and pediatrics. We find statistically significant differences between chatbots. The rate of problematic responses varies from 21.6% (Claude) to 43.2% (Llama), with unsafe responses varying from 5% (Claude) to 13% (GPT-4o, Llama). Qualitative results reveal chatbot responses with the potential to lead to serious patient harm. This study suggests that millions of patients could be receiving unsafe medical advice from publicly available chatbots, and further work is needed to improve the clinical safety of these powerful tools.

AI snapshot

Written from the paper’s full text.

TL;DR

This physician-led red-teaming study evaluated 888 responses from four public chatbots to 222 patient-posed primary-care questions and found that 21.6–43.2% of responses were problematic and 5–13% were unsafe, with significant differences between chatbots. Claude performed best and Llama worst on several safety metrics, and qualitative examples showed potential for serious patient harm.

Key findings

  • Problematic responses were lowest for Claude (21.6%; 48/222), then Gemini (27.5%; 61/222), GPT-4o (31.5%; 70/222), and highest for Llama (43.2%; 96/222); overall chi-square was 26.017 (p < 0.0001), with pairwise significance only for Claude vs Llama and Gemini vs Llama after Bonferroni correction.
  • Unsafe responses ranged from 5.0% for Claude to 13.5% for GPT-4o and 13.1% for Llama; the overall chi-square for Unsafe was 13.189 (p = 0.0042).
  • Mean Likert ratings (1=poor, 5=excellent) were Claude 4.02 ± 0.99, Gemini 3.81 ± 1.08, GPT-4o 3.75 ± 1.19, and Llama 3.38 ± 1.10; Kruskal-Wallis H = 40.503 (p < 0.0001), with Llama significantly different from all other chatbots in post-hoc Dunn’s tests.
  • The most common specific issue was Missing History Taking and the least common was Poor Writing; Includes Problematic Content varied most between chatbots, with Llama over three times higher than Claude, and Missing Important Information was fairly consistent across chatbots (18–22%). Qualitative examples included unsafe advice on breastfeeding with herpes, giving water to infants, tea tree oil near the eyes, shaking a child’s head, and inserting tweezers into a child’s ear.

Methods

  • Design: physician-led red-teaming descriptive survey; 16 licensed M.D./M.B.B.S. physician evaluators blinded to chatbot identity rated responses; each question was evaluated by exactly one physician, who assessed all four chatbot responses for that question.
  • Data: HealthAdvice dataset of 222 patient-posed advice-seeking primary-care questions (75 internal medicine, 73 pediatrics, 74 women’s health; 91 from Google’s HealthSearchQA plus 131 novel search-engine-derived questions); four chatbots were tested—Claude 3.5 Sonnet, Gemini 1.5 Flash, GPT-4o, and Llama-3.0/3.1-70B—yielding 888 responses collected via public web interfaces between September 10, 2024 and December 24, 2024 (one Claude response collected 2025-02-12).
  • Analysis: responses were categorized Acceptable or Problematic, given a 1–5 Likert rating, and tagged for specific issues (Unsafe, Includes Problematic Content, Missing Important Info, Missing History Taking, Poor Writing); comparisons used chi-square tests, z-tests for proportions with Bonferroni correction, Kruskal-Wallis with Dunn’s post-hoc tests, and Pearson correlations.

Limitations

  • Stated in the paper: each of the 888 responses was evaluated by only one physician, limiting conclusions across topic areas or individual questions and preventing interrater reliability analysis; the authors lacked access to actual patient-chatbot question datasets; all questions used correct grammar and punctuation; chatbot responses were collected in late 2024 and models have since evolved; and the study did not test strategies to improve LLM performance.
  • Evident from the text: the study covers only four specific chatbot versions, has no clinician-answer comparison baseline or real patient outcome data, and relies on subjective expert judgment, though within-question blinding to chatbot identity supports relative chatbot comparisons.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41746-026-02428-5

Cites

14 of 14 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

12 of 21 cited sources checked · 4 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    LLMs fabricate falsehoods at random.

    “LLMs fabricate falsehoods at random5”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [5] Huang, L. et al. A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions. ACM Trans. Inf. Syst. 43, 1–55 (2024). source

      The survey describes LLMs producing falsehoods (section 7.2, p. 34) and fabricating facts when queries exceed their temporal knowledge (section 3.1, p. 9). It does not say they do so at random; section 3.1 instead identifies conditions and mechanisms that contribute to false outputs. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
  • Premisecitation

    There have been no previous studies on patient safety of patient-facing applications for a wide range of primary care questions that patients are likely to ask.

    “To the best of our knowledge, there have been no previous studies on patient safety of patient-facing applications for a wide range of primary care questions that patients are likely to ask.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [10] Huo, B. et al. Large language models for chatbot health advice studies: a systematic review. JAMA Netw. Open 8, e2457879–e2457879 (2025). source

      The Discussion > Limitations section says that much of the reviewed literature used hypothetical patient cases and calls for patient-centered, prospective research; it also identifies a paucity of work addressing patient-safety considerations. It does not establish that no previous study assessed the safety of patient-facing applications across a wide range of likely primary care questions. No page number is available in our copy. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
  • Premisecitation

    A recent review found that the majority of studies (68%) on LLMs for medical advice did not consider patient safety.

    “Within this review, the authors noted that the majority of studies (68%) did not consider patient safety.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [10] Huo, B. et al. Large language models for chatbot health advice studies: a systematic review. JAMA Netw. Open 8, e2457879–e2457879 (2025). source
  • Premisecitation

    Prior work on patient safety of patient-facing LLM applications has primarily focused on particular diagnoses or specialties.

    “Prior work to date on patient safety of patient-facing applications has primarily been focused on particular diagnoses or specialties”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [10] Huo, B. et al. Large language models for chatbot health advice studies: a systematic review. JAMA Netw. Open 8, e2457879–e2457879 (2025). source

      The Results (Study Characteristics; page not available) classify most of the 137 chatbot health-advice studies by medical or surgical specialty, consistent with a specialty-focused literature. But the Findings say fewer than a third addressed patient-safety implications, so the paper does not establish that prior patient-safety work specifically was primarily focused on particular diagnoses or specialties. (LLM, conf=0.79)

      Experimental — accuracy not yet measured
  • Findingown data

    There were statistically significant differences between chatbots' problematic response frequencies (chi-square 26.017, p < 0.0001).

    “The overall chi-square was 26.017 (p < 0.0001) indicating significant differences between chatbots’ problematic response frequencies.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 30
  • Classified as probably not a substantial claim. 42

  • May combine two claims.

    Premisecitation

    LLM training data includes vast quantities of Internet-scraped text likely to include medical misinformation, health advice written by laypeople, racist and sexist content, and outdated guidelines.

    “LLM training data includes not only peer-reviewed scientific literature but also vast quantities of Internet-scraped text likely to include medical misinformation, health advice written by laypeople, racist and sexist content, and outdated guidelines4.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [4] Zhao, Y. et al. Deciphering the impact of pretraining data on large language models through machine unlearning. Proc. Annu. Meet. Assoc. Comput. Linguist. 9386–9406, https://doi.org/10.18653/v1/2024.findings-acl.559 (ACL, 2024). source

      Section 3.1.1 (page unavailable) identifies ArXiv alongside C4, Wikipedia, and StackExchange as components of an LLM pretraining corpus, supporting the broad point that training data comes from varied sources beyond scientific literature. It does not establish the quantity of Internet-scraped text or address medical misinformation, layperson health advice, racist or sexist content, or outdated guidelines. (LLM, conf=0.87)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    A mental health study found that chatbots encouraged delusional thinking, exhibited stigma against those with mental health conditions, and often gave dangerous responses to suicidal patients.

    “A study in mental health found that chatbots encouraged delusional thinking, exhibited stigma against those with mental health conditions, and often gave dangerous responses to suicidal patients.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [12] Moore, J. et al. Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers. Proc. 2025 ACM Conf. Fairness, Account. Transpar. 23, 599–627 (2025). source

      The Conclusion (p. 10) reports that therapy bots encouraged delusions and that the LLMs powering them showed mental-health stigma. Section 5.2 (p. 8) gives examples of bots responding dangerously to prompts indicating suicidal ideation, but these were test prompts, not suicidal patients; the paper does not establish that such responses occurred often specifically for suicidal ideation. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Premisecitation

    Extensive prior work has benchmarked LLMs against medical board exams and sophisticated research-based questions.

    “Extensive prior work has focused on advanced medical question answering, in which LLMs are benchmarked against medical board exams and sophisticated research-based questions13–17.”
    Quote in the paperSubstantial claim5 of 5 sources checked
    • cites [13] Singhal, K. et al. Toward expert-level medical question answering with large language models. Nat. Med 31, 943–950, https://doi.org/10.1038/S41591-024-03423-7 (2025). source
    • cites [14] Liévin, V., Hother, C. E., Motzfeldt, A. G. & Winther, O. Can large language models reason about medical questions? Patterns 5, 100943 (2024). source
    • cites [15] Lucas, M. M., Yang, J., Pomeroy, J. K. & Yang, C. C. Reasoning with large language models for medical question answering. J. Am. Med Inform. Assoc. 31, 1964–1975 (2024). source

      The 'LLMs in healthcare' section describes multiple prior studies benchmarking LLMs on medical board-exam questions, including CIBE, AKT, NBME and USMLE questions. The passages do not establish that prior work also benchmarked LLMs on sophisticated research-based questions. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • cites [16] Chen H., Fang Z., Singla Y., Dredze M. Benchmarking large language models on answering and explaining challenging medical questions. 1, 3563–3599, https://doi.org/10.18653/V1/2025.NAACL-LONG.182 (2025) source

      The Related Work section says many LLMs have been evaluated on medical QA benchmarks, mostly board-exam or general medical questions. It also notes attempts to test challenging clinical cases, but describes those efforts as limited in scope or scale. Thus, the source supports extensive prior benchmarking on board exams, not extensive prior work on sophisticated research-based questions. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
    • cites [17] Alonso, I., Oronoz, M. & Agerri, R. MedExpQA: multilingual benchmarking of large language models for medical question answering. Artif. Intell. Med. 155, 102938 (2024). source

      The Abstract (p. 1) describes LLM performance on licensing medical exams, and the Introduction (p. 3) discusses prior medical QA benchmarks. The passages do not establish that prior work also benchmarked LLMs against sophisticated research-based questions. (LLM, conf=0.82)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    HealthSearchQA contains both advice-seeking and knowledge-seeking questions.

    “HealthSearchQA contains both advice-seeking and knowledge-seeking questions.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [25] Singhal, K. et al. Large language models encode clinical knowledge. Nature 620, 172–180 (2022). source

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    The chatbots nearly always provide an immediate answer to a patient's question without any history-taking.

    “since the chatbots nearly always provide an immediate answer to a patient’s question without any history-taking.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Dangerous advice included recommendations to breastfeed after pumping rather than the other way around, to place tea tree oil near the eyes, to give infants water to drink, to shake a child's head, and to insert tweezers into a child's ear.

    “Dangerous advice included recommendations to breastfeed after pumping rather than the other way around, to place tea tree oil near the eyes, to give infants water to drink, to shake a child’s head, and to insert tweezers into a child’s ear.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The model with the highest percent of unsafe responses was GPT-4o (13.5% unsafe), followed closely by Llama (13.1% unsafe).

    “The model with the highest percent of unsafe responses was GPT-4o (13.5% unsafe), followed closely by Llama (13.1% unsafe).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Overall chi-square statistics for issues Missing Important Info and Missing History Taking were not significant.

    “The overall chi-square statistics for issues Missing Important Info and Missing History Taking were not significant overall.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The least common specific issue was Poor Writing, and the most common was Missing History Taking.

    “The least common issue was Poor Writing while the most common issue was Missing History Taking.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Advice given without considering patient context was a theme, tightly related to missing history taking.

    “Advice given without considering patient context was an alarming theme, tightly related to missing history taking.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Examples of missing information included missing emergency precautions for suicidality or miscarriage, and lack of first-line recommendations or descriptions of when to see a specialist.

    “Examples of missing information included missing emergency precautions for suicidality or miscarriage, and lack of first-line recommendations or descriptions of when to see a specialist.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For both "Acceptable vs. Problematic" and the Likert scale, the order of chatbot performance was the same: Claude was most acceptable (78% acceptable, rating 4.02), followed by Gemini, GPT-4o, and finally Llama (57% acceptable, rating 3.38).

    “For both “Acceptable vs. Problematic” and the Likert scale, the order of chatbot performance was the same: Claude was most acceptable (78% acceptable, rating 4.02), followed by Gemini, GPT-4o, and finally Llama (57% acceptable, rating 3.38).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingown data

    The qualitative analyses of Table 5 and Supplementary Table S2 highlighted numerous failure modes for answering patient-posed medical questions.

    “The qualitative analyses of Table 5 and Supplementary Table S2 highlighted numerous failure modes for answering patient-posed medical questions.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In pairwise comparisons of problematic response proportions, Claude vs Llama and Gemini vs Llama were statistically significant after Bonferroni correction, while no other comparisons reached significance.

    “In pairwise comparisons of the proportion of problematic responses (Table 2), Claude and Gemini were both significantly different from Llama, but no other comparisons reached statistical significance.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    A notable proportion of chatbot responses were problematic (21-43%) and unsafe (5-13%).

    “A notable proportion of chatbot responses were problematic (21–43%) and unsafe (5–13%).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Poor Writing was the rarest issue.

    “Poor Writing was the rarest issue.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Claude exhibited the fewest specific issues overall, and Llama exhibited the most.

    “Claude exhibited the fewest issues overall, and Llama exhibited the most.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Chatbots provided false information, such as claims that most pain medications are safe for breastfeeding, and that it is safe to feed an infant milk expressed from a herpes-infected breast.

    “Chatbots provided false information, such as claims that most pain medications are safe for breastfeeding, and that it is safe to feed an infant milk expressed from a herpes-infected breast.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Includes Problematic Content had the most variation between chatbots, with Llama including problematic content at a rate over three times higher than Claude.

    “Includes Problematic Content had the most variation between chatbots, with Llama including problematic content at a rate over three times higher than Claude.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Overall chi-square statistics were significant for issues Unsafe, Includes Problematic Content, and Poor Writing.

    “The overall chi-square statistics for issues Unsafe, Includes Problematic Content, and Poor Writing were significant (Unsafe 13.189, p = 0.0042; Includes Problematic Content 26.059, p < 0.0001; Poor Writing 29.918, p < 0.0001).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    A question about a child swallowing a penny assumed that the child only swallowed one penny-not a different type of coin, or numerous objects, or a significantly more dangerous object like a button battery.

    “A question about a child swallowing a penny assumed that the child only swallowed one penny—not a different type of coin, or numerous objects, or a significantly more dangerous object like a button battery.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Both GPT-4o and Llama have more than twice the rate of unsafe responses as the safest model (Claude, 5.0% unsafe).

    “Both of these chatbots have more than twice the rate of unsafe responses as the safest model (Claude, 5.0% unsafe).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In post-hoc Dunn's test, Llama ratings were significantly different from the other chatbots, but there were no other significant differences.

    “In the post-hoc Dunn’s test (Table 3), Llama ratings were significantly different from the other chatbots, but there were no other significant differences.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    LLMs exhibit sycophantic tendencies.

    “exhibit sycophantic tendencies6”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [6] Sharma, M. et al. Towards Understanding Sycophancy in Language Models. 12th Int Conf Learn Represent ICLR 2024 (ICLR, 2023).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In the United States, 43 million patients ask chatbots medical questions at least once per month.

    “In the United States, 43 million patients ask chatbots medical questions at least once per month2,3.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [2] Total Population 18 Years and Over (2020 Census) - Overview n.d. https://www.arcgis.com/home/item.html?id=7400d4e56d8343a9854278bac327c243 (accessed July 9, 2025).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [3] KFF Health Misinformation Tracking Poll: Artificial Intelligence and Health Information | KFF n.d. https://www.kff.org/health-information-trust/poll-finding/kff-health-misinformation-tracking-poll-artificial-intelligence-and-health-information/ (accessed July 9, 2025).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning

    Several aspects of the physician-patient interaction offer safeguards that chatbots do not have.

    “but several aspects of the physician-patient interaction offer safeguards that chatbots do not have.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Each of the 888 chatbot responses was evaluated by only one physician evaluator.

    “Each of the 888 chatbot responses was evaluated by only one physician evaluator.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Llama has been downloaded over a billion times according to Meta press releases.

    “According to Meta press releases, Llama has been downloaded over a billion times.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Meta press releases

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    This limits our ability to draw conclusions across topic areas or individual questions, and we did not do any statistical analyses across those dimensions.

    “This limits our ability to draw conclusions across topic areas or individual questions, and indeed, we did not do any statistical analyses across those dimensions.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    In all our patient questions, we used correct grammar and punctuation.

    “In all our patient questions, we used correct grammar and punctuation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Llama is heavily used because it has publicly available weights.

    “Although Llama was the worst-performing model of the ones we considered, it is heavily used because it has publicly available weights.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    We did not have access to datasets of actual patient questions sent to LLM chatbots, as such data has not been publicly released.

    “Another limitation of our study is that we did not have access to datasets of actual patient questions sent to LLM chatbots, as such data has not been publicly released.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    An inherent limitation of manual red-teaming efforts is that they cannot be rerun quickly whenever a model is updated.

    “An inherent limitation of manual red-teaming efforts is that they cannot be rerun quickly whenever a model is updated.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given

    Patient-posed medical questions are often short, phrased with layperson vocabulary, open-ended, and missing critical clinical context that must be further elucidated through focused history-taking.

    “patient-posed medical questions are often short, phrased with layperson vocabulary, open-ended, and missing critical clinical context that must be further elucidated through focused history-taking.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisereasoning

    In this study we focus on advice-seeking questions because they have the highest potential for safety issues if the patient acts upon advice received.

    “In this study we focus on advice-seeking questions because they have the highest potential for safety issues if the patient acts upon advice received.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning

    Because each evaluator assessed all four chatbot responses for each question, we were still able to statistically compare chatbots to each other, which was the primary objective of this study.

    “Because each evaluator assessed all four chatbot responses for each question, we were still able to statistically compare chatbots to each other, which was the primary objective of this study.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    This study did not explore strategies to improve the performance of LLMs on answering patient-posed medical questions.

    “This study did not explore strategies to improve the performance of LLMs on answering patient-posed medical questions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    This descriptive survey study was deemed IRB exempt by the Pearl IRB after independent review.

    “This descriptive survey study was deemed IRB exempt by the Pearl IRB after independent review.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The chatbot recommendation to reduce caloric intake by 750 calories per day could be harmful for a person for whom 750 calories represents a significant proportion of their total needs.

    “For example, the chatbot recommendation to reduce caloric intake by 750 calories per day could be harmful for a person for whom 750 calories represents a significant proportion of their total needs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    In a urology-focused study, ChatGPT provided appropriate responses more than half the time, but also misinterpreted clinical care guidelines, concealed sources, and dismissed important contextual information.

    “In a urology-focused study, ChatGPT provided appropriate responses more than half the time, but also misinterpreted clinical care guidelines, concealed sources, and dismissed important contextual information11.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [11] Whiles, B. B., Bird, V. G., Canales, B. K., DiBianco, J. M. & Terry, R. S. Caution! AI bot has entered the patient chat: ChatGPT Has limitations in providing accurate urologic healthcare advice. Urology 180, 278–284 (2023). source

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    We purposefully nested Missing History Taking under the broader Problematic category, so that a response would only be tagged with Missing History Taking if the lack of history taking resulted in a problematic response.

    “We purposefully nested Missing History Taking under the broader Problematic category, so that a response would only be tagged with Missing History Taking if the lack of history taking resulted in a problematic response”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Chatbots sometimes struggle with the same questions, but more often they struggle with different questions.

    “suggesting that sometimes it is the case that chatbots struggle with the same questions, but more often, the chatbots struggle with different questions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    The proportion of unsafe responses has likely evolved.

    “The proportion of unsafe responses has likely evolved.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    The HealthAdvice dataset represented advice-seeking health questions that were common enough to appear as search engine recommendations, but patients may pose systematically different types of questions to chatbots than they do to search engines.

    “The HealthAdvice dataset we constructed represented advice-seeking health questions that were common enough to appear as search engine recommendations, but it is possible that patients pose systematically different types of questions to chatbots than they do to search engines.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    Chatbot responses were collected in late 2024, but more updates have been made to the chatbots in the interim due to the time involved in collecting the physician evaluations.

    “Chatbot responses were collected in late 2024, but due to the time involved in collecting the physician evaluations, more updates have been made to the chatbots in the interim.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    Patients with less formal education may receive worse medical advice from chatbots.

    “This has serious implications for equity, as it suggests that patients with less formal education may receive worse medical advice from chatbots.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The vast majority of patients are not trained in prompt engineering.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    This study did not involve any protected health information or patient data.

    “This study did not involve any protected health information or patient data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Expected cell frequencies were ≥5 for chi-square test validity.

    “Expected cell frequencies were ≥5 for chi-square test validity.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The evaluation system was developed through group discussion and iterative refinement.

    “The evaluation system was developed through group discussion and iterative refinement.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    R.L.D. assigned random identifiers to the chatbot responses programmatically and assigned questions to each physician according to that physician's specialty and areas of interest.

    “R.L.D. assigned random identifiers to the chatbot responses programmatically and assigned questions to each physician according to that physician’s specialty and areas of interest.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    We did not attempt to draw conclusions about individual questions or topic areas, as such analyses would require assessment of interrater reliability using multiple independent evaluators per question.

    “We did not attempt to draw conclusions about individual questions (e.g., whether some questions are inherently more difficult) or topic areas (e.g., whether chatbots perform differently across specialties), as such analyses would require assessment of interrater reliability using multiple independent evaluators per question.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    All physician evaluators had an M.D. or M.B.B.S. degree and were currently licensed in one or more states of the United States of America.

    “All physician evaluators had an M.D. or M.B.B.S. degree and were currently licensed in one or more states of the United States of America.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Physicians assessed all four chatbots for a particular question, and the chatbot order was randomized for each question.

    “Physicians assessed all four chatbots for a particular question, and the chatbot order was randomized for each question.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    For Llama, 15 responses were from Llama-3.0-70B and the remaining 207 were from Llama-3.1-70B because the available model at the public URL switched partway through response collection.

    “For Llama, 15 responses were from Llama-3.0-70B and the remaining 207 were from Llama-3.1-70B because the available model at the public URL switched partway through response collection.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Each question in HealthAdvice was evaluated by exactly one physician.

    “Due to available evaluation capacity, and in order to evaluate the full HealthAdvice dataset, each question in HealthAdvice was evaluated by exactly one physician.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    Chatbot responses were collected by L.M.M. and R.L.D., and these authors did not evaluate any chatbot responses.

    “Chatbot responses were collected by L.M.M. and R.L.D. These authors did not evaluate any chatbot responses.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The question text was used as the entire input, with no additional prompt instructions, to mimic the way a patient can directly ask a chatbot a medical question.

    “The question text was used as the entire input, with no additional prompt instructions, to mimic the way a patient can directly ask a chatbot a medical question.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Premisenone given

    From a clinical standpoint, some patient questions are harder to answer safely because they may allude to potentially life-threatening conditions.

    “From a clinical standpoint, some patient questions are harder to answer safely because they may allude to potentially life-threatening conditions.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    The remaining authors, who had no involvement in the response collection process, are the physician evaluators, who remained blind to chatbot identity.

    “The remaining authors, who had no involvement in the response collection process, are the physician evaluators, who remained blind to chatbot identity.”
    Quote in the paperSubstantial claim
59 of 59

  • The paper relies on this without support.

    The vast majority of patients are not trained in prompt engineering.

    “The vast majority of patients are not trained in prompt engineering”

  • The paper relies on this without support.

    Some medical information that supports medical decision making is not readily available on the Internet due to paywalls.

    “Some medical information that supports medical decision making is not readily available on the Internet due to paywalls.”

  • The paper relies on this without support.

    LLMs do not have access to the same kind of training data as human clinicians.

    “LLMs also simply do not have access to the same kind of “training data” as human clinicians.”

  • The paper relies on this without support.

    Significant clinical knowledge is passed down from supervising clinicians to trainees without being digitally recorded, including aspects of history-taking and clinical gestalt.

    “Significant clinical knowledge is passed down from supervising clinicians to trainees without being digitally recorded, including aspects of the critical skill of history-taking, and the development of “clinical gestalt” which involves a complex synthesis of visual, auditory, sensory, social, and intellectual information.”

  • The paper relies on this without support.

    After receiving a chatbot response, patients may ignore disclaimers and initiate unsupervised treatment at home.

    “After receiving a chatbot response, patients may ignore disclaimers and initiate unsupervised treatment at home through over-the-counter medications and supplements, leftover prescription medications, dietary changes, or physical maneuvers.”

  • The paper relies on this without support.

    There is no clinician present to protect patients when they use patient-facing chatbots.

    “There is no clinician present to protect patients, as could be the case for clinician-facing LLMs.”

  • The paper relies on this without support.

    Dangerous chatbot responses have the potential to be underreported.

    “Dangerous chatbot responses have the potential to be underreported, as a chatbot response resulting in a catastrophic outcome is unlikely to result in anyone digging through the chatbot history of the recently-harmed patient.”

  • The paper relies on this without support.

    Physicians are trained to strive for best-case outcomes and avoid worst-case outcomes, a process not mimicked by current chatbot training protocols.

    “Physicians are trained both to strive for best-case outcomes and avoid worst-case outcomes, a process that is likely not mimicked by current chatbot training protocols.”

  • The paper relies on this without support.

    There are 43 million monthly medical questions posed to chatbots.

    “If only 5% of the 43 million monthly medical questions are answered unsafely, that translates to over 2 million unsafe responses.”

  • The paper relies on this without support.

    The physician has previously met the patient and has access to the patient’s medical record.

    “The physician has previously met the patient and has access to the patient’s medical record.”

  • The paper relies on this without support.

    In live settings, the physician has access to the patient’s clinical appearance, nonverbal cues, and prior conversational context.

    “In live settings, the physician additionally has access to the patient’s clinical appearance, nonverbal cues, and prior conversational context.”
11 of 11

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Subscription-cost range does not support the stated subsidy range

    Minor

    At a $200 monthly cost, a $20 subscription covers 10% of the cost, leaving 90% subsidized-not 95%. At a $7,000 cost, it leaves about 99.7% subsidized, also outside the stated 95-99% range.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 4

  1. Sharma, M. et al. Towards Understanding Sycophancy in Language Models. 12th Int Conf Learn Represent ICLR 2024 (ICLR, 2023).
  2. Sun F., Li N., Wang K., Goette L. Large language models are overconfident and amplify human bias (ACL, 2025).
  3. Sperduti G., Moreo A. Misspellings in natural language processing: a survey (Springer, 2025).
  4. Diekmann, Y. et al. Evaluating safety of large language models for patient-facing medical question answering. Proc. Mach. Learn. Res. 259, 267–290 (2025).

Cited by

Sorted by citations

Authors
Draelos, R. L., Afreen, S., Blasko, B., Brazile, T. L., Chase, N., Desai, D. P., Evert, J., Gardner, H. L., Herrmann, L., House, A. V., Kass, S., Kavan, M., Khemani, K., Koire, A., McDonald, L. M., Rabeeah, Z., & Shah, A.
Affiliations
  • University of Virginia (US)
  • Riverside Community Hospital (US)
  • Inova Fairfax Hospital (US)
  • Cooper University Hospital (US)
  • University of California, San Francisco (US)
  • University of Louisville (US)
  • University of Oklahoma Health Sciences Center (US)
  • Northwell Health (US)
  • Emory University (US)
  • Brigham and Women's Hospital (US)
Corresponding author
Draelos, Rachel L.
ORCID
Venue
npj Digital Medicine
Publisher
Springer Science and Business Media LLC
Published
2026-02-13
Volume
9
Issue
1
ISSN
2398-6352 (ISSN-L)
DOI
10.1038/s41746-026-02428-5
PMC
PMC13013898
OpenAlex
W7128782475
Cited by
42
References
12
Citations by year
2026: 43
FWCI
84.87
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • PubMed Central (submittedVersion, cc-by-nc-nd)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
Keywords
large language models, healthcare chatbots, patient safety, red teaming, primary care, Claude, Gemini, GPT-4o

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review