CoChat
Sign Up
Cited by 92Artificial Intelligence in Healthcare and EducationMachine Learning in HealthcareEthics and Social Impacts of AI

Why we need to be careful with LLMs in medicine

Open access
Bélisle-Pipon, Jean-Christophe·2024·Frontiers in Medicine
DOI 10.3389/fmed.2024.1495582PMC PMC11652181
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models (LLMs), the core of many generative AI (genAI) tools, are gaining attention for their potential applications in healthcare. These applications are wide-ranging, including tasks such as assisting with diagnostic processes, streamlining patient communication, and providing decision support to healthcare professionals. Their ability to process and generate large volumes of text makes them promising tools for managing medical documentation and enhancing the efficiency of clinical workflows (Harrer, 2023). LLMs offer a distinct advantage in that they are relatively straightforward to use, particularly since the introduction of ChatGPT-3.5, and they exhibit a notable alignment with human language and communication patterns, facilitating more natural interactions (Ayers et al., 2023) and acceptance of the LLMs' conclusions (Shekar et al., 2024). LLMs operate by predicting the next word in a sequence based on statistical correlations identified in large datasets (Patil et al., 2021;Schubert et al., 2023). However, while these models are effective at producing text that appears coherent and contextually appropriate, they do so without a genuine understanding of meaning or context. This limitation is particularly significant in healthcare, where accuracy is critical. Unlike human cognition, which is driven by a complex array of goals and behaviors, LLMs are narrowly focused on text generation. This focus can lead to the production of plausible sounding but inaccurate information, a phenomenon referred to as "AI hallucination" (OpenAI et al., 2024). In high-stakes environments like prediction, triaging, diagnosis, monitoring, or patient care, these inaccuracies can have serious consequences.While numerous articles across various Frontiers journals discuss LLMs, relatively few focus on AI hallucinations as a central issue. For example, Jin et al. (2023) in Frontiers in Medicine note that "While LLMs like ChatGPT offer tremendous potential in ophthalmology, addressing the challenges of AI hallucination and misinformation is paramount." Similarly, Giorgino et al. (2023) in Frontiers in Surgery emphasize that "The responsible use of this tool must be based on an awareness of its limitations and biases. Foremost among these is the dangerous concept of AI hallucination." Beyond the realm of healthcare, Williams (2024) in Frontiers in Education observes that "The concept of AI hallucination gained widespread attention around 2022, coinciding with the rise of LLMs such as ChatGPT. Users noticed these chatbots often generated random falsehoods in their responses, seemingly indifferent to relevance or accuracy." Williams (2024) continues by stressing that the "term AI hallucination has been criticized for its anthropomorphic connotations, as it likens human perception to the behavior of language models." Despite these critical discussions, they remain sparse compared to the many articles praising LLMs in medicine, highlighting the need for greater engagement in addressing the limitations of these technologies. This imbalance highlights the need for greater emphasis on mitigating the risks posed by these models. Building on this concern, Hicks, Humphries, and Slater (2024) challenge conventional thinking in their paper "ChatGPT is Bullshit." They assert that the inaccuracies produced by LLMs should not simply be labeled as "hallucinations," but as "bullshit," a term based on philosopher Harry Frankfurt's (2009) work. According to this perspective, "bullshit" reflects a disregard for accuracy, which poses serious challenges for the use of genAI in healthcare. By reconceptualizing LLMs in healthcare as "bullshiting" instead of "hallucinating," this paper aims to provide a perspective on the risks these tools pose in critical applications. It explores practical solutions such as layered LLM architectures and improved XAI methods, and emphasizes the urgency of implementing tailored oversight mechanisms to counterbalance the political and industry push for AI deregulation in sensitive domains like medicine.LLMs generate text by predicting the next word based on large datasets. While they produce human-like text, they don't inherently understand or verify its accuracy, acting as "prop-oriented make-believe tools" (Mallory, 2023). Their errors are not the result of technical glitches that can be resolved with better data or refined algorithms but stem from their fundamental nature-they do not evaluate evidence or reason in the human sense. This critical distinction between LLMs' statistical processing and human reasoning can lead to misconceptions, particularly when LLMs are portrayed or perceived as capable of human-like cognition. While LLMs can generate accurate and contextually relevant text, their outputs are based on statistical correlations, not genuine comprehension. As Bender et al. (2021) famously argued, LLMs, which generate word sequences based on learned patterns, function as "stochastic parrots." In

AI snapshot

Written from the paper’s full text.

TL;DR

This perspective argues that LLM errors in medicine should be reframed from "hallucinations" to "bullshit" (per Hicks et al. and Frankfurt), because LLMs are fundamentally indifferent to truth rather than merely misperceiving. It reviews proposed fixes—layered LLMs, explainable AI, regulation, and neurosymbolic AI—and concludes none fully resolves the underlying epistemic limitation, urging cautious, supervised use.

Key findings

  • LLMs generate text by predicting the next word from statistical correlations and "do not evaluate evidence or reason in the human sense"; their errors are therefore not technical glitches fixable with better data or algorithms but stem from their fundamental nature.
  • The paper endorses Hicks et al.'s claim that "ChatGPT is bullshit" rather than hallucinating, arguing the "hallucination" label "trivializes the severity of the issue" and "feeds into overblown hype."
  • Layered LLM architectures (one model cross-validating another) may incrementally improve accuracy but introduce error propagation, coordination complexity, diminishing returns, and risk of over-dependence on AI; XAI "does not address the core limitation" since explanations reflect statistical patterns, not genuine reasoning.
  • Neurosymbolic AI is presented as a promising alternative combining neural adaptability with logical precision, but it "is not a panacea," facing challenges in scalability, interpretability, and handling complex real-world medical data.
  • The paper flags a political risk: expected Trump administration deregulation and lobbying by BSA | The Software Alliance (representing OpenAI, Microsoft) could weaken safeguards, potentially producing "an AI regulation winter."

Methods

  • Design: a perspective/commentary article (Frontiers in Medicine), not an empirical study; it synthesizes and argues from cited literature and policy documents.
  • Data/sample: no primary data; sources include prior publications (e.g., Hicks et al. 2024; Bender et al. 2021; Harrer 2023), the White House AI Bill of Rights, the EU AI Act, and news/policy reporting.
  • Analysis: conceptual argument and normative reasoning; no statistical or qualitative analysis is reported.

Limitations

  • Stated in the paper: neurosymbolic AI's reliance on logical structures "may not fully capture the nuances of probabilistic and ambiguous information common in medicine," and it faces scalability and interpretability challenges.
  • Stated in the paper: disclosure alone is insufficient and can shift burden onto users, and layered LLM approaches risk error propagation and over-dependence.
  • Evident from the text: this is a non-empirical perspective with no data, sample, or formal analysis, so its claims rest on argument and cited sources rather than original evidence; the paper also does not quantify the risks it describes.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3389/fmed.2024.1495582

Cites

32 of 32 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 17 cited sources checked · 0 supported · 0 flagged

  • Interpretationreasoning

    Misunderstanding the nature of LLM outputs poses significant risks.

    “Consider the implications of using genAI to provide medical advice or assist in diagnosing patients—if the nature of its outputs is misunderstood, it poses significant risks.”
    Quote in the paperNot a substantial claim
  • Premisereasoning

    LLMs are fundamentally agnostic to empirical reality.

    “While some, like Downes et al. (33), challenge this view, suggesting that LLMs can produce sensible answers by leveraging higher-level structural information inherent in their design, the fact remains that LLMs remain fundamentally agnostic to empirical reality.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [33] Downes, 2024, LLMs are not just next token predictors, arXiv source

      Not checked in the free run.

  • Interpretationcitation

    XAI does not address the core limitation: LLMs depend on statistical patterns, not genuine reasoning or evidence evaluation.

    “However, XAI does not address the core limitation: LLMs depend on statistical patterns, not genuine reasoning or evidence evaluation (23).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [23] Titus, 2024, Does ChatGPT have semantic understanding? A problem with the statistics-of-occurrence strategy, Cogn Syst Res. source

      Not checked in the free run.

  • Interpretationreasoning

    There is a significant risk of misinformation infiltrating clinical workflows if the model's goal is to generate the most statistically likely response rather than the correct one.

    “If the goal of the model is to generate the most statistically likely response rather than the correct or most appropriate one, there is a significant risk of misinformation infiltrating clinical workflows.”
    Quote in the paperNot a substantial claim
  • Interpretationreasoning

    BSA's focus on removing barriers to innovation risks deprioritizing critical safeguards.

    “While the group acknowledges the importance of international governance and standards, its focus on removing barriers to innovation risks deprioritizing critical safeguards (such as government-imposed ethical AI standards and oversight mechanisms).”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 13
  • May combine two claims. 9

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation

    Generative AI is an experimental technology not yet ready for primetime (Harrer).

    “As stated by Harrer (1): “Health buyers beware: generative AI is an experimental technology not yet ready for primetime.””
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Harrer (1) Harrer, 2023, Attention is not all you need: the complicated case of ethically using large language models in healthcare and medicine, eBioMedicine. source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Neurosymbolic AI faces challenges in scalability, interpretability, and handling the complexity of real-world medical data.

    “Despite its promise, neurosymbolic AI is not a panacea. It faces challenges in scalability, interpretability, and handling the complexity of real-world medical data (27).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [27] 2024, From statistical relational to neurosymbolic artificial intelligence: a survey, Artif Intell. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationcitation

    Entrusting users with the responsibility of fact-checking AI outputs without giving them the resources or assurances of accuracy exposes the field to potentially dangerous mistakes.

    “Entrusting users with the responsibility of fact-checking AI outputs without giving them the resources or assurances of accuracy exposes the field to potentially dangerous mistakes, as well as to arguably lead to AI ethics dumping, so to offload such responsibility to downstream users (28).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [28] Bélisle-Pipon, 2024, Ethics dumping in artificial intelligence, Front Artif Intell. source

      Not checked in the free run.

    • Rests on an assumption: Most users, especially healthcare professionals, lack the time or expertise to verify every piece of AI-generated information.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Such a move could weaken efforts to mitigate the inherent risks of deploying LLMs and flawed AI systems in high-stakes domains like healthcare.

    “Such a move could weaken efforts to mitigate the inherent risks of deploying LLMs and flawed AI systems in high-stakes domains like healthcare.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    LLM errors are not the result of technical glitches that can be resolved with better data or refined algorithms but stem from their fundamental nature-they do not evaluate evidence or reason in the human sense.

    “Their errors are not the result of technical glitches that can be resolved with better data or refined algorithms but stem from their fundamental nature—they do not evaluate evidence or reason in the human sense.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation

    Attributing semantic understanding to LLMs when we are not warranted in doing so could have serious social and ethical implications (Titus).

    “Titus (23) convincingly stated that “Attributing semantic understanding to these systems when we are not warranted in doing so could have serious social and ethical implications related to anthropormorphizing (sic) these systems or over-trusting their ability to produce meaningful or truthful responses.””
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Titus (23) Titus, 2024, Does ChatGPT have semantic understanding? A problem with the statistics-of-occurrence strategy, Cogn Syst Res. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    Reframing AI errors from being seen as harmless "hallucinations" to recognizing them as dangerous "bullshit" is a critical reframing of how to approach the integration of AI into healthcare.

    “Reframing AI errors from being seen as harmless “hallucinations” to recognizing them as dangerous “bullshit” is more than just a shift in terminology—it is a critical reframing of how to approach the integration of AI into healthcare.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Premisecitation

    LLMs are nearing their developmental ceiling, and further investment in them risks diminishing returns (Wadhwa).

    “As Wadhwa (37) suggests, LLMs are nearing their developmental ceiling, and further investment in them risks diminishing returns.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Wadhwa (37)

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    Disclosure of AI limitations is not enough in itself as it is also conducive to problems, particularly by shifting the burden onto users.

    “However, disclosure is not enough in itself as it is also conducive to problems, particularly by shifting the burden onto users.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Employing multiple layers of LLMs introduces significant complexity, including the risk of error propagation and challenges associated with coordination of multiple models.

    “However, this methodology introduces significant complexity, including the risk of error propagation and the challenges associated with the coordination of multiple models.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    AI should serve as a supplementary tool, with all outputs rigorously validated by human experts before being applied in any clinical setting (White House AI Bill of Rights).

    “This is basically what the Notice and Explanation section of the White House's AI Bill of Rights (17) requires: “Medical professionals should not use AI as a standalone source of information or advice. Instead, AI should serve as a supplementary tool, with all outputs rigorously validated by human experts before being applied in any clinical setting.””
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [17] Blueprint for an AI Bill of Rights2022

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    Neurosymbolic AI's reliance on logical structures may not fully capture the nuances of probabilistic and ambiguous information common in medicine.

    “Moreover, its reliance on logical structures may not fully capture the nuances of probabilistic and ambiguous information common in medicine.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    LLMs are not trying to convey information; they are bullshitting (Hicks et al.).

    “As Hicks et al. (10) point out: “ChatGPT is not trying to communicate something they believe or perceive. Their inaccuracy is not due to misperception or hallucination. As we have pointed out, they are not trying to convey information at all. They are bullshitting.””
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Hicks et al. (10) Hicks, 2024, ChatGPT is bullshit, Ethics Inf Technol. source

      Not checked in the free run.

  • May combine two claims.

    Premisereasoning

    LLMs produce text that appears coherent and contextually appropriate but without genuine understanding of meaning or context.

    “However, while these models are effective at producing text that appears coherent and contextually appropriate, they do so without a genuine understanding of meaning or context.”
    • Also stated in: § Understanding AI's “bullshit” · plain
      “While they produce human-like text, they don't inherently understand or verify its accuracy, acting as “prop-oriented make-believe tools” (8).”
    • Also stated in: § Understanding AI's “bullshit” · plain
      “While LLMs can generate accurate and contextually relevant text, their outputs are based on statistical correlations, not genuine comprehension.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationreasoning

    The use of the term "hallucination" to describe when LLMs generate factually incorrect statements trivializes the severity of the issue.

    “The use of the term “hallucination” to describe when LLMs generate factually incorrect statements trivializes the severity of the issue.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Calling chatbot inaccuracies "hallucinations" feeds into overblown hype, suggests solutions that might not work, and could lead to misguided AI alignment efforts (Hicks et al.).

    “As Hicks et al. (10) note, “Calling chatbot inaccuracies ‘hallucinations' feeds into overblown hype about their abilities among technology cheerleaders, and could lead to unnecessary consternation among the general public. It also suggests solutions to the inaccuracy problems which might not work, and could lead to misguided efforts at AI alignment amongst specialists.””
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Hicks et al. (10) Hicks, 2024, ChatGPT is bullshit, Ethics Inf Technol. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Relying on LLMs without addressing their fundamental flaws presents significant risks in the medical field.

    “In the medical field, where evidence-based decision-making is paramount, relying on these models without addressing their fundamental flaws presents significant risks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    Without shared responsibility efforts, the drive for deregulation may exacerbate the risks posed by LLMs, particularly their tendency to produce plausible yet inaccurate or misleading outputs.

    “Without such efforts, the drive for deregulation may exacerbate the risks posed by LLMs, particularly their tendency to produce plausible yet inaccurate or misleading outputs.”
    Quote in the paperNot a substantial claim
18 of 18

  • The paper relies on this without support.

    LLMs can generate different answers to the same question on different occasions (lack reproducibility).

    “A key concern with LLMs in medical applications is their lack of reproducibility. Unlike traditional software systems, where identical inputs yield consistent outputs, LLMs can generate different answers to the same question on different occasions.”

  • The paper relies on this without support.

    Most users, especially healthcare professionals, lack the time or expertise to verify every piece of AI-generated information.

    “The implications for the health sector are significant, most users (especially healthcare professionals) lack the time or expertise to verify every piece of AI-generated information, especially in high-stake environments where the margin for error is slim, but the consequences significant.”
2 of 2

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Hallucination, in which LLMs generate confident but factually incorrect outputs, has been documented in clinical contexts with direct implications for patient safety.

    “Hallucination, in which LLMs generate confident but factually incorrect outputs, has been documented in clinical contexts with direct implications for patient safety”

    From A generative AI multi-agent framework with integrated XAI governance for cancer diagnostics: from multi-omics interpretation to lifestyle risk stratification (2026) · cites this paper as Bélisle-Pipon, 2024

    The Introduction (page not supplied) describes hallucinations as plausible-sounding but inaccurate information and says such inaccuracies can have serious consequences in diagnosis and patient care. The Discussion (page not supplied) likewise identifies risks to safe care, but the passages describe potential harms rather than documenting hallucinations occurring in clinical practice. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Hallucinated content in clinical settings can vary in severity, ranging from minor factual inconsistencies to unsupported treatment recommendations that may directly harm patients.

    “hallucinated content can vary in severity, ranging from minor factual inconsistencies to unsupported treatment recommendations that may directly harm patients”

    From A generative AI multi-agent framework with integrated XAI governance for cancer diagnostics: from multi-omics interpretation to lifestyle risk stratification (2026) · cites this paper as Bélisle-Pipon, 2024

    The Introduction describes hallucinations as plausible but inaccurate information that can have serious consequences in patient care. The section “Understanding AI’s ‘bullshit’” adds that acting on flawed medical advice could lead to improper treatments and serious consequences for patients. The paper does not describe a severity range beginning with minor factual inconsistencies or specifically identify unsupported treatment recommendations. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • LLMs tend to present fabrications with an authoritative tone, mixing them with factual information, making errors difficult to detect.

    “This risk is amplified by the tendency for LLMs to present these fabrications with an authoritative tone, mixing them with factual information, in a way that makes errors difficult to detect, presenting a potential threat to patient safety.7,11”

    From Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts (2026) · cites this paper as [11]

    The Introduction describes plausible-sounding inaccuracies, and the Discussion says fluent, coherent responses can conceal that they are not grounded in verified facts, posing risks in medical care (page numbers unavailable). This supports the claim that convincing presentation can make errors hard to recognize, but our copy does not specifically say that LLMs mix fabrications with factual information. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • LLMs are optimized for fluency and coherence rather than factual information.

    “These errors are fundamental to the design of LLMs, which are optimized for fluency and coherence rather than factual information.11”

    From Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts (2026) · cites this paper as [11]

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Bélisle-Pipon, J.-C.
Venue
Frontiers in Medicine
Publisher
Frontiers Media SA
Published
2024-12-04
Volume
11
Pages
1495582-1495582
ISSN
2296-858X (ISSN-L)
DOI
10.3389/fmed.2024.1495582
PMC
PMC11652181
OpenAlex
W4405021649
Semantic Scholar
abe3e48b4a1963094662f2efafb1de5c469fe2c5
Cited by
91
References
32
Citations by year
2026: 50 · 2025: 40 · 2024: 1
FWCI
8.51
Type
journal article
Language
en
Open access
Licence
Locations
  • Frontiers in Medicine (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
Keywords
large language models, AI hallucination, diagnosis, clinical decision support, clinical documentation, clinical workflow, patient communication, ChatGPT, generative AI in medicine, responsible AI use, patient safety, human-AI collaboration

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review