Cited by 159Artificial Intelligence in Healthcare and EducationEthics and Social Impacts of AIExplainable Artificial Intelligence (XAI)

Large Language Models and User Trust: Consequence of Self-Referential Learning Loop and the Deskilling of Health Care Professionals

Open access
Choudhury, Avishek, Chaudhry, Zaira·2024·Journal of Medical Internet Research

Published version of a preprint

Abstract

As the health care industry increasingly embraces large language models (LLMs), understanding the consequence of this integration becomes crucial for maximizing benefits while mitigating potential pitfalls. This paper explores the evolving relationship among clinician trust in LLMs, the transition of data sources from predominantly human-generated to artificial intelligence (AI)–generated content, and the subsequent impact on the performance of LLMs and clinician competence. One of the primary concerns identified in this paper is the LLMs’ self-referential learning loops, where AI-generated content feeds into the learning algorithms, threatening the diversity of the data pool, potentially entrenching biases, and reducing the efficacy of LLMs. While theoretical at this stage, this feedback loop poses a significant challenge as the integration of LLMs in health care deepens, emphasizing the need for proactive dialogue and strategic measures to ensure the safe and effective use of LLM technology. Another key takeaway from our investigation is the role of user expertise and the necessity for a discerning approach to trusting and validating LLM outputs. The paper highlights how expert users, particularly clinicians, can leverage LLMs to enhance productivity by off-loading routine tasks while maintaining a critical oversight to identify and correct potential inaccuracies in AI-generated content. This balance of trust and skepticism is vital for ensuring that LLMs augment rather than undermine the quality of patient care. We also discuss the risks associated with the deskilling of health care professionals. Frequent reliance on LLMs for critical tasks could result in a decline in health care providers’ diagnostic and thinking skills, particularly affecting the training and development of future professionals. The legal and ethical considerations surrounding the deployment of LLMs in health care are also examined. We discuss the medicolegal challenges, including liability in cases of erroneous diagnoses or treatment advice generated by LLMs. The paper references recent legislative efforts, such as The Algorithmic Accountability Act of 2023, as crucial steps toward establishing a framework for the ethical and responsible use of AI-based technologies in health care. In conclusion, this paper advocates for a strategic approach to integrating LLMs into health care. By emphasizing the importance of maintaining clinician expertise, fostering critical engagement with LLM outputs, and navigating the legal and ethical landscape, we can ensure that LLMs serve as valuable tools in enhancing patient care and supporting health care professionals. This approach addresses the immediate challenges posed by integrating LLMs and sets a foundation for their maintainable and responsible use in the future.

AI snapshot

Written from the paper’s full text.

TL;DR

This conceptual paper argues that growing clinician trust in large language models (LLMs) may combine with a shift from human-generated to AI-generated training data to create a self-referential learning loop that degrades LLM performance and contributes to clinician deskilling. It calls for critical oversight, transparency, and legal/ethical frameworks so LLMs augment rather than undermine patient care.

Key findings

  • The paper identifies a potential self-referential learning loop in which AI-generated content feeds back into LLM training, threatening data diversity, entrenching biases, increasing homogeneity, and ultimately hindering LLM performance.
  • User expertise shapes trust: subject matter experts such as doctors can critically evaluate and correct LLM outputs, while users lacking expertise are more vulnerable to persuasive but incorrect AI-generated content.
  • Deskilling risk is described as more pronounced for future health care professionals, especially medical students who may over-delegate educational and analytical tasks to AI, potentially attenuating critical thinking and diagnostic skill development.
  • The paper discusses medicolegal challenges, including liability allocation among clinicians, institutions, and AI providers, and references The Algorithmic Accountability Act of 2023 as a step toward responsible AI use.

Methods

  • Conceptual/theoretical investigation of the dynamics among clinician trust in LLMs, evolving training data sources, clinician competency, and LLM performance over time.
  • No empirical sample, dataset, or statistical analysis is reported; the argument is illustrated through Figure 1 and references to prior literature.
  • The paper synthesizes concerns about self-referential learning loops, user trust, deskilling, and accountability rather than testing hypotheses.

Limitations

  • The authors state that the self-referential loop and related phenomena are theoretical at this stage and not yet fully realized, representing anticipated challenges.
  • The paper does not report empirical validation, quantitative outcomes, or tested interventions.
  • Evident from the text: the discussion is speculative and relies on conceptual reasoning and cited literature rather than primary data.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.2196/56764

Cites

36 of 36 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 9 cited sources checked · 0 supported · 0 flagged

  • Interpretationreasoning

    When there is disagreement between a clinician's judgment and the LLM output, precautionary bias can lead the clinician, having developed reliance on the LLM, to doubt their own expertise and perceive the LLM as the safer alternative for decision-making.

    “The clinician, having developed a reliance on the LLM due to positive past experiences, might doubt their own expertise and perceive LLM to be the safer alternative for decision-making.”
    Quote in the paperSubstantial claim
  • Interpretationreasoning

    In health care contexts, the risk of self-referential loops can shift toward the reinforcement and entrenchment of specific clinical approaches and schools of thought, reflecting biases in curated data sets.

    “the risk of self-referential loops in health care contexts can shift toward a different concern, the reinforcement and entrenchment of specific clinical approaches and schools of thought.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The sources of LLM training data will increasingly shift from predominantly human-generated content to AI-generated content.
  • Interpretationcitation

    When LLMs such as Med-PaLM are specifically fine-tuned and tailored for health care applications, the severity of self-referential loop issues can be mitigated through stringent quality assurance measures.

    “when these LLMs, such as the Medical Pathways Language Model (Med-PaLM) [32], are specifically fine-tuned and tailored for health care applications, the severity of these issues can be mitigated through stringent quality assurance measures.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [32] source

      Not checked in the free run.

  • Interpretationcitation

    The concern about deskilling is more pronounced for the next generation of health care professionals, particularly medical students, whose over-delegation of tasks to AI could attenuate the development of critical analytical skills and comprehensive understanding of medical concepts.

    “the concern is more pronounced for the next generation of health care professionals, particularly medical students who might increasingly use AI for educational tasks and learning activities where over delegating tasks to AI could attenuate the development of critical analytical skills and a comprehensive understanding of medical concepts, traditionally cultivated through deep engagement with the”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [33] source

      Not checked in the free run.

    • cites [34] source

      Not checked in the free run.

    • Rests on an assumption: The integration and adoption of LLMs in health care will continue to deepen over time.
  • Interpretationreasoning

    When LLM recommendations align with a clinician's initial diagnosis or treatment plan, confirmation bias can lead to a narrowed diagnostic vision in which alternative diagnoses or treatments are not sufficiently considered.

    “This confirmation bias can lead to a narrowed diagnostic vision, where alternative diagnoses or treatments are not sufficiently considered.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 13
  • May combine two claims. 8
  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    LLM performance is temporal and will change as new data are fed to its algorithm.

    “Its performance is temporal and will change as new data are fed to its algorithm.”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    LLMs' self-referential learning loops, in which AI-generated content feeds into the learning algorithms, threaten the diversity of the data pool, potentially entrench biases, and reduce the efficacy of LLMs.

    “One of the primary concerns identified in this paper is the LLMs’ self-referential learning loops, where AI-generated content feeds into the learning algorithms, threatening the diversity of the data pool, potentially entrenching biases, and reducing the efficacy of LLMs.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The sources of LLM training data will increasingly shift from predominantly human-generated content to AI-generated content.
  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    Expert users, particularly clinicians, can leverage LLMs to enhance productivity by off-loading routine tasks while maintaining critical oversight to identify and correct inaccuracies in AI-generated content.

    “The paper highlights how expert users, particularly clinicians, can leverage LLMs to enhance productivity by off-loading routine tasks while maintaining a critical oversight to identify and correct potential inaccuracies in AI-generated content.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Transparency in LLMs can allow clinicians to verify errors and review outputs effectively.

    “Transparency can allow clinicians to verify errors and review outputs effectively.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S4 (based).

    Premisereasoning

    Clinicians, operating at the interface of LLM recommendations and patient care, are generally seen as the final decision-makers and thus bear the primary moral and legal responsibility for the outcomes of those decisions.

    “Clinicians, operating at the interface of LLM recommendations and patient care, are generally seen as the final decision-makers, thus bearing the primary moral and legal responsibility for the outcomes of those decisions.”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    The problem of a self-referential loop in AI-generated content, particularly in health care, has not yet materialized but represents a likely challenge as generative AI continues to proliferate.

    “While this problem of a self-referential loop in AI-generated content, particularly in the health care industry, has not yet materialized, it represents a likely challenge as generative AI continues to proliferate.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The sources of LLM training data will increasingly shift from predominantly human-generated content to AI-generated content.
  • May combine two claims.

    Interpretationcitation

    A self-referential loop in LLMs can lead to the propagation of biases, increased homogeneity in generated data, and ultimately hindered performance.

    “The consequence of a self-referential loop in LLMs can lead to several problematic outcomes, including the propagation of biases [30], increased homogeneity in generated data, and ultimately, hindered performance.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [30] source

      Not checked in the free run.

    • Rests on an assumption: The sources of LLM training data will increasingly shift from predominantly human-generated content to AI-generated content.
  • May combine two claims.

    Premisecitation

    Blind trust in LLMs can introduce two critical cognitive biases, precautionary bias and confirmation bias, both of which alter clinician behavior in the presence of agreement or disagreement between human judgment and LLM outputs.

    “Blind trust in LLMs can introduce 2 critical cognitive biases, precautionary [28] and confirmation bias [29], both of which alter clinician behavior in the presence of agreement or disagreement between human judgment and LLM outputs.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [28] source

      Not checked in the free run.

    • cites [29] Oswald, ME, 2004, Cognitive Illusions: A Handbook on Fallacies and Biases in Thinking, Judgement and Memory

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationreasoning

    The expertise of subject-matter expert users acts as a safeguard against potential errors, ensuring that the AI's assistance enhances productivity without introducing risk.

    “The expertise of such users acts as a safeguard against potential errors, ensuring that the AI’s assistance enhances productivity without introducing risk.”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    Trust in LLMs is a spectrum influenced by the user's subject-matter expertise and the effort the user is willing to invest in ensuring the accuracy of the outcomes.

    “Thus, trust in LLMs can be seen as a spectrum influenced by the user’s expertise, and the effort they are willing to invest in ensuring the accuracy of the outcomes.”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Frequent reliance on LLMs for critical tasks could result in a decline in health care providers' diagnostic and thinking skills, particularly affecting the training and development of future professionals.

    “Frequent reliance on LLMs for critical tasks could result in a decline in health care providers’ diagnostic and thinking skills, particularly affecting the training and development of future professionals.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The integration and adoption of LLMs in health care will continue to deepen over time.
  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    The Algorithmic Accountability Act of 2023 and the Artificial Intelligence Accountability Act represent a critical legislative step toward ensuring the responsible use of algorithms.

    “The Algorithmic Accountability Act of 2023 and Artificial Intelligence Accountability Act [35,36] represent a critical legislative step toward ensuring the responsible use of algorithms.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [35] H.R.3369—118th congress (2023-2024)Congress20232024-04-02https://www.congress.gov/bill/118th-congress/house-bill/3369

      Not checked in the free run.

    • cites [36] Algorithmic accountability act of 2023Office of U.S. Senator Ron Wyden20232024-04-02https://www.congress.gov/bill/118th-congress/senate-bill/2892/text

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisereasoning

    LLMs operate stochastically and can produce different responses under seemingly similar conditions, complicating their reliability.

    “LLMs can produce different responses under seemingly similar conditions, complicating their reliability.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    LLMs can analyze vast data sets to identify patterns or treatment outcomes not immediately apparent to human clinicians, offering insights that can lead to more accurate diagnoses and personalized treatment plans.

    “For instance, LLMs can analyze vast data sets to identify patterns or treatment outcomes that may not be immediately apparent to human clinicians, thereby offering insights that can lead to more accurate diagnoses and personalized treatment plans.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Premisecitation

    Despite producing human-like text, LLMs do not possess consciousness, comprehension, or the ability to discern the truthfulness of their outputs.

    “Despite their ability to produce human-like text, LLMs do not possess consciousness, comprehension, or the ability to discern the truthfulness of their outputs.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    If health care professionals must meticulously check each AI-generated output for accuracy, the time saved through automation may be offset by the time spent on verification.

    “If health care professionals are required to meticulously check each AI-generated output for accuracy, the time saved through automation may be offset by the time spent on verification.”
    Quote in the paperNot a substantial claim
16 of 16
  • The paper relies on this without support.

    The integration and adoption of LLMs in health care will continue to deepen over time.

    “As the health care industry increasingly embraces large language models (LLMs), understanding the consequence of this integration becomes crucial for maximizing benefits while mitigating potential pitfalls.”
  • The paper relies on this without support.

    The sources of LLM training data will increasingly shift from predominantly human-generated content to AI-generated content.

    “This paper explores the evolving relationship among clinician trust in LLMs, the transition of data sources from predominantly human-generated to artificial intelligence (AI)–generated content, and the subsequent impact on the performance of LLMs and clinician competence.”
  • The paper relies on this without support.

    Clinicians' trust in AI will escalate over successive time points while their clinical skills decline.

    “The first panel reveals a timeline that shows an inverse correlation between clinicians’ escalating trust in AI and the preservation of clinical skills over successive time points (T1 to Tn), signaling an increase in AI reliance and a decrease in skill retention.”
3 of 3

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Oswald, ME, 2004, Cognitive Illusions: A Handbook on Fallacies and Biases in Thinking, Judgement and Memory, 79

Cited by

Sorted by citations

Authors
Choudhury, A., & Chaudhry, Z.
Affiliations
West Virginia University (US)
Corresponding author
Choudhury, Avishek
Venue
Journal of Medical Internet Research
Published
2024-04-25
Volume
26
Pages
e56764
ISSN
1438-8871 (ISSN-L)
OpenAlex
W4392976364
Cited by
159
References
36
Citations by year
2026: 67 · 2025: 74 · 2024: 18
FWCI
14.79
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Journal of Medical Internet Research (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University) (cc-by)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords
large language models, clinician trust in AI, algorithmic bias, medicolegal liability, clinical competence
MeSH terms
Artificial Intelligence, Health Personnel, Humans, Language, Learning, Trust

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review