Preventing unrestricted and unmonitored AI experimentation in healthcare through transparency and accountability
Open accessThe DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 20 of the 49 sources its claims cite were checked for support.
Abstract
The integration of large language models (LLMs) into electronic health records offers potential benefits but raises significant ethical, legal, and operational concerns, including unconsented data use, lack of governance, and AI-related malpractice accountability. Sycophancy, feedback loop bias, and data reuse risk amplifying errors without proper oversight. To safeguard patients, especially the vulnerable, clinicians must advocate for patient-centered education, ethical practices, and robust oversight to prevent harm.
Overview
CoChat Summary
## TL;DR
This paper is a viewpoint arguing that the rapid integration of LLMs into EHRs and other healthcare workflows is happening without adequate transparency, oversight, consent, or accountability. The authors call for stronger governance, disclosure, and clinician leadership to reduce risks such as bias, privacy harms, and unclear malpractice liability.
## Key findings
- The paper states that integrating LLMs into EHR systems can involve “largely unconsented collection and utilization” of patient data and that there is “no established ethical or legal requirement” in the U.S. to obtain informed consent for medical AI use in clinical practice.
- It highlights that “fewer than 30% of AI tools” have had their performance and effectiveness thoroughly scrutinized, indicating a major post-deployment oversight gap.
- The authors note that LLM outputs can be biased by patient demographics, can lead to stereotypical presentations, and that ChatGPT-generated assessments/plans have been linked to recommendations for more expensive procedures, potentially worsening disparities and costs.
- They argue that many commercial LLMs lack transparency about architecture, training methods, and datasets, making risk assessment harder than with open-source models that disclose more details.
- The paper describes conflict-of-interest concerns when similar LLMs are used by both hospitals and insurers: insurers may use them to reduce payouts, while medical institutions may use them to optimize billing and reimbursements.
## Methods
- Not a primary empirical study; this is a narrative perspective/commentary paper making an argument for governance and transparency in healthcare AI.
- The paper draws on examples from EHR vendors, insurance litigation, and cited prior studies on bias, hallucinations, and AI oversight.
- Its main analysis is a conceptual and ethical review of risks, regulatory gaps, and accountability issues across clinical care, quality improvement, and insurance use cases.
## Limitations
- The paper does not report original data, a sample, or quantitative effect estimates; this is evident from the text.
- It does not present a formal systematic review or stated search strategy; this is evident from the text.
- The paper’s claims are largely argumentative and based on examples and cited literature rather than direct evaluation of specific healthcare deployments; this is evident from the text.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
11 Valid · 10 Not valid · 16 Not checked · 4 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“the integration of AI and large language models (LLMs) into hospital EHR systems currently lacks a defined oversight framework (Table 1)2.”
- Also stated in: § Lack of transparency and oversight in EHR AI integration · plain
“However, the integration of AI and large language models (LLMs) into hospital EHR systems currently lacks a defined oversight framework (Table 1)2.”
§ Preventing unrestricted and unmonitored AI experimentation in healthcare through transparency and accountability > Lack of transparency and oversight in EHR AI integration
What the sources say
- Partly supportedcites [2] Meskó, B. & Topol, E. J. The imperative for regulatory oversight of large language models (or generative AI) in healthcare. NPJ Digit. Med. 6, 1–6 (2023). sourceExperimental — accuracy not yet measured
The regulatory-challenges section says LLMs do not yet have a clear regulatory category, and the Conclusions call for guidance on how healthcare organizations can deploy them. That supports the broader oversight concern, but the paper does not specifically say that integration into hospital EHR systems lacks a defined oversight framework. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Lack of transparency and oversight in EHR AI integration · plain
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Their opacity is exacerbated by the industry’s practice of maintaining proprietary control over these models, limiting access to the underlying algorithms and training data4.”
§ Preventing unrestricted and unmonitored AI experimentation in healthcare through transparency and accountability > Introduction
What the sources say
- Partly supportedcites [4] Sun, C.-E., Oikarinen, T. & Weng, T.-W. Crafting large language models for enhanced interpretability. Preprint at https://doi.org/10.48550/arXiv.2407.04307 (2024). sourceExperimental — accuracy not yet measured
The Introduction (p. 1) describes LLMs as inherently opaque and says this makes them harder to debug and improve. It does not attribute that opacity to proprietary industry practices or say that restricted access to algorithms and training data exacerbates it; searches for those terms and close paraphrases found no supporting passage. (LLM, conf=0.86)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The addition of large language models (LLM) into healthcare operations, including electronic health record (EHR) systems represents a transformative level of adoption and integration, changing how enterprise systems interact with algorithms; permitting the largely unconsented collection and utilization of previously unmined private data originating from the care of patients1,2.”
§ Preventing unrestricted and unmonitored AI experimentation in healthcare through transparency and accountability > Introduction
What the sources say
- Not checked — no full text of the sourcecites [1] Bharathi Mohan, G. et al. An analysis of large language models: their impact and potential applications. Knowl. Inf. Syst. https://doi.org/10.1007/s10115-024-02120-8 (2024). source
- Partly supportedcites [2] Meskó, B. & Topol, E. J. The imperative for regulatory oversight of large language models (or generative AI) in healthcare. NPJ Digit. Med. 6, 1–6 (2023). sourceExperimental — accuracy not yet measured
The Introduction and “The LLM era in the practice of medicine” describe healthcare uses of LLMs, including clinical documentation and work with patient records. The “Data privacy and security” section identifies risks to sensitive information, but our copy does not say that integration with EHR systems permits largely unconsented collection or use of previously unmined patient data. (LLM, conf=0.86)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Additionally, clinical documentation produced by LLMs influences clinician decision-making through anchoring and automation biases14,15.”
§ Preventing unrestricted and unmonitored AI experimentation in healthcare through transparency and accountability > Emerging risks and bias in LLM outputs
What the sources say
- Partly supportedcites [14] Nori, H., King, N., McKinney, S. M., Carignan, D. & Horvitz, E. Capabilities of GPT-4 on Medical Challenge Problems. Preprint at https://doi.org/10.48550/arXiv.2303.13375 (2023). sourceExperimental — accuracy not yet measured
In “From Benchmarks to Real World” (pp. 19–20), the paper warns that machine recommendations can influence decision makers and that over-reliance on LLM-generated recommendations poses risks. It does not establish that LLM-produced clinical documentation influences clinicians through anchoring or automation bias specifically. (LLM, conf=0.84)
- Partly supportedcites [15] Chen, S. et al. The effect of using a large language model to respond to patient messages. Lancet Digit Health 6, e379–e381 (2024). sourceExperimental — accuracy not yet measured
The discussion says physicians’ responses changed with LLM assistance, suggesting automation bias and anchoring, and that physicians might adopt the LLM’s assessment. The study examined draft responses to patient messages in a simulation, not clinical documentation generally; it presents the effect on decision-making as a potential risk rather than an established general effect. (LLM, conf=0.86)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Although often not specifically trained for medicine, LLMs are now being utilized for a broader range of tasks, including medical language translation, patient communication, privacy protection, and workflow enhancement.”
§ Preventing unrestricted and unmonitored AI experimentation in healthcare through transparency and accountability > Expanding applications and ethical challenges of LLMs in healthcare While developers have provided high-level information about structure and functionality, critical elements such as data cards, model cards, and bias cards—proposed by AI researchers—are not always shared alongside a model. Although often not specifically trained for medicine, LLMs are now being utilized for a broader range of tasks, including medical language translation, patient communication, privacy protection, and workflow enhancement. They are also being integrated into CDSS and automating processes like discharge planning, coding, and billing6,27,28. Despite their potential, issues such as the propagation of race-based medicine by LLMs have not yet been addressed in EHR implementations27. However, several EHR vendors are moving forward with integrating generative AI across their platforms29. For example, Epic announced plans to roll out more than 100 generative AI projects such as auto-adverse drug reaction tagging, patient-friendly report summaries, a hospital billing coding assistant and an “explain my bill” AI agent at the 2024 HIMSS conference30. MEDITECH is collaborating with Google to leverage an LLM for drafting documentation notes that providers can edit before EHR incorporation. Similarly, Oracle Health’s AI “Clinical Digital Assistant” uses digital information and patient-provider conversations to draft notes and propose follow-up actions. AI is expanding EHR data sources by incorporating new types of data acquired from treatment rooms equipped with ambient monitoring systems. For example, technologies like the OR Black Box can capture detailed point-of-care data, including video, audio, and patient physiology, which can be utilized by LLMs for enhanced healthcare analysis31,32.
What the sources say
- Not checked yetcites [6] Nelson H. How 4 EHR vendors are leveraging generative AI in clinical workflows. TechTarget: Health IT and EHR. 2023 Dec 1. Available from: https://www.techtarget.com/searchhealthit/feature/How-4-EHR-Vendors-Are-Leveraging-Generative-AI-in-Clinical-Workflows. Accessed 04 Sep 2024.
- Partly supportedcites [27] Omiye, J. A., Lester, J. C., Spichak, S., Rotemberg, V. & Daneshjou, R. Large language models propagate race-based medicine. NPJ Digit. Med. 6, 1–4 (2023). sourceExperimental — accuracy not yet measured
The Abstract (page unavailable) says LLMs are being integrated into healthcare systems, with some connecting to electronic health records. It does not describe their use for medical language translation, patient communication, or privacy protection, and an EHR connection alone does not establish workflow enhancement. (LLM, conf=0.84)
- Could not verifycites [28] Gichoya, J. W. et al. AI recognition of patient race in medical imaging: a modelling study. Lancet Digit. Health 4, e406–e414 (2022). sourceExperimental — accuracy not yet measured
The Discussion (p. not available) examines deep-learning models that identify patient race from medical images, not uses of LLMs. Searches for medical language translation, patient communication, privacy protection, and workflow enhancement found no passage addressing the attributed claim. (LLM, conf=0.76)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims · 5 with their cited sources checked
Create a free account to see all 41 claims and to verify the 4 references still open.
- It may state two separate claims as one. 12
- Probably not a substantial claim. 7
Citations
Cites
High-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
The potential for artificial intelligence in healthcare
Davenport, T., & Kalakota, R. (2019) · Future Healthcare Journal · doi:10.7861/futurehosp.6-2-94 · Cited by 3,800
Predicting the future - big data, machine learning, and clinical medicine
Obermeyer, Z., & Emanuel, E. J. (2016) · New England Journal of Medicine · doi:10.1056/nejmp1606181 · Cited by 3,652
Explainability for artificial intelligence in healthcare: a multidisciplinary perspective
the Precise4Q consortium, Amann, J., Blasimme, A., Vayena, E., Frey, D., & Madai, V. I. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-01332-6 · Cited by 2,203
The rise of artificial intelligence in healthcare applications
Bohr, A., & Memarzadeh, K. (2020) · Artificial Intelligence in Healthcare · doi:10.1016/b978-0-12-818438-7.00002-2 · Cited by 1,636
Unintended consequences of machine learning in medicine
Cabitza, F., Rasoini, R., & Gensini, G. F. (2017) · JAMA · doi:10.1001/jama.2017.7797 · Cited by 1,066
The imperative for regulatory oversight of large language models (or generative AI) in healthcare
Meskó, B., & Topol, E. J. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00873-0 · Cited by 1,047
AI recognition of patient race in medical imaging: a modelling study
Gichoya, J. W., Banerjee, I., Bhimireddy, A. R., Burns, J. L., Celi, L. A., Chen, L.-C., Correa, R., Dullerud, N., Ghassemi, M., Huang, S.-C., et al. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00063-2 · Cited by 611
Self-citationCapabilities of GPT-4 on Medical Challenge Problems
Nori, H., King, N., McKinney, S. M., Carignan, D., & Horvitz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13375 · Cited by 509
Research works printed but not linked: 1
- Lee S. No safety without standards: defining protocols for AI red-teaming disclosures. Tech Policy Press. 2024 Mar 26. Available from: https://www.techpolicy.press/no-safety-without-standards-defining-protocols-for-ai-redteaming-disclosures/. Accessed 12 Aug 2024.
Cited by
The Importance of AI Data Governance in Large Language Models
(2025) · doi:10.3390/bdcc9060147 · Cited by 58
“…requires the protection of patient data in healthcare by closely monitoring the process through transparency and accountability to prevent harm; therefore, data encryption, masking, and hashing can protect patient health information within the use of a conceptual data governance framework [217].”
Li, H., Fu, J., & Python, A. (2025) · Journal of Medical Internet Research · doi:10.2196/71916 · Cited by 68
“The lack of transparency and accountability mechanisms continue to raise public concerns about the potential risks of widespread integration of LLMs with digital health [132].”
Moreno-Sánchez, P. A., Ser, J. D., van Gils, M., & Hernesniemi, J. A. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103812 · Cited by 78
Salvi, S., Vu, G. T., Gurupur, V. P., & King, C. (2025) · Electronics · doi:10.3390/electronics14163278 · Cited by 28
The perils of politeness: how large language models may amplify medical misinformation
Rosen, K. L., Sui, M., Heydari, K., Enichen, E. J., & Kvedar, J. C. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02135-7 · Cited by 41
Metadata
- Authors
- Comeau, D. S., Bitterman, D. S., & Celi, L. A.
- Affiliations
- Beth Israel Deaconess Medical Center (US)
- Brigham and Women's Hospital (US)
- Corresponding author
- Comeau, Donnella S.
- Venue
- npj Digital Medicine
- Published
- 2025-01-18
- Volume
- 8
- Issue
- 1
- Pages
- 42-42
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC11743128
- OpenAlex
- W4406556816
- Cited by
- 44
- References
- 32
- Citations by year
- 2026: 26 · 2025: 18
- FWCI
- 2.60
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by-nc-nd
- creativecommons.org/licenses/by-nc-nd/4.0 (tdm)
- creativecommons.org/licenses/by-nc-nd/4.0 (vor)
- Locations
- npj Digital Medicine (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Institutes of Health (5r01eb017205-07, R01-EB017205, EB017205, NIBIB R01 EB017205)
- National Cancer Institute (U54CA274516, R01CA294033)
- National Institute of Biomedical Imaging and Bioengineering (R01-EB017205, EB017205)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- large language models, electronic health records, AI governance, transparency, accountability, patient consent, data reuse, vulnerable patients, patient education, human oversight of AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Preventing unrestricted and unmonitored AI experimentation in healthcare through transparency and accountability. https://app.cochat.ai/papers/10.1038/s41746-025-01443-2
- The paper
- Comeau, D. S., Bitterman, D. S., & Celi, L. A. (2025). Preventing unrestricted and unmonitored AI experimentation in healthcare through transparency and accountability. npj Digital Medicine. https://doi.org/10.1038/s41746-025-01443-2
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.