CoChat

Preventing unrestricted and unmonitored AI experimentation in healthcare through transparency and accountability

Open access

The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 20 of the 49 sources its claims cite were checked for support.

Abstract

The integration of large language models (LLMs) into electronic health records offers potential benefits but raises significant ethical, legal, and operational concerns, including unconsented data use, lack of governance, and AI-related malpractice accountability. Sycophancy, feedback loop bias, and data reuse risk amplifying errors without proper oversight. To safeguard patients, especially the vulnerable, clinicians must advocate for patient-centered education, ethical practices, and robust oversight to prevent harm.

Overview

CoChat Summary

## TL;DR

This paper is a viewpoint arguing that the rapid integration of LLMs into EHRs and other healthcare workflows is happening without adequate transparency, oversight, consent, or accountability. The authors call for stronger governance, disclosure, and clinician leadership to reduce risks such as bias, privacy harms, and unclear malpractice liability.

## Key findings

  • The paper states that integrating LLMs into EHR systems can involve “largely unconsented collection and utilization” of patient data and that there is “no established ethical or legal requirement” in the U.S. to obtain informed consent for medical AI use in clinical practice.
  • It highlights that “fewer than 30% of AI tools” have had their performance and effectiveness thoroughly scrutinized, indicating a major post-deployment oversight gap.
  • The authors note that LLM outputs can be biased by patient demographics, can lead to stereotypical presentations, and that ChatGPT-generated assessments/plans have been linked to recommendations for more expensive procedures, potentially worsening disparities and costs.
  • They argue that many commercial LLMs lack transparency about architecture, training methods, and datasets, making risk assessment harder than with open-source models that disclose more details.
  • The paper describes conflict-of-interest concerns when similar LLMs are used by both hospitals and insurers: insurers may use them to reduce payouts, while medical institutions may use them to optimize billing and reimbursements.

## Methods

  • Not a primary empirical study; this is a narrative perspective/commentary paper making an argument for governance and transparency in healthcare AI.
  • The paper draws on examples from EHR vendors, insurance litigation, and cited prior studies on bias, hallucinations, and AI oversight.
  • Its main analysis is a conceptual and ethical review of risks, regulatory gaps, and accountability issues across clinical care, quality improvement, and insurance use cases.

## Limitations

  • The paper does not report original data, a sample, or quantitative effect estimates; this is evident from the text.
  • It does not present a formal systematic review or stated search strategy; this is evident from the text.
  • The paper’s claims are largely argumentative and based on examples and cited literature rather than direct evaluation of specific healthcare deployments; this is evident from the text.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41746-025-01443-2

Cites

33 of 33 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

11 Valid · 10 Not valid · 16 Not checked · 4 Own results and interpretations

20 more claims · 5 with their cited sources checked

Create a free account to see all 41 claims and to verify the 4 references still open.

See all claims — free account
  • It may state two separate claims as one. 12
  • Probably not a substantial claim. 7

Create a free account to see these claims.

Citations

Cites

Show all 33 references

Research works printed but not linked: 1

  1. Lee S. No safety without standards: defining protocols for AI red-teaming disclosures. Tech Policy Press. 2024 Mar 26. Available from: https://www.techpolicy.press/no-safety-without-standards-defining-protocols-for-ai-redteaming-disclosures/. Accessed 12 Aug 2024.

Cited by

See all 44

Metadata

Authors
Comeau, D. S., Bitterman, D. S., & Celi, L. A.
Affiliations
  • Beth Israel Deaconess Medical Center (US)
  • Brigham and Women's Hospital (US)
Corresponding author
Comeau, Donnella S.
Venue
npj Digital Medicine
Published
2025-01-18
Volume
8
Issue
1
Pages
42-42
ISSN
2398-6352 (ISSN-L)
OpenAlex
W4406556816
Cited by
44
References
32
Citations by year
2026: 26 · 2025: 18
FWCI
2.60
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • National Institutes of Health (5r01eb017205-07, R01-EB017205, EB017205, NIBIB R01 EB017205)
  • National Cancer Institute (U54CA274516, R01CA294033)
  • National Institute of Biomedical Imaging and Bioengineering (R01-EB017205, EB017205)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
Keywords
large language models, electronic health records, AI governance, transparency, accountability, patient consent, data reuse, vulnerable patients, patient education, human oversight of AI

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Preventing unrestricted and unmonitored AI experimentation in healthcare through transparency and accountability. https://app.cochat.ai/papers/10.1038/s41746-025-01443-2
The paper
Comeau, D. S., Bitterman, D. S., & Celi, L. A. (2025). Preventing unrestricted and unmonitored AI experimentation in healthcare through transparency and accountability. npj Digital Medicine. https://doi.org/10.1038/s41746-025-01443-2

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.