The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
The effect of using a large language model to respond to patient messages
Open accessAbstract
The relentless increase in administrative responsibilities, amplified by electronic health record (EHR) systems, has diverted clinician attention from direct patient care, fuelling burnout.1 In response, large language models (LLMs) are being adopted to streamline clinical and administrative tasks. Notably, Epic is currently leveraging OpenAI's ChatGPT models, including GPT-4, for electronic messaging via online portals.2 The volume of patient portal messaging has escalated in the past 5–10 years,3 and general-purpose LLMs are being deployed to manage this burden.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures

Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Additionally, clinical documentation produced by LLMs influences clinician decision-making through anchoring and automation biases14,15.”
From Preventing unrestricted and unmonitored AI experimentation in healthcare through transparency and accountability (2025) · cites this paper as [15]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This gap between knowledge retrieval and logical reasoning in medicine6,7 leads to a particularly concerning public health risk: The rapid generation and dissemination of false information is particularly critical in high-stakes fields like medicine.”
From When helpfulness backfires: LLMs and the risk of false medical information due to sycophantic behavior (2025) · cites this paper as [6]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Chen et al12 evaluated the harm risk of LLM responses to patient messages, observing a risk of severe harm in 7.1% of responses and death in 0.6%.”
From Evaluating the Quality and Safety of Ambient Digital Scribe Platforms Using Simulated Ambulatory Encounters (2025) · cites this paper as [12]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Similar to studies showing increased empathetic communication phrasing from LLMs to patient queries, this study indicates that LLMs may influence physicians to better consider human factors in their management reasoning.26–28”
From Large Language Model Influence on Management Reasoning: A Randomized Controlled Trial (2024) · cites this paper as [28]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · cited 1×
“Assessments of encoded general biomedical knowledge, such as performance on medical board exams, are a first step toward clinical applications, 5 but should not be used as surrogates for the clinical expertise and acumen needed to care for patients.”
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083 · cited 1×
“Assessments of encoded general biomedical knowledge, such as performance on medical board exams, are a first step toward clinical applications, 5 but should not be used as surrogates for the clinical expertise and acumen needed to care for patients.”
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613
Unintended consequences of machine learning in medicine
Cabitza, F., Rasoini, R., & Gensini, G. F. (2017) · JAMA · doi:10.1001/jama.2017.7797 · Cited by 1,066 · cited 2×
“…for new approaches for evaluation and monitoring, especially as trust in LLMs builds and clinicians become less vigilant and more reliant on LLMs. 8 In our study, harmful content was often associated with inadequate recognition or communication of the scenario’s acuity, rather than errors in…”
The imperative for regulatory oversight of large language models (or generative AI) in healthcare
Meskó, B., & Topol, E. J. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00873-0 · Cited by 1,047 · cited 1×
“10 Physicians and institutions must exercise caution as the health-care industry embraces these advanced technologies, because it is imperative to balance their innovative potential with a commitment to patient safety and care quality.”
Adler-Milstein, J., Zhao, W., Willard-Grace, R., Knox, M., & Grumbach, K. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz220 · Cited by 356
Use of Artificial Intelligence Chatbots for Cancer Treatment Information
Chen, S., Kann, B. H., Foote, M. B., Aerts, H. J. W. L., Savova, G. K., Mak, R. H., & Bitterman, D. S. (2023) · JAMA Oncology · doi:10.1001/jamaoncol.2023.2954 · Cited by 251
Self-citationHuman Factors Challenges for the Safe Use of Artificial Intelligence in Patient Care
Sujan, M., Furniss, D., Grundy, K., Grundy, H., Nelson, D., Elliott, M., White, S., Habli, I., & Reynolds, N. (2019) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2019-100081 · Cited by 144
Approaching autonomy in medical artificial intelligence
Bitterman, D. S., Aerts, H. J. W. L., & Mak, R. H. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30187-4 · Cited by 107 · cited 1×
“9 Moving forward, more transparency from EHR vendors and institutions about prompting methods are urgently needed for evaluations.”
Self-citationNath, B., Williams, B., Jeffery, M. M., O’Connell, R. T., Goldstein, R. L., Sinsky, C. A., & Melnick, E. R. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.31490 · Cited by 100
Cited by
The TRIPOD-LLM reporting guideline for studies using large language models
Gallifant, J., Afshar, M., Ameen, S., Aphinyanaphongs, Y., Chen, S., Cacciamani, G., Demner-Fushman, D., Dligach, D., Daneshjou, R., Fernandes, C., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03425-5 · Cited by 615
Self-citationGoh, E., Gallo, R. J., Strong, E., Weng, Y., Kerman, H., Freed, J. A., Cool, J. A., Kanjee, Z., Lane, K. P., Parsons, A. S., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03456-y · Cited by 167
“…data, extended data, supplementary information, acknowledgements, peer review information; details of author contributions and competing interests; and statements of data and code availability are available at https://doi.org/10.1038/s41591-024-03456-y. https://doi.org/10.1038/s41591-024-03456-y”
InfluentialFreyer, O., Wiest, I. C., Kather, J. N., & Gilbert, S. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00124-9 · Cited by 203
Retrieval-augmented generation for generative artificial intelligence in health care
Yang, R., Ning, Y., Keppo, E., Liu, M., Hong, C., Bitterman, D. S., Ong, J. C. L., Ting, D. S. W., & Liu, N. (2025) · npj Health Systems · doi:10.1038/s44401-024-00004-1 · Cited by 269
“Additionally, the utilization of generative AI could enhance the quality of health services for patients while alleviating the workload for clinicians 8 – 10 .”
Self-citationLarge Language Model–Based Responses to Patients’ In-Basket Messages
Small, W. R., Wiesenfeld, B. M., Brandfield-Harvey, B., Jonassen, Z., Mandal, S., Stevens, E. R., Major, V. J., Lostraglio, E., Szerencsy, A. C., Jones, S., et al. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.22399 · Cited by 130
Metadata
- Authors
- Chen, S., Guevara, M., Moningi, S., Hoebers, F., Elhalawani, H., Kann, B. H., Chipidza, F. E., Leeman, J., Aerts, H. J. W. L., Miller, T., Savova, G. K., Gallifant, J., Celi, L. A., Mak, R. H., Lustberg, M., Afshar, M., & Bitterman, D. S.
- Affiliations
- Brigham and Women's Hospital (US)
- Boston Children's Hospital (US)
- Massachusetts Institute of Technology (US)
- Beth Israel Deaconess Medical Center (US)
- Yale University (US)
- University of Wisconsin–Madison (US)
- Corresponding author
- Bitterman, Danielle S
- ORCID
- Venue
- The Lancet Digital Health
- Publisher
- Elsevier BV
- Published
- 2024-06
- Volume
- 6
- Issue
- 6
- Pages
- e379-e381
- ISSN
- 2589-7500 (ISSN-L)
- PMC
- PMC11829255
- OpenAlex
- W4395067201
- Cited by
- 147
- References
- 9
- Citations by year
- 2026: 58 · 2025: 65 · 2024: 23
- FWCI
- 13.65
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- creativecommons.org/licenses/by/4.0/ (vor)
- Locations
- The Lancet Digital Health (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion, cc-by)
- Research Publications (Maastricht University) (submittedVersion, cc-by)
- Funders
- European Research Council
- National Institutes of Health
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- large language models, patient portal messaging, chatbots, clinician burnout, administrative burden, electronic health records, GPT-4, ChatGPT
- MeSH terms
- Humans, Language
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The effect of using a large language model to respond to patient messages. https://app.cochat.ai/papers/10.1016/s2589-7500(24)00060-8
- The paper
- Chen, S., Guevara, M., Moningi, S., Hoebers, F., Elhalawani, H., Kann, B. H., Chipidza, F. E., Leeman, J., Aerts, H. J. W. L., Miller, T., Savova, G. K., Gallifant, J., Celi, L. A., Mak, R. H., Lustberg, M., Afshar, M., & Bitterman, D. S. (2024). The effect of using a large language model to respond to patient messages. The Lancet Digital Health. https://doi.org/10.1016/s2589-7500(24)00060-8
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.