CoChat

Leveraging large language models for generating responses to patient messages—a subjective analysis

Open access
Liu, Siru, McCoy, Allison B, Wright, Aileen P, Carew, Babatunde, Genkins, Julian Z, Huang, Sean S, Peterson, Josh F, Steitz, Bryan, Wright, Adam · 2024 · Journal of the American Medical Informatics Association
Cited by 86

The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.

Abstract

Objective: This study aimed to develop and assess the performance of fine-tuned large language models for generating responses to patient messages sent via an electronic health record patient portal. Materials and Methods: Utilizing a dataset of messages and responses extracted from the patient portal at a large academic medical center, we developed a model (CLAIR-Short) based on a pre-trained large language model (LLaMA-65B). In addition, we used the OpenAI API to update physician responses from an open-source dataset into a format with informative paragraphs that offered patient education while emphasizing empathy and professionalism. By combining with this dataset, we further fine-tuned our model (CLAIR-Long). To evaluate fine-tuned models, we used 10 representative patient portal questions in primary care to generate responses. We asked primary care physicians to review generated responses from our models and ChatGPT and rated them for empathy, responsiveness, accuracy, and usefulness. Results: The dataset consisted of 499 794 pairs of patient messages and corresponding responses from the patient portal, with 5000 patient messages and ChatGPT-updated responses from an online platform. Four primary care physicians participated in the survey. CLAIR-Short exhibited the ability to generate concise responses similar to provider’s responses. CLAIR-Long responses provided increased patient educational content compared to CLAIR-Short and were rated similarly to ChatGPT’s responses, receiving positive evaluations for responsiveness, empathy, and accuracy, while receiving a neutral rating for usefulness. Conclusion: This subjective analysis suggests that leveraging large language models to generate responses to patient messages demonstrates significant potential in facilitating communication between patients and healthcare providers.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/jamia/ocae052

Cites

18 of 18 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 18 references

Research works printed but not linked: 14

  1. Sorace, 2020, Quantifying the competitiveness of the electronic health record market and its implications for interoperability, Int J Med Inform, 136, 104037, 10.1016/j.ijmedinf.2019.104037 · DOI
  2. Tarver, 2018, Growth dynamics of patient-provider internet communication: trend analysis using the health information national trends survey (2003 to 2013), J Med Internet Res, 20, e109, 10.2196/jmir.7851 · DOI
  3. Akbar, 2021, Physicians’ electronic inbox work patterns and factors associated with high inbox work duration, J Am Med Inform Assoc, 28, 923, 10.1093/jamia/ocaa229 · DOI · Paper page
  4. Steitz, 2021, Association of immediate release of test results to patients with implications for clinical workflow, JAMA Netw Open, 4, e2129553, 10.1001/jamanetworkopen.2021.29553 · DOI · Paper page
  5. Sinsky, 2022, The electronic health record inbox: recommendations for relief, J Gen Intern Med, 37, 4002, 10.1007/s11606-022-07766-0 · DOI · Paper page
  6. Holmgren, 2022, Assessing the impact of the COVID-19 pandemic on clinician ambulatory electronic health record use, J Am Med Inform Assoc, 29, 453, 10.1093/jamia/ocab268 · DOI · Paper page
  7. Lieu, 2019, Primary care physicians’ experiences with and strategies for managing electronic messages, JAMA Netw Open, 2, e1918287, 10.1001/jamanetworkopen.2019.18287 · DOI · Paper page
  8. Adler-Milstein, 2020, Electronic health records and burnout: time spent on the electronic health record after hours and message volume associated with exhaustion but not with cynicism among primary care clinicians, J Am Med Inform Assoc, 27, 531, 10.1093/jamia/ocz220 · DOI · Paper page
  9. Kumah-Crystal, 2023, ChatGPT and the clinical informatics board examination: the end of unproctored maintenance of certification?, J Am Med Inform Assoc, 30, 1558, 10.1093/jamia/ocad104 · DOI · Paper page
  10. Cascella, 2023, Evaluating the feasibility of chatgpt in healthcare: an analysis of multiple clinical and research scenarios, J Med Syst, 47, 33, 10.1007/s10916-023-01925-4 · DOI · Paper page
  11. Friedman, 2009, A ‘Fundamental Theorem’ of biomedical informatics, J Am Med Inform Assoc, 16, 169, 10.1197/jamia.M3092 · DOI · Paper page
  12. Chambon, 2023, Automated deidentification of radiology reports combining transformer and “hide in plain sight” rule-based methods, J Am Med Inform Assoc, 30, 318, 10.1093/jamia/ocac219 · DOI · Paper page
  13. Heisey-Grove, 2020, Look who’s talking: application of a theory-based taxonomy to patient–clinician E-mail messages, Telemed e-Health, 26, 1345, 10.1089/tmj.2019.0192 · DOI
  14. Harris, 2009, Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support, J Biomed Inform, 42, 377, 10.1016/j.jbi.2008.08.010 · DOI · Paper page

Cited by

See all 86

Metadata

Authors
Liu, S., McCoy, A. B., Wright, A. P., Carew, B., Genkins, J. Z., Huang, S. S., Peterson, J. F., Steitz, B., & Wright, A.
Venue
Journal of the American Medical Informatics Association
Published
2024-03-18
Volume
31
Issue
6
Pages
1367-1379
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
OpenAlex
W4392928544
Cited by
86
References
18
FWCI
6.89
Type
journal article
Language
en
Open access
hybrid
Funders
NIH (R00LM014097-01, R01AG062499-01, R01LM013995-01)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Leveraging large language models for generating responses to patient messages—a subjective analysis. https://app.cochat.ai/papers/10.1093/jamia/ocae052
The paper
Liu, S., McCoy, A. B., Wright, A. P., Carew, B., Genkins, J. Z., Huang, S. S., Peterson, J. F., Steitz, B., & Wright, A. (2024). Leveraging large language models for generating responses to patient messages—a subjective analysis. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocae052

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.