Leveraging large language models for generating responses to patient messages—a subjective analysis
Open accessThe DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Abstract
Objective: This study aimed to develop and assess the performance of fine-tuned large language models for generating responses to patient messages sent via an electronic health record patient portal. Materials and Methods: Utilizing a dataset of messages and responses extracted from the patient portal at a large academic medical center, we developed a model (CLAIR-Short) based on a pre-trained large language model (LLaMA-65B). In addition, we used the OpenAI API to update physician responses from an open-source dataset into a format with informative paragraphs that offered patient education while emphasizing empathy and professionalism. By combining with this dataset, we further fine-tuned our model (CLAIR-Long). To evaluate fine-tuned models, we used 10 representative patient portal questions in primary care to generate responses. We asked primary care physicians to review generated responses from our models and ChatGPT and rated them for empathy, responsiveness, accuracy, and usefulness. Results: The dataset consisted of 499 794 pairs of patient messages and corresponding responses from the patient portal, with 5000 patient messages and ChatGPT-updated responses from an online platform. Four primary care physicians participated in the survey. CLAIR-Short exhibited the ability to generate concise responses similar to provider’s responses. CLAIR-Long responses provided increased patient educational content compared to CLAIR-Short and were rated similarly to ChatGPT’s responses, receiving positive evaluations for responsiveness, empathy, and accuracy, while receiving a neutral rating for usefulness. Conclusion: This subjective analysis suggests that leveraging large language models to generate responses to patient messages demonstrates significant potential in facilitating communication between patients and healthcare providers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A guideline of selecting and reporting intraclass correlation coefficients for reliability research
Koo, T. K., & Li, M. Y. (2016) · Journal of Chiropractic Medicine · doi:10.1016/j.jcm.2016.02.012 · Cited by 31,709
ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education
Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · doi:10.35542/osf.io/5er8f · Cited by 7,552
ChatGPT for good? On opportunities and challenges of large language models for education
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · Learning and Individual Differences · doi:10.1016/j.lindif.2023.102274 · Cited by 7,326 · cited 1×
“10 14–16 In particular, a recent study used ChatGPT to generate responses to 195 patient questions from social media forums.”
BackgroundLLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · cited 1×
“10 14–16 In particular, a recent study used ChatGPT to generate responses to 195 patient questions from social media forums.”
BackgroundKung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613 · cited 1×
“17 This study used an “out of the box” version of ChatGPT, but it is possible to further optimize large language models’ performance on specialized domains by fine-tuning them for specific tasks.”
MethodRecent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey
Min, B., Ross, H., Sulem, E., Veyseh, A. P. B., Nguyen, T. H., Sainz, O., Agirre, E., Heintz, I., & Roth, D. (2024) · ACM Computing Surveys · doi:10.1145/3605943 · Cited by 1,387 · cited 1×
“18 For instance, a recent study utilized 100 000 patient-doctor online conversations to fine-tune the open-source Large Language Model Meta AI (LLaMA)-7B model, which showed improved performance in similarity metric (e.g. BERTScore) in comparison to ChatGPT when answering patient questions.”
BackgroundArndt, B. G., Beasley, J. W., Watkinson, M. D., Temte, J. L., Tuan, W.-J., Sinsky, C. A., & Gilchrist, V. J. (2017) · The Annals of Family Medicine · doi:10.1370/afm.2121 · Cited by 972
Research works printed but not linked: 14
- Sorace, 2020, Quantifying the competitiveness of the electronic health record market and its implications for interoperability, Int J Med Inform, 136, 104037, 10.1016/j.ijmedinf.2019.104037 · DOI
- Tarver, 2018, Growth dynamics of patient-provider internet communication: trend analysis using the health information national trends survey (2003 to 2013), J Med Internet Res, 20, e109, 10.2196/jmir.7851 · DOI
- Akbar, 2021, Physicians’ electronic inbox work patterns and factors associated with high inbox work duration, J Am Med Inform Assoc, 28, 923, 10.1093/jamia/ocaa229 · DOI · Paper page
- Steitz, 2021, Association of immediate release of test results to patients with implications for clinical workflow, JAMA Netw Open, 4, e2129553, 10.1001/jamanetworkopen.2021.29553 · DOI · Paper page
- Sinsky, 2022, The electronic health record inbox: recommendations for relief, J Gen Intern Med, 37, 4002, 10.1007/s11606-022-07766-0 · DOI · Paper page
- Holmgren, 2022, Assessing the impact of the COVID-19 pandemic on clinician ambulatory electronic health record use, J Am Med Inform Assoc, 29, 453, 10.1093/jamia/ocab268 · DOI · Paper page
- Lieu, 2019, Primary care physicians’ experiences with and strategies for managing electronic messages, JAMA Netw Open, 2, e1918287, 10.1001/jamanetworkopen.2019.18287 · DOI · Paper page
- Adler-Milstein, 2020, Electronic health records and burnout: time spent on the electronic health record after hours and message volume associated with exhaustion but not with cynicism among primary care clinicians, J Am Med Inform Assoc, 27, 531, 10.1093/jamia/ocz220 · DOI · Paper page
- Kumah-Crystal, 2023, ChatGPT and the clinical informatics board examination: the end of unproctored maintenance of certification?, J Am Med Inform Assoc, 30, 1558, 10.1093/jamia/ocad104 · DOI · Paper page
- Cascella, 2023, Evaluating the feasibility of chatgpt in healthcare: an analysis of multiple clinical and research scenarios, J Med Syst, 47, 33, 10.1007/s10916-023-01925-4 · DOI · Paper page
- Friedman, 2009, A ‘Fundamental Theorem’ of biomedical informatics, J Am Med Inform Assoc, 16, 169, 10.1197/jamia.M3092 · DOI · Paper page
- Chambon, 2023, Automated deidentification of radiology reports combining transformer and “hide in plain sight” rule-based methods, J Am Med Inform Assoc, 30, 318, 10.1093/jamia/ocac219 · DOI · Paper page
- Heisey-Grove, 2020, Look who’s talking: application of a theory-based taxonomy to patient–clinician E-mail messages, Telemed e-Health, 26, 1345, 10.1089/tmj.2019.0192 · DOI
- Harris, 2009, Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support, J Biomed Inform, 42, 377, 10.1016/j.jbi.2008.08.010 · DOI · Paper page
Cited by
Current applications and challenges in large language models for patient care: a systematic review
Busch, F., Hoffmann, L., Rueger, C., van Dijk, E. H., Kader, R., Ortiz-Prado, E., Makowski, M. R., Saba, L., Hadamitzky, M., Kather, J. N., et al. (2025) · Communications Medicine · doi:10.1038/s43856-024-00717-2 · Cited by 358
Large language models in patient education: a scoping review of applications in medicine
Aydin, S., Karabacak, M., Vlachos, V., & Margetis, K. (2024) · Frontiers in Medicine · doi:10.3389/fmed.2024.1477898 · Cited by 288
Moulaei, K., Yadegari, A., Baharestani, M., Farzanbakhsh, S., Sabet, B., & Reza Afrash, M. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105474 · Cited by 244
Large language models for structured reporting in radiology: past, present, and future
Busch, F., Hoffmann, L., dos Santos, D. P., Makowski, M. R., Saba, L., Prucker, P., Hadamitzky, M., Navab, N., Kather, J. N., Truhn, D., et al. (2024) · European Radiology · doi:10.1007/s00330-024-11107-6 · Cited by 134
“Likewise, several studies have noted the provision of outdated medical information [47, 48].”
Luo, X., Tham, Y. C., Giuffrè, M., Ranisch, R., Daher, M., Lam, K., Eriksen, A. V., Hsu, C.-W., Ozaki, A., Moraes, F. Y. D., et al. (2025) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2025-113825 · Cited by 97
Wu, Y., Wu, M., Wang, C., Lin, J., Liu, J., & Liu, S. (2024) · JMIR Medical Informatics · doi:10.2196/54811 · Cited by 87
“Innovations in machine learning, natural language processing (NLP), and large language models (LLMs) are poised to significantly increase the intelligence and automation capabilities of EHR systems [73,74].”
BackgroundSelf-citationPrompt Engineering in Clinical Practice: Tutorial for Clinicians
Liu, J., Liu, F., Wang, C., & Liu, S. (2025) · Journal of Medical Internet Research · doi:10.2196/72644 · Cited by 76
Self-citationYousefi, F., Dehnavieh, R., Laberge, M., Gagnon, M.-P., Ghaemi, M. M., Nadali, M., & Azizi, N. (2025) · BMC Primary Care · doi:10.1186/s12875-025-02785-2 · Cited by 70
Liu, S., Wright, A. P., McCoy, A. B., Huang, S. S., Steitz, B., & Wright, A. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf059 · Cited by 44
Self-citationIannone, S., Kaur, A., & Johnson, K. B. (2026) · Journal of General Internal Medicine · doi:10.1007/s11606-025-09938-0 · Cited by 19
Meyer, B., Kfuri‐Rubens, R., Schmidt, G., Tariq, M., Riedel, C., Recker, F., Riedel, F., Kiechle, M., & Riedel, M. (2025) · International Journal of Gynecology & Obstetrics · doi:10.1002/ijgo.70251 · Cited by 10
“Several studies have already highlighted the potential of LLMs in patient education, such as answering individual patient questions, 21 improving educational materials, 22 and translating medical information into more accessible language for lay audiences.”
Metadata
- Authors
- Liu, S., McCoy, A. B., Wright, A. P., Carew, B., Genkins, J. Z., Huang, S. S., Peterson, J. F., Steitz, B., & Wright, A.
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2024-03-18
- Volume
- 31
- Issue
- 6
- Pages
- 1367-1379
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- OpenAlex
- W4392928544
- Cited by
- 86
- References
- 18
- FWCI
- 6.89
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Funders
- NIH (R00LM014097-01, R01AG062499-01, R01LM013995-01)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.