Cited by 306Artificial Intelligence in Healthcare and EducationHealth Literacy and Information AccessibilityText Readability and Simplification

Generative Artificial Intelligence to Transform Inpatient Discharge Summaries to Patient-Friendly Language and Format

Open access
Zaretsky, Jonah, Kim, Jeong Min, Baskharoun, Samuel, Zhao, Yunan, Austrian, Jonathan, Aphinyanaphongs, Yindalon, Gupta, Ravi, Blecker, Saul B. +1 more·2024·JAMA Network Open

Abstract

Importance: By law, patients have immediate access to discharge notes in their medical records. Technical language and abbreviations make notes difficult to read and understand for a typical patient. Large language models (LLMs [eg, GPT-4]) have the potential to transform these notes into patient-friendly language and format. Objective: To determine whether an LLM can transform discharge summaries into a format that is more readable and understandable. Design, Setting, and Participants: This cross-sectional study evaluated a sample of the discharge summaries of adult patients discharged from the General Internal Medicine service at NYU (New York University) Langone Health from June 1 to 30, 2023. Patients discharged as deceased were excluded. All discharge summaries were processed by the LLM between July 26 and August 5, 2023. Interventions: A secure Health Insurance Portability and Accountability Act-compliant platform, Microsoft Azure OpenAI, was used to transform these discharge summaries into a patient-friendly format between July 26 and August 5, 2023. Main Outcomes and Measures: Outcomes included readability as measured by Flesch-Kincaid Grade Level and understandability using Patient Education Materials Assessment Tool (PEMAT) scores. Readability and understandability of the original discharge summaries were compared with the transformed, patient-friendly discharge summaries created through the LLM. As balancing metrics, accuracy and completeness of the patient-friendly version were measured. Results: Discharge summaries of 50 patients (31 female [62.0%] and 19 male [38.0%]) were included. The median patient age was 65.5 (IQR, 59.0-77.5) years. Mean (SD) Flesch-Kincaid Grade Level was significantly lower in the patient-friendly discharge summaries (6.2 [0.5] vs 11.0 [1.5]; P < .001). PEMAT understandability scores were significantly higher for patient-friendly discharge summaries (81% vs 13%; P < .001). Two physicians reviewed each patient-friendly discharge summary for accuracy on a 6-point scale, with 54 of 100 reviews (54.0%) giving the best possible rating of 6. Summaries were rated entirely complete in 56 reviews (56.0%). Eighteen reviews noted safety concerns, mostly involving omissions, but also several inaccurate statements (termed hallucinations). Conclusions and Relevance: The findings of this cross-sectional study of 50 discharge summaries suggest that LLMs can be used to translate discharge summaries into patient-friendly language and formats that are significantly more readable and understandable than discharge summaries as they appear in electronic health records. However, implementation will require improvements in accuracy, completeness, and safety. Given the safety concerns, initial implementation will require physician review.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1001/jamanetworkopen.2024.0357

Cites

34 of 34 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Variability in LLM output quality, including hallucinations and incomplete contextualization, necessitates human oversight.

    “However, variability in output quality, including hallucinations and incomplete contextualization, necessitates human oversight.42”

    From Artificial Intelligence in Oncology: Practical Applications Across Clinical Care, Scholarship, and Translation. (2026) · cites this paper as [42]

    The Results report variable accuracy, hallucinations, omissions, and potential safety risks; Table 2 also gives an example of a summary that misleadingly described the patient's condition. The Methods say physicians reviewed the generated summaries, and the Discussion says accuracy and completeness must be addressed before deployment. The paper does not explicitly conclude that ongoing human oversight is necessary, so that prescription goes beyond its findings. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

2 of the 36 references couldn’t be matched to a known paper.

Research works printed but not linked: 1

  1. [31]Weiss BD. Health Literacy: A Manual for Clinicians. American Medical Association Foundation and American Medical Association; 2003: 51.

Cited by

Sorted by citations

Authors
Zaretsky, J., Kim, J. M., Baskharoun, S., Zhao, Y., Austrian, J., Aphinyanaphongs, Y., Gupta, R., Blecker, S. B., & Feldman, J.
Affiliations
  • NYU Langone Health (US)
  • NYU Grossman Long Island School of Medicine (US)
Corresponding author
Zaretsky, Jonah
Venue
JAMA Network Open
Published
2024-03-11
Volume
7
Issue
3
Pages
e240357
ISSN
2574-3805 (ISSN-L)
OpenAlex
W4392643126
Cited by
303
References
36
Citations by year
2026: 131 · 2025: 136 · 2024: 36
FWCI
28.33
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JAMA Network Open (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › General Health Professions › Health Literacy and Information Accessibility
  • Physical Sciences › Computer Science › Artificial Intelligence › Text Readability and Simplification
Keywords
discharge summaries, large language models, generative AI, readability, comprehensibility, patient education materials, clinical documentation, GPT-4, hallucinations, general internal medicine, patient access to medical records, health information literacy, Flesch-Kincaid Grade Level, PEMAT, natural language processing of clinical notes
MeSH terms
Adult, Aged, Artificial Intelligence, Cross-Sectional Studies, Female, Humans, Inpatients, Language, Male, Middle Aged, Patient Discharge, United States, Electronic Health Records

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review