CoChat
Sign Up
PreprintCited by 10Artificial Intelligence in Healthcare and EducationMachine Learning in HealthcareExplainable Artificial Intelligence (XAI)

Agentic LLM Workflows for Generating Patient-Friendly Medical Reports

Open access
Sudarshan, Malavikha, Shih, Sophie, Yee, Estella, Yang, Alina, Zou, John, Chen, Cathy, Zhou, Quan, Chen, Leon +2 more·2024·arXiv
DOI 10.48550/arxiv.2408.01112arXiv 2408.01112
Open this paper in CoChat — free accountOpen PDF

Abstract

The application of Large Language Models (LLMs) in healthcare is expanding rapidly, with one potential use case being the translation of formal medical reports into patient-legible equivalents. Currently, LLM outputs often need to be edited and evaluated by a human to ensure both factual accuracy and comprehensibility, and this is true for the above use case. We aim to minimize this step by proposing an agentic workflow with the Reflexion framework, which uses iterative self-reflection to correct outputs from an LLM. This pipeline was tested and compared to zero-shot prompting on 16 randomized radiology reports. In our multi-agent approach, reports had an accuracy rate of 94.94% when looking at verification of ICD-10 codes, compared to zero-shot prompted reports, which had an accuracy rate of 68.23%. Additionally, 81.25% of the final reflected reports required no corrections for accuracy or readability, while only 25% of zero-shot prompted reports met these criteria without needing modifications. These results indicate that our approach presents a feasible method for communicating clinical findings to patients in a quick, efficient and coherent manner whilst also retaining medical accuracy. The codebase is available for viewing at http://github.com/malavikhasudarshan/Multi-Agent-Patient-Letter-Generation.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper proposes an agentic Reflexion workflow to generate patient-friendly radiology reports and compares it with zero-shot prompting on 16 radiology reports. It reports higher ICD-10 accuracy and fewer required edits than zero-shot prompting.

Key findings

  • In the multi-agent approach, ICD-10 code verification accuracy was 94.94%, compared with 68.23% for zero-shot prompted reports.
  • 81.25% of final reflected reports required no corrections for accuracy or readability, while only 25% of zero-shot prompted reports met those criteria without needing modifications.
  • From 16 test radiology reports, zero-shot prompting led to 11/16 patient-friendly versions needing edits, while the agentic workflow resulted in only 3/16 requiring modification; the conclusion frames this as a reduction from 68.75% to 18.75%.
  • On average, accuracy was 26.71% better, readability scored 3.29% higher, and overall_score increased 17.51% in reflected letters compared with zero-shot letters.

Methods

  • Design: an agentic workflow using the Reflexion framework with iterative self-reflection, compared against zero-shot prompting. The pipeline extracts ICD-10 codes from the original report, generates patient-friendly letters (e.g., n=5), verifies ICD-10 codes against the simple-icd-10 database, calculates accuracy as validated identical ICD-10 codes divided by total original ICD-10 codes, quantifies readability using Flesch-Kincaid Grade Level, and computes overall_score = (readability * 0.3) + (accuracy * 0.7); Reflexion’s AlfWorld module iterates to optimize the score. The LLM used was OpenAI’s GPT-4o (gpt-4o-2024-05-13).
  • Data/sample: 16 test radiology reports of various modalities—CT, MR, and Ultrasound—ranging from 84 to 264 words and covering abdomen, pelvis, chest, head, and lumbar spine. One report was extracted from the SIIM FHIR server and results were pushed back; 15 other test reports were manually included in the input.
  • Analysis: reflected outputs were compared with zero-shot outputs on accuracy, readability, overall_score, and need for modification, where modification included changes in medical factuality (including ICD-10 codes), grammar, punctuation, tonality, and readability.

Limitations

  • Stated: ICD-10 description verification uses exact string matching, so synonymous definitions or minor word variants may be missed; fuzzy matching, Levenshtein distance, or K-Nearest Neighbors are suggested alternatives. The study assumes LLM-generated ICD-10 codes from GPT-4o are accurate and can be trusted, based on separate human validation tests. The authors call it an early prototype, note it does not have a 100% success rate and cannot eradicate verification, and list future needs for varied reading levels, languages, and medical fields; readability is currently standardized to a 6th-grade level, and the weighting system includes only readability and accuracy, not compassion (PERMA and CDE codes are considered future additions).
  • Evident from the text: the evaluation uses a small sample of 16 reports and a single LLM, which limits generalizability; the paper does not report patient comprehension or full clinical deployment outcomes.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2408.01112

Cites

28 of 28 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 18 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    In the multi-agent workflow, ICD-10 code verification accuracy was 94.94%, compared with 68.23% for zero-shot prompting.

    “In our multi-agent approach, reports had an accuracy rate of 94.94% when looking at verification of ICD-10 codes, compared to zero-shot prompted reports, which had an accuracy rate of 68.23%.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The ICD-10 codes generated by GPT-4o are accurate.
  • Findingown data

    Overall_score was 17.51% higher in reflected letters than in zero-shot generated letters.

    “This resulted in a 17.51% increase in overall_score in reflected letters vs zero-shot generated letters.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The ICD-10 codes generated by GPT-4o are accurate.
  • Findingown data

    The reflection agent improved medical accuracy by retaining ICD-10 codes in the final patient letter that zero-shot output sometimes omitted.

    “Our reflection agent increased the medical accuracy of reports, ensuring that ICD-10 codes were retained in the final patient letter which the zero-shot output sometimes missed.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The ICD-10 codes generated by GPT-4o are accurate.
  • Premisecitation

    For patients who receive abnormal or ambiguous radiology results before discussing them with a physician, the effect can be more harmful than helpful.

    “when abnormal or ambiguous results are received by patients before discussion with their physician, the impact can often be more harmful than beneficial [3].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [3]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Premisecitation

    The average American reads at a US 8th-grade level.

    “The average American's reading ability is equivalent to a US 8th-grade level [18].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [18]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims. 19
  • Classified as probably not a substantial claim. 4

  • May combine two claims.

    Findingown data

    The final reflected reports needed no corrections for accuracy or readability in 81.25% of cases, while zero-shot reports met that standard in 25% of cases.

    “Additionally, 81.25% of the final reflected reports required no corrections for accuracy or readability, while only 25% of zero-shot prompted reports met these criteria without needing modifications.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The ICD-10 codes generated by GPT-4o are accurate.

  • May combine two claims.

    Findingown data

    On average, reflected letters had 26.71% better accuracy and 3.29% higher readability than zero-shot letters.

    “On average, accuracy was 26.71% better, and readability scored 3.29% higher in the reflected patient letters, compared to the zero-shot letters (Fig. 4).”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The ICD-10 codes generated by GPT-4o are accurate.

  • May combine two claims.

    Findingown data

    In that example, the reflection agent's ICD-10 codes exactly matched the original medical report, whereas the zero-shot codes did not.

    “Additionally, the ICD-10 codes generated by the reflection agent precisely matched those from the original medical report, while the codes from the zero-shot report did not.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: The ICD-10 codes generated by GPT-4o are accurate.

  • May combine two claims.

    Findingown data

    Zero-shot prompts were sometimes unprofessional and used language that was too juvenile even when the prompt specified a US 6th-grade reading level.

    “Zero-shot prompts were sometimes not professional enough, and even when specified in the prompt that the reading level should match that of a US 6th grader's, the language used was too juvenile.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Among 16 test radiology reports, zero-shot prompting led to 11/16 patient-friendly versions needing edits, while the agentic workflow led to only 3/16 requiring modification.

    “From 16 test radiology reports, zero-shot prompting (using the same original prompt as given in our multi-agent workflow) led to 11/16 patient-friendly versions needing to be edited, whilst our agentic workflow resulted in only 3/16 reports that required modification.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The ICD-10 codes generated by GPT-4o are accurate.

  • May combine two claims.

    Findingown data

    With the same medical report, system prompt, and user prompt, the reflected output consistently scored higher on accuracy, readability, and overall_score.

    “When given an identical medical report, system prompt and user prompt, the reflected output consistently scored higher in terms of accuracy and readability, as well as in the overall_score measure.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The ICD-10 codes generated by GPT-4o are accurate.

  • May combine two claims.

    Findingown data

    In one example, zero-shot prompting registered only half of the desired ICD-10 codes, while the reflection agent generated all 4 ICD-10 codes.

    “With zero-shot prompting, only half of the desired ICD-10 codes were registered, whereas the reflection agent successfully generated all 4 ICD-10 codes.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The ICD-10 codes generated by GPT-4o are accurate.

  • May combine two claims.

    Findingown data

    Reflected agent outputs were consistently more concise, structured, and formal.

    “However, when the reflected agent was used, the final outputs seemed to be consistently more concise, structured and formal.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The 16 test reports had an average Flesch-Kincaid Grade Level of 11.03, equivalent to an 11th-grader's expected vocabulary.

    “Our 16 test reports had an average Flesch-Kincaid Grade Level of 11.03, corresponding to an 11th-grader's expected level of vocabulary.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Patient-friendly letters that simplify complex medical information have been explored as a response to this problem.

    “To address this, the creation of patient-friendly letters that simplify complex medical information has been explored [4].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [4]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    Using generative AI to create patient-friendly reports can make the process more efficient and give patients more meaningful, legible letters.

    “In applying generative artificial intelligence to the creation of patient-friendly reports, the pipeline can be made more efficient and patients can have access to more meaningful and legible letters [7,8].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [7]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [8]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    Agentic workflows permit reinforcement learning through reflection and can use chain-of-thought prompting by appending reflected feedback to the next prompt.

    “Agentic workflows allow for reinforcement learning through reflection [12], and can utilize chainof-thought prompting by appending reflected feedback at the end of the next prompt.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [12]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    A prior study of 97,052 radiology reports found that only 4% were at or below a US 8th-grade reading level, suggesting much of the information may be unintelligible to a significant portion of the population.

    “A previous study examining 97,052 radiology reports revealed that only 4% were at a reading level equal to or below the reading ability of a US 8th-grader [19], suggesting that much of this information may be unintelligible to a significant portion of the population.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [19]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The temperature was set to 0 to reduce variance.

    “The temperature is kept at 0 to minimize variance, and these codes are stored to be compared later.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Agentic workflows are iterative and include several intermediary steps beyond LLM prompting, unlike non-agentic or zero-shot/few-shot prompts that have a single input and output.

    “Agentic workflows are iterative and consist of several intermediary steps performed in addition to LLM prompting, as opposed to non-agentic or zero-shot/few-shot prompts which consist of a single input and a single output [12].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [12]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductcitation

    The ICD-10 codes were checked against the master ICD-10 code database using the simple-icd-10 package, and each code's description was retrieved and compared with the LLM's output.

    “These ICD-10 codes are verified against the master ICD-10 code database (using the simple-icd-10 package [15]) and the description for each code is also retrieved and compared against the LLM's output to see if they match.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [15]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    Greater patient access to jargon-filled medical reports can cause anxiety, misinterpretation, and confusion.

    “this increased level of accessibility can often lead to patient anxiety, misinterpretation and confusion when reading jargon-filled medical reports that they are not the primary audience for [2].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [2]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    The agentic workflow is a feasible way to communicate clinical findings to patients quickly, efficiently, and coherently while preserving medical accuracy.

    “These results indicate that our approach presents a feasible method for communicating clinical findings to patients in a quick, efficient and coherent manner whilst also retaining medical accuracy.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The ICD-10 codes generated by GPT-4o are accurate.

  • May combine two claims.

    Interpretationown data

    Using a multi-agent workflow reduced the share of LLM-generated reports requiring edits from 68.75% to 18.75%, which should substantially decrease time spent correcting medical accuracy and readability.

    “However, by significantly reducing the percentage of LLM-generated reports requiring edits-from 68.75% to 18.75%-through the incorporation of a multi-agent workflow, we can show that the time spent making changes in medical accuracy and readability will also be substantially decreased.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The ICD-10 codes generated by GPT-4o are accurate.

  • May combine two claims.

    Interpretationown data

    The approach significantly improves the quality of patient-legible letters but does not achieve 100% success and cannot completely eliminate the need for verification.

    “Although our approach does significantly improve the quality of patient-legible letters, it does not have an 100% success rate, and therefore cannot eradicate the need for verification completely.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The ICD-10 codes generated by GPT-4o are accurate.

  • May combine two claims.

    Interpretationown data

    The development has strong potential to streamline clinical workflows, reduce burdens on medical professionals and administrators, and improve patient experience by providing clearer, more accurate medical information.

    “This development has strong potential to streamline clinical workflows, lessen the burden on medical professionals and administrators, and improve the patient experience by swiftly providing clearer and more accurate medical information.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: The ICD-10 codes generated by GPT-4o are accurate.

  • May combine two claims.

    Interpretationown data

    The method improves report-generation efficiency and helps make healthcare information more accessible and understandable to patients.

    “Our method not only enhances the efficiency of report generation but also contributes to the overall goal of making healthcare information more accessible and understandable to patients.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The ICD-10 codes generated by GPT-4o are accurate.
22 of 22

  • The authors state this as an assumption.

    The ICD-10 codes generated by GPT-4o are accurate.

    “One assumption we make is the accuracy of the ICD-10 codes generated by the LLM model (GPT-4o).”
1 of 1

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Conflicting counts of unedited zero-shot reports

    Major

    The abstract says 25% of the 16 zero-shot reports needed no modifications, or 4 reports. The results say 11 of 16 needed editing, leaving 5 reports (31.25%) that needed none. Because the results use a broader definition of editing, they cannot leave more reports unedited than the abstract does.

1 of 1

Cites

Flagged first, then by citations

4 of the 32 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Related

Authors
Sudarshan, M., Shih, S., Yee, E., Yang, A., Zou, J., Chen, C., Zhou, Q., Chen, L., Singhal, C., & Shih, G.
Affiliations
  • University of California, Berkeley (US)
  • John Brown University (US)
  • New York University (US)
  • Weill Cornell Medicine (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2408.01112
arXiv
2408.01112
OpenAlex
W4406443929
Semantic Scholar
38b2753ffea03020feceb86d23c1c17a12f1d29c
Cited by
8
References
32
Citations by year
2026: 1 · 2025: 6 · 2024: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords
large language models, agentic workflows, self-reflection, multi-agent systems, radiology reports, readability, clinical communication, zero-shot prompting

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review