The DOI matches its Crossref record; 91 of the 91 references it cites were found in a registry or an index; 37 of the 61 sources its claims cite were checked for support.
Adapted large language models can outperform medical experts in clinical text summarization
Open accessPublished version of a preprint
Abstract
Adapted LLMs can perform text summarization of clinical notes such as radiology reports, progress notes, and discharge summaries, sometimes outperforming medical experts on certain criteria.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
40 Own result · 27 Own interpretation · 20 Own method · 13 Partly supported · 7 No basis given · 6 No full text of the sources · 5 Supported · 1 Supported so far · 1 Couldn’t confirm · 1 Sources not found · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Although LLM benchmarks for general natural language processing (NLP) tasks exist20,21, they do not evaluate performance on relevant clinical tasks.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
What the sources say
- Not matchedcites [20] Liang, P. et al. Holistic evaluation of language models. Transact. Mach. Learn. Res. (in the press).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites [21] Zheng, L. et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Preprint at https://doi.org/10.48550/arXiv.2306.05685 (2023). source
The citation follows the statement that LLM benchmarks for general NLP tasks exist. Section 2.1 (Motivation) lists several such benchmarks, and Section 5 discusses standardized LLM benchmarks. The citation is not attached to the subsequent claim about clinical tasks. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Currently, clinicians spend a substantial amount of time summarizing vast amounts of textual information—whether it be compiling diagnostic reports, writing progress notes or synthesizing a patient’s treatment history across different specialists1–3.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
What the sources say
- Not checked — no full text of the sourcecites [1] Golob, J. F. Jr, Como, J. J. & Claridge, J. A. The painful truth: the documentation burden of a trauma surgeon. J. Trauma Acute Care Surg. 80, 742–747 (2016). source
- Partly supportedcites [2] Arndt, B. G. et al. Tethered to the EHR: primary care physician workload assessment using EHR event log data and time–motion observations. Ann. Fam. Med. 15, 419–426 (2017). sourceExperimental — accuracy not yet measured
The Time Spent on Various Tasks table (p. 4) reports 84 minutes per day on documentation and 60 minutes reviewing notes among the family medicine physicians studied. Study Limitations (p. 6) also describes reviewing prior notes before writing a progress note. The paper does not establish how much of that time was spent summarizing vast amounts of text, compiling diagnostic reports, or synthesizing histories across specialists. (LLM, conf=0.86)
- Supportedcites [3] Fleming, S. L. et al. MedAlign: a clinician-generated dataset for instruction following with electronic medical records. Preprint at https://doi.org/10.48550/arXiv.2308.14089 (2023). source
The introductory passage (section marked “+”; page not specified) describes clinicians’ information-related tasks as a significant burden, citing 10 hours a week spent generating documentation. It specifically gives summarizing a patient’s treatment history across specialists as an example, supporting the citing sentence’s overall point. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Every source the paper cites for this claim was checked and supports it.
In the paper
“Other studies employ human experts to evaluate synthesized abstracts, demonstrating that NLP metrics are not sufficient to measure summary quality56.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
What the sources say
- Supportedcites [56] Tang, L. et al. Evaluating large language models on medical evidence summarization. NPJ Digit. Med. 6, 158 (2023). source
The Methods section (Design of the human evaluation) says six domain experts evaluated generated summaries. The Discussion says automatic metrics fell short of measuring factual inconsistency, potential medical harmfulness, and human preference, making human evaluation essential. The supplied passages have section labels but no page numbers. (LLM, conf=0.94)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Quantitative assessments with syntactic, semantic and conceptual NLP metrics reveal trade-offs between models and adaptation methods.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A clinical reader study with 10 physicians evaluated summary completeness, correctness and conciseness; in most cases, summaries from our best-adapted LLMs were deemed either equivalent (45%) or superior (36%) compared with summaries from medical experts.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Although FLAN-T5 was more competitive for syntactic metrics, such as BLEU42, this model is constrained to a shorter context length of 512 (Extended Data Table 2).”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Head-to-head model comparison.
What the sources say
- Not checked — no full text of the sourcecites [42] Papineni, K., Roukos, S., Ward, T. & Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proc. of the 40th Annual Meeting of the Association for Computational Linguistics. https://dl.acm.org/doi/pdf/10.3115/1073083.1073135 (Association for Computing Machinery, 2002). source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“When considering conciseness scores (Fig. 4b), radiology reports were the only task in which physicians did not prefer the best model’s summaries to the medical experts.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“FLAN-T5 emerged as the best-performing model with QLoRA.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Comparison of adaptation strategies.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, the metrics BERTScore and MEDCON correlated most strongly with physician preference for correctness.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We see a clear example in Extended Data Fig. 3 where the model mistakenly generated (hallucinated) several absent conditions, such as eosinophilia.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Analyzing reader study results
boostedSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 58
- It may state two separate claims as one. 40
- The quote only partly states this claim. 1
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Large language models (LLMs) have gained remarkable traction, leading to widespread adoption of models such as ChatGPT17, which excel at information retrieval, nuanced understanding and text generation18,19.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
What the sources say
- Partly supportedcites [17] Brown, T. et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33 https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf (NeurIPS, 2020). sourceExperimental — accuracy not yet measured
The Conclusion (p. 40) reports strong performance on many NLP tasks and high-quality text generation, supporting the generation part of the claim. But the paper evaluates GPT-3 rather than ChatGPT; its closed-book question answering does not test information retrieval (Section 3.2, p. 13), and its NLI results describe important limits to language understanding (Section 3.8, p. 20). The separate citation attached to ChatGPT’s adoption is not judged here. (LLM, conf=0.88)
- Supportedcites [18] Zhao, W. X. et al. A survey of large language models. Preprint at https://doi.org/10.48550/arXiv.2303.18223 (2023). source
The Introduction (p. 2) describes ChatGPT’s global impact and LLMs’ integration into search platforms and content-creation tools. The Key concepts section (p. 3) describes their ability to process and produce natural language across a wide range of tasks, supporting the broad characterization of their capabilities. (LLM, conf=0.84)
- Partly supportedcites [19] Bubeck, S. et al. Sparks of artificial general intelligence: early experiments with GPT-4. Preprint at https://doi.org/10.48550/arXiv.2303.12712 (2023). sourceExperimental — accuracy not yet measured
The Introduction describes GPT-4’s comprehension and text-generation capabilities, and §2.1 says ChatGPT shows some understanding. But the paper does not establish that these models excel at information retrieval: its discussion of Figure 5.1 says current knowledge is hard for language models and describes GPT-4 using external search tools to overcome that limitation. The widespread-adoption claim is attached to a different citation (17). (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“These tasks divert attention from direct patient care, leading to worse outcomes for patients and decreased job satisfaction for clinicians2,14–16.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
What the sources say
- Partly supportedcites [2] Arndt, B. G. et al. Tethered to the EHR: primary care physician workload assessment using EHR event log data and time–motion observations. Ann. Fam. Med. 15, 419–426 (2017). sourceExperimental — accuracy not yet measured
The Introduction (p. 1) links documentation and other EHR tasks to less face-to-face patient time and clinician dissatisfaction. The paper does not establish that documentation tasks lead to worse patient outcomes; its discussion of poorer clinical outcomes concerns electronic communication among team members and patients with coronary artery disease (Communication, p. 6). (LLM, conf=0.88)
- Not checked — no full text of the sourcecites [14] Shanafelt, T. D. et al. Relationship between clerical burden and characteristics of the electronic environment with physician burnout and professional satisfaction. Mayo Clin. Proc. 91, 836–848 (2016). source
- Partly supportedcites [15] Robinson, K. E. & Kersey, J. A. Novel electronic health record (EHR) education intervention in large healthcare organization improves quality, efficiency, time, and impact on burnout. Medicine (Baltimore) 97, e12319 (2018). sourceExperimental — accuracy not yet measured
The Introduction (page not provided) says physicians spend substantial time on EHR and desk work relative to direct patient contact, and links EHR documentation to impaired patient communication and increased career dissatisfaction. It does not establish that documentation tasks lead to worse patient outcomes; the Discussion addresses patient care but does not make that causal claim. (LLM, conf=0.87)
- Partly supportedcites [16] Toussaint, W. et al. Design considerations for high impact, automated echocardiogram analysis. Preprint at https://doi.org/10.48550/arXiv.2006.06292 (2020). sourceExperimental — accuracy not yet measured
In Design Discoveries (p. 2), the paper says cardiologists spend 4–5 hours a day annotating echocardiograms and suggests automation could free them to focus more attention on sick patients. It does not characterize this work as documentation or say that it causes worse patient outcomes or decreased clinician job satisfaction. (LLM, conf=0.78)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Similarly, documentation responsibilities for nurses can consume up to 60% of their time and account for considerable work stress11–13.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
What the sources say
- Partly supportedcites [11] Khamisa, N., Peltzer, K. & Oldenburg, B. Burnout in relation to specific contributing factors and health outcomes among nurses: a systematic review. Int. J. Environ. Res. Public Health 10, 2214–2240 (2013). sourceExperimental — accuracy not yet measured
The Results (§3.2–3.3; page unavailable) identify workload as a prominent stressor among nurses, supporting the general point about work stress. They do not identify documentation as the source of that stress or report that it consumes up to 60% of nurses’ time; searches for documentation, paperwork, and the time figure found no such passage. (LLM, conf=0.80)
- Not checked — no full text of the sourcecites [12] Duffy, W. J., Kharasch, M. S. & Du, H. Point of care documentation impact on the nurse–patient interaction. Nurs. Adm. Q. 34, E1–E10 (2010). source
- Not checked — no full text of the sourcecites [13] Chang, C.-P., Lee, T.-T., Liu, C.-H. & Mills, M. E. Nurses’ experiences of an initial and reimplemented electronic health record use. Comput. Inform. Nurs. 34, 183–190 (2016). source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The widespread adoption of electronic health records has expanded clinical documentation workload, directly contributing to increasing stress and clinician burnout7–9.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
What the sources say
- Not matchedcites [7] Gesner, E., Gazarian, P. & Dykes, P. The burden and burnout in documenting patient care: an integrative literature review. Stud. Health Technol. Inform. 21, 1194–1198 (2019).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites [8] Ratwani, R. M. et al. A usability and safety analysis of electronic health records: a multi-center study. J. Am. Med. Inform. Assoc. 25, 1197–1201 (2018). sourceExperimental — accuracy not yet measured
The Background and significance section says EHR usability problems create burdensome workflows and inefficiencies that contribute to clinician frustration. It does not say that widespread EHR adoption increased clinical documentation workload or directly caused increasing stress and burnout. The Discussion examines task time, clicks, and errors, not those broader outcomes; page numbers are unavailable in our copy. (LLM, conf=0.84)
- Not checked — no full text of the sourcecites [9] Ehrenfeld, J. M. & Wanderer, J. P. Technology as friend or foe? Do electronic health records increase burnout? Curr. Opin. Anaesthesiol. 31, 357–360 (2018). source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Many previous clinical NLP studies rely primarily on quantitative metrics for evaluation41,52–54.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
What the sources say
- Partly supportedcites [41] Yim, W.W. et al. Aci-bench: a novel ambient clinical intelligence dataset for benchmarking automatic visit note generation. Sci. Data https://doi.org/10.1038/s41597-023-02487-3 (2023). sourceExperimental — accuracy not yet measured
In Technical Validation > Baseline experiments (page unavailable), the paper calls automatic evaluation metrics “widely-used” and evaluates its own clinical-note generation models with quantitative scores. It does not establish that many previous clinical NLP studies relied primarily on quantitative rather than other forms of evaluation. (LLM, conf=0.82)
- Supportedcites [52] Ma, C. et al. ImpressionGPT: an iterative optimizing framework for radiology report summarization with chatGPT. Preprint at https://doi.org/10.48550/arXiv.2304.08448 (2023). source
The Dataset and Evaluation Metrics section (p. 8) uses ROUGE and factual-consistency scores and notes that earlier studies reported factual-consistency metrics. The Comparison with Other Methods section (p. 9) evaluates numerous earlier radiology-report summarization methods using ROUGE and factual-consistency metrics, supporting the broad statement about quantitative evaluation in prior clinical NLP work. (LLM, conf=0.87)
- Not checked — no full text of the sourcecites [53] Wei, S. et al. Medical question summarization with entity-driven contrastive learning. Preprint at https://doi.org/10.48550/arXiv.2304.07437 (2023). source
- Supportedcites [54] Manakul, P. et al. CUED at ProbSum 2023: Hierarchical ensemble of summarization models. In The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks 516–523 (Association for Computational Linguistics, 2023). source
The Experiments tables report the paper’s clinical summarization results using numerical ROUGE scores, and the Limitations section explicitly identifies ROUGE-L as its assessment metric. This supports citing the paper as an example of a clinical NLP study evaluated primarily with quantitative metrics; the paper alone does not establish how many other studies do so. (LLM, conf=0.87)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Previous work has demonstrated potential across clinical NLP tasks22,23, adapting to the medical domain by training a new model24,25, fine-tuning an existing model26,27 or supplying task-specific examples in the model prompt27,28.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
What the sources say
- Supportedcites [22] Wornow, M. et al. The shaky foundations of large language models and foundation models for electronic health records. NPJ Digit. Med. 6, 135 (2023). sourceExperimental — accuracy not yet measured
The “State of published clinical FMs—CLaMs” section describes models trained on clinical or biomedical text and evaluated on clinical NLP tasks. “Benefits of clinical FMs” describes fine-tuning and few-shot adaptation, while the evaluation discussion contrasts fine-tuning with prompting. The paper cautions that performance on these tasks does not establish clinical deployment value, but the claim asserts only potential. (LLM, conf=0.86)
- Not checked — no full text of the sourcecites [23] Thirunavukarasu, A. J. et al. Large language models in medicine. Nat. Med. 29, 1930–1940 (2023). source
- Partly supportedcites [24] Singhal, K. et al. Large language models encode clinical knowledge. Nature https://doi.org/10.1038/s41586-023-06291-2 (2023). sourceExperimental — accuracy not yet measured
The Methods > Modelling passages describe adapting a model to medical question-answering tasks with task-specific examples in prompts and instruction prompt tuning. They also say the authors used prompt tuning instead of full-model fine-tuning, keeping the model’s parameters frozen. Thus, this source supports the prompting part of the attributed claim, but not the suggestion that this study fine-tuned the existing model itself. (LLM, conf=0.86)
- Supportedcites [25] Tu, T. et al. Towards generalist biomedical AI. Preprint at https://doi.org/10.48550/arXiv.2307.14334 (2023). sourceExperimental — accuracy not yet measured
The citation is attached to fine-tuning an existing model and supplying task-specific examples in prompts, not to training a new model. The 'Dataset and preprocessing' section describes fine-tuning and adding one-shot exemplars to task prompts; '5.1 Evaluation on MultiMedBench' reports evaluation across biomedical tasks, including language-only tasks. No page numbers are available in our copy. (LLM, conf=0.91)
- Supportedcites [26] Toma, A. et al. Clinical Camel: an open-source expert-level medical language model with dialogue-based knowledge encoding. Preprint at https://doi.org/10.48550/arXiv.2305.12031 (2023). sourceExperimental — accuracy not yet measured
The Introduction describes Clinical Camel as a medical model fine-tuned from LLaMA-2 and reports promising medical-benchmark results. Section 4 evaluates it in five-shot settings, supporting the cited point about supplying task-specific examples. The citations for training a new model and for the sentence’s opening claim are separate from this source. (LLM, conf=0.88)
- Supportedcites [27] Van Veen, D. et al. RadAdapt: radiology report summarization via lightweight domain adaptation of large language models. In 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks 449–460 (Association for Computational Linguistics, 2023). sourceExperimental — accuracy not yet measured
Citation 27 is attached to fine-tuning an existing model and supplying task-specific examples in prompts. RadAdapt’s Section 3.2 describes both methods, and Section 5.1 reports that fine-tuning outperformed discrete prompting while adding in-context examples improved performance for almost every model studied. (LLM, conf=0.92)
boostedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Prompt phrasing and model temperature can have a considerable effect on LLM output, as demonstrated in the literature45,46 and in Fig. 2.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
What the sources say
- Not matchedcites [45] Strobelt, H. et al. Interactive and visual prompt engineering for ad-hoc task adaptation with large language models. IEEE Trans. Vis. Comput. Graph. 29, 1146–1156 (2022).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites [46] Wang, J. et al. Prompt engineering for healthcare: methodologies and applications. Preprint at https://doi.org/10.48550/arXiv.2304.14670 (2023). sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) says prompt design can significantly affect model performance, and Automated Prompts describes improved performance from prompt paraphrasing. The copy does not address model temperature (searched: “temperature”), so it supports the prompt-phrasing part but not the temperature part. (LLM, conf=0.87)
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“considering that Alpaca is known to perform worse than our other open-source autoregressive models, Vicuna and Llama-2 (refs. 21,35)”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Impact of domain-specific fine-tuning.
What the sources say
- Partly supportedcites [21] Zheng, L. et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Preprint at https://doi.org/10.48550/arXiv.2306.05685 (2023). sourceExperimental — accuracy not yet measured
Table 8 (Position bias results; page not provided) reports lower MT-Bench scores for Alpaca-7B and Alpaca-13B (4.54 and 4.53) than for the listed Vicuna variants (5.95–6.39). The passages do not compare Alpaca with Llama-2; the LLaMA models listed are not identified as Llama-2. (LLM, conf=0.87)
- Not matchedcites [35] The Vicuna Team. Vicuna: an open-source chatbot impressing GPT-4 with 90%* ChatGPT quality. https://lmsys.org/blog/2023-03-30-vicuna/ (2023).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“For correctness, the metrics BERTScore44 (measuring semantics) and MEDCON41 (measuring medical concepts) correlated most strongly with reader preference; meanwhile, the BLEU42 metric (measuring syntax) correlated most with completeness and least with conciseness.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Connecting quantitative and clinical evaluations
What the sources say
- Not matchedcites [44] Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q. & Artzi, Y. BERTScore: evaluating text generation with BERT. International Conference on Learning Representations. https://openreview.net/forum?id=SkeHuCVFDr (2020).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Could not verifycites [41] Yim, W.W. et al. Aci-bench: a novel ambient clinical intelligence dataset for benchmarking automatic visit note generation. Sci. Data https://doi.org/10.1038/s41597-023-02487-3 (2023). sourceExperimental — accuracy not yet measured
Technical Validation > Baseline experiments (page unavailable) describes BERTScore and medcon as evaluation metrics, but does not report correlations with reader preference, correctness, completeness, or conciseness. Searches for those terms and BLEU found no passage addressing the claimed comparisons; the paper discusses BLEURT instead. (LLM, conf=0.78)
- Not checked — no full text of the sourcecites [42] Papineni, K., Roukos, S., Ward, T. & Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proc. of the 40th Annual Meeting of the Association for Computational Linguistics. https://dl.acm.org/doi/pdf/10.3115/1073083.1073135 (Association for Computing Machinery, 2002). source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
Every source the paper cites for this claim was checked and supports it.
In the paper
“To address this, there have been recent releases of expert evaluations for adjacent clinical NLP tasks3,55.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
What the sources say
- Supportedcites [3] Fleming, S. L. et al. MedAlign: a clinician-generated dataset for instruction following with electronic medical records. Preprint at https://doi.org/10.48550/arXiv.2308.14089 (2023). source
The Clinical dataset description says MedAlign includes clinician evaluations of LLM-generated outputs, and the Clinician Evaluation of LLM Responses section describes clinicians judging and ranking those outputs. The Discussion and Conclusion says the authors share the benchmark dataset with the research community, making MedAlign an example of the cited kind of expert evaluation. The copy does not give a page number for these passages. (LLM, conf=0.88)
- Supportedcites [55] Yu, F. et al. Evaluating progress in automatic chest x-ray radiology report generation. Patterns (N Y) 4, 100802 (2023). source
The Results section, “Alignment between automated metrics and radiologists,” reports radiologist error scores for 50 studies and says they are shown in Figure S3. The paper therefore presents an expert evaluation of the adjacent clinical NLP task of chest X-ray report generation. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“This simple approach provides the model with context, enhancing LLM performance for a particular task or domain27,28.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Adaptation methods
What the sources say
- Supportedcites [27] Van Veen, D. et al. RadAdapt: radiology report summarization via lightweight domain adaptation of large language models. In 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks 449–460 (Association for Computational Linguistics, 2023). source
The Related Work section describes prompting as providing context for a task. Section 5.1 reports that adding in-context examples improved radiology report summarization performance for almost every model tested, with T5 as the exception. (LLM, conf=0.90)
- Supportedcites [28] Mathur, Y. et al. SummQA at MEDIQA-Chat 2023: in-context learning with GPT-4 for medical summarization. Preprint at https://doi.org/10.48550/arXiv.2306.17384 (2023). source
Section 2 (Background) describes giving models task instructions or examples in context. Section 5.5 reports that, for the paper's medical summarization task, generated summaries generally improved as more relevant in-context examples were provided. No page numbers are supplied in our copy's passages. (LLM, conf=0.90)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“This quantization negligibly impacts performance40; as such, we used QLoRA for all model training.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Adaptation methods
What the sources say
- Supportedcites [40] Dettmers, T., Pagnoni, A., Holtzman, A. & Zettlemoyer, L. QLoRA: efficient finetuning of quantized LLMs. Preprint at https://doi.org/10.48550/arXiv.2305.14314 (2023). source
The Summary reports that 4-bit QLoRA with NF4 matched 16-bit finetuning performance on the evaluated benchmarks, with no observed degradation relative to full finetuning. This supports the claim in the context of QLoRA model training; the paper separately notes that 4-bit quantization for inference alone can degrade performance. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“We further note the potential for LLMs to be biased58,59.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
What the sources say
- Supportedcites [58] Omiye, J. A., Lester, J. C., Spichak, S., Rotemberg, V. & Daneshjou, R. Large language models propagate race-based medicine. NPJ Digit. Med. 6, 195 (2023). source
The Abstract reports that all four LLMs studied produced examples of race-based medicine, directly supporting the potential for LLM bias. It also says LLMs may amplify biases present in their training data. (LLM, conf=0.95)
- Not checked — no full text of the sourcecites [59] Zack, T. et al. Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study. Lancet Digit. Health 6, e12–e22 (2024). source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Note that choosing ‘relevant’ in-context examples has been shown to outperform choosing examples at random69.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Adaptation methods
What the sources say
- Partly supportedcites [69] Nie, F., Chen, M., Zhang, Z. & Cheng, X. Improving few-shot performance of language models via nearest neighbor calibration. Preprint at https://doi.org/10.48550/arXiv.2212.02216 (2022). sourceExperimental — accuracy not yet measured
The Experimental Setup (p. 5) uses randomly sampled in-context demonstrations as a baseline, and the Analysis (p. 7) reports improvements from retrieving similar support samples. But those samples calibrate predictions through a nearest-neighbor classifier; the paper does not show that selecting relevant examples *as in-context demonstrations* outperforms selecting random demonstrations. (LLM, conf=0.86)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“As we are not the first to employ these datasets, Extended Data Table 4 contains quantitative metric scores from other works25,27,41,52–54 that developed methods specific to each individual summarization task.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Data
What the sources say
- Partly supportedcites [25] Tu, T. et al. Towards generalist biomedical AI. Preprint at https://doi.org/10.48550/arXiv.2307.14334 (2023). sourceExperimental — accuracy not yet measured
The Additional information section says the paper used a previously published MIMIC-III radiology-report summarization dataset. But Section 5.1 describes Med-PaLM M as jointly fine-tuned across tasks and distinguishes it from prior specialist models; it does not support describing this paper’s method as specific to an individual summarization task. Our copy provides no page numbers for these passages. (LLM, conf=0.86)
- Supportedcites [27] Van Veen, D. et al. RadAdapt: radiology report summarization via lightweight domain adaptation of large language models. In 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks 449–460 (Association for Computational Linguistics, 2023). source
The Data section (4.1) identifies MIMIC-III as RadAdapt’s primary dataset and says it also evaluates on MIMIC-CXR. Section 3.2 describes adaptation methods applied to radiology report summarization, so the paper serves as an example of prior work using these datasets for a specific summarization task. (LLM, conf=0.88)
- Supportedcites [41] Yim, W.W. et al. Aci-bench: a novel ambient clinical intelligence dataset for benchmarking automatic visit note generation. Sci. Data https://doi.org/10.1038/s41597-023-02487-3 (2023). source
The paper’s “Baseline experiments” section says its authors used the aci-bench dataset to benchmark several note-generation methods and report quantitative results. It therefore serves as an example of prior work employing one of the datasets for a specific summarization task. The citing paper’s Extended Data Table 4 is its own presentation of those scores. (LLM, conf=0.90)
- Supportedcites [52] Ma, C. et al. ImpressionGPT: an iterative optimizing framework for radiology report summarization with chatGPT. Preprint at https://doi.org/10.48550/arXiv.2304.08448 (2023). source
The Introduction (p. 2) says the authors developed ImpressionGPT and evaluated it on MIMIC-CXR and OpenI. The comparison section (p. 9) reports its quantitative metrics alongside other radiology-report summarization methods on those datasets. This supports citing the paper as an earlier work using the datasets and developing a method for that summarization task. (LLM, conf=0.91)
- Not checked — no full text of the sourcecites [53] Wei, S. et al. Medical question summarization with entity-driven contrastive learning. Preprint at https://doi.org/10.48550/arXiv.2304.07437 (2023). source
- Supportedcites [54] Manakul, P. et al. CUED at ProbSum 2023: Hierarchical ensemble of summarization models. In The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks 516–523 (Association for Computational Linguistics, 2023). source
The Introduction (§1; page not supplied) identifies this paper as work on the ProbSum medical problem-list summarization task. The Experimental Setup and Ensemble Methods (§4.1 and §4.3; pages not supplied) describe its use of the task data, task-specific method, and quantitative ROUGE results. It therefore serves as an example of the prior task-specific work cited in the sentence. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“inspired by the AHRQ harm scale43.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Connecting errors to medical harm
What the sources say
- Supportedcites [43] Walsh, K. E. et al. Measuring harm in healthcare: optimizing adverse event review. Med. Care 55, 436–441 (2017). source
The Methods (Rater Assessment of Harm Severity) describe the revised AHRQ Harm Scale and its six harm-severity options. The citation identifies the scale that inspired the citing authors’ supplemental reader study; it need not establish the citing authors’ own motivation. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Compared to training model weights from scratch, LoRA is much more efficient with respect to both computational requirements and the volume of training data required.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Adaptation methods
What the sources say
- Partly supportedcites [70] Hu, E. et al. LoRA: low-rank adaptation of large language models. Preprint at https://doi.org/10.48550/arXiv.2106.09685 (2021). sourceExperimental — accuracy not yet measured
The Introduction (p. 2) says LoRA makes training more efficient, and §4.2 (p. 5) reports lower GPU memory use and faster training than full fine-tuning. Those comparisons are with adapting a pretrained model, not training one from scratch. The low-data experiments (§F.3, p. 22) do not establish that LoRA requires less training data than training from scratch. (LLM, conf=0.86)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The ensuing safety analysis highlights challenges faced by both LLMs and medical experts, as we connect errors to potential medical harm and categorize types of fabricated information.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Consider example 1 in Extended Data Fig. 4, in which the input question mentioned ‘diabetes and neuropathy’. The model mirrored this phrasing verbatim, whereas the medical expert interpreted it as ‘diabetic neuropathy’.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Seq2seq models (FLAN-T5 and FLAN-UL2) performed very well on syntactical metrics, such as BLEU, but worse on others (Fig. 3d)”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“medical expert summaries were preferred in only a minority of cases (19%), whereas, in a majority, the best model was either non-inferior (45%) or preferred (36%).”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Analyzing reader study results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In terms of trade-offs between lightweight adaptation methods, while QLoRA fine-tuning performed similarly for some cases, ICL was the best overall, especially when including proprietary models GPT-3.5 and GPT-4.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“pooled results across 10 physicians (Fig. 4b) demonstrate that summaries from the best-adapted model (GPT-4 using ICL) were more complete and contained fewer errors compared to medical expert summaries”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Analyzing reader study results
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In example 2 of Extended Data Fig. 4, the model simply reformulated the input question about tests and their locations, whereas the medical expert inferred a broader query about tests and treatments.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, our best-adapted model achieved non-inferior results to medical experts while performing a basic search across 1–2 options for each task instruction (Extended Data Table 1).”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Regarding conciseness, the best model performed significantly better than medical experts (P < 0.001) overall and on two tasks, whereas, for radiology reports, it performed similarly to medical experts.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Analyzing reader study results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that the model committed misinterpretations, inaccuracies and hallucinations on 6%, 2% and 5% of samples, respectively, compared to 9%, 4% and 12%, respectively, by medical experts.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Fabricated information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although ICL and QLoRA were competitive for open-source models, proprietary models GPT-3.5 and GPT-4 far outperformed other models and methods given sufficient in-context examples.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Effect of context length for ICL.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Regarding correctness, the best model generated significantly fewer errors (P < 0.001) compared to medical expert summaries (Fig. 4b) overall and on two of three summarization tasks.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Analyzing reader study results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observed that the best model summaries were more complete, on average, than medical expert summaries, achieving statistical significance across all three summarization tasks with P < 0.001 (Fig. 4b).”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Analyzing reader study results
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“After searching over temperature values {0.1, 0.5, 0.9} using GPT-3.5, Fig. 2 demonstrates that the lowest value, 0.1, performed best.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Constructing prompt anatomy
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results of this harm study (Fig. 4d) indicate that the medical expert summaries would have both a higher likelihood (14%) and a higher extent (22%) of possible harm compared to the summaries from the best model (12% and 16%, respectively).”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Analyzing reader study results
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Among open-source models, seq2seq models performed better than autoregressive models (Llama-2 and Vicuna) on radiology reports but worse on patient questions and progress notes (Fig. 3c).”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Model performance generally improved with more context.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although the model was more complete than the expert in generating its progress note summary, it also missed historical context (a history of hypertension).”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Analyzing reader study results
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Figure 5 provides an example in which the model’s summary includes correct information that readers deemed not important.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Analyzing reader study results
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Per Extended Data Table 1, this task entailed the largest token count across our six datasets for both the input (dialogue) and the target (assessment).”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Data
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Due to the wide variety of these questions, MeQSum exhibited the highest lexical variance of our datasets (Extended Data Table 1).”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Data
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“because the best model and method uses a handful of examples via ICL”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The benefit of larger models is explored in Extended Data Fig. 5, which found this improvement marginal for Llama-2 (13B) compared to Llama-2 (7B).”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > LLMs
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“radiology report summaries from medical experts occasionally recommend further studies or refer to prior studies—for example, ‘… not significantly changed from prior’ in Fig. 5.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Recent data indicate that physicians can expend up to 2 hours on documentation for each hour of patient interaction10.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
What the sources say
- Not checked — no full text of the sourcecites [10] Sinsky, C. et al. Allocation of physician time in ambulatory practice: a time and motion study in 4 specialties. Ann. Intern. Med. 165, 753–760 (2016). source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“Even for experienced physicians with a high level of expertise, this intricate task naturally introduces the possibility for errors, which can be detrimental in healthcare where precision is paramount4–6.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
What the sources say
- Not checked — no full text of the sourcecites [4] Yackel, T. R. & Embi, P. J. Unintended errors with EHR-based result management: a case series. J. Am. Med. Inform. Assoc. 17, 104–107 (2010). source
- Not checked — no full text of the sourcecites [5] Bowman, S. Impact of electronic health record systems on information integrity: quality and safety implications. Perspect. Health Inf. Manag. 10, 1c (2013).
- Not checked — no full text of the sourcecites [6] Gershanik, E. F., Lacson, R. & Khorasani, R. Critical finding capture in the impression section of radiology reports. AMIA Annu. Symp. Proc. 2011, 465–469 (2011).
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This suggests that integrating LLMs into clinical workflows could alleviate documentation burden, allowing clinicians to focus more on patient care.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We note that the model’s summaries are more concise while concurrently being more complete.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Analyzing reader study results
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This could be attributed to the relatively vague length specification in the radiology reports instruction—that is, ‘…with minimal text’—whereas the other two task instructions quantify length.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This illustrates the value of relevant context in achieving better outcomes for the target task.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Constructing prompt anatomy
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Governance aside, ICL provides many benefits: (1) model weights are fixed, hence enabling queries of pre-existing LLMs, and (2) adaptation is feasible with even a few examples, whereas fine-tuning methods, such as QLoRA, typically require hundreds or thousands of examples.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Note that seq2seq models are often constrained to much shorter context length than autoregressive models, such as GPT-4, because seq2seq models require the memory-intensive step of encoding the input sequence into a fixed-size context vector.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The evidence from our reader study suggests that adapting LLMs can outperform medical experts in terms of completeness, correctness and conciseness.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In both cases, the model’s summaries leaned toward literalness, a trait that readers sometimes favored and sometimes did not.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This instance underscores the point that even medical experts, not just LLMs, can hallucinate.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Analyzing reader study results
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The ensuing clinical reader study demonstrates that LLM summaries are often preferred over medical expert summaries due to higher scores for completeness, correctness and conciseness.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Given that BLEU measures sequence overlap, this result seems reasonable, as more text provides more ‘surface area’ for overlap; more text also reduces the brevity penalty that BLEU applies on generated sequences, which are shorter than the reference42.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
What the sources say
- Not checked — no full text of the sourcecites [42] Papineni, K., Roukos, S., Ward, T. & Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proc. of the 40th Annual Meeting of the Association for Computational Linguistics. https://dl.acm.org/doi/pdf/10.3115/1073083.1073135 (Association for Computing Machinery, 2002). source
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This highlights the model’s tendency toward a literal approach without interpretation, which may be either advantageous or limiting.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
hedgedSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Addressing this limitation presents an opportunity to accelerate the process of clinical text summarization, hence alleviating documentation burden and improving patient care.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We first highlight the importance of ‘prompt engineering’, or modifying and tuning the input prompt to improve model performance.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the low magnitude of correlation values (approximately 0.2) underscores the need to go beyond NLP metrics with a clinical reader study when assessing clinical readiness.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Connecting quantitative and clinical evaluations
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Regarding blinding, for the clinical reader study, we presented, in a blinded manner, the A/B comparison of summaries from the model and human experts.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Statistics and reproducibility
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“it is not possible to verify whether a particular open-source dataset was included in model training.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When human evaluation is not feasible, Fig. 6 suggests that syntactic metrics are better at measuring completeness, whereas semantic and conceptual metrics are better at measuring correctness.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Compared to x-rays, MRIs and CT scans capture more information at a higher resolution. This usually leads to longer reports (Extended Data Table 1), rendering MIMIC-III a more challenging summarization dataset than Open-i or MIMIC-CXR.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Data
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“This study used only retrospective, de-identified data that fell outside the scope of institutional review board oversight.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Ethics approval
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The dialogue task was excluded owing to the unwieldiness of a reader parsing many lengthy transcribed conversations and paragraphs”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Reader study
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To obfuscate any formatting differences between A and B, we applied simple post-processing to standardize capitalization, punctuation, newline characters, etc.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Statistics and reproducibility
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For the quantitative experiments, our sample size of 250 was chosen to maximize the number of samples given constraints of dataset size and cost of computing resources. We deem this sufficient as it enabled reproducible results when running the same experiments with a different set of samples.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Statistics and reproducibility
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Although our datasets do not contain demographic information”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, our use of six datasets enhances the robustness of our experiments.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Statistics and reproducibility
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“This study adhered to the principles outlined in the Declaration of Helsinki.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Ethics approval
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“for the quantitative experiments, we randomly selected 250 test samples and then randomly divided remaining samples into training and validation sets.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Statistics and reproducibility
plainn = 250- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Informed consent was obtained from each physician before their participation.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Ethics approval
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Because the LLM temperature parameter is near zero, outputs have very little randomness and are, thus, reproducible.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Statistics and reproducibility
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“No data were excluded from the analyses.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Statistics and reproducibility
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These instances are out of scope for the LLM, as it does not have access to prior studies nor the purview to make recommendations.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Given the critical nature of nuanced medical tasks, such as summarization, that typically have no objective solutions, including human experts in the evaluation process beyond quantitative metrics is crucial to demonstrate clinical readiness.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For the clinical reader study, our sample size of 100 comparisons per reader per task was chosen to maximize the number of comparisons that could be made in a reasonable amount of time, which we estimated as 10 h. We deem this sufficient as it enabled statistically significant results for many combinations of tasks and attributes.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Statistics and reproducibility
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This suggests that incorporating LLM-generated candidate summaries could reduce documentation load, potentially leading to decreased clinician strain and improved patient care.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
In the paper
“Although three of our datasets (MIMIC-CXR, MIMIC-III and ProbSum) require PhysioNet57 access to ensure safe data usage by third parties, this is no guarantee against data leakage.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
What the sources say
- Not checked — no full text of the sourcecites [57] Johnson, A., Pollard, T. & Mark, R. MIMIC-III Clinical Database Demo (version 1.4). PhysioNet https://doi.org/10.13026/C2HM2Q (2019). source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“we did not consider the inherently context-specific nature of summarization.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The quote only partly states this claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Note that we reduced the temperature parameter of the LLMs to be near zero, hence reducing randomness in the generated output summaries.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Statistics and reproducibility
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For the clinical reader study, we randomly selected 100 samples for comparison; these samples and the A/B comparisons within each sample are displayed in random order.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Statistics and reproducibility
plainn = 100- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“we chose the smallest dataset––which consequently required the lowest cost of computational resources––and re-ran a set of experiments that rendered very similar quantitative metric scores. This provided confidence moving forward with datasets that required higher computational resources.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Statistics and reproducibility
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Although three of our datasets (MIMIC-CXR, MIMIC-III and ProbSum) require PhysioNet57 access to ensure safe data usage by third parties”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
What the sources say
- Not checked — no full text of the sourcecites [57] Johnson, A., Pollard, T. & Mark, R. MIMIC-III Clinical Database Demo (version 1.4). PhysioNet https://doi.org/10.13026/C2HM2Q (2019). source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
The effectiveness of large language models on a diverse range of clinical summarization tasks remains unproven.
“Although large language models (LLMs) have shown promise in natural language processing (NLP) tasks, their effectiveness on a diverse range of clinical summarization tasks remains unproven.”
The paper relies on this without support.
Adapting LLMs to summarize a diverse set of clinical tasks has not been thoroughly explored, and non-inferiority to medical experts has not been achieved.
“However, adapting LLMs to summarize a diverse set of clinical tasks has not been thoroughly explored nor has non-inferiority to medical experts been achieved.”
The paper relies on this without support.
Fabricated information poses a substantial obstacle to clinical integration of LLMs because medical applications require accuracy.
“Fabricated information, or factually incorrect text, poses a substantial obstacle to the clinical integration of LLMs given the critical need for accuracy in medical applications.”
The authors state this as an assumption.
Samples with similar A/B summaries in completeness and correctness were assumed to contribute no harm.
“the subset of samples with similar A/B summaries (in completeness and correctness) were assumed to contribute no harm.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Most have been tested only with text-based, low-stakes interactions,33–38”
From Review of Artificial Intelligence for Clinical Use in Alzheimer's Disease and Related Dementias (2026) · cites this paper as [34]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The first four models were selected because they were among the most widely used LLMs at the time of study inception (January 2024), and prior research found that LLaMA 2 and GPT models outperformed humans in medical text summarization [19], while Claude models demonstrated greater faithfulness in book summaries than GPT [20].”
From Generalization bias in large language model summarization of scientific research (2025) · cites this paper as [19]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Scientists use LLM-assisted writing because LLMs can improve grammar, rhetoric, and overall readability of their texts, help translate to English, and quickly generate summaries (33, 34).”
From Delving into LLM-assisted writing in biomedical publications through excess vocabulary (2025) · cites this paper as [33]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Our finding challenges previous studies on clinical text summarization which reported that ‘summaries from our best-adapted LLMs (including GPT-3.5, GPT-4, and LLaMA 2) are preferable to human summaries in terms of completeness and correctness’ [19, p. 3].”
From Generalization bias in large language model summarization of scientific research (2025) · cites this paper as [19]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Most LLM studies have depended on human lead semi-quantitative assessment of coherence, comprehensiveness, harmfulness and factual inconsistencies as well as comparing LLM with human based summaries [5, 13, 14].”
From A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population (2024) · cites this paper as [5]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Automated metrics do not correlate with quality and human input to score coherence, inconsistencies, comprehensiveness and harmfulness are semi-quantitative and will differ based on the use care in which LLM are applied [5, 13].”
From A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population (2024) · cites this paper as [5]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Most studies applying LLM on clinical notes have been conducted in the US without individual consent, using de-identified data, with only one study from France utilising retrospective MRI reports without identifiable data [5, 6, 13, 14, 17–20].”
From A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population (2024) · cites this paper as [5]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Despite comparable performance with clinician-written summaries (Veen et al., 2024), hallucinations manifest as fabricated or distorted clinical statements that are hard to detect (Asgari et al., 2025; Kim et al., 2025).”
From Hallucination Detection-Guided Preference Optimization for Clinical Summarization (2026) · cites this paper as Veen et al., 2024
The cited source says what the claim attributes to it.
In the citing paper
“LLMs are capable of processing large volumes of text and generating human-like summaries and interpretations [2].”
From Evaluating Medical Text Summaries Using Automatic Evaluation Metrics and LLM-as-a-Judge Approach: A Pilot Study (2025) · cites this paper as [2]
The cited source says what the claim attributes to it.
In the citing paper
“Domain-specific LLMs, fine-tuned on medical corpora or using knowledge graph embeddings, perform well in clinical text summarization [10, 11], note generation [12, 13], and medical question answering [14].”
From Enhancing hepatopathy clinical trial efficiency: a secure, large language model-powered pre-screening pipeline (2025) · cites this paper as [10]
The cited source says what the claim attributes to it.
In the citing paper
“A study by Van Veen et al. reported that medically adapted LLM can summarize text better than physicians [23].”
From Evaluating the Capability of Large Language Model Chatbots for Generating Plain Language Summaries in Radiology (2025) · cites this paper as [23]
The cited source says what the claim attributes to it.
In the citing paper
“Our system employs local language models to preserve data privacy, which comes at some cost to performance – the local model may not match that of commercial models58,59.”
From A large language model digital patient system enhances ophthalmology history taking skills (2025) · cites this paper as [59]
The cited source says what the claim attributes to it.
In the citing paper
“Many of the studies on LLM have been conducted using large anonymised US datasets [5–7].”
From A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population (2024) · cites this paper as [5]
The cited source says what the claim attributes to it.
In the citing paper
“Large language models (LLM) such as ChatGPT, GPT-4 API and Llama2 can interrogate and summarise large volumes of medical notes to create succinct summaries of radiology reports, internal medical progress notes and patient dialogue [5–8].”
From A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population (2024) · cites this paper as [5]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Our system employs local language models to preserve data privacy, which comes at some cost to performance – the local model may not match that of commercial models58,59.”
From A large language model digital patient system enhances ophthalmology history taking skills (2025) · cites this paper as [59]
Experimental — accuracy not yet measuredIn the citing paper
“LLMs have shown expert-level performance in clinical text summarization when appropriate prompting strategies are employed3.”
From Towards evaluating and building versatile large language models for medicine (2025) · cites this paper as [3]
In the citing paper
“Therefore, related works have instrumented quantitative (completeness, correctness, conciseness) and qualitative (content, language, structure) measurements to dissect the riveting characters that distinguish synthesized clinical reports from human reports46–49.”
From Towards a holistic framework for multimodal LLM in 3D brain CT radiology report generation (2025) · cites this paper as [46]
Citations
Cites
A guideline of selecting and reporting intraclass correlation coefficients for reliability research
Koo, T. K., & Li, M. Y. (2016) · Journal of Chiropractic Medicine · doi:10.1016/j.jcm.2016.02.012 · Cited by 31,709 · cited 1×
“We estimated intra-reader correlation based on a mean-rating, fixed-agreement, two-may mixed-effects model 90 using the Pingouin package 91 .”
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Goldberger, A. L., Amaral, L. A. N., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark, R. G., Mietus, J. E., Moody, G. B., Peng, C.-K., & Stanley, H. E. (2000) · Circulation · doi:10.1161/01.cir.101.23.e215 · Cited by 14,809
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 13,453
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
International Journal of Environmental Research and Public Health
Casariego Vales, E., A.M., C.-C., J.L., L., R.M., M.-P., D., O.-B., M., O.-C., P., R.-S., & F.J., S. (2013) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph · Cited by 7,742
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254 · cited 1×
“In addition, beyond the scope of this manuscript, there is further potential to improve accuracy through incorporating checks by human operators and checks by other LLMs or using a model ensemble 47,48 .”
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · cited 2×
“Previous work has demonstrated potential across clinical NLP tasks 22,23 , adapting to the medical domain by training a new model 24,25 , fine-tuning an existing model 26,27 or supplying task-specific examples in the model prompt 27,28 .”
Research works printed but not linked: 9
- Gesner, E., Gazarian, P. & Dykes, P. The burden and burnout in documenting patient care: an integrative literature review. Stud. Health Technol. Inform. 21, 1194–1198 (2019).
- Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q. & Artzi, Y. BERTScore: evaluating text generation with BERT. International Conference on Learning Representations. https://openreview.net/forum?id=SkeHuCVFDr (2020).
- Lin, C.-Y. ROUGE: a package for automatic evaluation of summaries. In Text Summarization Branches Out 74–81 (Association for Computational Linguistics, 2004).
- Lehman, E. et al. Do we still need clinical language models? In Proceedings of Machine Learning Research 209, 578–597 (Conference on Health, Inference, and Learning, 2023).
- Demner-Fushman, D., Ananiadou, S. & Cohen, K. B. The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks. https://aclanthology.org/2023.bionlp-1 (Association for Computational Linguistics, 2023).
- Gao, Y., Miller, T., Afshar, M. & Dligach, D. BioNLP Workshop 2023 Shared Task 1A: Problem List Summarization (version 1.0.0). PhysioNet. https://doi.org/10.13026/1z6g-ex18 (2023). · DOI
- Yim, W., Ben Abacha, A., Snider, N., Adams, G. & Yetisgen, M. Overview of the MEDIQA-Sum task at ImageCLEF 2023: summarization and classification of doctor–patient conversations. In CEUR Workshop Proceedings https://ceur-ws.org/Vol-3497/paper-109.pdf (2023). · DOI · Paper page
- Loshchilov, I. & Hutter, F. Decoupled weight decay regularization. In International Conference on Learning Representations https://openreview.net/forum?id=Bkg6RiCqY7 (2019)
- Okazaki, N. & Tsujii, J. Simple and efficient algorithm for approximate dictionary matching. In Proc. of the 23rd International Conference on Computational Linguistics https://aclanthology.org/C10-1096.pdf (Association for Computational Linguistics, 2010).
Cited by
Sorted by citations
Evaluation and mitigation of the limitations of large language models in clinical decision-making
Hager, P., Jungmann, F., Holland, R., Bhagat, K., Hubrecht, I., Knauer, M., Vielhauer, J., Makowski, M., Braren, R., Kaissis, G., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03097-1 · Cited by 798
“LLMs can summarize reports 3–5 , generate reports 2,4 , serve as diagnostic assistants 21,27 and could ultimately autonomously diagnose patients.”
A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation
Asgari, E., Montaña-Brown, N., Dubois, M., Khalil, S., Balloch, J., Yeung, J. A., & Pimenta, D. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01670-7 · Cited by 423
“In fact, the use of LLMs for clinical documentation, especially clinical note generation 4 or consultation summarisation 5,6 , is an active area of research.”
A generalist vision-language foundation model for diverse biomedical tasks
Zhang, K., Zhou, R., Adhikarla, E., Yan, Z., Liu, Y., Yu, J., Liu, Z., Chen, X., Davison, B. D., Ren, H., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03185-2 · Cited by 333
“Although metrics such as CIDEr and ROUGE-L can measure the agreement between generated content and a gold standard, and are commonly used for model selection to further assess clinical applicability 35 , ensuring the factual accuracy of these outputs remains a concern.”
Delving into LLM-assisted writing in biomedical publications through excess vocabulary
Kobak, D., González-Márquez, R., Horvát, E.-Á., & Lause, J. (2024) · arXiv · doi:10.48550/arxiv.2406.07016 · Cited by 214
A review of deep learning for brain tumor analysis in MRI
Dorfner, F. J., Patel, J., Kalpathy–Cramer, J., Gerstner, E. R., & Bridge, C. P. (2025) · npj Precision Oncology · doi:10.1038/s41698-024-00789-2 · Cited by 265
“…medicine, with DL models having demonstrated effectiveness on tasks such as medical text summarization, prediction of future lung cancer risk, prediction of SARS-CoV-2 infectivity and variant evolution, identi fi cation of new antibiotics, and assessment of mammography for breast cancer 1 – 5 .”
Delving into LLM-assisted writing in biomedical publications through excess vocabulary
Kobak, D., González-Márquez, R., Horvát, E.-Á., & Lause, J. (2025) · Science Advances · doi:10.1126/sciadv.adt3813 · Cited by 213
“Scientists use LLM-assisted writing because LLMs can improve grammar, rhetoric, and overall readability of their texts, help translate to English, and quickly generate summaries (Van Veen et al., 2024; Zhang et al., 2024).”
Goh, E., Gallo, R. J., Strong, E., Weng, Y., Kerman, H. M., Freed, J. A., Cool, J. A., Kanjee, Z., Lane, K. P., Parsons, A. S., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03456-y · Cited by 216
“As an exploratory evaluation, a single blinded reviewer (A.R.) rated all responses for potential harm using a similar methodology to a previously published assessment 34 .”
Self-citationVrdoljak, J., Boban, Z., Vilović, M., Kumrić, M., & Božić, J. (2025) · Healthcare · doi:10.3390/healthcare13060603 · Cited by 202
“While BART achieved higher scores in automated metrics, such as ROUGE and BERTScore, ChatGPT’s summaries were favored more by human medical experts in manual evaluations, demonstrating better readability and overall quality [38].”
InfluentialAlkhalaf, M., Yu, P., Yin, M., & Deng, C. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104662 · Cited by 156
Large language models for structured reporting in radiology: past, present, and future
Busch, F., Hoffmann, L., dos Santos, D. P., Makowski, M. R., Saba, L., Prucker, P., Hadamitzky, M., Navab, N., Kather, J. N., Truhn, D., et al. (2024) · European Radiology · doi:10.1007/s00330-024-11107-6 · Cited by 134
Generalization bias in large language model summarization of scientific research
Peters, U., & Chin-Yee, B. (2025) · Royal Society Open Science · doi:10.1098/rsos.241776 · Cited by 126
Large Language Model Capabilities in Perioperative Risk Prediction and Prognostication
Chung, P., Fong, C. T., Walters, A. M., Aghaeepour, N., Yetişgen, M., & O’Reilly-Shah, V. N. (2024) · JAMA Surgery · doi:10.1001/jamasurg.2024.1621 · Cited by 140
Generative artificial intelligence in medicine
Teo, Z. L., Thirunavukarasu, A. J., Elangovan, K., Cheng, H., Moova, P., Soetikno, B., Nielsen, C., Pollreisz, A., Ting, D. S. J., Morris, R. J. T., et al. (2025) · Nature Medicine · doi:10.1038/s41591-025-03983-2 · Cited by 119
“GAI exhibits remarkable summarization ability, with one study demonstrating superiority to clinicians in terms of quality and efficiency 106 .”
Self-citationAI-Generated Clinical Summaries Require More Than Accuracy
Goodman, K. E., Yi, P. H., & Morgan, D. J. (2024) · JAMA · doi:10.1001/jama.2024.0555 · Cited by 107
The generative era of medical AI
Fahrner, J., Chen, E., Topol, E. J., & Rajpurkar, P. (2025) · Cell · doi:10.1016/j.cell.2025.05.018 · Cited by 120
Merlin: a computed tomography vision–language foundation model and dataset
Blankemeier, L., Kumar, A., Cohen, J., Liu, J., Liu, L., Veen, D. V., Gardezi, S. J. S., Yu, H., Paschali, M., Chen, Z., et al. (2026) · Nature · doi:10.1038/s41586-026-10181-8 · Cited by 186
Self-citationFoundation Models in Radiology: What, How, Why, and Why Not
Paschali, M., Chen, Z., Blankemeier, L., Varma, M., Youssef, A., Blüthgen, C., Langlotz, C. P., Gatidis, S., & Chaudhari, A. S. (2025) · Radiology · doi:10.1148/radiol.240597 · Cited by 134
Self-citationPhysician- and Large Language Model–Generated Hospital Discharge Summaries
Williams, C. Y. K., Subramanian, C. R., Ali, S. S., Apolinario, M., Askin, E., Barish, P. N., Cheng, M., Deardorff, W. J., Donthi, N., Ganeshan, S., et al. (2025) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2025.0821 · Cited by 127
A comprehensive survey of large language models and multimodal large language models in medicine
Xiao, H., Zhou, F., Liu, X., Liu, T., Ли, Ж., Liu, X., & Huang, X. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102888 · Cited by 154
Reichenpfader, D., Müller, H., & Denecke, K. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01219-0 · Cited by 110
“In the broad clinical domain, these recent, generative LLMs show impressive capabilities, partly outperforming clinicians in test settings regarding, e.g., medical summary generation 90 , prediction of clinical outcomes 91 and answering of clinical questions 92 93 .”
Reliability of LLMs as medical assistants for the general public: a randomized preregistered study
Bean, A. M., Payne, R. E., Parsons, G., Kirk, H. R., Ciro, J., Mosquera-Gómez, R., Sara Hincapié, M., Ekanayaka, A. S., Tarassenko, L., Rocher, L., et al. (2026) · Nature Medicine · doi:10.1038/s41591-025-04074-y · Cited by 130
“The specific cases are described in brief in Extended Data Table 6, and the full texts and gold-standard relevant conditions are included in Supplementary Information.”
InfluentialChen, S., Gao, M., Sasse, K., Hartvigsen, T., Anthony, B., Fan, L., Aerts, H., Gallifant, J., & Bitterman, D. S. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02008-z · Cited by 112
Enhancing medical AI with retrieval-augmented generation: A mini narrative review
Gargari, O. K., & Habibi, G. (2025) · Digital Health · doi:10.1177/20552076251337177 · Cited by 141
Cheong, I., Xia, K., Feng, K. J. K., Chen, Q. Z., & Zhang, A. X. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3659048 · Cited by 91
Application of large language models in medicine
Liu, F., Zhou, H., Gu, B., Zou, X., Huang, J., Wu, J., Li, Y., Chen, S. S., Hua, Y., Zhou, P., et al. (2025) · Nature Reviews Bioengineering · doi:10.1038/s44222-025-00279-5 · Cited by 94
“These tasks include medical text summarization 63,64 , medical text generation 28 and medical text simplification 65 .”
Developing and evaluating large language model–generated emergency medicine handoff notes
Hartman, V., Zhang, X., Poddar, R., McCarty, M., Fortenko, A., Sholle, E., Sharma, R., Campion, T., & Steel, P. A. D. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.48723 · Cited by 76
Scientific evidence for clinical text summarization using large language models: scoping review
Bednarczyk, L., Reichenpfader, D., Gaudet-Blavignac, C., Ette, A. K., Zaghir, J., Zheng, Y., Bensahla, A., Bjelogrlic, M., & Lovis, C. (2025) · Journal of Medical Internet Research · doi:10.2196/68998 · Cited by 83
Towards a holistic framework for multimodal LLM in 3D brain CT radiology report generation
Li, C.-Y., Chang, K.-J., Yang, C.-F., Wu, H.-Y., Chen, W., Bansal, H., Chen, L., Yang, Y.-P., Chen, Y.-C., Chen, S.-P., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-57426-0 · Cited by 75
“To 35-37 By adopting similar designs with objective linguistic criteria, we found that both reviewer success rate and answer alternation reasons (“Suspicious wording” and “Both not mentioning critical features”) were associated with the writing style (“Familiarity and voice” and "Specificity or…”
Clinical entity augmented retrieval for clinical information extraction
López, I., Swaminathan, A., Vedula, K. S., Narayanan, S., Haredasht, F. N., Ma, S. P., Liang, A. S., Tate, S., Maddali, M. V., Gallo, R. J., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-024-01377-1 · Cited by 97
Abstractive text summarization: State of the art, challenges, and improvements
Shakil, H., Farooq, A., & Kalita, J. K. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128255 · Cited by 100
Evaluating large language models and agents in healthcare: key challenges in clinical applications
Chen, X., Xiang, J., Lu, S., Liu, Y., He, M., & Shi, D. (2025) · Intelligent Medicine · doi:10.1016/j.imed.2025.03.002 · Cited by 97
Jung, K.-H. (2025) · Healthcare Informatics Research · doi:10.4258/hir.2025.31.2.114 · Cited by 84
“Research using models such as GPT-4 indicates that LLM-generated summaries can match or even surpass human-produced summaries in completeness and accuracy for particular tasks, while accomplishing this significantly faster [15].”
The potential for large language models to transform cardiovascular medicine
Quer, G., & Topol, E. J. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00151-1 · Cited by 76
Introduction to large language models (LLMs) for dementia care and research
Treder, M. S., Lee, S., & Tsvetanov, K. A. (2024) · Frontiers in Dementia · doi:10.3389/frdem.2024.1385303 · Cited by 63
Prompt Engineering in Clinical Practice: Tutorial for Clinicians
Liu, J., Liu, F., Wang, C., & Liu, S. (2025) · Journal of Medical Internet Research · doi:10.2196/72644 · Cited by 76
“To address this, researchers have developed techniques like zero-shot, few-shot, and chain-of-thought prompting to optimize LLM performance [12-15].”
Generative AI in Medicine — Evaluating Progress and Challenges
Maddox, T. M., Embí, P. J., Gerhart, J., Goldsack, J. C., Parikh, R. B., & Sarich, T. C. (2025) · New England Journal of Medicine · doi:10.1056/nejmsb2503956 · Cited by 69
LLMIF: Augmented Large Language Model for Fuzzing IoT Devices
Wang, J., Yu, L., & Luo, X. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00211 · Cited by 75
Artificial intelligence to improve cardiovascular population health
Meder, B., Asselbergs, F. W., & Ashley, E. A. (2025) · European Heart Journal · doi:10.1093/eurheartj/ehaf125 · Cited by 72
Foundation models and intelligent decision-making: Progress, challenges, and perspectives
Huang, J., Xu, Y., Wang, Q., Wang, Q., Liang, X., Wang, F., Zhang, Z., Wei, W., Zhang, B., Huang, L., et al. (2025) · The Innovation · doi:10.1016/j.xinn.2025.100948 · Cited by 87
Giuffrè, M., Kresevic, S., Pugliese, N., You, K., & Shung, D. L. (2024) · Liver International · doi:10.1111/liv.15974 · Cited by 62
Medical Ethics of Large Language Models in Medicine
Ong, J. C. L., Chang, Y., Wasswa, W., Butte, A. J., Shah, N. H., Chew, L. S. T., Liu, N., Doshi‐Velez, F., Lu, W., Savulescu, J., et al. (2024) · NEJM AI · doi:10.1056/aira2400038 · Cited by 62
Prompt injection attacks on vision language models in oncology
Clusmann, J., Ferber, D., Wiest, I. C., Schneider, C. V., Brinker, T. J., Foersch, S., Truhn, D., & Kather, J. N. (2025) · Nature Communications · doi:10.1038/s41467-024-55631-x · Cited by 99
“Despite our fi ndings pointing to relevant security threats, integrating LLMs in hospitals holds tremendous promise for patient empowerment, reduction of documentation burden, and guideline-based clinician support 4,7,37 .”
A scoping review on generative AI and large language models in mitigating medication related harm
Ong, J. C. L., Chen, M. H., Ng, N., Elangovan, K., Tan, N. Y. T., Jin, L., Xie, Q., Ting, D. S. W., Rodriguez-Monguio, R., Bates, D. W., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01565-7 · Cited by 64
Assessment of Large Language Models in Clinical Reasoning: A Novel Benchmarking Study
McCoy, L. G., Swamy, R., Sagar, N., Wang, M., Bacchi, S., Fong, J. M. N., Tan, N. C., Tan, K., Buckley, T. A., Brodeur, P. G., et al. (2025) · NEJM AI · doi:10.1056/aidbp2500120 · Cited by 81
Ong, J. C. L., Jin, L., Elangovan, K., Lim, G. Y. S., Lim, D. Y. Z., Sng, G. G. R., Ke, Y. H., Tung, J. Y. M., Zhong, R. J., Koh, C. M. Y., et al. (2025) · Cell Reports Medicine · doi:10.1016/j.xcrm.2025.102323 · Cited by 58
Lee, C., Britto, S. S., & Diwan, K. (2024) · Cureus · doi:10.7759/cureus.73994 · Cited by 69
Large Language Models and Large Multimodal Models in Medical Imaging: A Primer for Physicians
Bradshaw, T., Tie, X., Warner, J., Hu, J., Li, Q., & Li, X. (2025) · Journal of Nuclear Medicine · doi:10.2967/jnumed.124.268072 · Cited by 60
“Clinical text summarization using LLMs has been extensively studied ( 4 , 42 , 44 , 45 ).”
Automated generation of discharge summaries: leveraging large language models with clinical data
Ganzinger, M., Kunz, N., Fuchs, P., Lyu, C. K., Loos, M., Dugas, M., & Pausch, T. M. (2025) · Scientific Reports · doi:10.1038/s41598-025-01618-7 · Cited by 59
Buess, L., Keicher, M., Navab, N., Maier, A., & Arasteh, S. T. (2025) · Biomedical Engineering Letters · doi:10.1007/s13534-025-00497-1 · Cited by 79
“Recent studies further demonstrate their impact, showing that AI-generated draft radiology reports can reduce reporting time by about 25% while maintaining diagnostic accuracy [6], thus addressing workload challenges in clinical practice [7].”
Large‐Language‐Model‐Based AI Agent for Organic Semiconductor Device Research
Zhang, Q., Hu, Y., Yan, J., Zhang, H., Xie, X., Zhu, J., Li, H., Niu, X., Li, L., Sun, Y., et al. (2024) · Advanced Materials · doi:10.1002/adma.202405163 · Cited by 53
HANNA: hard-constraint neural network for consistent activity coefficient prediction
Specht, T., Nagda, M. K., Fellenz, S., Mandt, S., Hasse, H., & Jirasek, F. (2024) · Chemical Science · doi:10.1039/d4sc05115g · Cited by 61
“5,6 These models are universal and highly flexible function approximators, 7 which perform best if they have large amounts of training data.”
Towards evaluating and building versatile large language models for medicine
Wu, C., Qiu, P., Liu, J., Gu, H., Li, N., Zhang, Y., Wang, Y., & Xie, W. (2025) · npj Digital Medicine · doi:10.1038/s41746-024-01390-4 · Cited by 52
Efficiency and Quality of Generative AI–Assisted Radiograph Reporting
Huang, J., Wittbrodt, M. T., Teague, C. N., Karl, E., Galal, G., Thompson, M., Chapa, A. R., Chiu, M.-L., Herynk, B. A., Linchangco, R., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.13921 · Cited by 60
The evaluation illusion of large language models in medicine
Agrawal, M., Chen, I. Y., Gulamali, F., & Joshi, S. D. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01963-x · Cited by 70
Large language model agents can use tools to perform clinical calculations
Goodell, A. J., Chu, S. N., Rouholiman, D., & Chu, L. F. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01475-8 · Cited by 48
“1forbothmodelstoevaluatethemost-likelyresponse 38 while allowing for some variability for self-correction (discussed below).”
Riaz, I. B., Harmon, S. A., Chen, Z., Naqvi, S. A. A., & Liang Cheng, C. (2024) · American Society of Clinical Oncology Educational Book · doi:10.1200/edbk_438516 · Cited by 49
Ma, Y., Zheng, S., Yang, Z., Pan, H., & Hong, J. (2025) · International Journal of Production Research · doi:10.1080/00207543.2025.2472298 · Cited by 65
“LLM is expected to reduce documentation burden for clinical (Van Veen et al. 2024).”
Large language models in biomedicine and healthcare
Zhou, J., Li, H., Chen, S., Chen, Z., Han, Z., & Gao, X. (2025) · Yearbook of pediatric endocrinology · doi:10.1038/s44387-025-00047-1 · Cited by 74
Holistic evaluation of large language models for medical tasks with MedHELM
Bedi, S., Cui, H., Fuentes, M., Unell, A., Wornow, M., Banda, J. M., Kotecha, N., Keyes, T., Mai, Y., Oez, M., et al. (2026) · Nature Medicine · doi:10.1038/s41591-025-04151-2 · Cited by 59
Self-citationVerlingue, L., Boyer, C., Olgiati, L., Brutti-Mairesse, C., Morel, D., & Blay, J. (2024) · The Lancet Regional Health - Europe · doi:10.1016/j.lanepe.2024.101064 · Cited by 50
A strategy for cost-effective large language model use at health system-scale
Klang, E., Apakama, D. U., Abbott, E. E., Vaid, A., Lampert, J. M., Sakhuja, A., Freeman, R., Charney, A. W., Reich, D. L., Kraft, M., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01315-1 · Cited by 56
Markowitz, D. M. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae387 · Cited by 68
Guo, Y., Li, T., Gong, B., Hu, Y., Wang, S., Yang, L., & Zheng, C. (2024) · Advanced Science · doi:10.1002/advs.202408069 · Cited by 49
Menezes, M. C. S., Hoffmann, A. F., Tan, A. L. M., Nalbandyan, M., Omenn, G. S., Mazzotti, D. R., Hernández-Arango, A., Visweswaran, S., Venkatesh, S., Mandl, K. D., et al. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00246-2 · Cited by 45
Savage, C. H., Kanhere, A., Parekh, V. S., Langlotz, C. P., Joshi, A., Huang, H., & Doo, F. X. (2025) · Radiology · doi:10.1148/radiol.241073 · Cited by 61
Self-citationExamining human-AI interaction in real-world healthcare beyond the laboratory
Wekenborg, M., Gilbert, S., & Kather, J. N. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01559-5 · Cited by 54
“While a variety of AI applications like large language models (LLMs) for clinical summarization 19 may offer smoother integration and fewer disruptions compared to the transition from paper charts to EHRs, it remains essential to evaluate their impact in real-world clinical practice rather than…”
Large language models: an overview of foundational architectures, recent trends, and a new taxonomy
Mienye, I. D., Jere, N., Obaido, G., Ogunruku, O. O., Esenogho, E., & Modisane, C. (2025) · Discover Applied Sciences · doi:10.1007/s42452-025-07668-w · Cited by 46
Evaluating large language models for drafting emergency department encounter summaries
Williams, C. Y. K., Bains, J., Tang, T., Patel, K., Lucas, A. N., Chen, F., Miao, B. Y., Butte, A. J., & Kornblith, A. E. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000899 · Cited by 41
ESMO guidance on the use of Large Language Models in Clinical Practice (ELCAP)
Wong, E. Y. T., Verlingue, L., Aldea, M., Franzoi, M. A., Umeton, R., Halabi, S., Harbeck, N., Indini, A., Prelaj, A., Romano, E., et al. (2025) · Annals of Oncology · doi:10.1016/j.annonc.2025.09.001 · Cited by 44
Expert evaluation of large language models for clinical dialogue summarization
Navarro, D. F., Coiera, E. W., Hambly, T. W., Triplett, Z., Asif, N., Susanto, A. P., Chowdhury, A., Azcoaga-Lorenzo, A., Dras, M., & Berkovsky, S. (2025) · Scientific Reports · doi:10.1038/s41598-024-84850-x · Cited by 48
“More recently, studies applied to the context of clinical summarization had also appeared 35,36 , showcasing the benefits of this approach in different types of medical text such as radiology reports 37 and clinical dialogues.”
Alba, C., Xue, B., Abraham, J., Kannampallil, T., & Lu, C. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01489-2 · Cited by 52
“These efforts include automating medical reports [15], creating personalized treatments [16], and generating summaries [17] or recommendations [14] from clinical notes, respectively.”
InfluentialLuo, P.-W. (2025) · American Journal of Clinical and Experimental Urology · doi:10.62347/uiap7979 · Cited by 39
Collaborative, T. C. (2025) · BMJ · doi:10.1136/bmj-2024-083305 · Cited by 52
“Additionally, given the diversity of models and their continuous learning and adaptation, 2 changes in model development and deployment can substantially influence performance, 34 necessitating the disclosure of technical details.”
A large language model digital patient system enhances ophthalmology history taking skills
Luo, M.-J., Bi, S., Pang, J., Liu, L., Tsui, C.-K., Lai, Y., Chen, W., Yang, Y., Xu, K., Zhao, L., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01841-6 · Cited by 39
“Our system employs local language models to preserve data privacy, which comes at some cost to performance – the local model may not match that of commercial models 58,59 .”
Best Practices for Large Language Models in Radiology
Blüthgen, C., Veen, D. V., Zakka, C., Link, K. E., Fanous, A. H., Daneshjou, R., Frauenfelder, T., Langlotz, C. P., Gatidis, S., & Chaudhari, A. S. (2025) · Radiology · doi:10.1148/radiol.240528 · Cited by 48
Self-citationProspects for AI clinical summarization to reduce the burden of patient chart review
Lee, C., Vogt, K. A., & Kumar, S. (2024) · Frontiers in Digital Health · doi:10.3389/fdgth.2024.1475092 · Cited by 49
Benchmarking Large Language Models in Evidence-Based Medicine
Li, J., Deng, Y., Sun, Q., Zhu, J., Tian, Y., Li, J., & Zhu, T. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3483816 · Cited by 35
“Van et al. [35] highlighted ICL’s effectiveness in improving clinical text summarization, applicable to diverse tasks such as radiology and doctor-patient dialogues.”
Evaluating large language models in echocardiography reporting: opportunities and challenges
Chao, C., Banerjee, I., Arsanjani, R., Ayoub, C., Tseng, A. S., Delbrouck, J.-B., Kane, G. C., López-Jiménez, F. A., Attia, Z. I., Oh, J. K., et al. (2024) · European Heart Journal - Digital Health · doi:10.1093/ehjdh/ztae086 · Cited by 39
Self-citationAI solutions to the radiology workforce shortage
Jing, A. B., Garg, N., Zhang, J., & Brown, J. J. (2025) · npj Health Systems · doi:10.1038/s44401-025-00023-6 · Cited by 40
Evaluating Large Language Models for Drafting Emergency Department Discharge Summaries
Williams, C. Y. K., Bains, J., Tang, T., Patel, K., Lucas, A. N., Chen, F., Miao, B. Y., Butte, A. J., & Kornblith, A. E. (2024) · doi:10.1101/2024.04.03.24305088 · Cited by 35
Few shot learning for phenotype-driven diagnosis of patients with rare genetic diseases
Alsentzer, E., Li, M. M., Kobren, S. N., Noori, A., Undiagnosed Diseases Network, Kohane, I. S., & Zitnik, M. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01749-1 · Cited by 54
Mohammadabadi, S. M. S., Kara, B. C., Eyüpoğlu, C., Uzay, C., Tosun, M. S., & Karakuş, O. (2025) · Electronics · doi:10.3390/electronics14183580 · Cited by 71
Adhikary, P. K., Srivastava, A., Kumar, S., Singh, S. M., Manuja, P., Gopinath, J. K., Krishnan, V., Gupta, S., Deb, K. S., & Chakraborty, T. (2024) · JMIR Mental Health · doi:10.2196/57306 · Cited by 49
Gin, B. C., O’Sullivan, P., Hauer, K. E., Abdulnour, R.-E. E., Mackenzie, M. E., ten Cate, O., & Boscardin, C. (2024) · Academic Medicine · doi:10.1097/acm.0000000000005930 · Cited by 51
Hu, D., Zhang, S., Liu, Q., Xiaofeng, Z., & Liu, B. (2025) · Journal of Medical Internet Research · doi:10.2196/65547 · Cited by 43
“Impression summarization refers to using models to simulate physicians to condense lengthy and detailed findings into concise and informative impressions [5-7], which is a key application of text summarization in the medical field [8].”
Evaluating Hospital Course Summarization by an Electronic Health Record–Based Large Language Model
Small, W. R., Austrian, J. S., O’Donnell, L., Burk‐Rafel, J., Hochman, K. A., Goodman, A. J., Zaretsky, J., Martin, J. A., Johnson, S. B., Major, V. J., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.26339 · Cited by 40
Large Language Model Assistant for Emergency Department Discharge Documentation
Song, J. W., Park, J., Kim, J. H., & You, S. C. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.38427 · Cited by 42
A dataset and benchmark for hospital course summarization with adapted large language models
Aali, A., Van Veen, D., Arefeen, Y. I., Hom, J., Bluethgen, C., Reis, E. P., Gatidis, S., Clifford, N., Daws, J., Tehrani, A. S., et al. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae312 · Cited by 32
Self-citationEconomics and Equity of Large Language Models: Health Care Perspective
Nagarajan, R., Kondo, M., Salas, F., Sezgın, E., Yao, Y., Klotzman, V., Godambe, S. A., Khan, N., Limon, A., Stephenson, G. S., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/64226 · Cited by 39
Understanding natural language: Potential application of large language models to ophthalmology
Yang, Z., Wang, D., Zhou, F., Song, D., Zhang, Y., Jiang, J., Kong, K., Liu, X., Qiao, Y., Chang, R. T., et al. (2024) · Asia-Pacific Journal of Ophthalmology · doi:10.1016/j.apjo.2024.100085 · Cited by 38
Abdel‐Rehim, A., Zenil, H., Orhobor, O. I., Fisher, M., Collins, R. J., Bourne, E., Fearnley, G. W., Tate, E., Smith, H. X., Soldatova, L., et al. (2025) · Journal of The Royal Society Interface · doi:10.1098/rsif.2024.0674 · Cited by 52
“Despite the clear potential of LLMs for hypothesis generation, their utility for hypothesis generation has been little investigated.”
The MI-CLAIM-GEN checklist for generative artificial intelligence in health
Miao, B. Y., Chen, I. Y., Williams, C. Y. K., Davidson, J., García-Agúndez, A., Sun, S., Zack, T. I., Saria, S., Arnaout, R., Quer, G., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03470-0 · Cited by 43
GPT-5 and open-weight large language models: Advances in reasoning, transparency, and control
León, M. (2025) · Information Systems · doi:10.1016/j.is.2025.102620 · Cited by 33
The potential of large language models to advance precision oncology
Liang, S., Zhang, J., Liu, X., Huang, Y., Shao, J., Liu, X., Li, W., Wang, G., & Wang, C. (2025) · EBioMedicine · doi:10.1016/j.ebiom.2025.105695 · Cited by 35
“64 In the abovementioned breast cancer-speci fi c investigation, in addition to decision-making, the ability of ChatGPT-3.”
Sohn, J.-S., Lee, E., Kim, J., Oh, H.-K., & Kim, E. J. (2025) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2025.1628216 · Cited by 41
Du, X.-S., Zhou, Z., Wang, Y.-F., Chuang, Y.-W., Li, Y.-M., Yang, R., Zhang, W., Wang, X., Chen, X.-Y., Guan, H., et al. (2024) · medRxiv · doi:10.1101/2024.08.11.24311828 · Cited by 36
“35,36,40 The findings were as follows: 1) The Artificial Intelligence Performance Instrument (AIPI) correlated with the Ottawa Clinic Assessment Tool (OCAT) when managing cases in otolaryngology–head and neck surgery ( ρ(cid:2) = (cid:2) 0.495); 2) BERTScore correlated with the quality score…”
BackgroundMethodInfluentialAlsanosi, S., & Padmanabhan, S. (2024) · Healthcare · doi:10.3390/healthcare12070788 · Cited by 38
So, J.-H., Chang, J., Kim, E., Na, J., Choi, J., Sohn, J.-Y., Kim, B.-H., & Chu, S. H. (2024) · JMIR Formative Research · doi:10.2196/58418 · Cited by 31
“Alongside these findings, growing evidence suggests that LLMs can perform exceptionally well on clinical tasks beyond structured clinical question answering, such as clinical text summarization, when appropriate techniques are used to align the models [28,29].”
Large language models for clinical decision support in gastroenterology and hepatology
Wiest, I. C., Bhat, M., Clusmann, J., Schneider, C. V., Jiang, X., & Kather, J. N. (2025) · Nature Reviews Gastroenterology & Hepatology · doi:10.1038/s41575-025-01108-1 · Cited by 34
“At the hypothesis generation stage, LLMs can streamline the analysis of medical free text, such as summarizing patient-provided documents or notes within EHRs, substantially reducing the time-intensive manual review required to extract insights 24 .”
Doo, F. X., Savani, D., Kanhere, A., Carlos, R. C., Joshi, A., Yi, P. H., & Parekh, V. S. (2024) · Radiology · doi:10.1148/radiol.240320 · Cited by 25
Metadata
- Authors
- Van Veen, D., Van Uden, C., Blankemeier, L., Delbrouck, J.-B., Aali, A., Bluethgen, C., Pareek, A., Polacin, M., Reis, E. P., Seehofnerová, A., Rohatgi, N., Hosamani, P., Collins, W., Ahuja, N., Langlotz, C. P., Hom, J., Gatidis, S., Pauly, J., & Chaudhari, A. S.
- Affiliations
- Stanford University (US)
- The University of Texas at Austin (US)
- Copenhagen University Hospital (DK)
- University of Zurich (CH)
- Hospital Israelita Albert Einstein (BR)
- Stanford Medicine (US)
- ORCID
- Venue
- Nature Medicine
- Published
- 2024-02-27
- Volume
- 30
- Issue
- 4
- Pages
- 1134-1142
- ISSN
- 1078-8956 (ISSN-L)
- 1546-170X
- PMC
- PMC11479659
- OpenAlex
- W4392193048
- Cited by
- 891
- References
- 80
- Citations by year
- 2026: 341 · 2025: 404 · 2024: 144
- FWCI
- 82.70
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Licence
- Locations
- Nature Medicine (publishedVersion)
- Research Square (acceptedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Zurich Open Repository and Archive (University of Zurich) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- PubMed Central (submittedVersion, cc-by)
- Zurich Open Repository and Archive (University of Zurich) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Zurich Open Repository and Archive (University of Zurich)
- Funders
- Foundation for the National Institutes of Health (R01 AR077604, R01 EB002524, R01 AR079431, P41 EB027060)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- large language models, clinical text summarization, LLM adaptation, electronic health records, radiology reports, progress notes, doctor-patient dialogue, patient questions, summary quality, safety analysis, clinical documentation burden
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
