The DOI matches its Crossref record; 91 of the 91 references it cites were found in a registry or an index; 37 of the 61 sources its claims cite were checked for support.
Adapted large language models can outperform medical experts in clinical text summarization
Open accessPublished version of a preprint
Abstract
Adapted LLMs can perform text summarization of clinical notes such as radiology reports, progress notes, and discharge summaries, sometimes outperforming medical experts on certain criteria.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
40 Own result · 27 Own interpretation · 20 Own method · 13 Partly supported · 7 No basis given · 6 No full text of the sources · 5 Supported · 1 Supported so far · 1 Couldn’t confirm · 1 Sources not found · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Although LLM benchmarks for general natural language processing (NLP) tasks exist20,21, they do not evaluate performance on relevant clinical tasks.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
What the sources say
- Not matchedcites [20] Liang, P. et al. Holistic evaluation of language models. Transact. Mach. Learn. Res. (in the press).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites [21] Zheng, L. et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Preprint at https://doi.org/10.48550/arXiv.2306.05685 (2023). source
The citation follows the statement that LLM benchmarks for general NLP tasks exist. Section 2.1 (Motivation) lists several such benchmarks, and Section 5 discusses standardized LLM benchmarks. The citation is not attached to the subsequent claim about clinical tasks. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Currently, clinicians spend a substantial amount of time summarizing vast amounts of textual information—whether it be compiling diagnostic reports, writing progress notes or synthesizing a patient’s treatment history across different specialists1–3.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
What the sources say
- Not checked — no full text of the sourcecites [1] Golob, J. F. Jr, Como, J. J. & Claridge, J. A. The painful truth: the documentation burden of a trauma surgeon. J. Trauma Acute Care Surg. 80, 742–747 (2016). source
- Partly supportedcites [2] Arndt, B. G. et al. Tethered to the EHR: primary care physician workload assessment using EHR event log data and time–motion observations. Ann. Fam. Med. 15, 419–426 (2017). sourceExperimental — accuracy not yet measured
The Time Spent on Various Tasks table (p. 4) reports 84 minutes per day on documentation and 60 minutes reviewing notes among the family medicine physicians studied. Study Limitations (p. 6) also describes reviewing prior notes before writing a progress note. The paper does not establish how much of that time was spent summarizing vast amounts of text, compiling diagnostic reports, or synthesizing histories across specialists. (LLM, conf=0.86)
- Supportedcites [3] Fleming, S. L. et al. MedAlign: a clinician-generated dataset for instruction following with electronic medical records. Preprint at https://doi.org/10.48550/arXiv.2308.14089 (2023). source
The introductory passage (section marked “+”; page not specified) describes clinicians’ information-related tasks as a significant burden, citing 10 hours a week spent generating documentation. It specifically gives summarizing a patient’s treatment history across specialists as an example, supporting the citing sentence’s overall point. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Every source the paper cites for this claim was checked and supports it.
In the paper
“Other studies employ human experts to evaluate synthesized abstracts, demonstrating that NLP metrics are not sufficient to measure summary quality56.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
What the sources say
- Supportedcites [56] Tang, L. et al. Evaluating large language models on medical evidence summarization. NPJ Digit. Med. 6, 158 (2023). source
The Methods section (Design of the human evaluation) says six domain experts evaluated generated summaries. The Discussion says automatic metrics fell short of measuring factual inconsistency, potential medical harmfulness, and human preference, making human evaluation essential. The supplied passages have section labels but no page numbers. (LLM, conf=0.94)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Quantitative assessments with syntactic, semantic and conceptual NLP metrics reveal trade-offs between models and adaptation methods.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A clinical reader study with 10 physicians evaluated summary completeness, correctness and conciseness; in most cases, summaries from our best-adapted LLMs were deemed either equivalent (45%) or superior (36%) compared with summaries from medical experts.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Although FLAN-T5 was more competitive for syntactic metrics, such as BLEU42, this model is constrained to a shorter context length of 512 (Extended Data Table 2).”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Head-to-head model comparison.
What the sources say
- Not checked — no full text of the sourcecites [42] Papineni, K., Roukos, S., Ward, T. & Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proc. of the 40th Annual Meeting of the Association for Computational Linguistics. https://dl.acm.org/doi/pdf/10.3115/1073083.1073135 (Association for Computing Machinery, 2002). source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“When considering conciseness scores (Fig. 4b), radiology reports were the only task in which physicians did not prefer the best model’s summaries to the medical experts.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“FLAN-T5 emerged as the best-performing model with QLoRA.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Comparison of adaptation strategies.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, the metrics BERTScore and MEDCON correlated most strongly with physician preference for correctness.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We see a clear example in Extended Data Fig. 3 where the model mistakenly generated (hallucinated) several absent conditions, such as eosinophilia.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Analyzing reader study results
boostedSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 58
- It may state two separate claims as one. 40
- The quote only partly states this claim. 1
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Large language models (LLMs) have gained remarkable traction, leading to widespread adoption of models such as ChatGPT17, which excel at information retrieval, nuanced understanding and text generation18,19.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
What the sources say
- Partly supportedcites [17] Brown, T. et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33 https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf (NeurIPS, 2020). sourceExperimental — accuracy not yet measured
The Conclusion (p. 40) reports strong performance on many NLP tasks and high-quality text generation, supporting the generation part of the claim. But the paper evaluates GPT-3 rather than ChatGPT; its closed-book question answering does not test information retrieval (Section 3.2, p. 13), and its NLI results describe important limits to language understanding (Section 3.8, p. 20). The separate citation attached to ChatGPT’s adoption is not judged here. (LLM, conf=0.88)
- Supportedcites [18] Zhao, W. X. et al. A survey of large language models. Preprint at https://doi.org/10.48550/arXiv.2303.18223 (2023). source
The Introduction (p. 2) describes ChatGPT’s global impact and LLMs’ integration into search platforms and content-creation tools. The Key concepts section (p. 3) describes their ability to process and produce natural language across a wide range of tasks, supporting the broad characterization of their capabilities. (LLM, conf=0.84)
- Partly supportedcites [19] Bubeck, S. et al. Sparks of artificial general intelligence: early experiments with GPT-4. Preprint at https://doi.org/10.48550/arXiv.2303.12712 (2023). sourceExperimental — accuracy not yet measured
The Introduction describes GPT-4’s comprehension and text-generation capabilities, and §2.1 says ChatGPT shows some understanding. But the paper does not establish that these models excel at information retrieval: its discussion of Figure 5.1 says current knowledge is hard for language models and describes GPT-4 using external search tools to overcome that limitation. The widespread-adoption claim is attached to a different citation (17). (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“These tasks divert attention from direct patient care, leading to worse outcomes for patients and decreased job satisfaction for clinicians2,14–16.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
What the sources say
- Partly supportedcites [2] Arndt, B. G. et al. Tethered to the EHR: primary care physician workload assessment using EHR event log data and time–motion observations. Ann. Fam. Med. 15, 419–426 (2017). sourceExperimental — accuracy not yet measured
The Introduction (p. 1) links documentation and other EHR tasks to less face-to-face patient time and clinician dissatisfaction. The paper does not establish that documentation tasks lead to worse patient outcomes; its discussion of poorer clinical outcomes concerns electronic communication among team members and patients with coronary artery disease (Communication, p. 6). (LLM, conf=0.88)
- Not checked — no full text of the sourcecites [14] Shanafelt, T. D. et al. Relationship between clerical burden and characteristics of the electronic environment with physician burnout and professional satisfaction. Mayo Clin. Proc. 91, 836–848 (2016). source
- Partly supportedcites [15] Robinson, K. E. & Kersey, J. A. Novel electronic health record (EHR) education intervention in large healthcare organization improves quality, efficiency, time, and impact on burnout. Medicine (Baltimore) 97, e12319 (2018). sourceExperimental — accuracy not yet measured
The Introduction (page not provided) says physicians spend substantial time on EHR and desk work relative to direct patient contact, and links EHR documentation to impaired patient communication and increased career dissatisfaction. It does not establish that documentation tasks lead to worse patient outcomes; the Discussion addresses patient care but does not make that causal claim. (LLM, conf=0.87)
- Partly supportedcites [16] Toussaint, W. et al. Design considerations for high impact, automated echocardiogram analysis. Preprint at https://doi.org/10.48550/arXiv.2006.06292 (2020). sourceExperimental — accuracy not yet measured
In Design Discoveries (p. 2), the paper says cardiologists spend 4–5 hours a day annotating echocardiograms and suggests automation could free them to focus more attention on sick patients. It does not characterize this work as documentation or say that it causes worse patient outcomes or decreased clinician job satisfaction. (LLM, conf=0.78)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Similarly, documentation responsibilities for nurses can consume up to 60% of their time and account for considerable work stress11–13.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
What the sources say
- Partly supportedcites [11] Khamisa, N., Peltzer, K. & Oldenburg, B. Burnout in relation to specific contributing factors and health outcomes among nurses: a systematic review. Int. J. Environ. Res. Public Health 10, 2214–2240 (2013). sourceExperimental — accuracy not yet measured
The Results (§3.2–3.3; page unavailable) identify workload as a prominent stressor among nurses, supporting the general point about work stress. They do not identify documentation as the source of that stress or report that it consumes up to 60% of nurses’ time; searches for documentation, paperwork, and the time figure found no such passage. (LLM, conf=0.80)
- Not checked — no full text of the sourcecites [12] Duffy, W. J., Kharasch, M. S. & Du, H. Point of care documentation impact on the nurse–patient interaction. Nurs. Adm. Q. 34, E1–E10 (2010). source
- Not checked — no full text of the sourcecites [13] Chang, C.-P., Lee, T.-T., Liu, C.-H. & Mills, M. E. Nurses’ experiences of an initial and reimplemented electronic health record use. Comput. Inform. Nurs. 34, 183–190 (2016). source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The widespread adoption of electronic health records has expanded clinical documentation workload, directly contributing to increasing stress and clinician burnout7–9.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
What the sources say
- Not matchedcites [7] Gesner, E., Gazarian, P. & Dykes, P. The burden and burnout in documenting patient care: an integrative literature review. Stud. Health Technol. Inform. 21, 1194–1198 (2019).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites [8] Ratwani, R. M. et al. A usability and safety analysis of electronic health records: a multi-center study. J. Am. Med. Inform. Assoc. 25, 1197–1201 (2018). sourceExperimental — accuracy not yet measured
The Background and significance section says EHR usability problems create burdensome workflows and inefficiencies that contribute to clinician frustration. It does not say that widespread EHR adoption increased clinical documentation workload or directly caused increasing stress and burnout. The Discussion examines task time, clicks, and errors, not those broader outcomes; page numbers are unavailable in our copy. (LLM, conf=0.84)
- Not checked — no full text of the sourcecites [9] Ehrenfeld, J. M. & Wanderer, J. P. Technology as friend or foe? Do electronic health records increase burnout? Curr. Opin. Anaesthesiol. 31, 357–360 (2018). source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Many previous clinical NLP studies rely primarily on quantitative metrics for evaluation41,52–54.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
What the sources say
- Partly supportedcites [41] Yim, W.W. et al. Aci-bench: a novel ambient clinical intelligence dataset for benchmarking automatic visit note generation. Sci. Data https://doi.org/10.1038/s41597-023-02487-3 (2023). sourceExperimental — accuracy not yet measured
In Technical Validation > Baseline experiments (page unavailable), the paper calls automatic evaluation metrics “widely-used” and evaluates its own clinical-note generation models with quantitative scores. It does not establish that many previous clinical NLP studies relied primarily on quantitative rather than other forms of evaluation. (LLM, conf=0.82)
- Supportedcites [52] Ma, C. et al. ImpressionGPT: an iterative optimizing framework for radiology report summarization with chatGPT. Preprint at https://doi.org/10.48550/arXiv.2304.08448 (2023). source
The Dataset and Evaluation Metrics section (p. 8) uses ROUGE and factual-consistency scores and notes that earlier studies reported factual-consistency metrics. The Comparison with Other Methods section (p. 9) evaluates numerous earlier radiology-report summarization methods using ROUGE and factual-consistency metrics, supporting the broad statement about quantitative evaluation in prior clinical NLP work. (LLM, conf=0.87)
- Not checked — no full text of the sourcecites [53] Wei, S. et al. Medical question summarization with entity-driven contrastive learning. Preprint at https://doi.org/10.48550/arXiv.2304.07437 (2023). source
- Supportedcites [54] Manakul, P. et al. CUED at ProbSum 2023: Hierarchical ensemble of summarization models. In The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks 516–523 (Association for Computational Linguistics, 2023). source
The Experiments tables report the paper’s clinical summarization results using numerical ROUGE scores, and the Limitations section explicitly identifies ROUGE-L as its assessment metric. This supports citing the paper as an example of a clinical NLP study evaluated primarily with quantitative metrics; the paper alone does not establish how many other studies do so. (LLM, conf=0.87)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Previous work has demonstrated potential across clinical NLP tasks22,23, adapting to the medical domain by training a new model24,25, fine-tuning an existing model26,27 or supplying task-specific examples in the model prompt27,28.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
What the sources say
- Supportedcites [22] Wornow, M. et al. The shaky foundations of large language models and foundation models for electronic health records. NPJ Digit. Med. 6, 135 (2023). sourceExperimental — accuracy not yet measured
The “State of published clinical FMs—CLaMs” section describes models trained on clinical or biomedical text and evaluated on clinical NLP tasks. “Benefits of clinical FMs” describes fine-tuning and few-shot adaptation, while the evaluation discussion contrasts fine-tuning with prompting. The paper cautions that performance on these tasks does not establish clinical deployment value, but the claim asserts only potential. (LLM, conf=0.86)
- Not checked — no full text of the sourcecites [23] Thirunavukarasu, A. J. et al. Large language models in medicine. Nat. Med. 29, 1930–1940 (2023). source
- Partly supportedcites [24] Singhal, K. et al. Large language models encode clinical knowledge. Nature https://doi.org/10.1038/s41586-023-06291-2 (2023). sourceExperimental — accuracy not yet measured
The Methods > Modelling passages describe adapting a model to medical question-answering tasks with task-specific examples in prompts and instruction prompt tuning. They also say the authors used prompt tuning instead of full-model fine-tuning, keeping the model’s parameters frozen. Thus, this source supports the prompting part of the attributed claim, but not the suggestion that this study fine-tuned the existing model itself. (LLM, conf=0.86)
- Supportedcites [25] Tu, T. et al. Towards generalist biomedical AI. Preprint at https://doi.org/10.48550/arXiv.2307.14334 (2023). sourceExperimental — accuracy not yet measured
The citation is attached to fine-tuning an existing model and supplying task-specific examples in prompts, not to training a new model. The 'Dataset and preprocessing' section describes fine-tuning and adding one-shot exemplars to task prompts; '5.1 Evaluation on MultiMedBench' reports evaluation across biomedical tasks, including language-only tasks. No page numbers are available in our copy. (LLM, conf=0.91)
- Supportedcites [26] Toma, A. et al. Clinical Camel: an open-source expert-level medical language model with dialogue-based knowledge encoding. Preprint at https://doi.org/10.48550/arXiv.2305.12031 (2023). sourceExperimental — accuracy not yet measured
The Introduction describes Clinical Camel as a medical model fine-tuned from LLaMA-2 and reports promising medical-benchmark results. Section 4 evaluates it in five-shot settings, supporting the cited point about supplying task-specific examples. The citations for training a new model and for the sentence’s opening claim are separate from this source. (LLM, conf=0.88)
- Supportedcites [27] Van Veen, D. et al. RadAdapt: radiology report summarization via lightweight domain adaptation of large language models. In 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks 449–460 (Association for Computational Linguistics, 2023). sourceExperimental — accuracy not yet measured
Citation 27 is attached to fine-tuning an existing model and supplying task-specific examples in prompts. RadAdapt’s Section 3.2 describes both methods, and Section 5.1 reports that fine-tuning outperformed discrete prompting while adding in-context examples improved performance for almost every model studied. (LLM, conf=0.92)
boostedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Prompt phrasing and model temperature can have a considerable effect on LLM output, as demonstrated in the literature45,46 and in Fig. 2.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
What the sources say
- Not matchedcites [45] Strobelt, H. et al. Interactive and visual prompt engineering for ad-hoc task adaptation with large language models. IEEE Trans. Vis. Comput. Graph. 29, 1146–1156 (2022).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites [46] Wang, J. et al. Prompt engineering for healthcare: methodologies and applications. Preprint at https://doi.org/10.48550/arXiv.2304.14670 (2023). sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) says prompt design can significantly affect model performance, and Automated Prompts describes improved performance from prompt paraphrasing. The copy does not address model temperature (searched: “temperature”), so it supports the prompt-phrasing part but not the temperature part. (LLM, conf=0.87)
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“considering that Alpaca is known to perform worse than our other open-source autoregressive models, Vicuna and Llama-2 (refs. 21,35)”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Impact of domain-specific fine-tuning.
What the sources say
- Partly supportedcites [21] Zheng, L. et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Preprint at https://doi.org/10.48550/arXiv.2306.05685 (2023). sourceExperimental — accuracy not yet measured
Table 8 (Position bias results; page not provided) reports lower MT-Bench scores for Alpaca-7B and Alpaca-13B (4.54 and 4.53) than for the listed Vicuna variants (5.95–6.39). The passages do not compare Alpaca with Llama-2; the LLaMA models listed are not identified as Llama-2. (LLM, conf=0.87)
- Not matchedcites [35] The Vicuna Team. Vicuna: an open-source chatbot impressing GPT-4 with 90%* ChatGPT quality. https://lmsys.org/blog/2023-03-30-vicuna/ (2023).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“For correctness, the metrics BERTScore44 (measuring semantics) and MEDCON41 (measuring medical concepts) correlated most strongly with reader preference; meanwhile, the BLEU42 metric (measuring syntax) correlated most with completeness and least with conciseness.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Connecting quantitative and clinical evaluations
What the sources say
- Not matchedcites [44] Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q. & Artzi, Y. BERTScore: evaluating text generation with BERT. International Conference on Learning Representations. https://openreview.net/forum?id=SkeHuCVFDr (2020).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Could not verifycites [41] Yim, W.W. et al. Aci-bench: a novel ambient clinical intelligence dataset for benchmarking automatic visit note generation. Sci. Data https://doi.org/10.1038/s41597-023-02487-3 (2023). sourceExperimental — accuracy not yet measured
Technical Validation > Baseline experiments (page unavailable) describes BERTScore and medcon as evaluation metrics, but does not report correlations with reader preference, correctness, completeness, or conciseness. Searches for those terms and BLEU found no passage addressing the claimed comparisons; the paper discusses BLEURT instead. (LLM, conf=0.78)
- Not checked — no full text of the sourcecites [42] Papineni, K., Roukos, S., Ward, T. & Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proc. of the 40th Annual Meeting of the Association for Computational Linguistics. https://dl.acm.org/doi/pdf/10.3115/1073083.1073135 (Association for Computing Machinery, 2002). source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
Every source the paper cites for this claim was checked and supports it.
In the paper
“To address this, there have been recent releases of expert evaluations for adjacent clinical NLP tasks3,55.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
What the sources say
- Supportedcites [3] Fleming, S. L. et al. MedAlign: a clinician-generated dataset for instruction following with electronic medical records. Preprint at https://doi.org/10.48550/arXiv.2308.14089 (2023). source
The Clinical dataset description says MedAlign includes clinician evaluations of LLM-generated outputs, and the Clinician Evaluation of LLM Responses section describes clinicians judging and ranking those outputs. The Discussion and Conclusion says the authors share the benchmark dataset with the research community, making MedAlign an example of the cited kind of expert evaluation. The copy does not give a page number for these passages. (LLM, conf=0.88)
- Supportedcites [55] Yu, F. et al. Evaluating progress in automatic chest x-ray radiology report generation. Patterns (N Y) 4, 100802 (2023). source
The Results section, “Alignment between automated metrics and radiologists,” reports radiologist error scores for 50 studies and says they are shown in Figure S3. The paper therefore presents an expert evaluation of the adjacent clinical NLP task of chest X-ray report generation. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“This simple approach provides the model with context, enhancing LLM performance for a particular task or domain27,28.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Adaptation methods
What the sources say
- Supportedcites [27] Van Veen, D. et al. RadAdapt: radiology report summarization via lightweight domain adaptation of large language models. In 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks 449–460 (Association for Computational Linguistics, 2023). source
The Related Work section describes prompting as providing context for a task. Section 5.1 reports that adding in-context examples improved radiology report summarization performance for almost every model tested, with T5 as the exception. (LLM, conf=0.90)
- Supportedcites [28] Mathur, Y. et al. SummQA at MEDIQA-Chat 2023: in-context learning with GPT-4 for medical summarization. Preprint at https://doi.org/10.48550/arXiv.2306.17384 (2023). source
Section 2 (Background) describes giving models task instructions or examples in context. Section 5.5 reports that, for the paper's medical summarization task, generated summaries generally improved as more relevant in-context examples were provided. No page numbers are supplied in our copy's passages. (LLM, conf=0.90)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“This quantization negligibly impacts performance40; as such, we used QLoRA for all model training.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Adaptation methods
What the sources say
- Supportedcites [40] Dettmers, T., Pagnoni, A., Holtzman, A. & Zettlemoyer, L. QLoRA: efficient finetuning of quantized LLMs. Preprint at https://doi.org/10.48550/arXiv.2305.14314 (2023). source
The Summary reports that 4-bit QLoRA with NF4 matched 16-bit finetuning performance on the evaluated benchmarks, with no observed degradation relative to full finetuning. This supports the claim in the context of QLoRA model training; the paper separately notes that 4-bit quantization for inference alone can degrade performance. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“We further note the potential for LLMs to be biased58,59.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
What the sources say
- Supportedcites [58] Omiye, J. A., Lester, J. C., Spichak, S., Rotemberg, V. & Daneshjou, R. Large language models propagate race-based medicine. NPJ Digit. Med. 6, 195 (2023). source
The Abstract reports that all four LLMs studied produced examples of race-based medicine, directly supporting the potential for LLM bias. It also says LLMs may amplify biases present in their training data. (LLM, conf=0.95)
- Not checked — no full text of the sourcecites [59] Zack, T. et al. Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study. Lancet Digit. Health 6, e12–e22 (2024). source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Note that choosing ‘relevant’ in-context examples has been shown to outperform choosing examples at random69.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Adaptation methods
What the sources say
- Partly supportedcites [69] Nie, F., Chen, M., Zhang, Z. & Cheng, X. Improving few-shot performance of language models via nearest neighbor calibration. Preprint at https://doi.org/10.48550/arXiv.2212.02216 (2022). sourceExperimental — accuracy not yet measured
The Experimental Setup (p. 5) uses randomly sampled in-context demonstrations as a baseline, and the Analysis (p. 7) reports improvements from retrieving similar support samples. But those samples calibrate predictions through a nearest-neighbor classifier; the paper does not show that selecting relevant examples *as in-context demonstrations* outperforms selecting random demonstrations. (LLM, conf=0.86)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“As we are not the first to employ these datasets, Extended Data Table 4 contains quantitative metric scores from other works25,27,41,52–54 that developed methods specific to each individual summarization task.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Data
What the sources say
- Partly supportedcites [25] Tu, T. et al. Towards generalist biomedical AI. Preprint at https://doi.org/10.48550/arXiv.2307.14334 (2023). sourceExperimental — accuracy not yet measured
The Additional information section says the paper used a previously published MIMIC-III radiology-report summarization dataset. But Section 5.1 describes Med-PaLM M as jointly fine-tuned across tasks and distinguishes it from prior specialist models; it does not support describing this paper’s method as specific to an individual summarization task. Our copy provides no page numbers for these passages. (LLM, conf=0.86)
- Supportedcites [27] Van Veen, D. et al. RadAdapt: radiology report summarization via lightweight domain adaptation of large language models. In 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks 449–460 (Association for Computational Linguistics, 2023). source
The Data section (4.1) identifies MIMIC-III as RadAdapt’s primary dataset and says it also evaluates on MIMIC-CXR. Section 3.2 describes adaptation methods applied to radiology report summarization, so the paper serves as an example of prior work using these datasets for a specific summarization task. (LLM, conf=0.88)
- Supportedcites [41] Yim, W.W. et al. Aci-bench: a novel ambient clinical intelligence dataset for benchmarking automatic visit note generation. Sci. Data https://doi.org/10.1038/s41597-023-02487-3 (2023). source
The paper’s “Baseline experiments” section says its authors used the aci-bench dataset to benchmark several note-generation methods and report quantitative results. It therefore serves as an example of prior work employing one of the datasets for a specific summarization task. The citing paper’s Extended Data Table 4 is its own presentation of those scores. (LLM, conf=0.90)
- Supportedcites [52] Ma, C. et al. ImpressionGPT: an iterative optimizing framework for radiology report summarization with chatGPT. Preprint at https://doi.org/10.48550/arXiv.2304.08448 (2023). source
The Introduction (p. 2) says the authors developed ImpressionGPT and evaluated it on MIMIC-CXR and OpenI. The comparison section (p. 9) reports its quantitative metrics alongside other radiology-report summarization methods on those datasets. This supports citing the paper as an earlier work using the datasets and developing a method for that summarization task. (LLM, conf=0.91)
- Not checked — no full text of the sourcecites [53] Wei, S. et al. Medical question summarization with entity-driven contrastive learning. Preprint at https://doi.org/10.48550/arXiv.2304.07437 (2023). source
- Supportedcites [54] Manakul, P. et al. CUED at ProbSum 2023: Hierarchical ensemble of summarization models. In The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks 516–523 (Association for Computational Linguistics, 2023). source
The Introduction (§1; page not supplied) identifies this paper as work on the ProbSum medical problem-list summarization task. The Experimental Setup and Ensemble Methods (§4.1 and §4.3; pages not supplied) describe its use of the task data, task-specific method, and quantitative ROUGE results. It therefore serves as an example of the prior task-specific work cited in the sentence. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“inspired by the AHRQ harm scale43.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Connecting errors to medical harm
What the sources say
- Supportedcites [43] Walsh, K. E. et al. Measuring harm in healthcare: optimizing adverse event review. Med. Care 55, 436–441 (2017). source
The Methods (Rater Assessment of Harm Severity) describe the revised AHRQ Harm Scale and its six harm-severity options. The citation identifies the scale that inspired the citing authors’ supplemental reader study; it need not establish the citing authors’ own motivation. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Compared to training model weights from scratch, LoRA is much more efficient with respect to both computational requirements and the volume of training data required.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Adaptation methods
What the sources say
- Partly supportedcites [70] Hu, E. et al. LoRA: low-rank adaptation of large language models. Preprint at https://doi.org/10.48550/arXiv.2106.09685 (2021). sourceExperimental — accuracy not yet measured
The Introduction (p. 2) says LoRA makes training more efficient, and §4.2 (p. 5) reports lower GPU memory use and faster training than full fine-tuning. Those comparisons are with adapting a pretrained model, not training one from scratch. The low-data experiments (§F.3, p. 22) do not establish that LoRA requires less training data than training from scratch. (LLM, conf=0.86)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The ensuing safety analysis highlights challenges faced by both LLMs and medical experts, as we connect errors to potential medical harm and categorize types of fabricated information.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Consider example 1 in Extended Data Fig. 4, in which the input question mentioned ‘diabetes and neuropathy’. The model mirrored this phrasing verbatim, whereas the medical expert interpreted it as ‘diabetic neuropathy’.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Seq2seq models (FLAN-T5 and FLAN-UL2) performed very well on syntactical metrics, such as BLEU, but worse on others (Fig. 3d)”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“medical expert summaries were preferred in only a minority of cases (19%), whereas, in a majority, the best model was either non-inferior (45%) or preferred (36%).”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Analyzing reader study results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In terms of trade-offs between lightweight adaptation methods, while QLoRA fine-tuning performed similarly for some cases, ICL was the best overall, especially when including proprietary models GPT-3.5 and GPT-4.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“pooled results across 10 physicians (Fig. 4b) demonstrate that summaries from the best-adapted model (GPT-4 using ICL) were more complete and contained fewer errors compared to medical expert summaries”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Analyzing reader study results
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In example 2 of Extended Data Fig. 4, the model simply reformulated the input question about tests and their locations, whereas the medical expert inferred a broader query about tests and treatments.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, our best-adapted model achieved non-inferior results to medical experts while performing a basic search across 1–2 options for each task instruction (Extended Data Table 1).”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Regarding conciseness, the best model performed significantly better than medical experts (P < 0.001) overall and on two tasks, whereas, for radiology reports, it performed similarly to medical experts.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Analyzing reader study results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that the model committed misinterpretations, inaccuracies and hallucinations on 6%, 2% and 5% of samples, respectively, compared to 9%, 4% and 12%, respectively, by medical experts.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Fabricated information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although ICL and QLoRA were competitive for open-source models, proprietary models GPT-3.5 and GPT-4 far outperformed other models and methods given sufficient in-context examples.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Effect of context length for ICL.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Regarding correctness, the best model generated significantly fewer errors (P < 0.001) compared to medical expert summaries (Fig. 4b) overall and on two of three summarization tasks.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Analyzing reader study results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observed that the best model summaries were more complete, on average, than medical expert summaries, achieving statistical significance across all three summarization tasks with P < 0.001 (Fig. 4b).”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Analyzing reader study results
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“After searching over temperature values {0.1, 0.5, 0.9} using GPT-3.5, Fig. 2 demonstrates that the lowest value, 0.1, performed best.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Constructing prompt anatomy
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results of this harm study (Fig. 4d) indicate that the medical expert summaries would have both a higher likelihood (14%) and a higher extent (22%) of possible harm compared to the summaries from the best model (12% and 16%, respectively).”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Analyzing reader study results
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Among open-source models, seq2seq models performed better than autoregressive models (Llama-2 and Vicuna) on radiology reports but worse on patient questions and progress notes (Fig. 3c).”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Model performance generally improved with more context.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although the model was more complete than the expert in generating its progress note summary, it also missed historical context (a history of hypertension).”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Analyzing reader study results
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Figure 5 provides an example in which the model’s summary includes correct information that readers deemed not important.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Analyzing reader study results
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Per Extended Data Table 1, this task entailed the largest token count across our six datasets for both the input (dialogue) and the target (assessment).”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Data
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Due to the wide variety of these questions, MeQSum exhibited the highest lexical variance of our datasets (Extended Data Table 1).”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Data
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“because the best model and method uses a handful of examples via ICL”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The benefit of larger models is explored in Extended Data Fig. 5, which found this improvement marginal for Llama-2 (13B) compared to Llama-2 (7B).”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > LLMs
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“radiology report summaries from medical experts occasionally recommend further studies or refer to prior studies—for example, ‘… not significantly changed from prior’ in Fig. 5.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Recent data indicate that physicians can expend up to 2 hours on documentation for each hour of patient interaction10.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
What the sources say
- Not checked — no full text of the sourcecites [10] Sinsky, C. et al. Allocation of physician time in ambulatory practice: a time and motion study in 4 specialties. Ann. Intern. Med. 165, 753–760 (2016). source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“Even for experienced physicians with a high level of expertise, this intricate task naturally introduces the possibility for errors, which can be detrimental in healthcare where precision is paramount4–6.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
What the sources say
- Not checked — no full text of the sourcecites [4] Yackel, T. R. & Embi, P. J. Unintended errors with EHR-based result management: a case series. J. Am. Med. Inform. Assoc. 17, 104–107 (2010). source
- Not checked — no full text of the sourcecites [5] Bowman, S. Impact of electronic health record systems on information integrity: quality and safety implications. Perspect. Health Inf. Manag. 10, 1c (2013).
- Not checked — no full text of the sourcecites [6] Gershanik, E. F., Lacson, R. & Khorasani, R. Critical finding capture in the impression section of radiology reports. AMIA Annu. Symp. Proc. 2011, 465–469 (2011).
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This suggests that integrating LLMs into clinical workflows could alleviate documentation burden, allowing clinicians to focus more on patient care.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We note that the model’s summaries are more concise while concurrently being more complete.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Analyzing reader study results
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This could be attributed to the relatively vague length specification in the radiology reports instruction—that is, ‘…with minimal text’—whereas the other two task instructions quantify length.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This illustrates the value of relevant context in achieving better outcomes for the target task.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Constructing prompt anatomy
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Governance aside, ICL provides many benefits: (1) model weights are fixed, hence enabling queries of pre-existing LLMs, and (2) adaptation is feasible with even a few examples, whereas fine-tuning methods, such as QLoRA, typically require hundreds or thousands of examples.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Note that seq2seq models are often constrained to much shorter context length than autoregressive models, such as GPT-4, because seq2seq models require the memory-intensive step of encoding the input sequence into a fixed-size context vector.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The evidence from our reader study suggests that adapting LLMs can outperform medical experts in terms of completeness, correctness and conciseness.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In both cases, the model’s summaries leaned toward literalness, a trait that readers sometimes favored and sometimes did not.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This instance underscores the point that even medical experts, not just LLMs, can hallucinate.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Analyzing reader study results
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The ensuing clinical reader study demonstrates that LLM summaries are often preferred over medical expert summaries due to higher scores for completeness, correctness and conciseness.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Given that BLEU measures sequence overlap, this result seems reasonable, as more text provides more ‘surface area’ for overlap; more text also reduces the brevity penalty that BLEU applies on generated sequences, which are shorter than the reference42.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
What the sources say
- Not checked — no full text of the sourcecites [42] Papineni, K., Roukos, S., Ward, T. & Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proc. of the 40th Annual Meeting of the Association for Computational Linguistics. https://dl.acm.org/doi/pdf/10.3115/1073083.1073135 (Association for Computing Machinery, 2002). source
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This highlights the model’s tendency toward a literal approach without interpretation, which may be either advantageous or limiting.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
hedgedSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Addressing this limitation presents an opportunity to accelerate the process of clinical text summarization, hence alleviating documentation burden and improving patient care.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We first highlight the importance of ‘prompt engineering’, or modifying and tuning the input prompt to improve model performance.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the low magnitude of correlation values (approximately 0.2) underscores the need to go beyond NLP metrics with a clinical reader study when assessing clinical readiness.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Connecting quantitative and clinical evaluations
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Regarding blinding, for the clinical reader study, we presented, in a blinded manner, the A/B comparison of summaries from the model and human experts.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Statistics and reproducibility
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“it is not possible to verify whether a particular open-source dataset was included in model training.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When human evaluation is not feasible, Fig. 6 suggests that syntactic metrics are better at measuring completeness, whereas semantic and conceptual metrics are better at measuring correctness.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Compared to x-rays, MRIs and CT scans capture more information at a higher resolution. This usually leads to longer reports (Extended Data Table 1), rendering MIMIC-III a more challenging summarization dataset than Open-i or MIMIC-CXR.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Data
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“This study used only retrospective, de-identified data that fell outside the scope of institutional review board oversight.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Ethics approval
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The dialogue task was excluded owing to the unwieldiness of a reader parsing many lengthy transcribed conversations and paragraphs”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Reader study
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To obfuscate any formatting differences between A and B, we applied simple post-processing to standardize capitalization, punctuation, newline characters, etc.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Statistics and reproducibility
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For the quantitative experiments, our sample size of 250 was chosen to maximize the number of samples given constraints of dataset size and cost of computing resources. We deem this sufficient as it enabled reproducible results when running the same experiments with a different set of samples.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Statistics and reproducibility
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Although our datasets do not contain demographic information”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, our use of six datasets enhances the robustness of our experiments.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Statistics and reproducibility
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“This study adhered to the principles outlined in the Declaration of Helsinki.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Ethics approval
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“for the quantitative experiments, we randomly selected 250 test samples and then randomly divided remaining samples into training and validation sets.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Statistics and reproducibility
plainn = 250- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Informed consent was obtained from each physician before their participation.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Ethics approval
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Because the LLM temperature parameter is near zero, outputs have very little randomness and are, thus, reproducible.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Statistics and reproducibility
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“No data were excluded from the analyses.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Statistics and reproducibility
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These instances are out of scope for the LLM, as it does not have access to prior studies nor the purview to make recommendations.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Given the critical nature of nuanced medical tasks, such as summarization, that typically have no objective solutions, including human experts in the evaluation process beyond quantitative metrics is crucial to demonstrate clinical readiness.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For the clinical reader study, our sample size of 100 comparisons per reader per task was chosen to maximize the number of comparisons that could be made in a reasonable amount of time, which we estimated as 10 h. We deem this sufficient as it enabled statistically significant results for many combinations of tasks and attributes.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Statistics and reproducibility
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This suggests that incorporating LLM-generated candidate summaries could reduce documentation load, potentially leading to decreased clinician strain and improved patient care.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
In the paper
“Although three of our datasets (MIMIC-CXR, MIMIC-III and ProbSum) require PhysioNet57 access to ensure safe data usage by third parties, this is no guarantee against data leakage.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
What the sources say
- Not checked — no full text of the sourcecites [57] Johnson, A., Pollard, T. & Mark, R. MIMIC-III Clinical Database Demo (version 1.4). PhysioNet https://doi.org/10.13026/C2HM2Q (2019). source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“we did not consider the inherently context-specific nature of summarization.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The quote only partly states this claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Note that we reduced the temperature parameter of the LLMs to be near zero, hence reducing randomness in the generated output summaries.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Statistics and reproducibility
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For the clinical reader study, we randomly selected 100 samples for comparison; these samples and the A/B comparisons within each sample are displayed in random order.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Statistics and reproducibility
plainn = 100- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“we chose the smallest dataset––which consequently required the lowest cost of computational resources––and re-ran a set of experiments that rendered very similar quantitative metric scores. This provided confidence moving forward with datasets that required higher computational resources.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Statistics and reproducibility
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Although three of our datasets (MIMIC-CXR, MIMIC-III and ProbSum) require PhysioNet57 access to ensure safe data usage by third parties”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
What the sources say
- Not checked — no full text of the sourcecites [57] Johnson, A., Pollard, T. & Mark, R. MIMIC-III Clinical Database Demo (version 1.4). PhysioNet https://doi.org/10.13026/C2HM2Q (2019). source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
The effectiveness of large language models on a diverse range of clinical summarization tasks remains unproven.
“Although large language models (LLMs) have shown promise in natural language processing (NLP) tasks, their effectiveness on a diverse range of clinical summarization tasks remains unproven.”
The paper relies on this without support.
Adapting LLMs to summarize a diverse set of clinical tasks has not been thoroughly explored, and non-inferiority to medical experts has not been achieved.
“However, adapting LLMs to summarize a diverse set of clinical tasks has not been thoroughly explored nor has non-inferiority to medical experts been achieved.”
The paper relies on this without support.
Fabricated information poses a substantial obstacle to clinical integration of LLMs because medical applications require accuracy.
“Fabricated information, or factually incorrect text, poses a substantial obstacle to the clinical integration of LLMs given the critical need for accuracy in medical applications.”
The authors state this as an assumption.
Samples with similar A/B summaries in completeness and correctness were assumed to contribute no harm.
“the subset of samples with similar A/B summaries (in completeness and correctness) were assumed to contribute no harm.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Most have been tested only with text-based, low-stakes interactions,33–38”
From Review of Artificial Intelligence for Clinical Use in Alzheimer's Disease and Related Dementias (2026) · cites this paper as [34]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The first four models were selected because they were among the most widely used LLMs at the time of study inception (January 2024), and prior research found that LLaMA 2 and GPT models outperformed humans in medical text summarization [19], while Claude models demonstrated greater faithfulness in book summaries than GPT [20].”
From Generalization bias in large language model summarization of scientific research (2025) · cites this paper as [19]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Scientists use LLM-assisted writing because LLMs can improve grammar, rhetoric, and overall readability of their texts, help translate to English, and quickly generate summaries (33, 34).”
From Delving into LLM-assisted writing in biomedical publications through excess vocabulary (2025) · cites this paper as [33]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Our finding challenges previous studies on clinical text summarization which reported that ‘summaries from our best-adapted LLMs (including GPT-3.5, GPT-4, and LLaMA 2) are preferable to human summaries in terms of completeness and correctness’ [19, p. 3].”
From Generalization bias in large language model summarization of scientific research (2025) · cites this paper as [19]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Most LLM studies have depended on human lead semi-quantitative assessment of coherence, comprehensiveness, harmfulness and factual inconsistencies as well as comparing LLM with human based summaries [5, 13, 14].”
From A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population (2024) · cites this paper as [5]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Automated metrics do not correlate with quality and human input to score coherence, inconsistencies, comprehensiveness and harmfulness are semi-quantitative and will differ based on the use care in which LLM are applied [5, 13].”
From A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population (2024) · cites this paper as [5]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Most studies applying LLM on clinical notes have been conducted in the US without individual consent, using de-identified data, with only one study from France utilising retrospective MRI reports without identifiable data [5, 6, 13, 14, 17–20].”
From A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population (2024) · cites this paper as [5]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Despite comparable performance with clinician-written summaries (Veen et al., 2024), hallucinations manifest as fabricated or distorted clinical statements that are hard to detect (Asgari et al., 2025; Kim et al., 2025).”
From Hallucination Detection-Guided Preference Optimization for Clinical Summarization (2026) · cites this paper as Veen et al., 2024
The cited source says what the claim attributes to it.
In the citing paper
“LLMs are capable of processing large volumes of text and generating human-like summaries and interpretations [2].”
From Evaluating Medical Text Summaries Using Automatic Evaluation Metrics and LLM-as-a-Judge Approach: A Pilot Study (2025) · cites this paper as [2]
The cited source says what the claim attributes to it.
In the citing paper
“Domain-specific LLMs, fine-tuned on medical corpora or using knowledge graph embeddings, perform well in clinical text summarization [10, 11], note generation [12, 13], and medical question answering [14].”
From Enhancing hepatopathy clinical trial efficiency: a secure, large language model-powered pre-screening pipeline (2025) · cites this paper as [10]
The cited source says what the claim attributes to it.
In the citing paper
“A study by Van Veen et al. reported that medically adapted LLM can summarize text better than physicians [23].”
From Evaluating the Capability of Large Language Model Chatbots for Generating Plain Language Summaries in Radiology (2025) · cites this paper as [23]
The cited source says what the claim attributes to it.
In the citing paper
“Our system employs local language models to preserve data privacy, which comes at some cost to performance – the local model may not match that of commercial models58,59.”
From A large language model digital patient system enhances ophthalmology history taking skills (2025) · cites this paper as [59]
The cited source says what the claim attributes to it.
In the citing paper
“Many of the studies on LLM have been conducted using large anonymised US datasets [5–7].”
From A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population (2024) · cites this paper as [5]
The cited source says what the claim attributes to it.
In the citing paper
“Large language models (LLM) such as ChatGPT, GPT-4 API and Llama2 can interrogate and summarise large volumes of medical notes to create succinct summaries of radiology reports, internal medical progress notes and patient dialogue [5–8].”
From A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population (2024) · cites this paper as [5]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Our system employs local language models to preserve data privacy, which comes at some cost to performance – the local model may not match that of commercial models58,59.”
From A large language model digital patient system enhances ophthalmology history taking skills (2025) · cites this paper as [59]
Experimental — accuracy not yet measuredIn the citing paper
“LLMs have shown expert-level performance in clinical text summarization when appropriate prompting strategies are employed3.”
From Towards evaluating and building versatile large language models for medicine (2025) · cites this paper as [3]
In the citing paper
“Therefore, related works have instrumented quantitative (completeness, correctness, conciseness) and qualitative (content, language, structure) measurements to dissect the riveting characters that distinguish synthesized clinical reports from human reports46–49.”
From Towards a holistic framework for multimodal LLM in 3D brain CT radiology report generation (2025) · cites this paper as [46]
Citations
Cites
A guideline of selecting and reporting intraclass correlation coefficients for reliability research
Koo, T. K., & Li, M. Y. (2016) · Journal of Chiropractic Medicine · doi:10.1016/j.jcm.2016.02.012 · Cited by 31,709 · cited 1×
“We estimated intra-reader correlation based on a mean-rating, fixed-agreement, two-may mixed-effects model 90 using the Pingouin package 91 .”
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Goldberger, A. L., Amaral, L. A. N., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark, R. G., Mietus, J. E., Moody, G. B., Peng, C.-K., & Stanley, H. E. (2000) · Circulation · doi:10.1161/01.cir.101.23.e215 · Cited by 14,809
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 13,453
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
International Journal of Environmental Research and Public Health
Casariego Vales, E., A.M., C.-C., J.L., L., R.M., M.-P., D., O.-B., M., O.-C., P., R.-S., & F.J., S. (2013) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph · Cited by 7,742
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254 · cited 1×
“In addition, beyond the scope of this manuscript, there is further potential to improve accuracy through incorporating checks by human operators and checks by other LLMs or using a model ensemble 47,48 .”
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · cited 2×
“Previous work has demonstrated potential across clinical NLP tasks 22,23 , adapting to the medical domain by training a new model 24,25 , fine-tuning an existing model 26,27 or supplying task-specific examples in the model prompt 27,28 .”
Research works printed but not linked: 9
- Gesner, E., Gazarian, P. & Dykes, P. The burden and burnout in documenting patient care: an integrative literature review. Stud. Health Technol. Inform. 21, 1194–1198 (2019).
- Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q. & Artzi, Y. BERTScore: evaluating text generation with BERT. International Conference on Learning Representations. https://openreview.net/forum?id=SkeHuCVFDr (2020).
- Lin, C.-Y. ROUGE: a package for automatic evaluation of summaries. In Text Summarization Branches Out 74–81 (Association for Computational Linguistics, 2004).
- Lehman, E. et al. Do we still need clinical language models? In Proceedings of Machine Learning Research 209, 578–597 (Conference on Health, Inference, and Learning, 2023).
- Demner-Fushman, D., Ananiadou, S. & Cohen, K. B. The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks. https://aclanthology.org/2023.bionlp-1 (Association for Computational Linguistics, 2023).
- Gao, Y., Miller, T., Afshar, M. & Dligach, D. BioNLP Workshop 2023 Shared Task 1A: Problem List Summarization (version 1.0.0). PhysioNet. https://doi.org/10.13026/1z6g-ex18 (2023). · DOI
- Yim, W., Ben Abacha, A., Snider, N., Adams, G. & Yetisgen, M. Overview of the MEDIQA-Sum task at ImageCLEF 2023: summarization and classification of doctor–patient conversations. In CEUR Workshop Proceedings https://ceur-ws.org/Vol-3497/paper-109.pdf (2023). · DOI · Paper page
- Loshchilov, I. & Hutter, F. Decoupled weight decay regularization. In International Conference on Learning Representations https://openreview.net/forum?id=Bkg6RiCqY7 (2019)
- Okazaki, N. & Tsujii, J. Simple and efficient algorithm for approximate dictionary matching. In Proc. of the 23rd International Conference on Computational Linguistics https://aclanthology.org/C10-1096.pdf (Association for Computational Linguistics, 2010).
Cited by
Sorted by citations
Items 701–800 of 856
On the purpose of artificial intelligence in critical care medicine
Drudi, C., Matta, S. C., Lee, H., O’Donoghue, S. C., D’Couto, H. T., Hamza, A., Gutiérrez, C. A., Nakasi, R., Byers, J. T., Tumukunde, M., et al. (2026) · Journal of Translational Critical Care Medicine · doi:10.1097/jtccm-d-25-00021
3-D 1T-nC-1T Ferroelectric-Based KV Cache for Efficient Long-Context LLM Inference
Xu, W., Luo, D., Deng, M., Cao, S., Zhong, S., Li, M. J., Huang, Q., & Huang, R. (2026) · IEEE Transactions on Electron Devices · doi:10.1109/ted.2026.3670094
Menopause and large language models: not quite ready for mid-life
Kransdorf, L. N. (2026) · Menopause The Journal of The North American Menopause Society · doi:10.1097/gme.0000000000002789
AI-driven smart patient retrieval for precision oncology
Wang, Y.-R. J., & Chaudhari, A. S. (2026) · Nature reviews. Cancer · doi:10.1038/s41568-026-00923-1
Self-citationLuo, J., Ma, J., Qiu, Y., Wang, T., Yang, Y., Qiu, G., Chen, H., Pang, J., & Wang, W. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02591-9
“This highlights the need for future optimization to balance readability with sentence-level complexity 20 – 22 .”
Sun, J., Ma, S., Fan, Y., & Washington, P. Y. (2026) · Applied Sciences · doi:10.3390/app16073388 · Cited by 1
Mittman, B. G., D’Aleo, G., Rothberg, M. B., Casacchia, N. J., & Albahra, S. (2026) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-026-01932-0 · Cited by 1
Agkatsev, S., Templeton, M. D., Lazaridis, L., et al. (2026) · Neuro-Oncology Advances · doi:10.1093/noajnl/vdag083
Benchmarking LLM decision support in inflammatory aneurysms: DeepSeek-R1, DeepSeek-V3 and ChatGPT-4o
Hao, J., & Zhang, C. (2026) · Clinical and Experimental Medicine · doi:10.1007/s10238-026-02045-2
Virtual medicine: medical AI in human health and diseases
Zhang, C., Wang, F.-X., Tang, X., Li, J.-L., Ding, N., Hong, Y., Song, P., Bai, L., & Su, J. (2026) · Military Medical Research · doi:10.1016/j.mmr.2026.100012 · Cited by 1
“For example, in the clinical text summary task, studies have shown that with the assistance of the GPT model, the completeness and accuracy of clinical text summaries can reach or even exceed the level of medical experts, significantly improving the efficiency of medical documents [127,128].”
Li, Y., Yan, B.-C., Zhang, R., Song, X. Z., Chen, L., & Cao, Z. (2026) · doi:10.2196/preprints.96304
Leveraging a Large Language Model to Generate Quality Improvement Feedback for Clinical Notes
Kim, C., Gelfinbein, J., Gencerliler, N., Jahan, N., Udaikumar, J., Heery, L., Goodman, A., Ng, S., Attard, J., Asha, S., et al. (2026) · Applied Clinical Informatics · doi:10.1055/a-2851-0739
Assessing Llm Hallucinations And The Reliability Of Using LLms For Automated Hallucination DetectionPreprint
Cheong, J., Chen, D. S., Leung, W., Chou, C., Corcoran, C. M., Kelly, S., Bearden, C. E., Cecchi, G., Baker, J. T., Kane, J. M., et al. (2026) · Research Square · doi:10.21203/rs.3.rs-9394250/v1
Agaronnik, N. D., Davis, J., Sounack, T., Manz, C. R., & Lindvall, C. J. (2026) · Journal of Palliative Medicine · doi:10.1177/10966218261438105
One Health: AI in Healthcare Summarization
Izuchukwu, C., Wimmer, H., & Powell, L. M. (2026) · Journal of Information Systems Applied Research and Analytics · doi:10.62273/tdbt3523
Lee, J. K., Choi, S., Park, S., Choi, Q., Hwang, S., & Cho, D. (2026) · Vox Sanguinis · doi:10.1111/vox.70280
“Consistent with prior analyses of domain-adapted models [28, 34], these findings suggest that, while enhanced reasoning enriches contextual interpretation, achieving concise and clinically usable responses still requires controlled alignment and prompt calibration.”
Ye, Y., Colombo, M., Meessen, J. M. T. A., Hajizadeh, N., Vore, L. D., Sigarchian, H. G., Peters, C., Krovvidi, S., Meresh, S., Hasson, L., et al. (2026) · European Heart Journal - Digital Health · doi:10.1093/ehjdh/ztag070 · Cited by 1
“11 To date, most studies that explored the performance of LLMs for paper selection in systematic review mainly focused on title/ abstract screening.”
Hyperbolic neurodynamics and graph convolutional networks in drug combination prediction
Li, J., Chen, J., Shang, Y., & Lei, X. (2026) · Physica A Statistical Mechanics and its Applications · doi:10.1016/j.physa.2026.131637
Generative Artificial Intelligence–Driven Voice Assistance for Patient Education in Ophthalmology
Jacobs, A., Anselmo, D., McHugh, R., Fernandez‐Garcia, I., Fang, C., Lasko, R., Mukherjee, S., & Holekamp, N. M. (2026) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2026.1307
Tang, W., Zhou, J., Chen, W., Xie, Y., Zhang, L., Chen, S., Hou, X., Wang, S., & Shi, H. Z. (2026) · Scientific Reports · doi:10.1038/s41598-026-50745-2 · Cited by 1
“While utilizing LLMs has become a viable approach to achieve this, the traditional fine-tuning pathway is often hindered by prohibitive annotation costs and specialized expertise requirements, limiting its widespread adoption in clinical settings 19–22 .”
Generative AI in Admission Notes and Diagnostic Completeness: A Pilot Study
Rodrigues, A. C., Misurac, J. M., Knake, L. A., Barker, K., & Blum, J. M. (2026) · Applied Clinical Informatics · doi:10.1055/a-2876-0998
Liu, J., Liu, S., & Wright, A. T. (2025) · Journal of Medical Internet Research · doi:10.2196/90061 · Cited by 1
Xu, R., Liu, J., Ye, H., Yang, T., Jiang, K., & Zhu, X. (2026) · Computational and Structural Biotechnology Journal · doi:10.34133/csbj.0132
“…significantly lower patient comprehension rates compared to traditional delivery methods [ 39 ] This creates a demand for AI-driven summarization—a task where adapted LLMs have been shown to generate summaries comparable or superior to those of medical experts for certain clinical texts [ 40 ].”
Bouterse, A., Pruneski, J. A., Oettl, F. C., Zsidai, B., Longo, U. G., Tischer, T., Hirschmann, M. T., & Samuelsson, K. (2026) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1002/ksa.70428
Patient-Centered Intelligent Healthcare
Ma, T., & Ye, C. (2026) · Springer eBooks · doi:10.1007/978-981-95-5585-7_23
Murphy, K. B., Schilder, B. M., & Skene, N. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1794934 · Cited by 4
Local citation relationship classification for recommendation: a SciBERT-based DualFocusNet approach
Lu, Y., Yuan, M., & Chen, M. (2026) · Scientometrics · doi:10.1007/s11192-026-05627-9
“Within operational workflows, LLMs like LLaMA-3 facilitate cross-document relation extraction, achieving 85% accuracy in linking citations to methodological reuse (Van Veen et al., 2024).”
Baâzaoui, A., Khadhraoui, H., & Barhoumi, W. (2026) · Lecture notes in business information processing · doi:10.1007/978-3-032-26836-5_25
Retrieval-Augmented Language Models for Checking Safety Consistency in Pharmaceutical Product Labels
Dupont, C., & Martin, J. (2025) · Pharmacophore · doi:10.51847/fzeuaxeluk
Iterative Dynamic Routing Framework for Medical Question Answering
Wang, C., Yang, X., Zhu, Y., Sun, M., Li, M., & Zhang, W. (2026) · Communications in computer and information science · doi:10.1007/978-981-95-7299-1_2
Patient-Facing Radiology Communication with LLMs: Calibration Deficit and the Metadata Paradox
Shin, C., Park, J. H., Kim, S. J., Lee, Y. H., & Lee, H. S. (2026) · Healthcare · doi:10.3390/healthcare14111490 · Cited by 2
“The safe integration of LLMs hinges on developing robust strategies that leverage their strengths while managing failure modes [10–12].”
Rey-Blanes, A., Moreno-Barea, F. J., Veredas-Morente, J., Vivas-Vargas, E., Gil-Garcia, F., & Veredas, F. J. (2026) · medRxiv · doi:10.64898/2026.05.25.26354027
Jurisdictional gerrymandering: The Authority of Problems without Solutions
Zilberstein, S. (2026) · Social Problems · doi:10.1093/socpro/spag024
Razzaghi, H., Nguyen, N., Pargi, M. K., et al. (2026) · medRxiv · doi:10.64898/2026.05.30.26354533
Chawla, R., Dahiya, S., Singh, G., & Fortino, G. (2026) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2026.3691181
“For conversational healthcare data, pilot studies using GPT-4 and LLaMA2 have achieved semantically accurate summaries of palliative care tele-consultations, though they still face hallucination challenges [32], [33], [34].”
Wang, Q., Yong, X. Y. X., Wang, Y., Zhongyan, L., Wang, X., & Gao, J. (2026) · Applied Soft Computing · doi:10.1016/j.asoc.2026.115605
An Agentic Workflow for Patient Medical History Summarization and Visualization
Xu, & Babaian, T. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-28570-6_15
Bian, J., Afshar, M., Scifres, C., Webber, E. C., Burton, D., Vawdrey, D., Wang, F., Melton, G. B., Shah, N. H., Patzer, R. E., et al. (2026) · npj Health Systems · doi:10.1038/s44401-026-00107-x
Transforming Rheumatology Practice
García-Agúndez, A., Creasman, M., Schmajuk, G., & Yazdany, J. (2026) · Rheumatic Disease Clinics of North America · doi:10.1016/j.rdc.2026.04.002 · Cited by 1
Zhu, J., Pan, W., Wang, Y., Yan, K., Fang, Z., & Yang, X. (2026) · Journal of Medical Internet Research · doi:10.2196/91222
“The LLMs—particularly DeepSeek—demonstrated outstanding performance in case characteristic summarization and differential diagnosis, reflecting their powerful information extraction and structured knowledge retrieval capabilities [17,18].”
Memory Bank Compression for Continual Adaptation of Large Language Models
Katraouras, T., & Rafailidis, D. (2026) · ACM Symposium on Applied Computing · doi:10.1145/3748522.3779742
Sun, Q., Wu, X., Chen, J., Zhang, Y., Zhou, C., & Zhu, Z. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1802570
Patel, A., Khand, Y., Vallamchetla, S. K., Li, P., Tao, C., & Cheung, J. (2026) · doi:10.2196/preprints.104092
Noda, R., Sakurada, T., Ichikawa, D., & Shibagaki, Y. (2026) · Clinical and Experimental Nephrology · doi:10.1007/s10157-026-02903-z
Dehkordi, M. K. H., Chen, S., Zaidenberg, A., Elhanan, G., Deek, F. P., & Perl, Y. (2026) · Big Data and Cognitive Computing · doi:10.3390/bdcc10060189
Hsu, F.-Y., Liu, S.-C., Mei, K.-Q., & Chang, Y.-J. (2026) · Designing Interactive Systems Conference · doi:10.1145/3800645.3813044
Ludwig, D., Wang, M., Buchanan, J. R., et al. (2026) · Drug Safety · doi:10.1007/s40264-026-01682-6
Limaye, D., Shah, R., & Jonelgadda, C. (2026) · Research Square · doi:10.21203/rs.3.rs-10047888/v1
Liu, W. T., Zhang, Z., Deng, F., & Xiao, Y. (2026) · Frontiers in Neuroscience · doi:10.3389/fnins.2026.1828237 · Cited by 1
Zheng, X.-C., Serra, Á., Chernov, I. S., Marchesi, M., Musvasva, E., & Doktorova, T. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1808378 · Cited by 1
“Recent large language models (LLMs) [10, 11, 12, 13, 14, 15] can read and synthesize massive unstructured corpora and, when domain-adapted, generate concise, source-grounded summaries of complex biomedical and medical text [16, 17, 18, 19, 20, 21, 22].”
McCradden, M. D., Tng, S., Jeyaseelan, D., Leane, C., Campbell, M., Braund, T., Earle-Bandaralage, L., Ebrahimi, M., Sharma, A., & Tang, J. (2026) · BMJ Open · doi:10.1136/bmjopen-2025-111656
LLM-Generated Summaries for Teachers: A Randomized Field Experiment in a Digital Learning Platform
Lim, W.-C. I., Worden, E., Sales, A., & Heffernan, N. T. (2026) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3774398.3811606
Neto, E. P. L., da Conceição, J. M., da Silva, N. F. F., et al. (2026) · doi:10.5753/sbcas.2026.21586
“Furthermore, recent studies demonstrate that 'adapted large language models can outperform medical experts in clinical text summarization' (VAN VEEN et al., 2024), suggesting that, with proper technical grounding and ethical oversight, these tools can significantly optimize documentation workflows…”
AI-CURA, an automated LLM workflow for high-accuracy genetic variant classification
Ma, W., Fong, G., Lai, J., Wu, H., Hue, S. P. Y., Ying, D., Chen, L., Tang, W., Preusch, C. B., Chu, A. T. W., et al. (2026) · Science Translational Medicine · doi:10.1126/scitranslmed.adz4172 · Cited by 1
Improving Emergency Care for Children With Medical Complexity
Dallas, A., & Pulcini, C. D. (2026) · Pediatric Emergency Care · doi:10.1097/pec.0000000000003548 · Cited by 1
Saha, D., Lin, Y., & Ghose, D. (2026) · ACM Transactions on Computing for Healthcare · doi:10.1145/3828166
bin Yusuf, C. H., & Ong, L.-Y. (2026) · Applied Sciences · doi:10.3390/app16136630
Sangariyavanich, E., Ponthongmak, W., Theera-Ampornpunt, N., et al. (2026) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2025-101997
Pelizzari, N., Savardi, M., Arrigoni, L., Gregori, M., Conti, A., Tedesco, M., Tratta, E., Ceretti, E., Covolo, L., Visconti, M., et al. (2026) · JMIR Research Protocols · doi:10.2196/95782
Medical question answering: A comprehensive multimodal and LLM-driven survey
Mhedhbi, E., Zhu, X., Ayaz, M., Khan, M., & Othmani, A. (2026) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2026.109541 · Cited by 1
Shin, C., Park, J. H., Kim, S. J., Lee, Y. H., & Lee, H. S. (2026) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-026-02109-5
Golchini, N., Mehandru, N., Alaa, A. M., & Molina, M. F. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001491
Development of an Agentic AI Framework for Microbiome Data Analysis
Chaijaroenmaitri, P., Chaladkan, C., Changaival, B., Puranachot, P., & Patumcharoenpol, P. (2026) · International Joint Conference on Computer Science and Software Engineering · doi:10.1109/jcsse68839.2026.11597058
Chen, C.-H., Tseng, P.-C., Dai, H.-J., et al. (2026) · JMIR Formative Research · doi:10.2196/94454
Xie, F., Rozema, E. J., Puttock, E. J., et al. (2026) · doi:10.2196/preprints.106749
Generative small language models in clinical NLP: applications, adaptation, and evaluation
Türkmen, H. (2026) · PeerJ Computer Science · doi:10.7717/peerj-cs.4000
An Ethical AI Approach for Disease Diagnosis Using Prompt-Based History Summarisation
Fathima, M. D., & Mohammed, M. (2026) · Journal of Studies in Science and Engineering · doi:10.53898/josse2026631
Michihata, N., Ishii, H., Tsujimura, H., Takano, N., Nomura, T., Takeuchi, Y., Kagawa, S., Asai, S., Kita, E., Komaru, A., et al. (2026) · Applied Clinical Informatics · doi:10.1055/a-2917-6695
Grassi, J., Pantiukhin, D., Kuznetsov, I., et al. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7162473
Heilmeyer, F., Lyssenko, L., Reinhard, T., Haverkamp, C., & Böhringer, D. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001461 · Cited by 1
Comparative Analysis of Medical Information Systems for Improved Patient Care and Data Precision
Andronic, D. T., Titu, S., & Titu, A. M. (2026) · International Conference on Electronics, Computers and Artificial Intelligence (ECAI) · doi:10.1109/ecai69016.2026.11613602
“Adapted language models can take over part of the documentation burden by summarising clinical text, freeing up time for care [25], and automated techniques are beginning to support the assessment and improvement of data quality [7].”
Shakeri, E., & Far, B. H. (2026) · Intelligence-Based Medicine · doi:10.1016/j.ibmed.2026.100442 · Cited by 3
Long-Term Memory Mechanism of Large Language Models for Personalized Medical Inquiry Service
Xu, F. (2026) · ITM Web of Conferences · doi:10.1051/itmconf/20268801023
Villegas, D. V., Solis, T. A., Omeñaca, A. T., Vinaixa, X. S., Catta-Preta, M. C., Fierro, J. M. M., Romeu, R., Bayés‐Genís, B., Rúbies-Feijoo, C., & Rocabruna, E. L. (2026) · Healthcare · doi:10.3390/healthcare14152290
Parambath, A., Pulpo, G., & Hartman, V. (2026) · JMIR Human Factors · doi:10.2196/95644 · Cited by 1
“Large language models can generate fluent summaries of long clinical text, yet in high-risk domains such as health care, trust depends not only on accuracy but also on verifiability [1,2].”
Bilimsel Metinlerde Anlamsal Özetleme: Derin Öğrenme Yaklaşımlarının Performans Analizi
Özcan, S., & Özcan, A. (2026) · Bilişim Teknolojileri Dergisi · doi:10.17671/gazibtd.1863483
Baştürk, B., & Onan, A. (2026) · Bilişim Teknolojileri Dergisi · doi:10.17671/gazibtd.1846644
Case-matched retrieval improves textual alignment of LLM-generated radiology impressions
Sorin, V., Collins, J. D., Hahn, L. D., Bratt, A. K., Klang, E., & Korfiatis, P. D. (2026) · PLoS ONE · doi:10.1371/journal.pone.0354688
“In radiology, they are being tested for report classification [3–6], clinical note summarization [7–9], structuring free text reports [10,11], and drafting reports, summaries, and impressions [12–14].”
Tian, X., Wang, Y., & Zhu, G. (2026) · IEEE Internet of Things Journal · doi:10.1109/jiot.2026.3719132
“They demonstrate near-human or even human-surpassing performance in tasks such as machine translation, question answering systems, and sentiment analysis [2], [3], [4], becoming a core driving force for artificial intelligence applications.”
Yu, S., He, Y., Li, P., Wu, C., Zhang, P., Li, J., & Tian, Y. (2026) · Intelligent Medicine · doi:10.1016/j.imed.2026.08.001
What We Left in the Chart: Rebuilding Clinical Context Using Large Language Models
Yılmaz, E. C., & Cardoza-Ochoa, D. R. (2026) · Radiology · doi:10.1148/radiol.261918
Retrieval-Augmented Generation for Clinical Decision Support in Gait Analysis: A Proof-of-Concept
Giarmatzis, G., Horst, F., Slijepčević, D., Aggelousis, N., & Papakostas, G. A. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-31526-7_18
Convergent Generative Adversarial Networks for Medical Report Generation With Visual Manifold Graph
Wang, Y., Xu, S., Zhou, K., & Li, D. (2026) · IEEE Access · doi:10.1109/access.2026.3721550
“A reliable system can help reduce the repetitive reporting workload, improve reporting consistency, and provide preliminary diagnostic support for clinicians, especially in high-throughput imaging scenarios [1].”
Large language model–based patient handoff tools relevant to intensive care: a scoping review
Bains, S. S., Kolesnikov, M. R., Bedrick, S., Mohan, V., & Gold, J. A. (2026) · Considerations in Medicine · doi:10.1136/conmed-2026-000046
da Silva, R. P., & Pazin‐Filho, A. (2026) · Artificial Intelligence in Emergency Medicine · doi:10.1016/j.aiemed.2026.100030
Khan, M. S. H., Koohsari, M. J., Li, J., Zhao, J., Luo, Y., Yasunaga, A., Oka, K., & Kaczynski, A. T. (2026) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph23081035
Koohsari, M. J., Loo, B. P. Y., Zhao, J., Li, J., Long, Y., Lu, Y., Oka, K., & Kaczynski, A. T. (2026) · Developments in the Built Environment · doi:10.1016/j.dibe.2026.101007
A Comprehensive Evaluation of the Phi Family of Small Language Models in Healthcare
Awasthi, R., Shree, S., Bhattad, A., Mishra, S., Singh, N., Khare, A., Edara, S. R., Ramachandran, S. P., Mahapatra, D., Cywinski, J. B., et al. (2026) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi69079.2026.00108
“[16]–[18].”
Supplementary Material for Evaluating AI-Based Mental Health Interventions
Sharma, A. (2026) · ACM eBooks · doi:10.1145/3749728.3749737
Human–AI Collaboration to Empower People Seeking Support
Sharma, A. (2026) · ACM eBooks · doi:10.1145/3749728.3749732
Evaluating AI-based Mental Health Interventions
Sharma, A. (2026) · ACM eBooks · doi:10.1145/3749728.3749733 · Cited by 1
(2026) · ACM eBooks · doi:10.1145/3749728.3749738
(2026) · ACM eBooks · doi:10.1145/3749728.3749739
Supplementary Materials for Human–AI Collaboration to Empower People Providing Support
Sharma, A. (2026) · ACM eBooks · doi:10.1145/3749728.3749735
Human–AI Collaboration to Empower People Providing Support
Sharma, A. (2026) · ACM eBooks · doi:10.1145/3749728.3749731
Peng, J., Tuo, Y., Wang, G., 韩瑞琴, Wang, Q., Wang, Y., Hu, Z., & Hu, X. (2026) · Visual Computing for Industry Biomedicine and Art · doi:10.1186/s42492-026-00228-y
“Research has focused on report generation, case summarization, and education, although the accuracy has declined for rare findings and nuanced descriptors [34].”
Sharma, A. (2026) · ACM eBooks · doi:10.1145/3749728.3749730
(2026) · ACM eBooks · doi:10.1145/3749728.3749729
Metadata
- Authors
- Van Veen, D., Van Uden, C., Blankemeier, L., Delbrouck, J.-B., Aali, A., Bluethgen, C., Pareek, A., Polacin, M., Reis, E. P., Seehofnerová, A., Rohatgi, N., Hosamani, P., Collins, W., Ahuja, N., Langlotz, C. P., Hom, J., Gatidis, S., Pauly, J., & Chaudhari, A. S.
- Affiliations
- Stanford University (US)
- The University of Texas at Austin (US)
- Copenhagen University Hospital (DK)
- University of Zurich (CH)
- Hospital Israelita Albert Einstein (BR)
- Stanford Medicine (US)
- ORCID
- Venue
- Nature Medicine
- Published
- 2024-02-27
- Volume
- 30
- Issue
- 4
- Pages
- 1134-1142
- ISSN
- 1078-8956 (ISSN-L)
- 1546-170X
- PMC
- PMC11479659
- OpenAlex
- W4392193048
- Cited by
- 891
- References
- 80
- Citations by year
- 2026: 341 · 2025: 404 · 2024: 144
- FWCI
- 82.70
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Licence
- Locations
- Nature Medicine (publishedVersion)
- Research Square (acceptedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Zurich Open Repository and Archive (University of Zurich) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- PubMed Central (submittedVersion, cc-by)
- Zurich Open Repository and Archive (University of Zurich) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Zurich Open Repository and Archive (University of Zurich)
- Funders
- Foundation for the National Institutes of Health (R01 AR077604, R01 EB002524, R01 AR079431, P41 EB027060)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- large language models, clinical text summarization, LLM adaptation, electronic health records, radiology reports, progress notes, doctor-patient dialogue, patient questions, summary quality, safety analysis, clinical documentation burden
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
