The DOI matches its Crossref record; 91 of the 91 references it cites were found in a registry or an index; 37 of the 61 sources its claims cite were checked for support.
Adapted large language models can outperform medical experts in clinical text summarization
Open accessPublished version of a preprint
Abstract
Adapted LLMs can perform text summarization of clinical notes such as radiology reports, progress notes, and discharge summaries, sometimes outperforming medical experts on certain criteria.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
40 Own result · 27 Own interpretation · 20 Own method · 13 Partly supported · 7 No basis given · 6 No full text of the sources · 5 Supported · 1 Supported so far · 1 Couldn’t confirm · 1 Sources not found · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Although LLM benchmarks for general natural language processing (NLP) tasks exist20,21, they do not evaluate performance on relevant clinical tasks.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
What the sources say
- Not matchedcites [20] Liang, P. et al. Holistic evaluation of language models. Transact. Mach. Learn. Res. (in the press).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites [21] Zheng, L. et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Preprint at https://doi.org/10.48550/arXiv.2306.05685 (2023). source
The citation follows the statement that LLM benchmarks for general NLP tasks exist. Section 2.1 (Motivation) lists several such benchmarks, and Section 5 discusses standardized LLM benchmarks. The citation is not attached to the subsequent claim about clinical tasks. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Currently, clinicians spend a substantial amount of time summarizing vast amounts of textual information—whether it be compiling diagnostic reports, writing progress notes or synthesizing a patient’s treatment history across different specialists1–3.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
What the sources say
- Not checked — no full text of the sourcecites [1] Golob, J. F. Jr, Como, J. J. & Claridge, J. A. The painful truth: the documentation burden of a trauma surgeon. J. Trauma Acute Care Surg. 80, 742–747 (2016). source
- Partly supportedcites [2] Arndt, B. G. et al. Tethered to the EHR: primary care physician workload assessment using EHR event log data and time–motion observations. Ann. Fam. Med. 15, 419–426 (2017). sourceExperimental — accuracy not yet measured
The Time Spent on Various Tasks table (p. 4) reports 84 minutes per day on documentation and 60 minutes reviewing notes among the family medicine physicians studied. Study Limitations (p. 6) also describes reviewing prior notes before writing a progress note. The paper does not establish how much of that time was spent summarizing vast amounts of text, compiling diagnostic reports, or synthesizing histories across specialists. (LLM, conf=0.86)
- Supportedcites [3] Fleming, S. L. et al. MedAlign: a clinician-generated dataset for instruction following with electronic medical records. Preprint at https://doi.org/10.48550/arXiv.2308.14089 (2023). source
The introductory passage (section marked “+”; page not specified) describes clinicians’ information-related tasks as a significant burden, citing 10 hours a week spent generating documentation. It specifically gives summarizing a patient’s treatment history across specialists as an example, supporting the citing sentence’s overall point. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Every source the paper cites for this claim was checked and supports it.
In the paper
“Other studies employ human experts to evaluate synthesized abstracts, demonstrating that NLP metrics are not sufficient to measure summary quality56.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
What the sources say
- Supportedcites [56] Tang, L. et al. Evaluating large language models on medical evidence summarization. NPJ Digit. Med. 6, 158 (2023). source
The Methods section (Design of the human evaluation) says six domain experts evaluated generated summaries. The Discussion says automatic metrics fell short of measuring factual inconsistency, potential medical harmfulness, and human preference, making human evaluation essential. The supplied passages have section labels but no page numbers. (LLM, conf=0.94)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Quantitative assessments with syntactic, semantic and conceptual NLP metrics reveal trade-offs between models and adaptation methods.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A clinical reader study with 10 physicians evaluated summary completeness, correctness and conciseness; in most cases, summaries from our best-adapted LLMs were deemed either equivalent (45%) or superior (36%) compared with summaries from medical experts.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Although FLAN-T5 was more competitive for syntactic metrics, such as BLEU42, this model is constrained to a shorter context length of 512 (Extended Data Table 2).”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Head-to-head model comparison.
What the sources say
- Not checked — no full text of the sourcecites [42] Papineni, K., Roukos, S., Ward, T. & Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proc. of the 40th Annual Meeting of the Association for Computational Linguistics. https://dl.acm.org/doi/pdf/10.3115/1073083.1073135 (Association for Computing Machinery, 2002). source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“When considering conciseness scores (Fig. 4b), radiology reports were the only task in which physicians did not prefer the best model’s summaries to the medical experts.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“FLAN-T5 emerged as the best-performing model with QLoRA.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Comparison of adaptation strategies.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, the metrics BERTScore and MEDCON correlated most strongly with physician preference for correctness.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We see a clear example in Extended Data Fig. 3 where the model mistakenly generated (hallucinated) several absent conditions, such as eosinophilia.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Analyzing reader study results
boostedSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 58
- It may state two separate claims as one. 40
- The quote only partly states this claim. 1
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Large language models (LLMs) have gained remarkable traction, leading to widespread adoption of models such as ChatGPT17, which excel at information retrieval, nuanced understanding and text generation18,19.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
What the sources say
- Partly supportedcites [17] Brown, T. et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33 https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf (NeurIPS, 2020). sourceExperimental — accuracy not yet measured
The Conclusion (p. 40) reports strong performance on many NLP tasks and high-quality text generation, supporting the generation part of the claim. But the paper evaluates GPT-3 rather than ChatGPT; its closed-book question answering does not test information retrieval (Section 3.2, p. 13), and its NLI results describe important limits to language understanding (Section 3.8, p. 20). The separate citation attached to ChatGPT’s adoption is not judged here. (LLM, conf=0.88)
- Supportedcites [18] Zhao, W. X. et al. A survey of large language models. Preprint at https://doi.org/10.48550/arXiv.2303.18223 (2023). source
The Introduction (p. 2) describes ChatGPT’s global impact and LLMs’ integration into search platforms and content-creation tools. The Key concepts section (p. 3) describes their ability to process and produce natural language across a wide range of tasks, supporting the broad characterization of their capabilities. (LLM, conf=0.84)
- Partly supportedcites [19] Bubeck, S. et al. Sparks of artificial general intelligence: early experiments with GPT-4. Preprint at https://doi.org/10.48550/arXiv.2303.12712 (2023). sourceExperimental — accuracy not yet measured
The Introduction describes GPT-4’s comprehension and text-generation capabilities, and §2.1 says ChatGPT shows some understanding. But the paper does not establish that these models excel at information retrieval: its discussion of Figure 5.1 says current knowledge is hard for language models and describes GPT-4 using external search tools to overcome that limitation. The widespread-adoption claim is attached to a different citation (17). (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“These tasks divert attention from direct patient care, leading to worse outcomes for patients and decreased job satisfaction for clinicians2,14–16.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
What the sources say
- Partly supportedcites [2] Arndt, B. G. et al. Tethered to the EHR: primary care physician workload assessment using EHR event log data and time–motion observations. Ann. Fam. Med. 15, 419–426 (2017). sourceExperimental — accuracy not yet measured
The Introduction (p. 1) links documentation and other EHR tasks to less face-to-face patient time and clinician dissatisfaction. The paper does not establish that documentation tasks lead to worse patient outcomes; its discussion of poorer clinical outcomes concerns electronic communication among team members and patients with coronary artery disease (Communication, p. 6). (LLM, conf=0.88)
- Not checked — no full text of the sourcecites [14] Shanafelt, T. D. et al. Relationship between clerical burden and characteristics of the electronic environment with physician burnout and professional satisfaction. Mayo Clin. Proc. 91, 836–848 (2016). source
- Partly supportedcites [15] Robinson, K. E. & Kersey, J. A. Novel electronic health record (EHR) education intervention in large healthcare organization improves quality, efficiency, time, and impact on burnout. Medicine (Baltimore) 97, e12319 (2018). sourceExperimental — accuracy not yet measured
The Introduction (page not provided) says physicians spend substantial time on EHR and desk work relative to direct patient contact, and links EHR documentation to impaired patient communication and increased career dissatisfaction. It does not establish that documentation tasks lead to worse patient outcomes; the Discussion addresses patient care but does not make that causal claim. (LLM, conf=0.87)
- Partly supportedcites [16] Toussaint, W. et al. Design considerations for high impact, automated echocardiogram analysis. Preprint at https://doi.org/10.48550/arXiv.2006.06292 (2020). sourceExperimental — accuracy not yet measured
In Design Discoveries (p. 2), the paper says cardiologists spend 4–5 hours a day annotating echocardiograms and suggests automation could free them to focus more attention on sick patients. It does not characterize this work as documentation or say that it causes worse patient outcomes or decreased clinician job satisfaction. (LLM, conf=0.78)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Similarly, documentation responsibilities for nurses can consume up to 60% of their time and account for considerable work stress11–13.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
What the sources say
- Partly supportedcites [11] Khamisa, N., Peltzer, K. & Oldenburg, B. Burnout in relation to specific contributing factors and health outcomes among nurses: a systematic review. Int. J. Environ. Res. Public Health 10, 2214–2240 (2013). sourceExperimental — accuracy not yet measured
The Results (§3.2–3.3; page unavailable) identify workload as a prominent stressor among nurses, supporting the general point about work stress. They do not identify documentation as the source of that stress or report that it consumes up to 60% of nurses’ time; searches for documentation, paperwork, and the time figure found no such passage. (LLM, conf=0.80)
- Not checked — no full text of the sourcecites [12] Duffy, W. J., Kharasch, M. S. & Du, H. Point of care documentation impact on the nurse–patient interaction. Nurs. Adm. Q. 34, E1–E10 (2010). source
- Not checked — no full text of the sourcecites [13] Chang, C.-P., Lee, T.-T., Liu, C.-H. & Mills, M. E. Nurses’ experiences of an initial and reimplemented electronic health record use. Comput. Inform. Nurs. 34, 183–190 (2016). source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The widespread adoption of electronic health records has expanded clinical documentation workload, directly contributing to increasing stress and clinician burnout7–9.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
What the sources say
- Not matchedcites [7] Gesner, E., Gazarian, P. & Dykes, P. The burden and burnout in documenting patient care: an integrative literature review. Stud. Health Technol. Inform. 21, 1194–1198 (2019).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites [8] Ratwani, R. M. et al. A usability and safety analysis of electronic health records: a multi-center study. J. Am. Med. Inform. Assoc. 25, 1197–1201 (2018). sourceExperimental — accuracy not yet measured
The Background and significance section says EHR usability problems create burdensome workflows and inefficiencies that contribute to clinician frustration. It does not say that widespread EHR adoption increased clinical documentation workload or directly caused increasing stress and burnout. The Discussion examines task time, clicks, and errors, not those broader outcomes; page numbers are unavailable in our copy. (LLM, conf=0.84)
- Not checked — no full text of the sourcecites [9] Ehrenfeld, J. M. & Wanderer, J. P. Technology as friend or foe? Do electronic health records increase burnout? Curr. Opin. Anaesthesiol. 31, 357–360 (2018). source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Many previous clinical NLP studies rely primarily on quantitative metrics for evaluation41,52–54.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
What the sources say
- Partly supportedcites [41] Yim, W.W. et al. Aci-bench: a novel ambient clinical intelligence dataset for benchmarking automatic visit note generation. Sci. Data https://doi.org/10.1038/s41597-023-02487-3 (2023). sourceExperimental — accuracy not yet measured
In Technical Validation > Baseline experiments (page unavailable), the paper calls automatic evaluation metrics “widely-used” and evaluates its own clinical-note generation models with quantitative scores. It does not establish that many previous clinical NLP studies relied primarily on quantitative rather than other forms of evaluation. (LLM, conf=0.82)
- Supportedcites [52] Ma, C. et al. ImpressionGPT: an iterative optimizing framework for radiology report summarization with chatGPT. Preprint at https://doi.org/10.48550/arXiv.2304.08448 (2023). source
The Dataset and Evaluation Metrics section (p. 8) uses ROUGE and factual-consistency scores and notes that earlier studies reported factual-consistency metrics. The Comparison with Other Methods section (p. 9) evaluates numerous earlier radiology-report summarization methods using ROUGE and factual-consistency metrics, supporting the broad statement about quantitative evaluation in prior clinical NLP work. (LLM, conf=0.87)
- Not checked — no full text of the sourcecites [53] Wei, S. et al. Medical question summarization with entity-driven contrastive learning. Preprint at https://doi.org/10.48550/arXiv.2304.07437 (2023). source
- Supportedcites [54] Manakul, P. et al. CUED at ProbSum 2023: Hierarchical ensemble of summarization models. In The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks 516–523 (Association for Computational Linguistics, 2023). source
The Experiments tables report the paper’s clinical summarization results using numerical ROUGE scores, and the Limitations section explicitly identifies ROUGE-L as its assessment metric. This supports citing the paper as an example of a clinical NLP study evaluated primarily with quantitative metrics; the paper alone does not establish how many other studies do so. (LLM, conf=0.87)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Previous work has demonstrated potential across clinical NLP tasks22,23, adapting to the medical domain by training a new model24,25, fine-tuning an existing model26,27 or supplying task-specific examples in the model prompt27,28.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
What the sources say
- Supportedcites [22] Wornow, M. et al. The shaky foundations of large language models and foundation models for electronic health records. NPJ Digit. Med. 6, 135 (2023). sourceExperimental — accuracy not yet measured
The “State of published clinical FMs—CLaMs” section describes models trained on clinical or biomedical text and evaluated on clinical NLP tasks. “Benefits of clinical FMs” describes fine-tuning and few-shot adaptation, while the evaluation discussion contrasts fine-tuning with prompting. The paper cautions that performance on these tasks does not establish clinical deployment value, but the claim asserts only potential. (LLM, conf=0.86)
- Not checked — no full text of the sourcecites [23] Thirunavukarasu, A. J. et al. Large language models in medicine. Nat. Med. 29, 1930–1940 (2023). source
- Partly supportedcites [24] Singhal, K. et al. Large language models encode clinical knowledge. Nature https://doi.org/10.1038/s41586-023-06291-2 (2023). sourceExperimental — accuracy not yet measured
The Methods > Modelling passages describe adapting a model to medical question-answering tasks with task-specific examples in prompts and instruction prompt tuning. They also say the authors used prompt tuning instead of full-model fine-tuning, keeping the model’s parameters frozen. Thus, this source supports the prompting part of the attributed claim, but not the suggestion that this study fine-tuned the existing model itself. (LLM, conf=0.86)
- Supportedcites [25] Tu, T. et al. Towards generalist biomedical AI. Preprint at https://doi.org/10.48550/arXiv.2307.14334 (2023). sourceExperimental — accuracy not yet measured
The citation is attached to fine-tuning an existing model and supplying task-specific examples in prompts, not to training a new model. The 'Dataset and preprocessing' section describes fine-tuning and adding one-shot exemplars to task prompts; '5.1 Evaluation on MultiMedBench' reports evaluation across biomedical tasks, including language-only tasks. No page numbers are available in our copy. (LLM, conf=0.91)
- Supportedcites [26] Toma, A. et al. Clinical Camel: an open-source expert-level medical language model with dialogue-based knowledge encoding. Preprint at https://doi.org/10.48550/arXiv.2305.12031 (2023). sourceExperimental — accuracy not yet measured
The Introduction describes Clinical Camel as a medical model fine-tuned from LLaMA-2 and reports promising medical-benchmark results. Section 4 evaluates it in five-shot settings, supporting the cited point about supplying task-specific examples. The citations for training a new model and for the sentence’s opening claim are separate from this source. (LLM, conf=0.88)
- Supportedcites [27] Van Veen, D. et al. RadAdapt: radiology report summarization via lightweight domain adaptation of large language models. In 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks 449–460 (Association for Computational Linguistics, 2023). sourceExperimental — accuracy not yet measured
Citation 27 is attached to fine-tuning an existing model and supplying task-specific examples in prompts. RadAdapt’s Section 3.2 describes both methods, and Section 5.1 reports that fine-tuning outperformed discrete prompting while adding in-context examples improved performance for almost every model studied. (LLM, conf=0.92)
boostedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Prompt phrasing and model temperature can have a considerable effect on LLM output, as demonstrated in the literature45,46 and in Fig. 2.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
What the sources say
- Not matchedcites [45] Strobelt, H. et al. Interactive and visual prompt engineering for ad-hoc task adaptation with large language models. IEEE Trans. Vis. Comput. Graph. 29, 1146–1156 (2022).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites [46] Wang, J. et al. Prompt engineering for healthcare: methodologies and applications. Preprint at https://doi.org/10.48550/arXiv.2304.14670 (2023). sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) says prompt design can significantly affect model performance, and Automated Prompts describes improved performance from prompt paraphrasing. The copy does not address model temperature (searched: “temperature”), so it supports the prompt-phrasing part but not the temperature part. (LLM, conf=0.87)
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“considering that Alpaca is known to perform worse than our other open-source autoregressive models, Vicuna and Llama-2 (refs. 21,35)”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Impact of domain-specific fine-tuning.
What the sources say
- Partly supportedcites [21] Zheng, L. et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Preprint at https://doi.org/10.48550/arXiv.2306.05685 (2023). sourceExperimental — accuracy not yet measured
Table 8 (Position bias results; page not provided) reports lower MT-Bench scores for Alpaca-7B and Alpaca-13B (4.54 and 4.53) than for the listed Vicuna variants (5.95–6.39). The passages do not compare Alpaca with Llama-2; the LLaMA models listed are not identified as Llama-2. (LLM, conf=0.87)
- Not matchedcites [35] The Vicuna Team. Vicuna: an open-source chatbot impressing GPT-4 with 90%* ChatGPT quality. https://lmsys.org/blog/2023-03-30-vicuna/ (2023).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“For correctness, the metrics BERTScore44 (measuring semantics) and MEDCON41 (measuring medical concepts) correlated most strongly with reader preference; meanwhile, the BLEU42 metric (measuring syntax) correlated most with completeness and least with conciseness.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Connecting quantitative and clinical evaluations
What the sources say
- Not matchedcites [44] Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q. & Artzi, Y. BERTScore: evaluating text generation with BERT. International Conference on Learning Representations. https://openreview.net/forum?id=SkeHuCVFDr (2020).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Could not verifycites [41] Yim, W.W. et al. Aci-bench: a novel ambient clinical intelligence dataset for benchmarking automatic visit note generation. Sci. Data https://doi.org/10.1038/s41597-023-02487-3 (2023). sourceExperimental — accuracy not yet measured
Technical Validation > Baseline experiments (page unavailable) describes BERTScore and medcon as evaluation metrics, but does not report correlations with reader preference, correctness, completeness, or conciseness. Searches for those terms and BLEU found no passage addressing the claimed comparisons; the paper discusses BLEURT instead. (LLM, conf=0.78)
- Not checked — no full text of the sourcecites [42] Papineni, K., Roukos, S., Ward, T. & Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proc. of the 40th Annual Meeting of the Association for Computational Linguistics. https://dl.acm.org/doi/pdf/10.3115/1073083.1073135 (Association for Computing Machinery, 2002). source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
Every source the paper cites for this claim was checked and supports it.
In the paper
“To address this, there have been recent releases of expert evaluations for adjacent clinical NLP tasks3,55.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
What the sources say
- Supportedcites [3] Fleming, S. L. et al. MedAlign: a clinician-generated dataset for instruction following with electronic medical records. Preprint at https://doi.org/10.48550/arXiv.2308.14089 (2023). source
The Clinical dataset description says MedAlign includes clinician evaluations of LLM-generated outputs, and the Clinician Evaluation of LLM Responses section describes clinicians judging and ranking those outputs. The Discussion and Conclusion says the authors share the benchmark dataset with the research community, making MedAlign an example of the cited kind of expert evaluation. The copy does not give a page number for these passages. (LLM, conf=0.88)
- Supportedcites [55] Yu, F. et al. Evaluating progress in automatic chest x-ray radiology report generation. Patterns (N Y) 4, 100802 (2023). source
The Results section, “Alignment between automated metrics and radiologists,” reports radiologist error scores for 50 studies and says they are shown in Figure S3. The paper therefore presents an expert evaluation of the adjacent clinical NLP task of chest X-ray report generation. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“This simple approach provides the model with context, enhancing LLM performance for a particular task or domain27,28.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Adaptation methods
What the sources say
- Supportedcites [27] Van Veen, D. et al. RadAdapt: radiology report summarization via lightweight domain adaptation of large language models. In 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks 449–460 (Association for Computational Linguistics, 2023). source
The Related Work section describes prompting as providing context for a task. Section 5.1 reports that adding in-context examples improved radiology report summarization performance for almost every model tested, with T5 as the exception. (LLM, conf=0.90)
- Supportedcites [28] Mathur, Y. et al. SummQA at MEDIQA-Chat 2023: in-context learning with GPT-4 for medical summarization. Preprint at https://doi.org/10.48550/arXiv.2306.17384 (2023). source
Section 2 (Background) describes giving models task instructions or examples in context. Section 5.5 reports that, for the paper's medical summarization task, generated summaries generally improved as more relevant in-context examples were provided. No page numbers are supplied in our copy's passages. (LLM, conf=0.90)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“This quantization negligibly impacts performance40; as such, we used QLoRA for all model training.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Adaptation methods
What the sources say
- Supportedcites [40] Dettmers, T., Pagnoni, A., Holtzman, A. & Zettlemoyer, L. QLoRA: efficient finetuning of quantized LLMs. Preprint at https://doi.org/10.48550/arXiv.2305.14314 (2023). source
The Summary reports that 4-bit QLoRA with NF4 matched 16-bit finetuning performance on the evaluated benchmarks, with no observed degradation relative to full finetuning. This supports the claim in the context of QLoRA model training; the paper separately notes that 4-bit quantization for inference alone can degrade performance. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“We further note the potential for LLMs to be biased58,59.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
What the sources say
- Supportedcites [58] Omiye, J. A., Lester, J. C., Spichak, S., Rotemberg, V. & Daneshjou, R. Large language models propagate race-based medicine. NPJ Digit. Med. 6, 195 (2023). source
The Abstract reports that all four LLMs studied produced examples of race-based medicine, directly supporting the potential for LLM bias. It also says LLMs may amplify biases present in their training data. (LLM, conf=0.95)
- Not checked — no full text of the sourcecites [59] Zack, T. et al. Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study. Lancet Digit. Health 6, e12–e22 (2024). source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Note that choosing ‘relevant’ in-context examples has been shown to outperform choosing examples at random69.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Adaptation methods
What the sources say
- Partly supportedcites [69] Nie, F., Chen, M., Zhang, Z. & Cheng, X. Improving few-shot performance of language models via nearest neighbor calibration. Preprint at https://doi.org/10.48550/arXiv.2212.02216 (2022). sourceExperimental — accuracy not yet measured
The Experimental Setup (p. 5) uses randomly sampled in-context demonstrations as a baseline, and the Analysis (p. 7) reports improvements from retrieving similar support samples. But those samples calibrate predictions through a nearest-neighbor classifier; the paper does not show that selecting relevant examples *as in-context demonstrations* outperforms selecting random demonstrations. (LLM, conf=0.86)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“As we are not the first to employ these datasets, Extended Data Table 4 contains quantitative metric scores from other works25,27,41,52–54 that developed methods specific to each individual summarization task.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Data
What the sources say
- Partly supportedcites [25] Tu, T. et al. Towards generalist biomedical AI. Preprint at https://doi.org/10.48550/arXiv.2307.14334 (2023). sourceExperimental — accuracy not yet measured
The Additional information section says the paper used a previously published MIMIC-III radiology-report summarization dataset. But Section 5.1 describes Med-PaLM M as jointly fine-tuned across tasks and distinguishes it from prior specialist models; it does not support describing this paper’s method as specific to an individual summarization task. Our copy provides no page numbers for these passages. (LLM, conf=0.86)
- Supportedcites [27] Van Veen, D. et al. RadAdapt: radiology report summarization via lightweight domain adaptation of large language models. In 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks 449–460 (Association for Computational Linguistics, 2023). source
The Data section (4.1) identifies MIMIC-III as RadAdapt’s primary dataset and says it also evaluates on MIMIC-CXR. Section 3.2 describes adaptation methods applied to radiology report summarization, so the paper serves as an example of prior work using these datasets for a specific summarization task. (LLM, conf=0.88)
- Supportedcites [41] Yim, W.W. et al. Aci-bench: a novel ambient clinical intelligence dataset for benchmarking automatic visit note generation. Sci. Data https://doi.org/10.1038/s41597-023-02487-3 (2023). source
The paper’s “Baseline experiments” section says its authors used the aci-bench dataset to benchmark several note-generation methods and report quantitative results. It therefore serves as an example of prior work employing one of the datasets for a specific summarization task. The citing paper’s Extended Data Table 4 is its own presentation of those scores. (LLM, conf=0.90)
- Supportedcites [52] Ma, C. et al. ImpressionGPT: an iterative optimizing framework for radiology report summarization with chatGPT. Preprint at https://doi.org/10.48550/arXiv.2304.08448 (2023). source
The Introduction (p. 2) says the authors developed ImpressionGPT and evaluated it on MIMIC-CXR and OpenI. The comparison section (p. 9) reports its quantitative metrics alongside other radiology-report summarization methods on those datasets. This supports citing the paper as an earlier work using the datasets and developing a method for that summarization task. (LLM, conf=0.91)
- Not checked — no full text of the sourcecites [53] Wei, S. et al. Medical question summarization with entity-driven contrastive learning. Preprint at https://doi.org/10.48550/arXiv.2304.07437 (2023). source
- Supportedcites [54] Manakul, P. et al. CUED at ProbSum 2023: Hierarchical ensemble of summarization models. In The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks 516–523 (Association for Computational Linguistics, 2023). source
The Introduction (§1; page not supplied) identifies this paper as work on the ProbSum medical problem-list summarization task. The Experimental Setup and Ensemble Methods (§4.1 and §4.3; pages not supplied) describe its use of the task data, task-specific method, and quantitative ROUGE results. It therefore serves as an example of the prior task-specific work cited in the sentence. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“inspired by the AHRQ harm scale43.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Connecting errors to medical harm
What the sources say
- Supportedcites [43] Walsh, K. E. et al. Measuring harm in healthcare: optimizing adverse event review. Med. Care 55, 436–441 (2017). source
The Methods (Rater Assessment of Harm Severity) describe the revised AHRQ Harm Scale and its six harm-severity options. The citation identifies the scale that inspired the citing authors’ supplemental reader study; it need not establish the citing authors’ own motivation. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Compared to training model weights from scratch, LoRA is much more efficient with respect to both computational requirements and the volume of training data required.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Adaptation methods
What the sources say
- Partly supportedcites [70] Hu, E. et al. LoRA: low-rank adaptation of large language models. Preprint at https://doi.org/10.48550/arXiv.2106.09685 (2021). sourceExperimental — accuracy not yet measured
The Introduction (p. 2) says LoRA makes training more efficient, and §4.2 (p. 5) reports lower GPU memory use and faster training than full fine-tuning. Those comparisons are with adapting a pretrained model, not training one from scratch. The low-data experiments (§F.3, p. 22) do not establish that LoRA requires less training data than training from scratch. (LLM, conf=0.86)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The ensuing safety analysis highlights challenges faced by both LLMs and medical experts, as we connect errors to potential medical harm and categorize types of fabricated information.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Consider example 1 in Extended Data Fig. 4, in which the input question mentioned ‘diabetes and neuropathy’. The model mirrored this phrasing verbatim, whereas the medical expert interpreted it as ‘diabetic neuropathy’.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Seq2seq models (FLAN-T5 and FLAN-UL2) performed very well on syntactical metrics, such as BLEU, but worse on others (Fig. 3d)”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“medical expert summaries were preferred in only a minority of cases (19%), whereas, in a majority, the best model was either non-inferior (45%) or preferred (36%).”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Analyzing reader study results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In terms of trade-offs between lightweight adaptation methods, while QLoRA fine-tuning performed similarly for some cases, ICL was the best overall, especially when including proprietary models GPT-3.5 and GPT-4.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“pooled results across 10 physicians (Fig. 4b) demonstrate that summaries from the best-adapted model (GPT-4 using ICL) were more complete and contained fewer errors compared to medical expert summaries”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Analyzing reader study results
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In example 2 of Extended Data Fig. 4, the model simply reformulated the input question about tests and their locations, whereas the medical expert inferred a broader query about tests and treatments.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, our best-adapted model achieved non-inferior results to medical experts while performing a basic search across 1–2 options for each task instruction (Extended Data Table 1).”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Regarding conciseness, the best model performed significantly better than medical experts (P < 0.001) overall and on two tasks, whereas, for radiology reports, it performed similarly to medical experts.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Analyzing reader study results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that the model committed misinterpretations, inaccuracies and hallucinations on 6%, 2% and 5% of samples, respectively, compared to 9%, 4% and 12%, respectively, by medical experts.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Fabricated information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although ICL and QLoRA were competitive for open-source models, proprietary models GPT-3.5 and GPT-4 far outperformed other models and methods given sufficient in-context examples.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Effect of context length for ICL.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Regarding correctness, the best model generated significantly fewer errors (P < 0.001) compared to medical expert summaries (Fig. 4b) overall and on two of three summarization tasks.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Analyzing reader study results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observed that the best model summaries were more complete, on average, than medical expert summaries, achieving statistical significance across all three summarization tasks with P < 0.001 (Fig. 4b).”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Analyzing reader study results
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“After searching over temperature values {0.1, 0.5, 0.9} using GPT-3.5, Fig. 2 demonstrates that the lowest value, 0.1, performed best.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Constructing prompt anatomy
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results of this harm study (Fig. 4d) indicate that the medical expert summaries would have both a higher likelihood (14%) and a higher extent (22%) of possible harm compared to the summaries from the best model (12% and 16%, respectively).”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Analyzing reader study results
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Among open-source models, seq2seq models performed better than autoregressive models (Llama-2 and Vicuna) on radiology reports but worse on patient questions and progress notes (Fig. 3c).”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Model performance generally improved with more context.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although the model was more complete than the expert in generating its progress note summary, it also missed historical context (a history of hypertension).”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Analyzing reader study results
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Figure 5 provides an example in which the model’s summary includes correct information that readers deemed not important.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Analyzing reader study results
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Per Extended Data Table 1, this task entailed the largest token count across our six datasets for both the input (dialogue) and the target (assessment).”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Data
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Due to the wide variety of these questions, MeQSum exhibited the highest lexical variance of our datasets (Extended Data Table 1).”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Data
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“because the best model and method uses a handful of examples via ICL”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The benefit of larger models is explored in Extended Data Fig. 5, which found this improvement marginal for Llama-2 (13B) compared to Llama-2 (7B).”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > LLMs
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“radiology report summaries from medical experts occasionally recommend further studies or refer to prior studies—for example, ‘… not significantly changed from prior’ in Fig. 5.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Recent data indicate that physicians can expend up to 2 hours on documentation for each hour of patient interaction10.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
What the sources say
- Not checked — no full text of the sourcecites [10] Sinsky, C. et al. Allocation of physician time in ambulatory practice: a time and motion study in 4 specialties. Ann. Intern. Med. 165, 753–760 (2016). source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“Even for experienced physicians with a high level of expertise, this intricate task naturally introduces the possibility for errors, which can be detrimental in healthcare where precision is paramount4–6.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
What the sources say
- Not checked — no full text of the sourcecites [4] Yackel, T. R. & Embi, P. J. Unintended errors with EHR-based result management: a case series. J. Am. Med. Inform. Assoc. 17, 104–107 (2010). source
- Not checked — no full text of the sourcecites [5] Bowman, S. Impact of electronic health record systems on information integrity: quality and safety implications. Perspect. Health Inf. Manag. 10, 1c (2013).
- Not checked — no full text of the sourcecites [6] Gershanik, E. F., Lacson, R. & Khorasani, R. Critical finding capture in the impression section of radiology reports. AMIA Annu. Symp. Proc. 2011, 465–469 (2011).
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This suggests that integrating LLMs into clinical workflows could alleviate documentation burden, allowing clinicians to focus more on patient care.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We note that the model’s summaries are more concise while concurrently being more complete.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Analyzing reader study results
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This could be attributed to the relatively vague length specification in the radiology reports instruction—that is, ‘…with minimal text’—whereas the other two task instructions quantify length.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This illustrates the value of relevant context in achieving better outcomes for the target task.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Constructing prompt anatomy
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Governance aside, ICL provides many benefits: (1) model weights are fixed, hence enabling queries of pre-existing LLMs, and (2) adaptation is feasible with even a few examples, whereas fine-tuning methods, such as QLoRA, typically require hundreds or thousands of examples.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Note that seq2seq models are often constrained to much shorter context length than autoregressive models, such as GPT-4, because seq2seq models require the memory-intensive step of encoding the input sequence into a fixed-size context vector.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The evidence from our reader study suggests that adapting LLMs can outperform medical experts in terms of completeness, correctness and conciseness.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In both cases, the model’s summaries leaned toward literalness, a trait that readers sometimes favored and sometimes did not.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This instance underscores the point that even medical experts, not just LLMs, can hallucinate.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Analyzing reader study results
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The ensuing clinical reader study demonstrates that LLM summaries are often preferred over medical expert summaries due to higher scores for completeness, correctness and conciseness.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Given that BLEU measures sequence overlap, this result seems reasonable, as more text provides more ‘surface area’ for overlap; more text also reduces the brevity penalty that BLEU applies on generated sequences, which are shorter than the reference42.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
What the sources say
- Not checked — no full text of the sourcecites [42] Papineni, K., Roukos, S., Ward, T. & Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proc. of the 40th Annual Meeting of the Association for Computational Linguistics. https://dl.acm.org/doi/pdf/10.3115/1073083.1073135 (Association for Computing Machinery, 2002). source
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This highlights the model’s tendency toward a literal approach without interpretation, which may be either advantageous or limiting.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
hedgedSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Addressing this limitation presents an opportunity to accelerate the process of clinical text summarization, hence alleviating documentation burden and improving patient care.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We first highlight the importance of ‘prompt engineering’, or modifying and tuning the input prompt to improve model performance.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the low magnitude of correlation values (approximately 0.2) underscores the need to go beyond NLP metrics with a clinical reader study when assessing clinical readiness.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Results > Connecting quantitative and clinical evaluations
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Regarding blinding, for the clinical reader study, we presented, in a blinded manner, the A/B comparison of summaries from the model and human experts.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Statistics and reproducibility
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“it is not possible to verify whether a particular open-source dataset was included in model training.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When human evaluation is not feasible, Fig. 6 suggests that syntactic metrics are better at measuring completeness, whereas semantic and conceptual metrics are better at measuring correctness.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Compared to x-rays, MRIs and CT scans capture more information at a higher resolution. This usually leads to longer reports (Extended Data Table 1), rendering MIMIC-III a more challenging summarization dataset than Open-i or MIMIC-CXR.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Data
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“This study used only retrospective, de-identified data that fell outside the scope of institutional review board oversight.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Ethics approval
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The dialogue task was excluded owing to the unwieldiness of a reader parsing many lengthy transcribed conversations and paragraphs”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Reader study
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To obfuscate any formatting differences between A and B, we applied simple post-processing to standardize capitalization, punctuation, newline characters, etc.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Statistics and reproducibility
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For the quantitative experiments, our sample size of 250 was chosen to maximize the number of samples given constraints of dataset size and cost of computing resources. We deem this sufficient as it enabled reproducible results when running the same experiments with a different set of samples.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Statistics and reproducibility
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Although our datasets do not contain demographic information”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, our use of six datasets enhances the robustness of our experiments.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Statistics and reproducibility
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“This study adhered to the principles outlined in the Declaration of Helsinki.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Ethics approval
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“for the quantitative experiments, we randomly selected 250 test samples and then randomly divided remaining samples into training and validation sets.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Statistics and reproducibility
plainn = 250- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Informed consent was obtained from each physician before their participation.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Ethics approval
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Because the LLM temperature parameter is near zero, outputs have very little randomness and are, thus, reproducible.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Statistics and reproducibility
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“No data were excluded from the analyses.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Statistics and reproducibility
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These instances are out of scope for the LLM, as it does not have access to prior studies nor the purview to make recommendations.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Given the critical nature of nuanced medical tasks, such as summarization, that typically have no objective solutions, including human experts in the evaluation process beyond quantitative metrics is crucial to demonstrate clinical readiness.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For the clinical reader study, our sample size of 100 comparisons per reader per task was chosen to maximize the number of comparisons that could be made in a reasonable amount of time, which we estimated as 10 h. We deem this sufficient as it enabled statistically significant results for many combinations of tasks and attributes.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Statistics and reproducibility
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This suggests that incorporating LLM-generated candidate summaries could reduce documentation load, potentially leading to decreased clinician strain and improved patient care.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
In the paper
“Although three of our datasets (MIMIC-CXR, MIMIC-III and ProbSum) require PhysioNet57 access to ensure safe data usage by third parties, this is no guarantee against data leakage.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
What the sources say
- Not checked — no full text of the sourcecites [57] Johnson, A., Pollard, T. & Mark, R. MIMIC-III Clinical Database Demo (version 1.4). PhysioNet https://doi.org/10.13026/C2HM2Q (2019). source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“we did not consider the inherently context-specific nature of summarization.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The quote only partly states this claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Note that we reduced the temperature parameter of the LLMs to be near zero, hence reducing randomness in the generated output summaries.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Statistics and reproducibility
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For the clinical reader study, we randomly selected 100 samples for comparison; these samples and the A/B comparisons within each sample are displayed in random order.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Statistics and reproducibility
plainn = 100- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“we chose the smallest dataset––which consequently required the lowest cost of computational resources––and re-ran a set of experiments that rendered very similar quantitative metric scores. This provided confidence moving forward with datasets that required higher computational resources.”
§ Adapted large language models can outperform medical experts in clinical text summarization > Methods > Statistics and reproducibility
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Although three of our datasets (MIMIC-CXR, MIMIC-III and ProbSum) require PhysioNet57 access to ensure safe data usage by third parties”
§ Adapted large language models can outperform medical experts in clinical text summarization > Discussion
What the sources say
- Not checked — no full text of the sourcecites [57] Johnson, A., Pollard, T. & Mark, R. MIMIC-III Clinical Database Demo (version 1.4). PhysioNet https://doi.org/10.13026/C2HM2Q (2019). source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
The effectiveness of large language models on a diverse range of clinical summarization tasks remains unproven.
“Although large language models (LLMs) have shown promise in natural language processing (NLP) tasks, their effectiveness on a diverse range of clinical summarization tasks remains unproven.”
The paper relies on this without support.
Adapting LLMs to summarize a diverse set of clinical tasks has not been thoroughly explored, and non-inferiority to medical experts has not been achieved.
“However, adapting LLMs to summarize a diverse set of clinical tasks has not been thoroughly explored nor has non-inferiority to medical experts been achieved.”
The paper relies on this without support.
Fabricated information poses a substantial obstacle to clinical integration of LLMs because medical applications require accuracy.
“Fabricated information, or factually incorrect text, poses a substantial obstacle to the clinical integration of LLMs given the critical need for accuracy in medical applications.”
The authors state this as an assumption.
Samples with similar A/B summaries in completeness and correctness were assumed to contribute no harm.
“the subset of samples with similar A/B summaries (in completeness and correctness) were assumed to contribute no harm.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Most have been tested only with text-based, low-stakes interactions,33–38”
From Review of Artificial Intelligence for Clinical Use in Alzheimer's Disease and Related Dementias (2026) · cites this paper as [34]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The first four models were selected because they were among the most widely used LLMs at the time of study inception (January 2024), and prior research found that LLaMA 2 and GPT models outperformed humans in medical text summarization [19], while Claude models demonstrated greater faithfulness in book summaries than GPT [20].”
From Generalization bias in large language model summarization of scientific research (2025) · cites this paper as [19]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Scientists use LLM-assisted writing because LLMs can improve grammar, rhetoric, and overall readability of their texts, help translate to English, and quickly generate summaries (33, 34).”
From Delving into LLM-assisted writing in biomedical publications through excess vocabulary (2025) · cites this paper as [33]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Our finding challenges previous studies on clinical text summarization which reported that ‘summaries from our best-adapted LLMs (including GPT-3.5, GPT-4, and LLaMA 2) are preferable to human summaries in terms of completeness and correctness’ [19, p. 3].”
From Generalization bias in large language model summarization of scientific research (2025) · cites this paper as [19]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Most LLM studies have depended on human lead semi-quantitative assessment of coherence, comprehensiveness, harmfulness and factual inconsistencies as well as comparing LLM with human based summaries [5, 13, 14].”
From A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population (2024) · cites this paper as [5]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Automated metrics do not correlate with quality and human input to score coherence, inconsistencies, comprehensiveness and harmfulness are semi-quantitative and will differ based on the use care in which LLM are applied [5, 13].”
From A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population (2024) · cites this paper as [5]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Most studies applying LLM on clinical notes have been conducted in the US without individual consent, using de-identified data, with only one study from France utilising retrospective MRI reports without identifiable data [5, 6, 13, 14, 17–20].”
From A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population (2024) · cites this paper as [5]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Despite comparable performance with clinician-written summaries (Veen et al., 2024), hallucinations manifest as fabricated or distorted clinical statements that are hard to detect (Asgari et al., 2025; Kim et al., 2025).”
From Hallucination Detection-Guided Preference Optimization for Clinical Summarization (2026) · cites this paper as Veen et al., 2024
The cited source says what the claim attributes to it.
In the citing paper
“LLMs are capable of processing large volumes of text and generating human-like summaries and interpretations [2].”
From Evaluating Medical Text Summaries Using Automatic Evaluation Metrics and LLM-as-a-Judge Approach: A Pilot Study (2025) · cites this paper as [2]
The cited source says what the claim attributes to it.
In the citing paper
“Domain-specific LLMs, fine-tuned on medical corpora or using knowledge graph embeddings, perform well in clinical text summarization [10, 11], note generation [12, 13], and medical question answering [14].”
From Enhancing hepatopathy clinical trial efficiency: a secure, large language model-powered pre-screening pipeline (2025) · cites this paper as [10]
The cited source says what the claim attributes to it.
In the citing paper
“A study by Van Veen et al. reported that medically adapted LLM can summarize text better than physicians [23].”
From Evaluating the Capability of Large Language Model Chatbots for Generating Plain Language Summaries in Radiology (2025) · cites this paper as [23]
The cited source says what the claim attributes to it.
In the citing paper
“Our system employs local language models to preserve data privacy, which comes at some cost to performance – the local model may not match that of commercial models58,59.”
From A large language model digital patient system enhances ophthalmology history taking skills (2025) · cites this paper as [59]
The cited source says what the claim attributes to it.
In the citing paper
“Many of the studies on LLM have been conducted using large anonymised US datasets [5–7].”
From A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population (2024) · cites this paper as [5]
The cited source says what the claim attributes to it.
In the citing paper
“Large language models (LLM) such as ChatGPT, GPT-4 API and Llama2 can interrogate and summarise large volumes of medical notes to create succinct summaries of radiology reports, internal medical progress notes and patient dialogue [5–8].”
From A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population (2024) · cites this paper as [5]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Our system employs local language models to preserve data privacy, which comes at some cost to performance – the local model may not match that of commercial models58,59.”
From A large language model digital patient system enhances ophthalmology history taking skills (2025) · cites this paper as [59]
Experimental — accuracy not yet measuredIn the citing paper
“LLMs have shown expert-level performance in clinical text summarization when appropriate prompting strategies are employed3.”
From Towards evaluating and building versatile large language models for medicine (2025) · cites this paper as [3]
In the citing paper
“Therefore, related works have instrumented quantitative (completeness, correctness, conciseness) and qualitative (content, language, structure) measurements to dissect the riveting characters that distinguish synthesized clinical reports from human reports46–49.”
From Towards a holistic framework for multimodal LLM in 3D brain CT radiology report generation (2025) · cites this paper as [46]
Citations
Cites
A guideline of selecting and reporting intraclass correlation coefficients for reliability research
Koo, T. K., & Li, M. Y. (2016) · Journal of Chiropractic Medicine · doi:10.1016/j.jcm.2016.02.012 · Cited by 31,709 · cited 1×
“We estimated intra-reader correlation based on a mean-rating, fixed-agreement, two-may mixed-effects model 90 using the Pingouin package 91 .”
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Goldberger, A. L., Amaral, L. A. N., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark, R. G., Mietus, J. E., Moody, G. B., Peng, C.-K., & Stanley, H. E. (2000) · Circulation · doi:10.1161/01.cir.101.23.e215 · Cited by 14,809
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 13,453
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
International Journal of Environmental Research and Public Health
Casariego Vales, E., A.M., C.-C., J.L., L., R.M., M.-P., D., O.-B., M., O.-C., P., R.-S., & F.J., S. (2013) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph · Cited by 7,742
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254 · cited 1×
“In addition, beyond the scope of this manuscript, there is further potential to improve accuracy through incorporating checks by human operators and checks by other LLMs or using a model ensemble 47,48 .”
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · cited 2×
“Previous work has demonstrated potential across clinical NLP tasks 22,23 , adapting to the medical domain by training a new model 24,25 , fine-tuning an existing model 26,27 or supplying task-specific examples in the model prompt 27,28 .”
Research works printed but not linked: 9
- Gesner, E., Gazarian, P. & Dykes, P. The burden and burnout in documenting patient care: an integrative literature review. Stud. Health Technol. Inform. 21, 1194–1198 (2019).
- Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q. & Artzi, Y. BERTScore: evaluating text generation with BERT. International Conference on Learning Representations. https://openreview.net/forum?id=SkeHuCVFDr (2020).
- Lin, C.-Y. ROUGE: a package for automatic evaluation of summaries. In Text Summarization Branches Out 74–81 (Association for Computational Linguistics, 2004).
- Lehman, E. et al. Do we still need clinical language models? In Proceedings of Machine Learning Research 209, 578–597 (Conference on Health, Inference, and Learning, 2023).
- Demner-Fushman, D., Ananiadou, S. & Cohen, K. B. The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks. https://aclanthology.org/2023.bionlp-1 (Association for Computational Linguistics, 2023).
- Gao, Y., Miller, T., Afshar, M. & Dligach, D. BioNLP Workshop 2023 Shared Task 1A: Problem List Summarization (version 1.0.0). PhysioNet. https://doi.org/10.13026/1z6g-ex18 (2023). · DOI
- Yim, W., Ben Abacha, A., Snider, N., Adams, G. & Yetisgen, M. Overview of the MEDIQA-Sum task at ImageCLEF 2023: summarization and classification of doctor–patient conversations. In CEUR Workshop Proceedings https://ceur-ws.org/Vol-3497/paper-109.pdf (2023). · DOI · Paper page
- Loshchilov, I. & Hutter, F. Decoupled weight decay regularization. In International Conference on Learning Representations https://openreview.net/forum?id=Bkg6RiCqY7 (2019)
- Okazaki, N. & Tsujii, J. Simple and efficient algorithm for approximate dictionary matching. In Proc. of the 23rd International Conference on Computational Linguistics https://aclanthology.org/C10-1096.pdf (Association for Computational Linguistics, 2010).
Cited by
Sorted by citations
Items 401–500 of 856
HEAL-Summ: a lightweight and ethical framework for accessible summarization of health information
Fisher, A., Srinivasan, K. K., Hillier, S. A., & Mago, V. (2025) · Frontiers in Public Health · doi:10.3389/fpubh.2025.1619274 · Cited by 2
‘Hey Siri, ask them out:’ investigating the use of artificial intelligence in date request messaging
Roaché, D. J., Sharabi, L. L., Hicks, L. L., & Hurley, M. M. (2025) · Behaviour and Information Technology · doi:10.1080/0144929x.2025.2559289 · Cited by 4
“Although evidence abounds showing how AI-MC may perform better than humans in some situations (e.g. sum-marising clinical texts; Van Veen et al. 2024), we assert that an objective measure of message quality may explain observed differences between AI-MC influence and human messages.”
Taxonomy Portraits: Deciphering the Hierarchical Relationships of Medical Large Language Models
Nagarajan, R., Klotzman, V., Kondo, M., Godambe, S. A., Gold, A., Henderson, J. J., & Martel, S. (2025) · JMIR Medical Informatics · doi:10.2196/72918 · Cited by 2
“Specific LLM health care applications include (1) virtual health assistants and language translation [3], (2) summarization of clinical narratives and ambient listening [4,5], (3) patient education [6], and (4) clinical trial matching [7].”
Mehta, N. N., Walker, E., Flester, E., Folk, G. A., Agnihotri, A. P., Nagel, I. D., Tran, M. D., Goldbaum, M. H., Borooah, S., & Scott, N. L. (2025) · Translational Vision Science & Technology · doi:10.1167/tvst.14.10.13 · Cited by 2
Li, W., & Li, Q. (2025) · Digital Health · doi:10.1177/20552076251393385 · Cited by 6
Large language models in clinical psychiatry: Applications and optimization strategies
Wang, Y., Li, M.-D., Wang, S.-H., Fang, Y., Sun, J., Lü, L., & Yan, W. (2025) · World Journal of Psychiatry · doi:10.5498/wjp.v15.i11.108199 · Cited by 5
Cui, H., Zhang, Z., Zhang, J., Du, C., Zhang, W., & Cui, H. (2025) · Aesthetic Plastic Surgery · doi:10.1007/s00266-025-05393-8 · Cited by 3
Vasilev, Y. A., Vladzymyrskyy, A. V., Omelyanskaya, O. V., Alymova, Y. A., Альфредовна, А. Д., Федоровна, Ш. Ю., Kodenko, M. R., Blokhin, I. A., & Reshetnikov, R. V. (2025) · AI · doi:10.3390/ai6110287 · Cited by 3
Wang, S., & Ye, Q. (2025) · JMIR Medical Informatics · doi:10.2196/73857 · Cited by 4
“Among these, the embedding average score and BERTScore use cosine similarity between vectors to calculate the relevance to the standard answers, which has proven to be an effective approach [32-34].”
Artificial Intelligence in Pediatric Ultrasound: An Update and Future Applications
Kuang, C., Jiang, Z., Wang, Y., Fang, A., Zhang, H., Liu, H., & Ye, Z. (2025) · BIO Integration · doi:10.15212/bioi-2025-0130 · Cited by 3
“LLM-based automated text processing can reduce human errors in report writing, shorten the time needed to generate diagnostic reports, and enhance the integration and analysis of multimodal data, especially in pediatric settings [30].”
DCMI: A Differential Calibration Membership Inference Attack Against Retrieval-Augmented Generation
Gao, X., Meng, X., Dong, Y., Li, Z., & Guo, S. (2025) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3719027.3765103 · Cited by 4
Yum, J.-E. I., Naqvi, S. A. A., Zhou, B., & Riaz, I. B. (2025) · JCO Clinical Cancer Informatics · doi:10.1200/cci-25-00134 · Cited by 4
Human versus machine: Can generative AI anticipate insect biological control outcomes?
Wyckhuys, K. A. G., Akutse, K. S., Amalin, D. M., Araj, S.-E., Beltran, M. J. B., Fekih, I. B., Calatayud, P., Cicero, L., Cokola, M. C., Colmenárez, Y. C., et al. (2025) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2025.111317 · Cited by 2
Devane, D., Pope, J. S., Byrne, P., Forde, E., O'Byrne, I., Woloshin, S., Culloty, E., Dahly, D., Elgersma, I. H., Munthe‐Kaas, H. M., et al. (2025) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2025.112102 · Cited by 5
Alqahtani, H. (2025) · BMC Oral Health · doi:10.1186/s12903-025-07573-z · Cited by 4
Rethinking sepsis prediction in the era of large language models
Wong, A., Singh, K., & Nemati, S. (2026) · npj Health Systems · doi:10.1038/s44401-025-00062-z · Cited by 2
Cutting-edge AI technologies in skin cancer applications
Diao, Y., Chen, X., Huang, Z., Tan, Q., Yang, M., Yu, H., Xu, Y., & Hu, X. (2026) · Cancer Letters · doi:10.1016/j.canlet.2026.218256 · Cited by 2
Lajmi, N., Patel, M., Obery, G., Dorge, A., Sharma, A., Halligan, J., & Lo, E. (2026) · JCO Clinical Cancer Informatics · doi:10.1200/cci-25-00244 · Cited by 2
“1,18,24 Yet, despite growing interest in LLM-based summarization tools, their clinical utility and clinician acceptance in oncology work fl ows remain largely untested.”
Performance of Large Language Models for Radiology Report Impression Generation: A Systematic Review
Ng, C. K. C., Sun, Z., & Te, I. K. H. (2026) · Technologies · doi:10.3390/technologies14020099 · Cited by 2
Bouterse, A., Pruneski, J. A., Oettl, F. C., Zsidai, B., Tischer, T., Longo, U. G., Seil, R., Hirschmann, M. T., & Samuelsson, K. (2026) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1002/ksa.70339 · Cited by 2
“Furthermore, incorporation of large language models (LLMs) into EHR systems has demonstrated the ability of AI to outperform physicians when tasked with sum-marising and transcribing clinical text [31, 71, 78].”
Khoruzhaya, A. N., Varyukhina, M. D., Erizhokov, R. A., Blokhin, I. A., Reshetnikov, R. V., Kodenko, M. R., Pamova, A. P., Burtsev, T., Arzamasov, K. M., Omelyanskaya, O. V., et al. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1761601 · Cited by 2
“%) to those produced by medical experts (3).”
Chen, Y., He, K., Qu, W., Ma, L., Su, X., Liu, X., Kang, Z., Li, F., Yan, J., Wang, Z., et al. (2026) · European Journal of Cancer · doi:10.1016/j.ejca.2026.116667 · Cited by 2
RAG-based architectures for drug side effect retrieval using compact LLMs
Nygren, S., Erdogan, O., Avci, P., Daniels, A., Rassool, R., Beheshti, A., & Galeano, D. (2026) · Scientific Reports · doi:10.1038/s41598-026-41495-2 · Cited by 2
CKD-EHR: Clinical Knowledge Distillation for Electronic Health Records
Wang, J., Ling, H., Zhang, L., Zhang, L., Wang, F., Gao, Y., & Li, Z. (2026) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2026.110090 · Cited by 2
Muasher-Kerwin, C., Hughes, M. C., & Econie, S. M. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-01088-4 · Cited by 3
Du, X., Anna, N., Oates, M., Wang, Y., Wang, X., Plasek, J. M., Aronson, S., Lebo, M. S., & Zhou, L. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2026.106424 · Cited by 4
Bias Patterns in the Application of LLMs for Clinical Decision Support
Poulain, R., Adiba, F. I., Fayyaz, H., & Beheshti, R. (2026) · Delaware Journal of Public Health · doi:10.32481/djph.2026.03.10 · Cited by 32
Comparison of AI-generated radiology impressions: a multi-stakeholder evaluation
Phadke, S., Suresh, N., Allen, Z., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02586-6 · Cited by 2
“Similar fi ndings have been reported in prior studies, suggesting that downstream clinicians may value explanatory context and narrative clarity more than brevity alone 7,8 .”
InfluentialLiu, Y., John, J., Sarkar, S., Zakkar, A., Kinkopf, P., Teo, P. T., & Abazeed, M. E. (2026) · JCO Clinical Cancer Informatics · doi:10.1200/cci-25-00284 · Cited by 2
Peters, U., Bertazzoli, A., DeJesus, J. M., van der Velden, G. J., & Chin-Yee, B. H. (2026) · Public Understanding of Science · doi:10.1177/09636625261425891 · Cited by 2
“Moreover, people are increasingly using chatbots powered by large language models (LLMs), 1 such as ChatGPT, to learn about scientific findings, because LLMs can quickly summarize complex information in accessible terms (Van Veen et al., 2024).”
Tracing the Pen: Electronic Health Records Amid the Rise of Generative AI
Nargesi, A. A., You, J. G.-T., Bitterman, D. S., Succi, M. D., Mishuris, R. G., Poon, E. G., & Landman, A. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02508-6 · Cited by 3
“LLMs have also shown an equivalent or superior performance to medical experts in chart summarization tasks, including those related to patient questions 38 .”
Responsible use of large language models in gastroenterology and hepatology
Khanna, S. (2026) · Therapeutic Advances in Gastroenterology · doi:10.1177/17562848261441693 · Cited by 2
Chow, M., Li, A., Quek, D. Y. J., Fan, E. B., & Sadasiv, M. S. (2026) · Postgraduate Medical Journal · doi:10.1093/postmj/qgag068 · Cited by 6
Use of AI agents to assess preoperative frailty in cancer patients
Lee, D. T., Yen, S., Smits, G. E., Matteson, D. S., Cohen, N., Klang, E., Nadkarni, G. N., & Marin, M. L. (2026) · npj Digital Surgery · doi:10.1038/s44484-026-00012-8 · Cited by 3
“Frontier LLMs have demonstrated impressive performance across a wide range of traditional health care tasks that can be automated, such as medical question and answering 6,7 , clinical report summarization 8 – 10 and data extraction from narrative reports 11 – 15 .”
Large Language Model Authorship in Ophthalmic Publications
Fortenbach, C. R., Wu, Y. S., Mungra, P. M., & Gelder, R. N. V. (2026) · Ophthalmology · doi:10.1016/j.ophtha.2026.05.037 · Cited by 2
Artificial Intelligence Note Summarization in the Emergency Department
You, A. X., Kahl, N. M., Patel, A., Cao, J., Castillo, E. M., Bedmutha, P. S., Chan, T., Singh, K., & Longhurst, C. (2026) · JAMA Network Open · doi:10.1001/jamanetworkopen.2026.18021 · Cited by 2
Generative Artificial Intelligence and Large Language Models in Clinical Oncology
Yu, Y., Zhao, Z., Wang, Z., Lin, R., Tan, Y., Chen, Y., Li, T., Baptista‐Hon, D. T., Zhang, X., Wu, C., et al. (2026) · MedComm · doi:10.1002/mco2.70833 · Cited by 2
Tutorial: guidance on the use of large language models for medical research
Jin, Q., Wan, N., Leaman, R., Tian, S., Wang, Z., Yang, Y., Wang, Z., Xiong, G., Lai, P.-T., Zhu, Q. L., et al. (2026) · Nature Protocols · doi:10.1038/s41596-026-01408-z · Cited by 15
Serapio, A., Chen, T. L., Tangsombatvisit, B., et al. (2026) · Radiology · doi:10.1148/radiol.253238 · Cited by 2
Safety and security of large language models in healthcare
Clusmann, J., Freyer, O., Ostermann, M., Ferber, D., Laleh, N. G., Hilgers, L., Kolbinger, F. R., Schneider, C. V., Downing, A., Wekenborg, M., et al. (2026) · Nature · doi:10.1038/s41586-026-10687-1 · Cited by 5
A Comprehensive Study of Automatic Text Summarization Techniques
Vaishali, Sehgal, G., & Dixit, P. (2024) · International Conference on Emerging Innovations and Advanced Computing (INNOCOMP) · doi:10.1109/innocomp63224.2024.00119 · Cited by 2
Enhancing Medical Summarization with Parameter Efficient Fine Tuning on Local CPUs
Yang, S. S. Z., Fye, G. M., Yap, W. C., & Yu, D. (2024) · International Conference on Electrical, Communication, and Computer Engineering (ICECCE) · doi:10.1109/icecce63537.2024.10823619 · Cited by 2
Empowering Meta-Analysis: Leveraging Large Language Models for Scientific Synthesis
Ahad, J. I., Sultan, R. M., Kaikobad, A., Rahman, F. A., Amin, M. R., Mohammed, N., & Rahman, S. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825310 · Cited by 6
Martin, E., Landon, B. E., Spetz, J., et al. (2025) · Health Affairs Scholar · doi:10.1093/haschl/qxaf006 · Cited by 2
“17 Meanwhile, artificial intelligence–based solutions, such as large language and machine learning models, have been shown to effectively summarize clinical note content 18 and to anno-tate and summarize transcripts from Objective Structural Clinical Examinations, 19 respectively.”
Natural Language Processing in Gastroenterology
Parasa, S., & Sridhar, A. R. M. (2025) · Gastrointestinal Endoscopy Clinics of North America · doi:10.1016/j.giec.2024.09.005 · Cited by 2
A Proposed LLM-Based Supported Treatment Framework for Intracerebral Hemorrhage
Kim, H.-J., Park, D., Lee, J.-J., Jeon, J. P., & Won, D.-O. (2025) · International Conference on Consumer Electronics · doi:10.1109/icce63647.2025.10930045 · Cited by 2
Clinical Text Summarization using NLP Pretrained Language Models: A Case Study of MIMIC-IV-Notes
Arokodare, O., Wimmer, H., & Du, J. (2025) · Journal of Information Systems Applied Research and Analytics · doi:10.62273/naka3054 · Cited by 4
Performance of Large Language Model-Generated Spanish Discharge Material
Pérez-Guerrero, E. J., Aali, A., Irizarry, E., Corso, N. K., Hom, J., Rodríguez, F., & Santiago, C. (2025) · Journal of General Internal Medicine · doi:10.1007/s11606-025-09758-2 · Cited by 2
Self-citationIndrakanti, A. K., Heierle, J. E., Muenger, H., Koch, A. T., Kaiser, P., Bach, M., Fiehler, J., Tsogkas, I., Guzman, R. G., Mutke, M. A., et al. (2025) · Clinical Neuroradiology · doi:10.1007/s00062-025-01538-z · Cited by 2
Tran, H. H.-V., Thu, A., Twayana, A. R., Fuertes, A., Gonzalez, M., Basta, M., James, M., Frishman, W. H., & Aronow, W. S. (2025) · Cardiology in Review · doi:10.1097/crd.0000000000001042 · Cited by 2
Dermatology 3.0: New Technologies Transforming the Management of Skin Conditions
Tan, J. K. L., Koh, M. J. A., & Navarrete‐Dechent, C. (2025) · JEADV Clinical Practice · doi:10.1002/jvc2.70089 · Cited by 2
Green, Z., Ashton, J. J., & Beattie, R. M. (2025) · Archives of Disease in Childhood · doi:10.1136/archdischild-2025-329505 · Cited by 2
Fine-Tuning LLaMA2 for Summarizing Discharge Notes: Evaluating the Role of Highlighted Information
Dehkordi, M. K. H., Perl, Y., Deek, F. P., & Liu, H. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc10010004 · Cited by 5
Liu, D., Hu, X., Xiao, C., Bai, J., Barandouzi, Z. A., Lee, S., Webster, C., Brock, L.-U. B., Lee, L., Bold, D., et al. (2024) · doi:10.2196/preprints.67914 · Cited by 2
Kadhim, A. Z., Green, Z., Nazari, I., Baker, J., George, M. R., Heinson, A., Stammers, M., Kipps, C., Beattie, R. M., Ashton, J. J., et al. (2025) · medRxiv · doi:10.1101/2025.03.07.25323569 · Cited by 2
Prabha, P. L., & Parvathy, M. (2025) · Journal of the Chinese Institute of Engineers · doi:10.1080/02533839.2025.2466647 · Cited by 2
“It can be subdivided into a number of input file-based summarizations in which (Fan et al. 2023) single-document and (Van Veen et al. 2024) multi-document summarizations are involved.”
Fine-Tuning LLaMA2 for Summarizing Discharge Notes: Evaluating the Role of Highlighted InformationPreprint
Dehkordi, M. K. H., Perl, Y., Deek, F. P., et al. (2025) · Research Square · doi:10.21203/rs.3.rs-7181141/v1 · Cited by 2
LLMs for Radiology Reports: From General Purpose to Lightweight Domain Adaptation
Huisman, M., & Rasoolzadeh, N. (2025) · Radiology · doi:10.1148/radiol.252524 · Cited by 2
Takayama, T., Sado, K., Suda, K., Tamura, H., Ueda‐Arakawa, N., Ishihara, K., Ueda, Y., Okamoto, K., Santos, L. H. O., Oshika, T., et al. (2026) · PLOS Digital Health · doi:10.1101/2025.10.06.25337211 · Cited by 3
“18 However, as both the input prompts and target outputs lengthen (e.g., discharge referrals and discharge summaries), known issues such as hallucinations and other errors become more prominent.”
Structured reasoning failures compromise LLM interpretation of clinical oncology notes
Kenaston, M. W., Ayub, U., Parmar, M., Kumar, P., Anjum, M. U., Naqvi, S. A. A., Chaudhuri, A. A., Zakharia, Y., Heath, E. I., Bekaii-Saab, T. S., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02951-5 · Cited by 2
Yunus, S., Hark, C., & Okumuş, F. (2024) · International Artificial Intelligence and Data Processing Symposium (IDAP) · doi:10.1109/idap64064.2024.10710907 · Cited by 1
An Introductory Module of Generative Artificial Intelligence in Medical Education
Cervantes, J. (2024) · Medical Science Educator · doi:10.1007/s40670-024-02218-2 · Cited by 2
Automatic Summarization of Life-Critical Situations by Generative AI
Sun, Y., & Li, X. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0840-9_5 · Cited by 2
Less Hallucination and More Factuality: Human Values Alignment in Text Summarization
Qiu, Z., & Xu, Y. (2024) · Communications in computer and information science · doi:10.1007/978-981-97-9671-7_14 · Cited by 1
Abstractive Summarization of Lectures and Lecture Segments Transcripts with BART
Alesh, Y., Aoudia, M., Abdulghani, O., Ali, O. A., & Talib, M. W. A. (2024) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-981-97-9255-9_4 · Cited by 2
Fine-Tuning Generative Pre-Trained Transformers for Clinical Dialogue Summarization
Ronan, I., & Tabirca, S. (2024) · International Conference on Frontiers of Information Technology (FIT) · doi:10.1109/fit63703.2024.10838420 · Cited by 1
Rosen, J. M. (2025) · Gastroenterology Clinics of North America · doi:10.1016/j.gtc.2025.01.001 · Cited by 1
ChatCEA: a knowledge-driven intelligent service agent for controlled environment agriculture
Xia, F., Pan, J., Zhong, R., Liu, W., Chen, T., Zhu, L., Yang, Y. X., Hua, G., & Lin, T. (2026) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2026.111733 · Cited by 2
Khurana, J., Zaki, H. A., Pavlick, E., Turbitt, J., McGee, H. A., Gupta, S., Datla, S. S. P., Eldeeb, S., Salazar‐Mather, T. P., Warrier, S. S., et al. (2026) · BMC Medical Education · doi:10.1186/s12909-026-09975-1 · Cited by 2
Green, K., Asthana, S., Ponce, O. J., Downey, J., & Watson, J. (2025) · JMIR Research Protocols · doi:10.2196/68511 · Cited by 2
“There is growing interest in generating medical summaries using LLMs to condense complex patient information from text or voice-recorded datasets [7].”
InfluentialKapur, I., Sharkey, C. N., & Ren, C. (2025) · Journal of Evidence-Based Social Work · doi:10.1080/26408066.2025.2517073 · Cited by 2
Feasibility of Automated Precharting using GPT-4 in New Specialty Referrals.
Liang, A. S., Banda, J. M., Savage, T., Pandya, A., Carey, R., Megwalu, U. C., Chang, M. T., Dash, D., Corbin, C. K., Sharma, A., et al. (2025) · Europe PMC (PubMed Central) · Cited by 1
Cost-Effective Medical Dialogue Summarization with Fine-Tuned Compact LLMs
Yang, L. H., Xu, C., & Wang, X. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-6963-9_18 · Cited by 1
Performance of Transformer-Based Methods on Restaurant Reviews Analysis
Karaduman, M., Baydemir, M. B., & Yıldırım, M. (2025) · Firat University Journal of Experimental and Computational Engineering · doi:10.62520/fujece.1632266 · Cited by 1
Lin, K.-S., Tsai, F., Kuo, N., et al. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-8892-0_15 · Cited by 1
Symptom Recognition in Medical Conversations Via multi- Instance Learning and Prompt
Wang, H., Bai, X.-F., Cui, X.-T., Chen, G., Fan, G.-M., Wei, G.-L., Zheng, Y.-P., Wu, J., & Gao, S.-S. (2025) · Journal of Medical Systems · doi:10.1007/s10916-025-02240-w · Cited by 2
“In the biomedical domain, models such as BioBERT [13] and related adaptations [ 14] have further improved performance by incorporat - ing large-scale medical text corpora.”
Meddeb, A., Bakhtiari, N., Rangus, I., Fetscher, L., Leguellec, B., Busch, F., Doucet, A., Hua, V. T., Verot-Nguyen, F., Paiusan, L. V., et al. (2026) · Journal of NeuroInterventional Surgery · doi:10.1136/jnis-2026-025429 · Cited by 1
Liu, Y., & Zhang, Y. (2025) · Physiotherapy Practice and Research · doi:10.1177/22130683251379638 · Cited by 2
“Studies have indicated that it can streamline medical documentation practices, greatly reducing clinicians ’ administrative burdens 17 and allowing greater focus on patient care.”
Patil, S. S., Ramteke, M., Verma, M., Chandra, T., & Rathore, A. S. (2025) · Neural Computing and Applications · doi:10.1007/s00521-025-11600-z · Cited by 1
“Extensive experiments comparing QLoRA with other PEFT methods highlight its superior efficiency, especially in terms of GPU memory usage and performance retention [14, 35–37].”
Bamoula, A., Abbaoui, W., Es-semlali, N., et al. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-032-01967-7_35 · Cited by 1
InstructStep: Fine-Grained Localization of Step Content and Relation in Instructional Video
He, W., Xu, W., Guo, P., Miao, Z., & Tian, Y. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3754999 · Cited by 1
Sun, Y., Chen, Z., Cai, H., & Lü, X. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.106171 · Cited by 1
Bains, B., Rapuri, S., Robitaille, E., Wang, J., Khera, A., Gómez, C., Reyes, E., Perry, C., Wilson, J., & Tracey, E. (2025) · JMIR Medical Informatics · doi:10.2196/80205 · Cited by 1
Jin, T., Yu, X., Ai, Z., & Guo, H. (2025) · Advanced ultrasound in diagnosis and therapy · doi:10.26599/audt.2025.250104 · Cited by 4
Li, S., Gong, J., & Molina, A. A. R. (2025) · ACM/IEEE International Conference on Connected Health: Applications, Systems and Engineering Technologies (CHASE) · doi:10.1145/3721201.3725440 · Cited by 1
Evaluating empathy in GPT-4-generated vs. physician-written emergency department discharge letters
Ben-Haim, G., Livne, A., Manor, U., Hochstein, D., Saban, M., Blaier, O., Iram, Y. A., Balzam, M. G., Lutenberg, A., Eyade, R., et al. (2025) · Digital Health · doi:10.1177/20552076251389992 · Cited by 2
Kulkarni Kale, U., & Vemulapalli, G. (2025) · The Cancer Journal · doi:10.1097/ppo.0000000000000799 · Cited by 1
Künstliche Intelligenz: Chancen und Grenzen
Clusmann, J., Schneider, C. V., Foersch, S., Truhn, D., & Kather, J. N. (2025) · Forum · doi:10.1007/s12312-025-01520-3 · Cited by 1
Large language models and their impact in medical imaging education
Zhu, J., & Cai, H. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.3433 · Cited by 2
Leveraging LLMs for Efficient Data Structure Standardization in Chinese Medical Examination Reports
Zhang, Y., Zhang, X., Wang, R., et al. (2025) · doi:10.1109/embc58623.2025.11254111 · Cited by 1
Large Language Modeling–Enabled Analysis of Atrial Fibrillation on Social Media
Parsa, S., Somani, S. S., Rogers, A. J., Hernandez‐Boussard, T., Jain, S. S., & Rodríguez, F. (2025) · Journal of the American Heart Association · doi:10.1161/jaha.125.043999 · Cited by 1
Park, S. M., Park, J. Y., Lee, T. S., Kim, J. H., Yoon, H., Kim, B. S., & Moon, B. S. (2025) · Archives of Pharmacal Research · doi:10.1007/s12272-025-01591-9 · Cited by 1
“AI-assisted reconstruction and quantification can shorten the scan duration, reduce patient burden, and support real-time pharmacokinetic modeling—benefits that are particularly relevant to early clinical and microdosing studies (Van Veen et al. 2024; Hu et al. 2025).”
Yogo, T. (2025) · Frontiers in Veterinary Science · doi:10.3389/fvets.2025.1690085 · Cited by 1
Eftee, S. Y., & Abrar, A. (2025) · Communications in computer and information science · doi:10.1007/978-3-032-11352-8_21 · Cited by 1
Mushtaq, S., & Veningston, K. (2026) · International Journal of Data Science and Analytics · doi:10.1007/s41060-025-00956-z · Cited by 2
Lu, J., Chiu, K. W.-H., Chan, C., Cheng, H.-M., Zhou, J., Justin Christopher, N. G., Tang, F. F. Y., Kan, W. K., Yu, P. L. H., & Seto, W. (2026) · European Journal of Radiology · doi:10.1016/j.ejrad.2026.112689 · Cited by 1
Lin, T., Wan, H., Liang, Y., Ming, J., Lu, J., & Guo, Z. (2026) · Frontiers in Immunology · doi:10.3389/fimmu.2025.1679496 · Cited by 1
Liao, W., Li, M., Ma, C., Han, Y., Wang, D., Liu, H., Wang, Y., Feng, Z., Wang, H., & Guan, Y. (2026) · Journal of Medical Internet Research · doi:10.2196/83188 · Cited by 2
Multi-Objective Optimization of Large Language Model Summaries for Clinical Text Narratives
Verma, D., Darlan, D., & Mallipeddi, R. (2026) · IEEE Access · doi:10.1109/access.2026.3656535 · Cited by 1
“Due to its training in diverse instruction-based datasets, the model excels in interpreting detailed instructions and delivering outputs that are contextually accurate and syntactically fluent [54], [55].”
Muasher-Kerwin, C., Hughes, M. C., Sanatizadeh, A., Mayer, J. F., & Alhoori, H. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-00879-z · Cited by 2
Metadata
- Authors
- Van Veen, D., Van Uden, C., Blankemeier, L., Delbrouck, J.-B., Aali, A., Bluethgen, C., Pareek, A., Polacin, M., Reis, E. P., Seehofnerová, A., Rohatgi, N., Hosamani, P., Collins, W., Ahuja, N., Langlotz, C. P., Hom, J., Gatidis, S., Pauly, J., & Chaudhari, A. S.
- Affiliations
- Stanford University (US)
- The University of Texas at Austin (US)
- Copenhagen University Hospital (DK)
- University of Zurich (CH)
- Hospital Israelita Albert Einstein (BR)
- Stanford Medicine (US)
- ORCID
- Venue
- Nature Medicine
- Published
- 2024-02-27
- Volume
- 30
- Issue
- 4
- Pages
- 1134-1142
- ISSN
- 1078-8956 (ISSN-L)
- 1546-170X
- PMC
- PMC11479659
- OpenAlex
- W4392193048
- Cited by
- 891
- References
- 80
- Citations by year
- 2026: 341 · 2025: 404 · 2024: 144
- FWCI
- 82.70
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Licence
- Locations
- Nature Medicine (publishedVersion)
- Research Square (acceptedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Zurich Open Repository and Archive (University of Zurich) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- PubMed Central (submittedVersion, cc-by)
- Zurich Open Repository and Archive (University of Zurich) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Zurich Open Repository and Archive (University of Zurich)
- Funders
- Foundation for the National Institutes of Health (R01 AR077604, R01 EB002524, R01 AR079431, P41 EB027060)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- large language models, clinical text summarization, LLM adaptation, electronic health records, radiology reports, progress notes, doctor-patient dialogue, patient questions, summary quality, safety analysis, clinical documentation burden
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
