Adapted large language models can outperform medical experts in clinical text summarization

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

LLMs such as ChatGPT have gained widespread adoption and excel at information retrieval, nuanced understanding, and text generation.

“Large language models (LLMs) have gained remarkable traction, leading to widespread adoption of models such as ChatGPT17, which excel at information retrieval, nuanced understanding and text generation18,19.”

• cites [17] Brown, T. et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33 https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf (NeurIPS, 2020).Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [18] Zhao, W. X. et al. A survey of large language models. Preprint at https://doi.org/10.48550/arXiv.2303.18223 (2023). sourceSupported
• cites [19] Bubeck, S. et al. Sparks of artificial general intelligence: early experiments with GPT-4. Preprint at https://doi.org/10.48550/arXiv.2303.12712 (2023). sourceNot checked yet

  Checked before — its earlier verdict is held back until a new check runs.

May combine two claims.

Documentation tasks divert attention from direct patient care, leading to worse patient outcomes and decreased clinician job satisfaction.

“These tasks divert attention from direct patient care, leading to worse outcomes for patients and decreased job satisfaction for clinicians2,14–16.”

• cites [2] Arndt, B. G. et al. Tethered to the EHR: primary care physician workload assessment using EHR event log data and time–motion observations. Ann. Fam. Med. 15, 419–426 (2017). sourcePartly supported

  The Introduction (p. 1) links documentation and other EHR tasks to less face-to-face patient time and clinician dissatisfaction. The paper does not establish that documentation tasks lead to worse patient outcomes; its discussion of poorer clinical outcomes concerns electronic communication among team members and patients with coronary artery disease (Communication, p. 6). (LLM, conf=0.88)

  Experimental — accuracy not yet measured
• cites [14] Shanafelt, T. D. et al. Relationship between clerical burden and characteristics of the electronic environment with physician burnout and professional satisfaction. Mayo Clin. Proc. 91, 836–848 (2016). sourceNot checked — no full text of the source
• cites [15] Robinson, K. E. & Kersey, J. A. Novel electronic health record (EHR) education intervention in large healthcare organization improves quality, efficiency, time, and impact on burnout. Medicine (Baltimore) 97, e12319 (2018). sourcePartly supported

  The Introduction (page not provided) says physicians spend substantial time on EHR and desk work relative to direct patient contact, and links EHR documentation to impaired patient communication and increased career dissatisfaction. It does not establish that documentation tasks lead to worse patient outcomes; the Discussion addresses patient care but does not make that causal claim. (LLM, conf=0.87)

  Experimental — accuracy not yet measured
• cites [16] Toussaint, W. et al. Design considerations for high impact, automated echocardiogram analysis. Preprint at https://doi.org/10.48550/arXiv.2006.06292 (2020). sourcePartly supported

  In Design Discoveries (p. 2), the paper says cardiologists spend 4–5 hours a day annotating echocardiograms and suggests automation could free them to focus more attention on sick patients. It does not characterize this work as documentation or say that it causes worse patient outcomes or decreased clinician job satisfaction. (LLM, conf=0.78)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Documentation responsibilities can consume up to 60% of nurses' time and account for considerable work stress.

“Similarly, documentation responsibilities for nurses can consume up to 60% of their time and account for considerable work stress11–13.”

• cites [11] Khamisa, N., Peltzer, K. & Oldenburg, B. Burnout in relation to specific contributing factors and health outcomes among nurses: a systematic review. Int. J. Environ. Res. Public Health 10, 2214–2240 (2013). sourcePartly supported

  The Results (§3.2–3.3; page unavailable) identify workload as a prominent stressor among nurses, supporting the general point about work stress. They do not identify documentation as the source of that stress or report that it consumes up to 60% of nurses’ time; searches for documentation, paperwork, and the time figure found no such passage. (LLM, conf=0.80)

  Experimental — accuracy not yet measured
• cites [12] Duffy, W. J., Kharasch, M. S. & Du, H. Point of care documentation impact on the nurse–patient interaction. Nurs. Adm. Q. 34, E1–E10 (2010). sourceNot checked — no full text of the source
• cites [13] Chang, C.-P., Lee, T.-T., Liu, C.-H. & Mills, M. E. Nurses’ experiences of an initial and reimplemented electronic health record use. Comput. Inform. Nurs. 34, 183–190 (2016). sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

Widespread adoption of electronic health records has increased clinical documentation workload, contributing to stress and clinician burnout.

“The widespread adoption of electronic health records has expanded clinical documentation workload, directly contributing to increasing stress and clinician burnout7–9.”

• cites [7] Gesner, E., Gazarian, P. & Dykes, P. The burden and burnout in documenting patient care: an integrative literature review. Stud. Health Technol. Inform. 21, 1194–1198 (2019).Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [8] Ratwani, R. M. et al. A usability and safety analysis of electronic health records: a multi-center study. J. Am. Med. Inform. Assoc. 25, 1197–1201 (2018). sourcePartly supported

  The Background and significance section says EHR usability problems create burdensome workflows and inefficiencies that contribute to clinician frustration. It does not say that widespread EHR adoption increased clinical documentation workload or directly caused increasing stress and burnout. The Discussion examines task time, clicks, and errors, not those broader outcomes; page numbers are unavailable in our copy. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites [9] Ehrenfeld, J. M. & Wanderer, J. P. Technology as friend or foe? Do electronic health records increase burnout? Curr. Opin. Anaesthesiol. 31, 357–360 (2018). sourceNot checked — no full text of the source

Classified as probably not a substantial claim.

Many previous clinical NLP studies rely primarily on quantitative metrics for evaluation.

“Many previous clinical NLP studies rely primarily on quantitative metrics for evaluation41,52–54.”

• cites [41] Yim, W.W. et al. Aci-bench: a novel ambient clinical intelligence dataset for benchmarking automatic visit note generation. Sci. Data https://doi.org/10.1038/s41597-023-02487-3 (2023). sourcePartly supported

  In Technical Validation > Baseline experiments (page unavailable), the paper calls automatic evaluation metrics “widely-used” and evaluates its own clinical-note generation models with quantitative scores. It does not establish that many previous clinical NLP studies relied primarily on quantitative rather than other forms of evaluation. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites [52] Ma, C. et al. ImpressionGPT: an iterative optimizing framework for radiology report summarization with chatGPT. Preprint at https://doi.org/10.48550/arXiv.2304.08448 (2023). sourceSupported
• cites [53] Wei, S. et al. Medical question summarization with entity-driven contrastive learning. Preprint at https://doi.org/10.48550/arXiv.2304.07437 (2023). sourceNot checked — no full text of the source
• cites [54] Manakul, P. et al. CUED at ProbSum 2023: Hierarchical ensemble of summarization models. In The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks 516–523 (Association for Computational Linguistics, 2023). sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Prior work has shown potential in clinical NLP tasks through domain adaptation by training new models, fine-tuning existing models, or providing task-specific examples in prompts.

“Previous work has demonstrated potential across clinical NLP tasks22,23, adapting to the medical domain by training a new model24,25, fine-tuning an existing model26,27 or supplying task-specific examples in the model prompt27,28.”

• cites [22] Wornow, M. et al. The shaky foundations of large language models and foundation models for electronic health records. NPJ Digit. Med. 6, 135 (2023). sourceSupported
  Experimental — accuracy not yet measured
• cites [23] Thirunavukarasu, A. J. et al. Large language models in medicine. Nat. Med. 29, 1930–1940 (2023). sourceNot checked — no full text of the source
• cites [24] Singhal, K. et al. Large language models encode clinical knowledge. Nature https://doi.org/10.1038/s41586-023-06291-2 (2023). sourcePartly supported

  The Methods > Modelling passages describe adapting a model to medical question-answering tasks with task-specific examples in prompts and instruction prompt tuning. They also say the authors used prompt tuning instead of full-model fine-tuning, keeping the model’s parameters frozen. Thus, this source supports the prompting part of the attributed claim, but not the suggestion that this study fine-tuned the existing model itself. (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites [25] Tu, T. et al. Towards generalist biomedical AI. Preprint at https://doi.org/10.48550/arXiv.2307.14334 (2023). sourceSupported
  Experimental — accuracy not yet measured
• cites [26] Toma, A. et al. Clinical Camel: an open-source expert-level medical language model with dialogue-based knowledge encoding. Preprint at https://doi.org/10.48550/arXiv.2305.12031 (2023). sourceSupported
  Experimental — accuracy not yet measured
• cites [27] Van Veen, D. et al. RadAdapt: radiology report summarization via lightweight domain adaptation of large language models. In 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks 449–460 (Association for Computational Linguistics, 2023). sourceSupported
  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Prompt phrasing and model temperature can considerably affect LLM output.

“Prompt phrasing and model temperature can have a considerable effect on LLM output, as demonstrated in the literature45,46 and in Fig. 2.”

• cites [45] Strobelt, H. et al. Interactive and visual prompt engineering for ad-hoc task adaptation with large language models. IEEE Trans. Vis. Comput. Graph. 29, 1146–1156 (2022).Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [46] Wang, J. et al. Prompt engineering for healthcare: methodologies and applications. Preprint at https://doi.org/10.48550/arXiv.2304.14670 (2023). sourcePartly supported

  The Introduction (page unavailable) says prompt design can significantly affect model performance, and Automated Prompts describes improved performance from prompt paraphrasing. The copy does not address model temperature (searched: “temperature”), so it supports the prompt-phrasing part but not the temperature part. (LLM, conf=0.87)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim.

Alpaca is known to perform worse than Vicuna and Llama-2 on relevant tasks.

“considering that Alpaca is known to perform worse than our other open-source autoregressive models, Vicuna and Llama-2 (refs. 21,35)”

• cites [21] Zheng, L. et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Preprint at https://doi.org/10.48550/arXiv.2306.05685 (2023). sourcePartly supported

  Table 8 (Position bias results; page not provided) reports lower MT-Bench scores for Alpaca-7B and Alpaca-13B (4.54 and 4.53) than for the listed Vicuna variants (5.95–6.39). The passages do not compare Alpaca with Llama-2; the LLaMA models listed are not identified as Llama-2. (LLM, conf=0.87)

  Experimental — accuracy not yet measured
• cites [35] The Vicuna Team. Vicuna: an open-source chatbot impressing GPT-4 with 90%* ChatGPT quality. https://lmsys.org/blog/2023-03-30-vicuna/ (2023).Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

For correctness, BERTScore and MEDCON correlated most strongly with reader preference; BLEU correlated most with completeness and least with conciseness.

“For correctness, the metrics BERTScore44 (measuring semantics) and MEDCON41 (measuring medical concepts) correlated most strongly with reader preference; meanwhile, the BLEU42 metric (measuring syntax) correlated most with completeness and least with conciseness.”

• cites [44] Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q. & Artzi, Y. BERTScore: evaluating text generation with BERT. International Conference on Learning Representations. https://openreview.net/forum?id=SkeHuCVFDr (2020).Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [41] Yim, W.W. et al. Aci-bench: a novel ambient clinical intelligence dataset for benchmarking automatic visit note generation. Sci. Data https://doi.org/10.1038/s41597-023-02487-3 (2023). sourceCould not verify

  Technical Validation > Baseline experiments (page unavailable) describes BERTScore and medcon as evaluation metrics, but does not report correlations with reader preference, correctness, completeness, or conciseness. Searches for those terms and BLEU found no passage addressing the claimed comparisons; the paper discusses BLEURT instead. (LLM, conf=0.78)

  Experimental — accuracy not yet measured
• cites [42] Papineni, K., Roukos, S., Ward, T. & Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proc. of the 40th Annual Meeting of the Association for Computational Linguistics. https://dl.acm.org/doi/pdf/10.3115/1073083.1073135 (Association for Computing Machinery, 2002). sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

There have been recent releases of expert evaluations for adjacent clinical NLP tasks.

“To address this, there have been recent releases of expert evaluations for adjacent clinical NLP tasks3,55.”

• cites [3] Fleming, S. L. et al. MedAlign: a clinician-generated dataset for instruction following with electronic medical records. Preprint at https://doi.org/10.48550/arXiv.2308.14089 (2023). sourceSupported
• cites [55] Yu, F. et al. Evaluating progress in automatic chest x-ray radiology report generation. Patterns (N Y) 4, 100802 (2023). sourceSupported

Classified as probably not a substantial claim.

In-context learning provides the model with context, enhancing LLM performance for a particular task or domain.

“This simple approach provides the model with context, enhancing LLM performance for a particular task or domain27,28.”

• cites [27] Van Veen, D. et al. RadAdapt: radiology report summarization via lightweight domain adaptation of large language models. In 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks 449–460 (Association for Computational Linguistics, 2023). sourceSupported
• cites [28] Mathur, Y. et al. SummQA at MEDIQA-Chat 2023: in-context learning with GPT-4 for medical summarization. Preprint at https://doi.org/10.48550/arXiv.2306.17384 (2023). sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

QLoRA's 4-bit quantization negligibly impacts performance.

“This quantization negligibly impacts performance40; as such, we used QLoRA for all model training.”

• cites [40] Dettmers, T., Pagnoni, A., Holtzman, A. & Zettlemoyer, L. QLoRA: efficient finetuning of quantized LLMs. Preprint at https://doi.org/10.48550/arXiv.2305.14314 (2023). sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Large language models have the potential to be biased.

“We further note the potential for LLMs to be biased58,59.”

• cites [58] Omiye, J. A., Lester, J. C., Spichak, S., Rotemberg, V. & Daneshjou, R. Large language models propagate race-based medicine. NPJ Digit. Med. 6, 195 (2023). sourceSupported
• cites [59] Zack, T. et al. Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study. Lancet Digit. Health 6, e12–e22 (2024). sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

Choosing relevant in-context examples outperforms choosing examples at random.

“Note that choosing ‘relevant’ in-context examples has been shown to outperform choosing examples at random69.”

• cites [69] Nie, F., Chen, M., Zhang, Z. & Cheng, X. Improving few-shot performance of language models via nearest neighbor calibration. Preprint at https://doi.org/10.48550/arXiv.2212.02216 (2022). sourcePartly supported

  The Experimental Setup (p. 5) uses randomly sampled in-context demonstrations as a baseline, and the Analysis (p. 7) reports improvements from retrieving similar support samples. But those samples calibrate predictions through a nearest-neighbor classifier; the paper does not show that selecting relevant examples *as in-context demonstrations* outperforms selecting random demonstrations. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Other works have previously employed these datasets and developed methods specific to each summarization task.

“As we are not the first to employ these datasets, Extended Data Table 4 contains quantitative metric scores from other works25,27,41,52–54 that developed methods specific to each individual summarization task.”

• cites [25] Tu, T. et al. Towards generalist biomedical AI. Preprint at https://doi.org/10.48550/arXiv.2307.14334 (2023). sourcePartly supported

  The Additional information section says the paper used a previously published MIMIC-III radiology-report summarization dataset. But Section 5.1 describes Med-PaLM M as jointly fine-tuned across tasks and distinguishes it from prior specialist models; it does not support describing this paper’s method as specific to an individual summarization task. Our copy provides no page numbers for these passages. (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites [27] Van Veen, D. et al. RadAdapt: radiology report summarization via lightweight domain adaptation of large language models. In 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks 449–460 (Association for Computational Linguistics, 2023). sourceSupported
• cites [41] Yim, W.W. et al. Aci-bench: a novel ambient clinical intelligence dataset for benchmarking automatic visit note generation. Sci. Data https://doi.org/10.1038/s41597-023-02487-3 (2023). sourceSupported
• cites [52] Ma, C. et al. ImpressionGPT: an iterative optimizing framework for radiology report summarization with chatGPT. Preprint at https://doi.org/10.48550/arXiv.2304.08448 (2023). sourceSupported
• cites [53] Wei, S. et al. Medical question summarization with entity-driven contrastive learning. Preprint at https://doi.org/10.48550/arXiv.2304.07437 (2023). sourceNot checked — no full text of the source
• cites [54] Manakul, P. et al. CUED at ProbSum 2023: Hierarchical ensemble of summarization models. In The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks 516–523 (Association for Computational Linguistics, 2023). sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

The supplemental reader study was inspired by the AHRQ harm scale.

“inspired by the AHRQ harm scale43.”

• cites [43] Walsh, K. E. et al. Measuring harm in healthcare: optimizing adverse event review. Med. Care 55, 436–441 (2017). sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

LoRA is much more efficient than training model weights from scratch in terms of computational requirements and training data volume.

“Compared to training model weights from scratch, LoRA is much more efficient with respect to both computational requirements and the volume of training data required.”

• cites [70] Hu, E. et al. LoRA: low-rank adaptation of large language models. Preprint at https://doi.org/10.48550/arXiv.2106.09685 (2021). sourcePartly supported

  The Introduction (p. 2) says LoRA makes training more efficient, and §4.2 (p. 5) reports lower GPU memory use and faster training than full fine-tuning. Those comparisons are with adapting a pretrained model, not training one from scratch. The low-data experiments (§F.3, p. 22) do not establish that LoRA requires less training data than training from scratch. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

May combine two claims.

Safety analysis showed that both LLMs and medical experts face challenges, connecting errors to potential medical harm and categorizing fabricated information.

“The ensuing safety analysis highlights challenges faced by both LLMs and medical experts, as we connect errors to potential medical harm and categorize types of fabricated information.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In one example, the model mirrored the input phrase 'diabetes and neuropathy' verbatim, while the medical expert interpreted it as 'diabetic neuropathy'.

“Consider example 1 in Extended Data Fig. 4, in which the input question mentioned ‘diabetes and neuropathy’. The model mirrored this phrasing verbatim, whereas the medical expert interpreted it as ‘diabetic neuropathy’.”

May combine two claims.

Seq2seq models (FLAN-T5 and FLAN-UL2) performed very well on syntactic metrics such as BLEU but worse on others.

“Seq2seq models (FLAN-T5 and FLAN-UL2) performed very well on syntactical metrics, such as BLEU, but worse on others (Fig. 3d)”

May combine two claims.

Medical expert summaries were preferred in 19% of cases, while the best model was non-inferior in 45% and preferred in 36%.

“medical expert summaries were preferred in only a minority of cases (19%), whereas, in a majority, the best model was either non-inferior (45%) or preferred (36%).”

May combine two claims.

ICL was the best overall adaptation method, especially when including proprietary models GPT-3.5 and GPT-4, although QLoRA performed similarly in some cases.

“In terms of trade-offs between lightweight adaptation methods, while QLoRA fine-tuning performed similarly for some cases, ICL was the best overall, especially when including proprietary models GPT-3.5 and GPT-4.”

May combine two claims.

Pooled results across 10 physicians showed that summaries from the best-adapted model (GPT-4 using ICL) were more complete and contained fewer errors than medical expert summaries.

“pooled results across 10 physicians (Fig. 4b) demonstrate that summaries from the best-adapted model (GPT-4 using ICL) were more complete and contained fewer errors compared to medical expert summaries”

Probably not a substantial claim: may not pass S3 (load-bearing).

In another example, the model reformulated the input question about tests and their locations, while the medical expert inferred a broader query about tests and treatments.

“In example 2 of Extended Data Fig. 4, the model simply reformulated the input question about tests and their locations, whereas the medical expert inferred a broader query about tests and treatments.”

May combine two claims.

The best-adapted model achieved non-inferior results to medical experts while using a basic search across 1-2 options for each task instruction.

“Overall, our best-adapted model achieved non-inferior results to medical experts while performing a basic search across 1–2 options for each task instruction (Extended Data Table 1).”

May combine two claims.

The best model performed significantly better than medical experts on conciseness overall and on two tasks (P < 0.001), but similarly on radiology reports.

“Regarding conciseness, the best model performed significantly better than medical experts (P < 0.001) overall and on two tasks, whereas, for radiology reports, it performed similarly to medical experts.”

May combine two claims.

The model committed misinterpretations, inaccuracies, and hallucinations on 6%, 2%, and 5% of samples, respectively, compared with 9%, 4%, and 12% by medical experts.

“We found that the model committed misinterpretations, inaccuracies and hallucinations on 6%, 2% and 5% of samples, respectively, compared to 9%, 4% and 12%, respectively, by medical experts.”

May combine two claims.

Given sufficient in-context examples, proprietary models GPT-3.5 and GPT-4 far outperformed other models and methods, although ICL and QLoRA were competitive for open-source models.

“Although ICL and QLoRA were competitive for open-source models, proprietary models GPT-3.5 and GPT-4 far outperformed other models and methods given sufficient in-context examples.”

May combine two claims.

The best model generated significantly fewer errors than medical expert summaries overall and on two of three summarization tasks (P < 0.001).

“Regarding correctness, the best model generated significantly fewer errors (P < 0.001) compared to medical expert summaries (Fig. 4b) overall and on two of three summarization tasks.”

May combine two claims.

The best model's summaries were more complete on average than medical expert summaries, with statistical significance across all three summarization tasks (P < 0.001).

“We observed that the best model summaries were more complete, on average, than medical expert summaries, achieving statistical significance across all three summarization tasks with P < 0.001 (Fig. 4b).”

Probably not a substantial claim: may not pass S3 (load-bearing).

For GPT-3.5, a temperature of 0.1 performed best among 0.1, 0.5, and 0.9.

“After searching over temperature values {0.1, 0.5, 0.9} using GPT-3.5, Fig. 2 demonstrates that the lowest value, 0.1, performed best.”

May combine two claims.

Medical expert summaries had a higher likelihood (14% vs 12%) and higher extent (22% vs 16%) of possible harm compared with the best model's summaries.

“The results of this harm study (Fig. 4d) indicate that the medical expert summaries would have both a higher likelihood (14%) and a higher extent (22%) of possible harm compared to the summaries from the best model (12% and 16%, respectively).”

May combine two claims.

Among open-source models, seq2seq models performed better than autoregressive models (Llama-2 and Vicuna) on radiology reports but worse on patient questions and progress notes.

“Among open-source models, seq2seq models performed better than autoregressive models (Llama-2 and Vicuna) on radiology reports but worse on patient questions and progress notes (Fig. 3c).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Model performance generally improved with more context.

“Model performance generally improved with more context.”

May combine two claims.

In the progress note example, the model missed historical context (a history of hypertension) despite being more complete than the expert.

“Although the model was more complete than the expert in generating its progress note summary, it also missed historical context (a history of hypertension).”

Probably not a substantial claim: may not pass S3 (load-bearing).

In one example, the model's summary included correct information that readers deemed not important.

“Figure 5 provides an example in which the model’s summary includes correct information that readers deemed not important.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The dialogue task had the largest token count across the six datasets for both input and target.

“Per Extended Data Table 1, this task entailed the largest token count across our six datasets for both the input (dialogue) and the target (assessment).”

Probably not a substantial claim: may not pass S3 (load-bearing).

MeQSum exhibited the highest lexical variance among the datasets used.

“Due to the wide variety of these questions, MeQSum exhibited the highest lexical variance of our datasets (Extended Data Table 1).”

Probably not a substantial claim: may not pass S3 (load-bearing).

The best model and method uses a handful of examples via in-context learning.

“because the best model and method uses a handful of examples via ICL”

Probably not a substantial claim: may not pass S3 (load-bearing).

For Llama-2, the improvement from 13B to 7B parameters is marginal.

“The benefit of larger models is explored in Extended Data Fig. 5, which found this improvement marginal for Llama-2 (13B) compared to Llama-2 (7B).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Medical expert radiology report summaries sometimes recommend further studies or refer to prior studies.

“radiology report summaries from medical experts occasionally recommend further studies or refer to prior studies—for example, ‘… not significantly changed from prior’ in Fig. 5.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Physicians may spend up to two hours on documentation for each hour of patient interaction.

“Recent data indicate that physicians can expend up to 2 hours on documentation for each hour of patient interaction10.”

• cites [10] Sinsky, C. et al. Allocation of physician time in ambulatory practice: a time and motion study in 4 specialties. Ann. Intern. Med. 165, 753–760 (2016). sourceNot checked — no full text of the source

May combine two claims.

Even experienced physicians can make errors in summarizing clinical text, which can be harmful in healthcare where precision is critical.

“Even for experienced physicians with a high level of expertise, this intricate task naturally introduces the possibility for errors, which can be detrimental in healthcare where precision is paramount4–6.”

• cites [4] Yackel, T. R. & Embi, P. J. Unintended errors with EHR-based result management: a case series. J. Am. Med. Inform. Assoc. 17, 104–107 (2010). sourceNot checked — no full text of the source
• cites [5] Bowman, S. Impact of electronic health record systems on information integrity: quality and safety implications. Perspect. Health Inf. Manag. 10, 1c (2013).Not checked — no full text of the source
• cites [6] Gershanik, E. F., Lacson, R. & Khorasani, R. Critical finding capture in the impression section of radiology reports. AMIA Annu. Symp. Proc. 2011, 465–469 (2011).Not checked — no full text of the source

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Integrating LLMs into clinical workflows could alleviate documentation burden and allow clinicians to focus more on patient care.

“This suggests that integrating LLMs into clinical workflows could alleviate documentation burden, allowing clinicians to focus more on patient care.”

May combine two claims.

The model's summaries were more concise while also being more complete.

“We note that the model’s summaries are more concise while concurrently being more complete.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The lack of preference for the model on radiology report conciseness could be due to the vague length specification in that instruction, unlike the quantified length in other tasks.

“This could be attributed to the relatively vague length specification in the radiology reports instruction—that is, ‘…with minimal text’—whereas the other two task instructions quantify length.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Relevant context improves outcomes for the target task.

“This illustrates the value of relevant context in achieving better outcomes for the target task.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

ICL has benefits: model weights are fixed, enabling queries of pre-existing LLMs, and adaptation is feasible with few examples, whereas QLoRA typically requires hundreds or thousands of examples.

“Governance aside, ICL provides many benefits: (1) model weights are fixed, hence enabling queries of pre-existing LLMs, and (2) adaptation is feasible with even a few examples, whereas fine-tuning methods, such as QLoRA, typically require hundreds or thousands of examples.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Seq2seq models are often constrained to shorter context lengths than autoregressive models because they encode the input sequence into a fixed-size context vector.

“Note that seq2seq models are often constrained to much shorter context length than autoregressive models, such as GPT-4, because seq2seq models require the memory-intensive step of encoding the input sequence into a fixed-size context vector.”

May combine two claims.

Evidence from the reader study suggests that adapting LLMs can outperform medical experts in completeness, correctness, and conciseness.

“The evidence from our reader study suggests that adapting LLMs can outperform medical experts in terms of completeness, correctness and conciseness.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In both cases, the model's summaries leaned toward literalness, which readers sometimes favored and sometimes did not.

“In both cases, the model’s summaries leaned toward literalness, a trait that readers sometimes favored and sometimes did not.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Even medical experts, not just LLMs, can hallucinate.

“This instance underscores the point that even medical experts, not just LLMs, can hallucinate.”

May combine two claims.

The clinical reader study demonstrates that LLM summaries are often preferred over medical expert summaries due to higher completeness, correctness, and conciseness scores.

“The ensuing clinical reader study demonstrates that LLM summaries are often preferred over medical expert summaries due to higher scores for completeness, correctness and conciseness.”

May combine two claims.

BLEU's correlation with completeness is reasonable because BLEU measures sequence overlap, and more text provides more overlap and reduces the brevity penalty.

“Given that BLEU measures sequence overlap, this result seems reasonable, as more text provides more ‘surface area’ for overlap; more text also reduces the brevity penalty that BLEU applies on generated sequences, which are shorter than the reference42.”

• cites [42] Papineni, K., Roukos, S., Ward, T. & Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proc. of the 40th Annual Meeting of the Association for Computational Linguistics. https://dl.acm.org/doi/pdf/10.3115/1073083.1073135 (Association for Computing Machinery, 2002). sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The model tends toward a literal approach without interpretation, which may be advantageous or limiting.

“This highlights the model’s tendency toward a literal approach without interpretation, which may be either advantageous or limiting.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Addressing the lack of clinical task benchmarks could accelerate clinical text summarization, alleviating documentation burden and improving patient care.

“Addressing this limitation presents an opportunity to accelerate the process of clinical text summarization, hence alleviating documentation burden and improving patient care.”

Classified as probably not a substantial claim.

Prompt engineering is important for improving model performance.

“We first highlight the importance of ‘prompt engineering’, or modifying and tuning the input prompt to improve model performance.”

Classified as probably not a substantial claim. May combine two claims.

The low magnitude of correlation values (approximately 0.2) indicates that NLP metrics alone are insufficient for assessing clinical readiness; a clinical reader study is needed.

“However, the low magnitude of correlation values (approximately 0.2) underscores the need to go beyond NLP metrics with a clinical reader study when assessing clinical readiness.”

Classified as probably not a substantial claim.

For the clinical reader study, the A/B comparison of summaries from the model and human experts was presented in a blinded manner.

“Regarding blinding, for the clinical reader study, we presented, in a blinded manner, the A/B comparison of summaries from the model and human experts.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

It is not possible to verify whether a particular open-source dataset was included in model training, especially for proprietary models.

“it is not possible to verify whether a particular open-source dataset was included in model training.”

May combine two claims.

When human evaluation is not feasible, syntactic metrics are better at measuring completeness, while semantic and conceptual metrics are better at measuring correctness.

“When human evaluation is not feasible, Fig. 6 suggests that syntactic metrics are better at measuring completeness, whereas semantic and conceptual metrics are better at measuring correctness.”

May combine two claims.

MIMIC-III is a more challenging summarization dataset than Open-i or MIMIC-CXR because MRI and CT scans capture more information at higher resolution, leading to longer reports.

“Compared to x-rays, MRIs and CT scans capture more information at a higher resolution. This usually leads to longer reports (Extended Data Table 1), rendering MIMIC-III a more challenging summarization dataset than Open-i or MIMIC-CXR.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The study used only retrospective, de-identified data that fell outside the scope of institutional review board oversight.

“This study used only retrospective, de-identified data that fell outside the scope of institutional review board oversight.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The dialogue task was excluded from the clinical reader study because parsing lengthy transcribed conversations and paragraphs was unwieldy.

“The dialogue task was excluded owing to the unwieldiness of a reader parsing many lengthy transcribed conversations and paragraphs”

Probably not a substantial claim: may not pass S3 (load-bearing).

Simple post-processing was applied to standardize capitalization, punctuation, and newline characters to obfuscate formatting differences between A and B.

“To obfuscate any formatting differences between A and B, we applied simple post-processing to standardize capitalization, punctuation, newline characters, etc.”

Classified as probably not a substantial claim. May combine two claims.

The quantitative experiments used a sample size of 250, chosen to maximize samples given dataset size and compute constraints, and deemed sufficient because it enabled reproducible results.

“For the quantitative experiments, our sample size of 250 was chosen to maximize the number of samples given constraints of dataset size and cost of computing resources. We deem this sufficient as it enabled reproducible results when running the same experiments with a different set of samples.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The datasets used in this study do not contain demographic information.

“Although our datasets do not contain demographic information”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Using six datasets enhances the robustness of the experiments.

“Furthermore, our use of six datasets enhances the robustness of our experiments.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The study adhered to the principles of the Declaration of Helsinki.

“This study adhered to the principles outlined in the Declaration of Helsinki.”

Classified as probably not a substantial claim. May combine two claims.

For the quantitative experiments, 250 test samples were randomly selected, and the remaining samples were randomly divided into training and validation sets.

“for the quantitative experiments, we randomly selected 250 test samples and then randomly divided remaining samples into training and validation sets.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Informed consent was obtained from each physician before participation.

“Informed consent was obtained from each physician before their participation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Because the LLM temperature parameter is near zero, outputs have very little randomness and are reproducible.

“Because the LLM temperature parameter is near zero, outputs have very little randomness and are, thus, reproducible.”

Probably not a substantial claim: may not pass S3 (load-bearing).

No data were excluded from the analyses.

“No data were excluded from the analyses.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The LLM is out of scope for recommendations or references to prior studies because it lacks access to prior studies and the purview to make recommendations.

“These instances are out of scope for the LLM, as it does not have access to prior studies nor the purview to make recommendations.”

Probably not a substantial claim: may not pass S2 (checkable).

Because nuanced medical tasks like summarization typically lack objective solutions, human expert evaluation beyond quantitative metrics is crucial to demonstrate clinical readiness.

“Given the critical nature of nuanced medical tasks, such as summarization, that typically have no objective solutions, including human experts in the evaluation process beyond quantitative metrics is crucial to demonstrate clinical readiness.”

May combine two claims.

The clinical reader study used 100 comparisons per reader per task, chosen to maximize comparisons in a reasonable time, and deemed sufficient because it enabled statistically significant results.

“For the clinical reader study, our sample size of 100 comparisons per reader per task was chosen to maximize the number of comparisons that could be made in a reasonable amount of time, which we estimated as 10 h. We deem this sufficient as it enabled statistically significant results for many combinations of tasks and attributes.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Using LLM-generated candidate summaries could reduce documentation burden, potentially decreasing clinician strain and improving patient care.

“This suggests that incorporating LLM-generated candidate summaries could reduce documentation load, potentially leading to decreased clinician strain and improved patient care.”

Probably not a substantial claim: may not pass S3 (load-bearing).

PhysioNet access for three datasets is no guarantee against data leakage.

“Although three of our datasets (MIMIC-CXR, MIMIC-III and ProbSum) require PhysioNet57 access to ensure safe data usage by third parties, this is no guarantee against data leakage.”

• cites [57] Johnson, A., Pollard, T. & Mark, R. MIMIC-III Clinical Database Demo (version 1.4). PhysioNet https://doi.org/10.13026/C2HM2Q (2019). sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

The study did not examine the context-specific nature of summarization, such as differences in preferences across specialties or individual clinicians.

“we did not consider the inherently context-specific nature of summarization.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The temperature parameter of the LLMs was reduced to near zero, reducing randomness in generated summaries.

“Note that we reduced the temperature parameter of the LLMs to be near zero, hence reducing randomness in the generated output summaries.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

For the clinical reader study, 100 samples were randomly selected for comparison, and the samples and A/B comparisons were displayed in random order.

“For the clinical reader study, we randomly selected 100 samples for comparison; these samples and the A/B comparisons within each sample are displayed in random order.”

May combine two claims.

The smallest dataset was re-run and produced very similar quantitative metric scores, providing confidence for higher-resource datasets.

“we chose the smallest dataset––which consequently required the lowest cost of computational resources––and re-ran a set of experiments that rendered very similar quantitative metric scores. This provided confidence moving forward with datasets that required higher computational resources.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Three datasets (MIMIC-CXR, MIMIC-III, and ProbSum) require PhysioNet access to ensure safe data usage by third parties.

“Although three of our datasets (MIMIC-CXR, MIMIC-III and ProbSum) require PhysioNet57 access to ensure safe data usage by third parties”

• cites [57] Johnson, A., Pollard, T. & Mark, R. MIMIC-III Clinical Database Demo (version 1.4). PhysioNet https://doi.org/10.13026/C2HM2Q (2019). sourceNot checked — no full text of the source