Understanding Faithfulness and Reasoning of Large Language Models on Plain Biomedical Summaries
Open accessAbstract
Generating plain biomedical summaries with Large Language Models (LLMs) can enhance the accessibility of biomedical knowledge to the public.However, how faithful the generated summaries are remains an open yet critical question.To address this, we propose FAREBIO, a benchmark dataset with expertannotated Faithfulness and Reasoning on plain Biomedical Summaries.This dataset consists of 175 plain summaries (1, 445 sentences) generated by seven different LLMs, paired with source articles.Using our dataset, we identify the performance gap of LLMs in generating faithful plain biomedical summaries and observe a negative correlation between abstractiveness and faithfulness.We also show that current faithfulness evaluation metrics do not work well in the biomedical domain and confirm the over-confident tendency of LLMs as faithfulness evaluators.To better understand the faithfulness judgements, we further benchmark LLMs in retrieving supporting evidence and show the gap of LLMs in reasoning faithfulness evaluation at different abstractiveness levels.Going beyond the binary faithfulness labels, coupled with the annotation of supporting sentences, our dataset could further contribute to the understanding of faithfulness evaluation and reasoning.1 8 Annotation details are provided in Appendix G. 9
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Even domain-adapted models frequently introduce hallucinations expressed with medically plausible terminology, requiring expert review.
“Even domain-adapted models frequently introduce hallucinations expressed with medically plausible terminology, requiring expert review (Hegselmann et al., 2024; Williams et al., 2025; Fang et al., 2024; Das et al., 2025), and underscoring the need for stricter factual reliability in clinical summarization than in general-domain tasks.”
From Hallucination Detection-Guided Preference Optimization for Clinical Summarization (2026) · cites this paper as Fang et al., 2024
Section 3.3 (page not provided) reports non-faithful hallucinations in plain biomedical summaries, supporting the general concern about factual reliability. It does not establish that domain-adapted models frequently hallucinate: it says the fine-tuned Flan-T5 had a low factual-hallucination rate. The passages also do not describe hallucinations as using medically plausible terminology or explicitly require expert review. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Part of this paper’s reference list could not be split into single entries.
Research works printed but not linked: 25
- Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. GPT-4 Technical Re- port. arXiv preprint arXiv:2303.08774. · arXiv
- Griffin Adams, Jason Zuckerg, and Noémie Elhadad. 2023b. A Meta-Evaluation of Faithfulness Metrics for Long-Form Hospital-Course Summarization. In Machine Learning for Healthcare Conference, pages 2-30.
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901.
- Meri Coleman and Ta Lin Liau. 1975. A computer readability formula designed for machine scoring. Journal of Applied Psychology, 60(2):283.
- Edgar Dale and Jeanne S Chall. 1948. A formula for predicting readability: Instructions. Educational re- search bulletin, pages 37-54.
- Tirthankar Ghosal, Philipp Mayr, Anita de Waard, Aakanksha Naik, Shannon Shen, Amanpreet Singh, Orion Weller, Yanxia Qin, and Yoonjoo Lee, editors. 2024. The 4th Workshop on Scholarly Document Pro- cessing. Association for Computational Linguistics, Bangkok, Thailand.
- Yue Guo, Tal August, Gondy Leroy, Trevor Cohen, and Lucy Lu Wang. 2023. APPLS: Evaluating Evaluation Metrics for Plain Language Summarization. arXiv preprint arXiv:2305.14341. · arXiv
- Yue Guo, Wei Qiu, Yizhong Wang, and Trevor Cohen. 2021. Automated Lay Language Summarization of Biomedical Scientific Reviews. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 35, pages 160-168.
- Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2023. A Survey on Hallucination in Large Language Mod- els: Principles, Taxonomy, Challenges, and Open Questions. arXiv preprint arXiv:2311.05232. · arXiv
- Sebastian Antony Joseph, Lily Chen, Jan Trienes, Han- nah Louisa Göke, Monika Coers, Wei Xu, Byron C Wallace, and Junyi Jessy Li. 2024. FactPICO: Factu- ality Evaluation for Plain Language Summarization of Medical Evidence. In The 62nd Annual Meeting of the Association for Computational Linguistics.
- Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large Lan- guage Models are Zero-Shot Reasoners. Advances in neural information processing systems, 35:22199- 22213.
- Bor-Chen Kuo, Pei-Chen Wu, and Chen-Huei Liao. 2024. GPT-3.5, GPT-4, Bard, and Claude's Per- formance on the Chinese Reading Comprehension Test. In Joint Proceedings of LAK 2024 Workshops, co-located with 14th International Conference on Learning Analytics and Knowledge, Kyoto, Japan.
- Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234-1240.
- Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
- Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. 2023. The flan collection: designing data and methods for effective instruction tuning. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org.
- Mary L McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica, 22(3):276-282.
- Arjun Panickssery, Samuel R Bowman, and Shi Feng. 2024. LLM Evaluators Recognize and Favor Their Own Generations. arXiv preprint arXiv:2404.13076. · arXiv
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the lim- its of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67.
- Sanjana Ramprasad, Kundan Krishna, Zachary Lipton, and Byron Wallace. 2024. Evaluating the factual- ity of zero-shot summarizers across varied domains. In Proceedings of the 18th Conference of the Euro- pean Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pages 50-59, St. Julian's, Malta. Association for Computational Linguistics.
- Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Fi- rat, Julian Schrittwieser, et al. 2024. Gemini 1.5: Un- locking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530. · arXiv
- Khaled Saab, Tao Tu, Wei-Hung Weng, Ryutaro Tanno, David Stutz, Ellery Wulczyn, Fan Zhang, Tim Strother, Chunjong Park, Elahe Vedadi, et al. 2024. Capabilities of gemini models in medicine. arXiv preprint arXiv:2404.18416. · arXiv
- Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open Founda- tion and Fine-Tuned Chat Models. arXiv preprint arXiv:2307.09288. · arXiv
- Andrew Trotman, Antti Puurula, and Blake Burgess. 2014. Improvements to BM25 and language mod- els examined. In Proceedings of the 2014 Aus- tralasian Document Computing Symposium, ADCS 2014, Melbourne, VIC, Australia, November 27-28, 2014, page 58. ACM. · DOI
- Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. In Inter- national Conference on Learning Representations.
- N/A A Custom Made Electrode Construct and Reliable Implantation Method That Allows for Long-Term Bilateral Deep Brain Stimulation in Mice https://pubmed.ncbi.nlm.nih.gov/32385967/ Animals; Deep Brain Stimulation* ; Electrodes, Im- planted; Mice; Parkinson Disease* / therapy; Repro- ducibility of Results; Subthalamic Nucleus* Diagnostic accuracy of automated occlusion detec- tion in CT angiography using e-CTA https://pubmed.ncbi.nlm.nih.gov/33527886/ Cerebral Angiography; Computed Tomography An- giography*; Humans; Predictive Value of Tests; Sensitivity and Specificity; Stroke* / diagnostic imaging Efficacy of life skills training on general health in students https://pubmed.ncbi.nlm.nih.gov/23922605/ N/A Core-Shell Silver/Polymeric Nanoparticles-Based Combinatorial Therapy against Breast Cancer In- vitro https://pubmed.ncbi.nlm.nih.gov/27491622/ Antineoplastic Agents / pharmacology*; Breast Neoplasms / drug therapy*; Cell Line; Cell Pro- liferation / drug effects; Cell Survival / drug effects; Doxorubicin / pharmacology*; Drug Therapy, Com- bination; Female; Humans; In Vitro Techniques; MCF-7 Cells; Nanoshells / chemistry; Polyethy- lene Glycols / chemistry; Polymers / chemistry*; Polyvinyl Alcohol / chemistry; Povidone / chem- istry; Silver / chemistry*
Cited by
Sorted by citations
Explainable Agentic Artificial Intelligence in Healthcare: A Scoping Review
Collaco, B. G., Prabha, S., Gomez-Cabello, C. A., Haider, S. A., Genovese, A., Wood, N. G., Gopala, N., Raman, R., Hester, E. O., & Forte, A. J. (2026) · Bioengineering · doi:10.3390/bioengineering13050513 · Cited by 3 · see citing works
You, Z., & Guo, Y. (2026) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2026.105019 · Cited by 2 · see citing works
Hallucination Detection-Guided Preference Optimization for Clinical Summarization
Seethakantha, S. K., Thai, D. N., Gudi, V. P., Tiwari, S., Matar, R., Mitra, A., Zhao, W., McCallum, A., & Salloum, W. (2026) · arXiv · doi:10.48550/arxiv.2605.28910
“…introduce hallucinations expressed with medically plausible terminology, requiring expert review (Hegselmann et al., 2024; Williams et al., 2025; Fang et al., 2024; Das et al., 2025), and underscoring the need for stricter factual reliability in clinical summarization than in general-domain…”
The evolving landscape of large language models and non-large language models in health care
Yang, R., Li, H., Wong, M. Y. H., Ke, Y., Li, X., Yu, K., Liao, J., Liew, J. C. K., Nair, S. V., Ong, J. C. L., et al. (2026) · npj Health Systems · doi:10.1038/s44401-026-00076-1
Intelligent Narrative Summaries and Risk Scoring of Laboratory Panels with Large Language Models
Hanumanthgari, R. R. (2026) · EPJ Web of Conferences · doi:10.1051/epjconf/202638100002
- Authors
- Fang, B., Dai, X., & Karimi, S.
- Affiliations
- Commonwealth Scientific and Industrial Research Organisation (AU)
- Venue
- Findings of the Association for Computational Linguistics: EMNLP 2024
- Published
- 2024
- Pages
- 9890-9911
- OpenAlex
- W4404780964
- Cited by
- 4
- References
- 0
- Citations by year
- 2026: 4
- FWCI
- 1.15
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- dx.doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- faithfulness evaluation, large language models, plain language summaries, overconfidence, reasoning faithfulness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
