CoChat
Sign Up
PreprintCited by 6Artificial Intelligence in Healthcare and EducationTopic ModelingMachine Learning in Healthcare

A Data-Centric Approach To Generate Faithful and High Quality Patient Summaries with Large Language Models

Open access
Hegselmann, Stefan, Shen, Shannon Zejiang, Gierse, Florian, Agrawal, Monica, Sontag, David, Jiang, Xiaoyi·2024·arXiv
DOI 10.48550/arxiv.2402.15422arXiv 2402.15422
Open this paper in CoChat — free accountOpen PDF

Abstract

Patients often face difficulties in understanding their hospitalizations, while healthcare workers have limited resources to provide explanations. In this work, we investigate the potential of large language models to generate patient summaries based on doctors' notes and study the effect of training data on the faithfulness and quality of the generated summaries. To this end, we release (i) a rigorous labeling protocol for errors in medical texts and (ii) a publicly available dataset of annotated hallucinations in 100 doctor-written and 100 generated summaries. We show that fine-tuning on hallucination-free data effectively reduces hallucinations from 2.60 to 1.55 per summary for Llama 2, while preserving relevant information. We observe a similar effect on GPT-4 (0.70 to 0.40), when the few-shot examples are hallucination-free. We also conduct a qualitative evaluation using hallucination-free and improved training data. We find that common quantitative metrics do not correlate well with faithfulness and quality. Finally, we test GPT-4 for automatic hallucination detection, which clearly outperforms common baselines.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2402.15422

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Training-time hallucination mitigation methods require large volumes of high-quality domain data and depend on factuality metrics that correlate weakly with clinical correctness, limiting their practical impact.

    “These methods require large volumes of high-quality domain data (Hegselmann et al., 2024) and depend on factuality metrics that correlate weakly with clinical correctness, limiting their practical impact.”

    From Hallucination Detection-Guided Preference Optimization for Clinical Summarization (2026) · cites this paper as Hegselmann et al., 2024

    The citation is attached to the claim that these methods require large volumes of high-quality domain data. The paper’s Related Work section says the opposite for sample-efficient LLMs: revising a small number of training examples is sufficient and feasible even manually. The later claim about factuality metrics is not the part attributed to this citation. (LLM, conf=0.90)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Even domain-adapted models frequently introduce hallucinations expressed with medically plausible terminology, requiring expert review.

    “Even domain-adapted models frequently introduce hallucinations expressed with medically plausible terminology, requiring expert review (Hegselmann et al., 2024; Williams et al., 2025; Fang et al., 2024; Das et al., 2025), and underscoring the need for stricter factual reliability in clinical summarization than in general-domain tasks.”

    From Hallucination Detection-Guided Preference Optimization for Clinical Summarization (2026) · cites this paper as Hegselmann et al., 2024

    Section 5.1 reports 114 hallucinations in 100 generated patient summaries, and §3.2 says the generating models had been trained for the study’s hallucination-reduction experiments. The Discussion (§6) emphasizes thorough evaluation by human domain experts. These passages support the concern about hallucinations and expert review, but do not establish that the hallucinations were expressed with medically plausible terminology. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

  • Clinical Text Summarization: Adapting Large Language Models Can Outperform Human Experts

    Veen, D. V., Uden, C. V., Blankemeier, L., Delbrouck, J.-B., Aali, A., Bluethgen, C., Pareek, A., Polacin, M., Reis, E. P., Seehofnerova, A., et al. (2023) · doi:10.21203/rs.3.rs-3483777/v1 · Cited by 892 · cited 8× · see citing works

    “Given the large burden of and repetition within clinical documentation, automated clinical summarization is of great interest across healthcare; generation targets include discharge summaries, radiology impression sections, patient-facing instructions, and problem lists (Van Veen et al., 2023b; Adams et al., 2022).”
    BackgroundResultInfluential

Flagged first, then by citations

Research works printed but not linked: 59

  1. Griffin Adams, Emily Alsentzer, Mert Ketenci, Jason Zucker, and Noémie Elhadad. What's in a Sum- mary? Laying the Groundwork for Advances in Hospital-Course Summarization. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguis- tics: Human Language Technologies, pages 4794- 4811, Online, June 2021. Association for Com- putational Linguistics. doi: 10.18653/v1/2021. naacl-main.382. URL https://aclanthology. org/2021.naacl-main.382. · DOI
  2. Griffin Adams, Han-Chin Shing, Qing Sun, Christo- pher Winestock, Kathleen McKeown, and Noémie Elhadad. Learning to Revise References for Faith- ful Summarization. In Yoav Goldberg, Zor- nitsa Kozareva, and Yue Zhang, editors, Find- ings of the Association for Computational Linguis- tics: EMNLP 2022, pages 4009-4027, Abu Dhabi, United Arab Emirates, December 2022. Associa- tion for Computational Linguistics. doi: 10.18653/ v1/2022.findings-emnlp.296. URL https:// aclanthology.org/2022.findings-emnlp.296. · DOI
  3. Griffin Adams, Jason Zucker, and Noémie El- hadad. A Meta-Evaluation of Faithfulness Met- rics for Long-Form Hospital-Course Summariza- tion, March 2023. URL http://arxiv.org/abs/ 2303.03948. arXiv:2303.03948 [cs. · arXiv
  4. Rahul Aralikatte, Shashi Narayan, Joshua Maynez, Sascha Rothe, and Ryan McDonald. Focus At- tention: Promoting Faithfulness and Diversity in Summarization. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Con- ference on Natural Language Processing (Volume 1: Long Papers), pages 6078-6095, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.474. URL https: //aclanthology.org/2021.acl-long.474. · DOI
  5. Yaara Artsi, Vera Sorin, Eli Konen, Benjamin S. Glicksberg, Girish Nadkarni, and Eyal Klang. Large language models in simplifying radio- logical reports: systematic review. preprint, Radiology and Imaging, January 2024. URL http://medrxiv.org/lookup/doi/10.1101/ 2024.01.05.24300884. D. Baker, De Roberts, Rg Newcombe, and Ka Fox. Evaluation of drug information for cardiology patients. British Journal of Clinical Phar- macology, 31(5):525-531, 1991. ISSN 1365- 2125. doi: 10.1111/j.1365-2125.1991.tb05574.x. URL https://onlinelibrary.wiley.com/doi/ abs/10.1111/j.1365-2125.1991.tb05574.x. · DOI
  6. Christoph Becker, Samuel Zumbrunn, Katha- rina Beck, Alessia Vincent, Nina Loretz, Jonas Müller, Simon A. Amacher, Rainer Schaefert, and Sabina Hunziker. Interventions to Im- prove Communication at Hospital Discharge and Rates of Readmission: A Systematic Review and Meta-analysis. JAMA Network Open, 4 (8):e2119346, August 2021. ISSN 2574-3805. doi: 10.1001/jamanetworkopen.2021.19346. URL https://jamanetwork.com/journals/ jamanetworkopen/fullarticle/2783547. · DOI
  7. Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The Long-Document Transformer, December 2020. URL http://arxiv.org/abs/ 2004.05150. arXiv:2004.05150 [cs]. · arXiv
  8. Pengshan Cai, Fei Liu, Adarsha Bajracharya, Joe Sills, Alok Kapoor, Weisong Liu, Dan Berlowitz, David Levy, Richeek Pradhan, and Hong Yu. Generation of Patient After-Visit Summaries to Support Physicians. In Proceedings of the 29th International Conference on Computational Lin- guistics, pages 6234-6247, Gyeongju, Republic of Korea, October 2022. International Commit- tee on Computational Linguistics. URL https: //aclanthology.org/2022.coling-1.544.
  9. Shuyang Cao and Lu Wang. CLIFF: Contrastive Learning for Improving Faithfulness and Factu- ality in Abstractive Summarization. In Proceed- ings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 6633- 6649, Online and Punta Cana, Dominican Repub- lic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main. 532. URL https://aclanthology.org/2021. emnlp-main.532. · DOI
  10. Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. Faithful to the original: fact-aware neural abstrac- tive summarization. In Proceedings of the Thirty- Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Sympo- sium on Educational Advances in Artificial Intel- ligence, AAAI'18/IAAI'18/EAAI'18, pages 4784- 4791, New Orleans, Louisiana, USA, February 2018. AAAI Press. ISBN 978-1-57735-800-8.
  11. Sarah K Doyle, James C Rippey, Angela Jacques, Alethea M Rea, Bronwyn N Kaiser, Suzanne M Miller, and Matthew HR Anstey. Effect of personalised, mobile-accessible discharge instruc- tions for patients leaving the emergency depart- ment: A randomised controlled trial. Emer- gency Medicine Australasia, 32(6):967-973, 2020. ISSN 1742-6723. doi: 10.1111/1742-6723.13516. URL https://onlinelibrary.wiley.com/doi/ abs/10.1111/1742-6723.13516. · DOI
  12. Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. SummEval: Re-evaluating Sum- marization Evaluation, February 2021. URL http: //arxiv.org/abs/2007.12626. arXiv:2007.12626 · arXiv
  13. Alex Federman, Erin Sarzynski, Cindy Brach, Paul Francaviglia, Jessica Jacques, Lina Jan- dorf, Angela Sanchez Munoz, Michael Wolf, and Joseph Kannry. Challenges optimizing the af- ter visit summary. International journal of medical informatics, 120:14-19, December 2018. ISSN 1386-5056. doi: 10.1016/j.ijmedinf.2018.09. 009. URL https://www.ncbi.nlm.nih.gov/pmc/ articles/PMC6326571/. · DOI
  14. Raymond Fok, Joseph Chee Chang, Tal August, Amy X. Zhang, and Daniel S. Weld. Qlar- ify: Bridging Scholarly Abstracts and Papers with Recursively Expandable Summaries, October 2023. URL http://arxiv.org/abs/2310.07581. arXiv:2310.07581 · arXiv
  15. Nunzia B. Giuse, Taneya Y. Koonce, Alan B. Storrow, Sheila V. Kusnoor, and Fei Ye. Us- ing Health Literacy and Learning Style Prefer- ences to Optimize the Delivery of Health In- formation. Journal of Health Communication, 17(sup3):122-140, October 2012. ISSN 1081- 0730, 1087-0415. doi: 10.1080/10810730.2012. 712610. URL http://www.tandfonline.com/ doi/abs/10.1080/10810730.2012.712610. · DOI
  16. Ary L. Goldberger, Luis A. N. Amaral, Leon Glass, Jeffrey M. Hausdorff, Plamen Ch. Ivanov, Roger G. Mark, Joseph E. Mietus, George B. Moody, Chung- Kang Peng, and H. Eugene Stanley. PhysioBank, PhysioToolkit, and PhysioNet: Components of a New Research Resource for Complex Physiologic Signals. Circulation, 101(23), June 2000. ISSN 0009-7322, 1524-4539. doi: 10.1161/01.CIR.101.23. e215. URL https://www.ahajournals.org/doi/ 10.1161/01.CIR.101.23.e215. · DOI · Paper page
  17. Hana L. Haver, Anuj K. Gupta, Emily B. Am- binder, Manisha Bahl, Eniola T. Oluyemi, Jean Jeudy, and Paul H. Yi. Evaluating the Use of ChatGPT to Accurately Simplify Patient- centered Information about Breast Cancer Preven- tion and Screening. Radiology: Imaging Cancer, 6(2):e230086, March 2024. doi: 10.1148/rycan. 230086. URL https://pubs.rsna.org/doi/abs/ 10.1148/rycan.230086. Publisher: Radiological Society of North America. · DOI
  18. Karen S. Hayes. Randomized Trial of Geragogy- Based Medication Instruction in the Emergency Department. Nursing Research, 47(4):211, August 1998. ISSN 0029-6562. URL https: //journals.lww.com/nursingresearchonline/ fulltext/1998/07000/randomized_trial_of_ geragogy_based_medication.
  19. Huan He, Sunyang Fu, Liwei Wang, Sijia Liu, Andrew Wen, and Hongfang Liu. MedTa- tor: a serverless annotation tool for corpus development. Bioinformatics, 38(6):1776-1778, March 2022. ISSN 1367-4803. doi: 10.1093/ bioinformatics/btab880. URL https://doi.org/ 10.1093/bioinformatics/btab880. · DOI
  20. Karl Moritz Hermann, Tomas Kocisky, Ed- ward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teach- ing Machines to Read and Comprehend. In Advances in Neural Information Process- ing Systems, volume 28.
  21. Curran Associates, Inc., 2015. URL https://proceedings. neurips.cc/paper_files/paper/2015/hash/ afdec7005cc9f14302cd0474fd0f3c96-Abstract. html.
  22. Leora I. Horwitz, John P. Moriarty, Christine Chen, Robert L. Fogerty, Ursula C. Brewster, Sandhya Kanade, Boback Ziaeian, Grace Y. Jenq, and Harlan M. Krumholz. Quality of Discharge Practices and Patient Understanding at an Academic Medical Center. JAMA Inter- nal Medicine, August 2013. ISSN 2168-6106. doi: 10.1001/jamainternmed.2013.9318. URL http://archinte.jamanetwork.com/article. aspx?doi=10.1001/jamainternmed.2013.9318. · DOI
  23. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models, October 2021. URL http: //arxiv.org/abs/2106.09685. arXiv:2106.09685 · arXiv
  24. Yichong Huang, Xiachong Feng, Xiaocheng Feng, and Bing Qin. The Factual Inconsistency Problem in Abstractive Text Summarization: A Survey, April 2023. URL http://arxiv.org/abs/2104.14839. arXiv:2104.14839 [cs. · arXiv
  25. Sosena Kebede, Hasan M. Shihab, Zackary D. Berger, Nina G. Shah, Hsin-Chieh Yeh, and Daniel J. Brotman. Patients' Understanding of Their Hos- pitalizations and Association With Satisfaction. JAMA Internal Medicine, 174(10):1698-1700, Oc- tober 2014. ISSN 2168-6106. doi: 10.1001/ jamainternmed.2014.3765. URL https://doi. org/10.1001/jamainternmed.2014.3765. · DOI
  26. Zeljko Kraljevic, Thomas Searle, Anthony Shek, Lukasz Roguski, Kawsar Noor, Daniel Bean, Au- relie Mascio, Leilei Zhu, Amos A. Folarin, An- gus Roberts, Rebecca Bendayan, Mark P. Richard- son, Robert Stewart, Anoop D. Shah, Wai Keong Wong, Zina Ibrahim, James T. Teo, and Richard J. B. Dobson. Multi-domain clinical natural language processing with MedCAT: The Medi- cal Concept Annotation Toolkit. Artificial In- telligence in Medicine, 117:102083, July 2021. ISSN 0933-3657. doi: 10.1016/j.artmed.2021. 102083. URL https://www.sciencedirect.com/ science/article/pii/S0933365721000762. · DOI
  27. Klaus Krippendorff. Content Analysis: An Introduc- tion to Its Methodology. SAGE Publications, May 2018. ISBN 978-1-5063-9567-8. Google-Books-ID: nE1aDwAAQBAJ.
  28. Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. Evaluating the Fac- tual Consistency of Abstractive Text Summa- rization. In Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP), pages 9332-9346, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main. 750. URL https://aclanthology.org/2020. emnlp-main.750. · DOI · Paper page
  29. Mike Lewis, Yinhan Liu, Naman Goyal, Mar- jan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettle- moyer. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In Dan Ju- rafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.703. URL https: //aclanthology.org/2020.acl-main.703. · DOI · Paper page
  30. Chin-Yew Lin. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https://aclanthology.org/W04-1013.
  31. R. Lin, R. Gallagher, M. Spinaze, H. Najoumian, C. Dennis, R. Clifton-Bligh, and G. Tofler. Effect of a patient-directed discharge letter on patient understanding of their hospitalisa- tion. Internal Medicine Journal, 44(9):851-857, 2014. ISSN 1445-5994. doi: 10.1111/imj. 12482. URL https://onlinelibrary.wiley. com/doi/abs/10.1111/imj.12482. · DOI
  32. Fangyu Liu, Ehsan Shareghi, Zaiqiao Meng, Marco Basaldella, and Nigel Collier. Self-Alignment Pre- training for Biomedical Entity Representations. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 4228-4238, On- line, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main. 334. URL https://aclanthology.org/2021. naacl-main.334. · DOI
  33. Laurens van der Maaten and Geoffrey Hinton. Vi- sualizing Data using t-SNE. Journal of Machine Learning Research, 9(86):2579-2605, 2008. ISSN 1533-7928. URL http://jmlr.org/papers/v9/ vandermaaten08a.html.
  34. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On Faithfulness and Factuality in Abstractive Summarization. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/ 2020.acl-main.173. URL https://aclanthology. org/2020.acl-main.173. · DOI · Paper page
  35. Abhika Mishra, Akari Asai, Vidhisha Balachandran, Yizhong Wang, Graham Neubig, Yulia Tsvetkov, and Hannaneh Hajishirzi. Fine-grained Halluci- nation Detection and Editing for Language Mod- els, January 2024. URL http://arxiv.org/abs/ 2401.06855. arXiv:2401.06855 [cs. · arXiv
  36. Francesco Moramarco, Alex Papadopoulos Korfiatis, Mark Perera, Damir Juric, Jack Flann, Ehud Re- iter, Anya Belz, and Aleksandar Savkov. Human Evaluation and Correlation with Automatic Met- rics in Consultation Note Generation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicen- cio, editors, Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguis- tics (Volume 1: Long Papers), pages 5739-5754, Dublin, Ireland, May 2022. Association for Com- putational Linguistics. doi: 10.18653/v1/2022. acl-long.394. URL https://aclanthology.org/ 2022.acl-long.394. · DOI · Paper page
  37. Stephanie K Mueller, Kyla Giannelli, Robert Boxer, and Jeffrey L Schnipper. Readability of pa- tient discharge instructions with and without the use of electronically available disease-specific tem- plates. Journal of the American Medical In- formatics Association, 22(4):857-863, July 2015. ISSN 1527-974X, 1067-5027. doi: 10.1093/ jamia/ocv005. URL https://academic.oup.com/ jamia/article/22/4/857/1746200. · DOI
  38. Feng Nan, Ramesh Nallapati, Zhiguo Wang, Ci- cero Nogueira dos Santos, Henghui Zhu, De- jiao Zhang, Kathleen McKeown, and Bing Xi- ang. Entity-level Factual Consistency of Abstrac- tive Text Summarization. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2727-2733, Online, April 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.eacl-main.235. URL https:// aclanthology.org/2021.eacl-main.235. · DOI
  39. Zhuk, and Barret Zoph. GPT-4 Technical Re- port, December 2023. URL http://arxiv.org/ abs/2303.08774. arXiv:2303.08774 · arXiv
  40. Sarita Pathak, Gregory Summerville, Celia P Ka- plan, Sarah S Nouri, and Leah S Karliner. Patient- Reported Use of the After Visit Summary in a Primary Care Internal Medicine Practice. Jour- nal of Patient Experience, 7(5):703-707, Octo- ber 2020. ISSN 2374-3735. doi: 10.1177/ 2374373519879286. URL https://www.ncbi.nlm. nih.gov/pmc/articles/PMC7705830/. · DOI
  41. Carswella Phillips. Relationships between work- load perception, burnout, and intent to leave among medical-surgical nurses. JBI Evidence Implementation, 18(2):265, June 2020. ISSN 2691- 3321. doi: 10.1097/XEB.0000000000000220. URL https://journals.lww.com/ijebh/ abstract/2020/06000/relationships_ between_workload_perception,.11.aspx. · DOI
  42. Isabel Segura-Bedmar, Paloma Martínez, and María Herrero-Zazo. SemEval-2013 Task 9 : Extrac- tion of Drug-Drug Interactions from Biomedical Texts (DDIExtraction 2013). In Suresh Manand- har and Deniz Yuret, editors, Second Joint Con- ference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 341-350, Atlanta, Geor- gia, USA, June 2013. Association for Computa- tional Linguistics. URL https://aclanthology. org/S13-2056.
  43. Sina J. Semnani, Violet Z. Yao, Heidi C. Zhang, and Monica S. Lam. WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipedia, October 2023. URL http://arxiv.org/abs/2305.14292. arXiv:2305.14292 [cs. · arXiv
  44. Craig Thomson and Ehud Reiter. A Gold Standard Methodology for Evaluating Accuracy in Data-To- Text Systems. In Brian Davis, Yvette Graham, John Kelleher, and Yaji Sripada, editors, Proceed- ings of the 13th International Conference on Nat- ural Language Generation, pages 158-168, Dublin, Ireland, December 2020. Association for Computa- tional Linguistics. URL https://aclanthology. org/2020.inlg-1.
  45. Craig Thomson and Ehud Reiter. Generation Chal- lenges: Results of the Accuracy Evaluation Shared Task, August 2021. URL http://arxiv.org/abs/ 2108.05644. arXiv:2108.05644 [cs. · arXiv
  46. Craig Thomson, Ehud Reiter, and Barkavi Sun- dararajan. Evaluating factual accuracy in complex data-to-text. Computer Speech & Language, 80:101482, May 2023. ISSN 0885- 2308. doi: 10.1016/j.csl.2023.101482. URL https://www.sciencedirect.com/science/ article/pii/S0885230823000013. · DOI
  47. Katherine Tian, Eric Mitchell, Huaxiu Yao, Christo- pher D. Manning, and Chelsea Finn. Fine- tuning Language Models for Factuality, November 2023. URL http://arxiv.org/abs/2311.08401. arXiv:2311.08401 · arXiv
  48. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucu- rull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ran- jan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sha- ran Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open Foundation and Fine-Tuned Chat Models, July 2023. URL http://arxiv.org/abs/2307. 09288. arXiv:2307.09288 [cs. · arXiv
  49. Dave Van Veen, Cara Van Uden, Maayane Attias, Anuj Pareek, Christian Bluethgen, Malgorzata Po- lacin, Wah Chiu, Jean-Benoit Delbrouck, Juan Manuel Zambrano Chaves, Curtis P. Langlotz, Ak- shay S. Chaudhari, and John Pauly. RadAdapt: Radiology Report Summarization via Lightweight Domain Adaptation of Large Language Models, May 2023. URL http://arxiv.org/abs/2305. 01146. arXiv:2305.01146 [cs. · arXiv
  50. Dave Van Veen, Cara Van Uden, Louis Blanke- meier, Jean-Benoit Delbrouck, Asad Aali, Chris- tian Bluethgen, Anuj Pareek, Malgorzata Polacin, Eduardo Pontes Reis, Anna Seehofnerová, Nidhi Rohatgi, Poonam Hosamani, William Collins, Neera Ahuja, Curtis P. Langlotz, Jason Hom, Ser- gios Gatidis, John Pauly, and Akshay S. Chaud- hari. Adapted large language models can out- perform medical experts in clinical text summa- rization. Nature Medicine, pages 1-9, Febru- ary 2024. ISSN 1546-170X. doi: 10.1038/ s41591-024-02855-5. URL https://www.nature. com/articles/s41591-024-02855-5. Publisher: Nature Publishing Group. · DOI · Paper page
  51. Alexander G. Watson, Jonathan V. McCoy, JoAnn Mathew, Daniel A. Gundersen, and Robert M. Eisenstein. Impact of physician workload on burnout in the emergency depart- ment. Psychology, Health & Medicine, 24(4): 414-428, April 2019. ISSN 1354-8506. doi: 10.1080/13548506.2018.1539236. URL https: //doi.org/10.1080/13548506.2018.1539236. Publisher: Taylor & Francis eprint: https://doi.org/10.1080/13548506.2018.1539236. · DOI
  52. Wei-Hung Weng, Yu-An Chung, and Peter Szolovits. Unsupervised Clinical Language Translation. In Proceedings of the 25th ACM SIGKDD Interna- tional Conference on Knowledge Discovery & Data Mining, KDD '19, pages 3121-3131, New York, NY, USA, July 2019. Association for Comput- ing Machinery. ISBN 978-1-4503-6201-6. doi: 10.1145/3292500.3330710. URL https://dl.acm. org/doi/10.1145/3292500.3330710. · DOI
  53. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Can- wen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. HuggingFace's Transformers: State-of-the-art Nat- ural Language Processing, July 2020. URL http: //arxiv.org/abs/1910.03771. arXiv:1910.03771 · arXiv
  54. Qianqian Xie, Jiayu Zhou, Yifan Peng, and Fei Wang. FactReranker: Fact-guided Reranker for Faithful Radiology Report Summarization, March 2023. URL http://arxiv.org/abs/2303.08335. arXiv:2303.08335 [cs. · arXiv
  55. Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. Optimizing Sta- tistical Machine Translation for Text Simplifica- tion. Transactions of the Association for Com- putational Linguistics, 4:401-415, 2016. doi: 10. 1162/tacl a 00107. URL https://aclanthology. org/Q16-1029. Place: Cambridge, MA Publisher: MIT Press. · DOI
  56. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. BERTScore: Eval- uating Text Generation with BERT, February 2020a. URL http://arxiv.org/abs/1904.09675. arXiv:1904.09675 [cs. · arXiv
  57. Yuhao Zhang, Derek Merck, Emily Tsai, Christo- pher D. Manning, and Curtis Langlotz. Opti- mizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics, pages 5108-5120, Online, July 2020b. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.458. URL https: //aclanthology.org/2020.acl-main.458. · DOI · Paper page
  58. Chunting Zhou, Graham Neubig, Jiatao Gu, Mona Diab, Francisco Guzmán, Luke Zettlemoyer, and Marjan Ghazvininejad. Detecting Hallucinated Content in Conditional Neural Sequence Gen- eration. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1393-1404, Online, August 2021. Associ- ation for Computational Linguistics. doi: 10. 18653/v1/2021.findings-acl.120. URL https:// aclanthology.org/2021.findings-acl.120. · DOI · Paper page
  59. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: Less Is More for Alignment. In Thirty-seventh Con- ference on Neural Information Processing Systems, November 2023. URL https://openreview.net/ forum?id=KBMOKmX2he.

Cited by

Sorted by citations

Authors
Hegselmann, S., Shen, S. Z., Gierse, F., Agrawal, M., Sontag, D., & Jiang, X.
Affiliations
  • University of Münster (DE)
  • MIT Computer Science and Artificial Intelligence Laboratory (US)
  • Duke University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2402.15422
arXiv
2402.15422
OpenAlex
W4392182480
Cited by
6
References
1
Citations by year
2025: 3 · 2024: 3
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University)
Funders
  • Deutsche Forschungsgemeinschaft (INST 211/667‐1)
  • Westfälische Wilhelms-Universität Münster
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
medical text summarization, hallucinations, faithfulness, training data quality, few-shot prompting, Llama 2, GPT-4

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review