CoChat
Sign Up
PreprintCited by 57Artificial Intelligence in Healthcare and EducationTopic ModelingMultimodal Machine Learning Applications

Faithful AI in Medicine: A Systematic Review with Large Language Models and Beyond

Open access
Xie, Qianqian, Schenck, Edward J., Yang, He S., Chen, Yong, Peng, Yifan, Wang, Fei·2023·medRxiv
DOI 10.21203/rs.3.rs-3661764/v1PMC PMC10723541
Open this paper in CoChat — free accountOpen PDF

Abstract

Objective While artificial intelligence (AI), particularly large language models (LLMs), offers significant potential for medicine, it raises critical concerns due to the possibility of generating factually incorrect information, leading to potential long-term risks and ethical issues. This review aims to provide a comprehensive overview of the faithfulness problem in existing research on AI in healthcare and medicine, with a focus on the analysis of the causes of unfaithful results, evaluation metrics, and mitigation methods. Materials and Methods Using PRISMA methodology, we sourced 5,061 records from five databases (PubMed, Scopus, IEEE Xplore, ACM Digital Library, Google Scholar) published between January 2018 to March 2023. We removed duplicates and screened records based on exclusion criteria. Results With 40 leaving articles, we conducted a systematic review of recent developments aimed at optimizing and evaluating factuality across a variety of generative medical AI approaches. These include knowledge-grounded LLMs, text-to-text generation, multimodality-to-text generation, and automatic medical fact-checking tasks. Discussion Current research investigating the factuality problem in medical AI is in its early stages. There are significant challenges related to data resources, backbone models, mitigation methods, and evaluation metrics. Promising opportunities exist for novel faithful medical AI research involving the adaptation of LLMs and prompt engineering. Conclusion This comprehensive review highlights the need for further research to address the issues of reliability and factuality in medical AI, serving as both a reference and inspiration for future research into the safe, ethical use of AI in medicine and healthcare.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.21203/rs.3.rs-3661764/v1

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Prior research found that generative AI created nonfactual or unfaithful data and outputs.

    “Prior research found that generative AI created nonfactual or unfaithful data and outputs [72,77].”

    From Generative AI in medical practice: in-depth exploration of privacy and security challenges (2024) · cites this paper as [77]

  • The context recall limitation pattern appeared in approximately 40% of cases.

    “This pattern appeared in approximately 40% of cases, underscoring retrieval challenges in medical RAG systems where the complexity of clinical terminology limits the effectiveness of standard mechanisms [57,58].”

    From Enhancing Medical Question Answering with LLMs via a Hybrid Retrieval-Augmented Generation Framework (2026) · cites this paper as [58]

    The Results section on medical text summarization reports nearly 30% factual errors in radiology summaries, not a 40% context-recall limitation. Searches for “40%,” “context recall,” and medical RAG retrieval challenges found no passage in our copy reporting the claimed pattern or rate. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 109 · Cited by 57

Research works printed but not linked: 72

  1. [1]Yu K.-H., Beam A. L. & Kohane I. S. Artificial intelligence in healthcare. Nat. biomedical engineering 2, 719– 731 (2018). 31015651
  2. [2]Topol E. J. High-performance medicine: the convergence of human and artificial intelligence. Nat. medicine 25, 44– 56 (2019). 30617339
  3. [3]Rajpurkar P., Chen E., Banerjee O. & Topol E. J. Ai in health and medicine. Nat. medicine 28, 31– 38 (2022). 35058619
  4. [4]LeCun Y., Bengio Y. & Hinton G. Deep learning. nature 521, 436– 444 (2015). 26017442
  5. [5]Wang B. Pre-trained language models in biomedical domain: A systematic survey. arXiv preprint arXiv:2110.05006 (2021). · arXiv
  6. [6]Kenton J. D. M.-W. C. & Toutanova L. K. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, 4171– 4186 (2019).
  7. [7]Vaswani A. Attention is all you need. Adv. neural information processing systems 30 (2017).
  8. [8]Agrawal M., Hegselmann S., Lang H., Kim Y. & Sontag D. Large language models are zero-shot clinical information extractors. arXiv preprint arXiv:2205.12689 (2022). · arXiv
  9. [9]Tiu E. Expert-level detection of pathologies from unannotated chest x-ray images via self-supervised learning. Nat. Biomed. Eng. 1– 8 (2022). 36109605 PMC9792370
  10. [10]Brown T. Language models are few-shot learners. Adv. neural information processing systems 33, 1877– 1901 (2020).
  11. [12]OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023). · arXiv
  12. [13]Bubeck S. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 (2023). · arXiv
  13. [14]Kung T. H. Performance of chatgpt on usmle: Potential for ai-assisted medical education using large language models. PLOS Digit. Heal. 2, e0000198 (2023). PMC9931230 36812645
  14. [17]Zhang Y., Merck D., Tsai E., Manning C. D. & Langlotz C. Optimizing the factual correctness of a summary: A study of summarizing radiology reports. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 5108– 5120 (2020).
  15. [18]Luo R. Biogpt: generative pre-trained transformer for biomedical text generation and mining. Briefings Bioinforma. 23 (2022). 36156661
  16. [19]Li W. Faithfulness in natural language generation: A systematic survey of analysis, evaluation and optimization methods. arXiv preprint arXiv:2203.05227 (2022). · arXiv
  17. [21]Lee J. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36, 1234– 1240 (2020). 31501885 PMC7703786
  18. [22]Peng Y., Yan S. & Lu Z. Transfer learning in biomedical natural language processing: an evaluation of bert and elmo on ten benchmarking datasets. arXiv preprint arXiv:1906.05474 (2019). · arXiv
  19. [23]Bengio S., Vinyals O., Jaitly N. & Shazeer N. Scheduled sampling for sequence prediction with recurrent neural networks. Adv. neural information processing systems 28 (2015).
  20. [24]Wang C. & Sennrich R. On exposure bias, hallucination and domain shift in neural machine translation. In 2020 Annual Conference of the Association for Computational Linguistics, 3544– 3552 (Association for Computational Linguistics (ACL), 2020).
  21. [25]Singhal K. Large language models encode clinical knowledge. arXiv preprint arXiv:2212.13138 (2022). · arXiv
  22. [26]Chowdhery A. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 (2022). · arXiv
  23. [27]Zakka C., Chaurasia A., Shad R. & Hiesinger W. Almanac: Knowledge-grounded language models for clinical medicine. arXiv preprint arXiv:2303.01229 (2023). · arXiv
  24. [28]Nori H., King N., McKinney S. M., Carignan D. & Horvitz E. Capabilities of gpt-4 on medical challenge problems. arXiv preprint arXiv:2303.13375 (2023). · arXiv
  25. [29]Afantenos S., Karkaletsis V. & Stamatopoulos P. Summarization from medical documents: a survey. Artif. intelligence medicine 33, 157– 177 (2005). 15811783
  26. [31]arXiv preprint arXiv:2304.08763 (2023). · arXiv
  27. [32]Luo Z., Xie Q. & Ananiadou S. Citationsum: Citation-aware graph contrastive learning for scientific paper summarization. In Proceedings of the ACM Web Conference 2023, 1843– 1852 (2023).
  28. [33]Xie Q., Bishop J. A., Tiwari P. & Ananiadou S. Pre-trained language models with domain knowledge for biomedical extractive summarization. Knowledge-Based Syst. 252, 109460 (2022).
  29. [34]Luo Z., Xie Q. & Ananiadou S. Readability controllable biomedical document summarization. In Findings of the Association for Computational Linguistics: EMNLP 2022, 4667– 4680 (2022).
  30. [35]Zhang Y., Ding D. Y., Qian T., Manning C. D. & Langlotz C. P. Learning to summarize radiology findings. In Proceedings of the Ninth International Workshop on Health Text Mining and Information Analysis, 204– 213 (2018).
  31. [36]Delbrouck J.-B., Varma M. & Langlotz C. P. Toward expanding the scope of radiology report summarization to multiple anatomies and modalities. arXiv preprint arXiv:2211.08584 (2022). · arXiv
  32. [38]Xie Q., Zhou J., Peng Y. & Wang F. Factreranker: Fact-guided reranker for faithful radiology report summarization. arXiv preprint arXiv:2303.08335 (2023). · arXiv
  33. [39]DeYoung J., Beltagy I., van Zuylen M., Kuehl B. & Wang L. Mŝ2: Multi-document summarization of medical studies. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 7494– 7513 (2021).
  34. [40]Lewis M. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 7871– 7880 (2020).
  35. [41]Wallace B. C., Saha S., Soboczenski F. & Marshall I. J. Generating (factual?) narrative summaries of rcts: Experiments with neural multi-document summarization. AMIA Summits on Transl. Sci. Proc. 2021, 605 (2021). PMC8378607 34457176
  36. [42]Alambo A., Banerjee T., Thirunarayan K. & Raymer M. Entity-driven fact-aware abstractive summarization of biomedical literature. In 2022 26th International Conference on Pattern Recognition (ICPR), 613– 620 (IEEE, 2022).
  37. [43]Yadav S., Gupta D., Abacha A. B. & Demner-Fushman D. Reinforcement learning for abstractive question summarization with question-aware semantic rewards. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), 249– 255 (2021).
  38. [44]Chintagunta B., Katariya N., Amatriain X. & Kannan A. Medically aware gpt-3 as a data generator for medical dialogue summarization. In Machine Learning for Healthcare Conference, 354– 372 (PMLR, 2021).
  39. [45]Liu F. Retrieve, reason, and refine: Generating accurate and faithful patient instructions. In Advances in Neural Information Processing Systems.
  40. [46]Zhang T., Kishore V., Wu F., Weinberger K. Q. & Artzi Y. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.
  41. [47]Jain S. Radgraph: Extracting clinical entities and relations from radiology reports. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1).
  42. [48]Otmakhova J., Verspoor K., Baldwin T. & Lau J. H. The patient is more dead than alive: exploring the current state of the multi-document summarisation of the biomedical literature. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 5098– 5111 (2022).
  43. [49]Otmakhova J., Verspoor K., Baldwin T., Yepes A. J. & Lau J. H. M3: Multi-level dataset for multi-document summarisation of medical studies. In Findings of the Association for Computational Linguistics: EMNLP 2022, 3887– 3901 (2022).
  44. [50]Adams G., Zucker J. & Elhadad N. A meta-evaluation of faithfulness metrics for long-form hospital-course summarization. arXiv preprint arXiv:2303.03948 (2023). PMC11441639 39350919 · arXiv
  45. [51]Deng M., Tan B., Liu Z., Xing E. & Hu Z. Compression, transduction, and creation: A unified framework for evaluating natural language generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 7580– 7605 (2021).
  46. [52]Laban P., Schnabel T., Bennett P. N. & Hearst M. A. Summac: Re-visiting nli-based models for inconsistency detection in summarization. Transactions Assoc. for Comput. Linguist. 10, 163– 177 (2022).
  47. [53]Lu J., Li J., Wallace B. C., He Y. & Pergola G. Napss: Paragraph-level medical text simplification via narrative prompting and sentence-matching summarization. arXiv preprint arXiv:2302.05574 (2023). · arXiv
  48. [54]Devaraj A., Wallace B. C., Marshall I. J. & Li J. J. Paragraph-level simplification of medical texts. In Proceedings of the conference. Association for Computational Linguistics. North American Chapter. Meeting, vol. 2021, 4972 (NIH Public Access, 2021). PMC9161242 35663507
  49. [55]Jeblick K. Chatgpt makes medicine easy to swallow: An exploratory case study on simplified radiology reports. arXiv preprint arXiv:2212.14882 (2022). PMC11126432 37794249 · arXiv
  50. [56]Lyu Q. Translating radiology reports into plain language using chatgpt and gpt-4 with prompt learning: Promising results, limitations, and potential. arXiv preprint arXiv:2303.09038 (2023). PMC10192466 37198498 · arXiv
  51. [58]Devaraj A., Sheffield W., Wallace B. C. & Li J. J. Evaluating factuality in text simplification. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 7331– 7345 (2022). PMC9671157 36404800
  52. [59]Nishino T. Reinforcement learning with imbalanced dataset for data-to-text medical report generation. In Findings of the Association for Computational Linguistics: EMNLP 2020, 2223– 2236 (2020).
  53. [60]Miura Y., Zhang Y., Tsai E., Langlotz C. & Jurafsky D. Improving factual completeness and consistency of image-to-text radiology report generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 5288– 5304 (2021).
  54. [61]Delbrouck J.-B. Improving the factual correctness of radiology report generation with semantic rewards. In Findings of the Association for Computational Linguistics: EMNLP 2022, 4348– 4360 (2022).
  55. [62]Nishino T. Factual accuracy is not enough: Planning consistent description order for radiology report generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 7123– 7138 (2022).
  56. [63]Liu G. Clinically accurate chest x-ray report generation. In Machine Learning for Healthcare Conference, 249– 269 (PMLR, 2019).
  57. [65]Papineni K., Roukos S., Ward T. & Zhu W.-J. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 311– 318 (2002).
  58. [66]Kotonya N. & Toni F. Explainable automated fact-checking for public health claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 7740– 7754 (2020).
  59. [67]Wadden D. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 7534– 7550 (2020).
  60. [68]Poliak A. Collecting verified covid-19 question answer pairs. In Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020 (2020).
  61. [69]Sarrouti M., Abacha A. B., M’rabet Y. & Demner-Fushman D. Evidence-based fact-checking of health-related claims. In Findings of the Association for Computational Linguistics: EMNLP 2021, 3499– 3512 (2021).
  62. [70]Saakyan A., Chakrabarty T. & Muresan S. Covid-fact: Fact extraction and verification of real-world claims on covid-19 pandemic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2116– 2129 (2021).
  63. [71]Mohr I., Wührl A. & Klinger R. Covert: A corpus of fact-checked biomedical covid-19 tweets. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, 244– 257 (2022).
  64. [72]Srba I. Monant medical misinformation dataset: Mapping articles to fact-checked claims. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2949– 2959 (2022).
  65. [73]Wadhwa S., Khetan V., Amir S. & Wallace B. Redhot: A corpus of annotated medical questions, experiences, and claims on social media. arXiv preprint arXiv:2210.06331 (2022). · arXiv
  66. [74]Beltagy I., Lo K. & Cohan A. Scibert: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 3615– 3620 (2019).
  67. [75]Wadden D. Multivers: Improving scientific claim verification with weak supervision and full-document context. In Findings of the Association for Computational Linguistics: NAACL 2022, 61– 76 (2022).
  68. [76]Beltagy I., Peters M. E. & Cohan A. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 (2020). · arXiv
  69. [77]Alsentzer E. Publicly available clinical bert embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, 72– 78 (2019).
  70. [79]Chowdhery A. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 (2022). · arXiv
  71. [80]Yuan Z., Liu Y., Tan C., Huang S. & Huang F. Improving biomedical pretrained language models with knowledge. In Proceedings of the 20th Workshop on Biomedical Language Processing, 180– 190 (2021).
  72. [81]Jha K. & Zhang A. Continual knowledge infusion into pre-trained biomedical language models. Bioinformatics 38, 494– 502 (2022). 34554186

Authors
Xie, Q., Schenck, E. J., Yang, H. S., Chen, Y., Peng, Y., & Wang, F.
Affiliations
  • Cornell University (US)
  • NewYork–Presbyterian Hospital (US)
  • Weill Cornell Medicine (US)
  • University of Pennsylvania (US)
Corresponding author
Wang, Fei
ORCID
Venue
medRxiv
Publisher
Springer Science and Business Media LLC
Published
2023-12-04
DOI
10.21203/rs.3.rs-3661764/v1
PMC
PMC10723541
OpenAlex
W4366580365
Cited by
57
References
109
Citations by year
2026: 7 · 2025: 17 · 2024: 25 · 2023: 8
Type
posted content
Language
en
Open access
Licence
Locations
  • medRxiv (acceptedVersion, cc-by-nc-nd)
  • Research Square (acceptedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (submittedVersion, cc-by-nc-nd)
  • PubMed Central (submittedVersion, cc-by)
Funders
  • National Science Foundation (1750326)
  • National Institutes of Health
  • National Institute on Aging (RF1AG072449, R01AG080991, R01AG080624, R01AG076448, R01AG076234)
  • National Heart, Lung, and Blood Institute (K23HL151876)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
large language models, faithfulness, factual accuracy, artificial intelligence in healthcare, text-to-text generation, evaluation metrics, clinical decision support

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review