CoChat
Sign Up
PreprintCited by 43Artificial Intelligence in Healthcare and EducationTopic ModelingInformation Retrieval and Search Behavior

How well do LLMs cite relevant medical references? An evaluation framework and analyses

Open access
Wu, Kevin, Wu, Eric, Cassasola, Ally, Zhang, Angela, Wei, Kevin, Nguyen, Teresa, Riantawan, Sith, Riantawan, Patricia Shi +2 more·2024·arXiv
DOI 10.48550/arxiv.2402.02008arXiv 2402.02008
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models (LLMs) are currently being used to answer medical questions across a variety of clinical domains. Recent top-performing commercial LLMs, in particular, are also capable of citing sources to support their responses. In this paper, we ask: do the sources that LLMs generate actually support the claims that they make? To answer this, we propose three contributions. First, as expert medical annotations are an expensive and time-consuming bottleneck for scalable evaluation, we demonstrate that GPT-4 is highly accurate in validating source relevance, agreeing 88% of the time with a panel of medical doctors. Second, we develop an end-to-end, automated pipeline called \textit{SourceCheckup} and use it to evaluate five top-performing LLMs on a dataset of 1200 generated questions, totaling over 40K pairs of statements and sources. Interestingly, we find that between ~50% to 90% of LLM responses are not fully supported by the sources they provide. We also evaluate GPT-4 with retrieval augmented generation (RAG) and find that, even still, around 30\% of individual statements are unsupported, while nearly half of its responses are not fully supported. Third, we open-source our curated dataset of medical questions and expert annotations for future evaluations. Given the rapid pace of LLM development and the potential harms of incorrect or outdated medical information, it is crucial to also understand and quantify their capability to produce relevant, trustworthy medical references.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2402.02008

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Even state-of-the-art proprietary LLMs perpetuate historic biases, cite inappropriate medical articles and fail to perform information-driven administrative tasks like medical coding.

    “Even state-of-the-art proprietary LLMs perpetuate historic biases41, cite inappropriate medical articles42 and fail to perform information-driven administrative tasks like medical coding43.”

    From Medical large language models are vulnerable to data-poisoning attacks (2025) · cites this paper as [42]

  • The literature review that forms the basis of research conception may include facts or citations that are inaccurate, irrelevant, or fake, i.e., hallucinations.

    “First, the literature review that forms the basis of research conception may include facts or citations that are inaccurate, irrelevant, or fake, i.e., “hallucinations” [5, 34–36].”

    From Autonomous artificial intelligence, scientific research, and human values (2026) · cites this paper as [36]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 9

  1. Brin, D., Sorin, V., Vaid, A., Soroush, A., Glicksberg, B. S., Charney, A. W., Nadkarni, G., and Klang, E. Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessments. Sci. Rep., 13(1):16492, October 2023.
  2. Gilbert, S., Harvey, H., Melvin, T., Vollebregt, E., and Wicks, P. Large language model AI chatbots require ap- proval as medical devices. Nat. Med., 29(10):2396-2398, October 2023.
  3. Gottlieb, S. and Silvis, L. How to safely integrate large language models into health care. JAMA Health Forum, 4(9):e233909, September 2023.
  4. Haupt, C. E. and Marks, M. AI-Generated medical Advice- GPT and beyond. JAMA, 329(16):1349-1350, April 2023.
  5. Maples, B., Cerit, M., Vishwanath, A., and Pea, R. Lone- liness and suicide mitigation for students using GPT3- enabled chatbots. npj Mental Health Research, 3(1):1-6, January 2024.
  6. Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., Jiang, X., Cobbe, K., Eloundou, T., Krueger, G., Button, K., Knight, M., Chess, B., and Schulman, J. WebGPT: Browser-assisted question-answering with human feed- back. December 2021.
  7. Qin, Y., Cai, Z., Jin, D., Yan, L., Liang, S., Zhu, K., Lin, Y., Han, X., Ding, N., Wang, H., Xie, R., Qi, F., Liu, Z., Sun, M., and Zhou, J. WebCPM: Interactive web search for Chinese long-form question answering. pp. 8968-8988, July 2023.
  8. Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., Payne, P., Seneviratne, M., Gamble, P., Kelly, C., Babiker, A., Schärli, N., Chowdhery, A., Mansfield, P., Demner- Fushman, D., Agüera Y Arcas, B., Webster, D., Corrado, G. S., Matias, Y., Chou, K., Gottweis, J., Tomasev, N., Liu, Y., Rajkomar, A., Barral, J., Semturs, C., Karthike- salingam, A., and Natarajan, V. Large language models encode clinical knowledge. Nature, 620(7972):172-180, August 2023a.
  9. Strong, E., DiGiammarino, A., Weng, Y., Kumar, A., Hosamani, P., Hom, J., and Chen, J. H. Chatbot vs medical student performance on Free-Response clinical reasoning examinations. JAMA Intern. Med., 183(9): 1028-1030, September 2023.

Cited by

Sorted by citations

Related

Authors
Wu, K., Wu, E., Cassasola, A., Zhang, A., Wei, K., Nguyen, T., Riantawan, S., Riantawan, P. S., Ho, D. E., & Zou, J.
Affiliations
  • Stanford University (US)
  • University of Southern California (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2402.02008
arXiv
2402.02008
OpenAlex
W4391590636
Cited by
41
References
0
Citations by year
2026: 4 · 2025: 6 · 2024: 31
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
large language models, citation evaluation, medical question answering, GPT-4, retrieval-augmented generation, expert annotation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review