Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index; 4 of the 10 sources its claims cite were checked for support.

An automated framework for assessing how well LLMs cite relevant medical references

Open access
Wu, Kevin, Wu, Eric, Wei, Kevin, Zhang, Angela, Casasola, Allison, Nguyen, Teresa, Riantawan, Sith, Shi, Patricia, Ho, Daniel, Zou, James · 2025 · Nature Communications

Abstract

As large language models (LLMs) are increasingly used to address health-related queries, it is crucial that they support their conclusions with credible references. While models can cite sources, the extent to which these support claims remains unclear. To address this gap, we introduce SourceCheckup, an automated agent-based pipeline that evaluates the relevance and supportiveness of sources in LLM responses. We evaluate seven popular LLMs on a dataset of 800 questions and 58,000 pairs of statements and sources on data that represent common medical queries. Our findings reveal that between 50% and 90% of LLM responses are not fully supported, and sometimes contradicted, by the sources they cite. Even for GPT-4o with Web Search, approximately 30% of individual statements are unsupported, and nearly half of its responses are not fully supported. Independent assessments by doctors further validate these results. Our research underscores significant limitations in current LLMs to produce trustworthy medical references. Assessing the degree to which medical large language models reliably convey existing, trustworthy knowledge is crucial. This study introduces SourceCheckup, an automated framework revealing that large language models frequently cite medical references that do not fully support, or even contradict, their responses, showing significant gaps in reliability for clinical use.

Overview

CoChat Summary

TL;DR

SourceCheckup is an automated agent-based pipeline for evaluating whether LLM responses to medical questions cite sources that actually support their statements. Across seven LLMs and 800 questions, it finds that 50–90% of responses are not fully supported by their cited sources, even for GPT-4o with web search.

Key findings

  • SourceCheckup showed 88.7% agreement with a consensus of three US-licensed medical experts on source verification, higher than the 86.1% average inter-doctor agreement; no statistically significant difference was found between model and doctor consensus (p = 0.21).
  • Among seven evaluated LLMs, GPT-4o (RAG) had the highest citation performance, but only 55% of its responses were fully supported; Gemini Ultra 1.0 (RAG) had 34.5% response-level support, and Gemini Pro (API) had fully supported responses about 10% of the time.
  • Models without web access produced valid URLs only 40–70% of the time; GPT-4o (API) produced valid URLs around 70% of the time, while Llama-2-70b and Meditron-7b produced citation URLs <5% and <1% of the time, respectively.
  • On HealthSearchQA, GPT-4o (RAG) had 100% citation URL validity, 75.7% statement-level support, and 38.4% response-level support; on Reddit r/AskDocs, response-level support was 31.0%.
  • SourceCleanup removed 34.7% (52/150) of unsupported statements entirely; of the remaining 98 modified statements, 85.7% (84/98) were supported after modification, for an aggregate 90.7% (136/150) removed or correctly edited.

Methods

  • SourceCheckup has four modules: question generation, LLM question answering, statement and URL source parsing, and source verification. GPT-4o was used as the backbone for question generation, parsing, and source verification, with validation against US-licensed medical doctors.
  • The dataset combined 400 real-world medical queries from Reddit’s r/AskDocs with 400 GPT-4o-generated questions from MayoClinic reference documents, yielding 800 questions and about 58,000 statement-source pairs; a clinician-annotated subset of 400 question/answer pairs was used.
  • Seven LLMs were evaluated: GPT-4o (RAG), GPT-4o (API), Claude v2.1 (API), Mistral Medium (API), Gemini Ultra 1.0 (RAG), Gemini Pro (API), and Mixtral-8x7b (API); open-source Llama-2-70b and Meditron-7b were also tested but excluded from main results due to low citation URL production. Metrics included source URL validity, statement-level support, and response-level support.

Limitations

  • The paper states that the automated pipeline can accrue errors in question generation, statement parsing, citation extraction, and source verification, leading to small fluctuations in final results.
  • The paper states that source verification can be ambiguous, as shown by lack of full agreement among the three medical doctors, so individual data points may be noisy.
  • The paper states that the 1-1 mapping between statements and sources does not count cases where a statement could be supported by aggregating multiple sources; however, after merging sources for previously unsupported GPT-4o (RAG) statements, 95.1% remained unsupported.
  • The paper also notes that URL analysis may be affected by US-based question sources, 404 errors, and paywall access differences across institutions.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41467-025-58551-6

Cites

45 of 45 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Validation of SourceCheckup against doctors and evaluation of LLMs using SourceCheckup. a Agreement between the Source Verification model and doctors on the task of source verification. We asked three medical doctors (D1, D2, and D3) to determine whether pairs of statements and source texts are supported or unsupported. We found that the Source Verification model has a higher agreement with the doctor consensus than the average agreement between doctors. The 95% confidence intervals are computed with the bootstrap method and are shown in the error bars, and the total sample size is N = 400. b Evaluation of the quality of source verification in LLMs on medical queries. Each model is evaluated on three metrics. Source URL Validity measures the proportion of generated URLs that return a valid webpage. Statement-level Support measures the percentage of statements that are supported by at least one source in the same response. Response-level Support measures the percentage of responses that have all their statements supported. Full numerical results are displayed in Supplementary Table 4. The 95% confidence intervals are computed with the bootstrap method and are shown in the error bars. The sample sizes used to compute each statistic are found in Supplementary Table 8.
Fig. 1 Validation of SourceCheckup against doctors and evaluation of LLMs using SourceCheckup. a Agreement between the Source Verification model and doctors on the task of source verification. We asked three medical doctors (D1, D2, and D3) to determine whether pairs of statements and source texts are supported or unsupported. We found that the Source Verification model has a higher agreement with the doctor consensus than the average agreement between doctors. The 95% confidence intervals are computed with the bootstrap method and are shown in the error bars, and the total sample size is N = 400. b Evaluation of the quality of source verification in LLMs on medical queries. Each model is evaluated on three metrics. Source URL Validity measures the proportion of generated URLs that return a valid webpage. Statement-level Support measures the percentage of statements that are supported by at least one source in the same response. Response-level Support measures the percentage of responses that have all their statements supported. Full numerical results are displayed in Supplementary Table 4. The 95% confidence intervals are computed with the bootstrap method and are shown in the error bars. The sample sizes used to compute each statistic are found in Supplementary Table 8.CC BY-NC-ND · PubMed Central
Fig. 2 Schematic of the SourceCheckup evaluation pipeline. To start, GPT-4o generates a question based on a given medical reference text. Each evaluated LLM produces a response based on this question, which includes the response text along with any URL sources. The LLM response is parsed for individual medical statements while the URL sources are downloaded. Finally, the Source Verification model is asked to determine whether a given medical statement is supported by the source text and to provide a reason for the decision.
Fig. 2 Schematic of the SourceCheckup evaluation pipeline. To start, GPT-4o generates a question based on a given medical reference text. Each evaluated LLM produces a response based on this question, which includes the response text along with any URL sources. The LLM response is parsed for individual medical statements while the URL sources are downloaded. Finally, the Source Verification model is asked to determine whether a given medical statement is supported by the source text and to provide a reason for the decision.CC BY-NC-ND · PubMed Central
Fig. 3 An example of the SourceCheckup evaluation framework based on a real response from GPT-4o (RAG). A question is generated based on the contents of a medical reference text. The question is posed to an LLM, and the response is parsed into statements and sources. Each statement-source pair is automatically scored by the Source Verification model as supported (i.e., the source contains evidence to support the statement) or not supported.
Fig. 3 An example of the SourceCheckup evaluation framework based on a real response from GPT-4o (RAG). A question is generated based on the contents of a medical reference text. The question is posed to an LLM, and the response is parsed into statements and sources. Each statement-source pair is automatically scored by the Source Verification model as supported (i.e., the source contains evidence to support the statement) or not supported.CC BY-NC-ND · PubMed Central

Verification

Claims the paper makes

4 Valid · 1 Not valid · 3 Not checked · 78 Own results and interpretations

39 more claims

Create a free account to see all 86 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 30
  • It may state two separate claims as one. 17

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 31 references

Research works printed but not linked: 3

  1. Proposed Taxonomy for Gender Bias in Text - ACL Anthology. Httpsaclanthologyorg › Httpsaclanthologyorg ›.
  2. Chen, H.-T., Xu, F., Arora, S. & Choi, E. Understanding Retrieval Augmentation for Long-Form Question Answering. (2023). Proc. 2024 Conf. Lang. Model. (COLM), (2024).
  3. Eriksen Alexander, V., Sören, M. öller & Jesper, Ryg Use of GPT-4 to diagnose complex clinical cases. NEJM AI 1, AIp2300031 (2023).

Cited by

See all 72

Metadata

Authors
Wu, K., Wu, E., Wei, K., Zhang, A., Casasola, A., Nguyen, T., Riantawan, S., Shi, P., Ho, D., & Zou, J.
Affiliations
  • Stanford University (US)
  • University of Southern California (US)
  • Loma Linda University (US)
Corresponding author
Zou, James
Venue
Nature Communications
Published
2025-04-16
Volume
16
Issue
1
Pages
3615-3615
ISSN
2041-1723 (ISSN-L)
OpenAlex
W4409505176
Cited by
72
References
31
Citations by year
2026: 52 · 2025: 20
FWCI
17.18
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Communications (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
large language models, citation evaluation, medical reference, source relevance, automated assessment, medical queries, GPT-4o, clinical reliability
MeSH terms
Humans, Language, Natural Language Processing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: An automated framework for assessing how well LLMs cite relevant medical references. https://app.cochat.ai/papers/10.1038/s41467-025-58551-6
The paper
Wu, K., Wu, E., Wei, K., Zhang, A., Casasola, A., Nguyen, T., Riantawan, S., Shi, P., Ho, D., & Zou, J. (2025). An automated framework for assessing how well LLMs cite relevant medical references. Nature Communications. https://doi.org/10.1038/s41467-025-58551-6

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.