The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index; 4 of the 10 sources its claims cite were checked for support.
An automated framework for assessing how well LLMs cite relevant medical references
Open accessAbstract
As large language models (LLMs) are increasingly used to address health-related queries, it is crucial that they support their conclusions with credible references. While models can cite sources, the extent to which these support claims remains unclear. To address this gap, we introduce SourceCheckup, an automated agent-based pipeline that evaluates the relevance and supportiveness of sources in LLM responses. We evaluate seven popular LLMs on a dataset of 800 questions and 58,000 pairs of statements and sources on data that represent common medical queries. Our findings reveal that between 50% and 90% of LLM responses are not fully supported, and sometimes contradicted, by the sources they cite. Even for GPT-4o with Web Search, approximately 30% of individual statements are unsupported, and nearly half of its responses are not fully supported. Independent assessments by doctors further validate these results. Our research underscores significant limitations in current LLMs to produce trustworthy medical references. Assessing the degree to which medical large language models reliably convey existing, trustworthy knowledge is crucial. This study introduces SourceCheckup, an automated framework revealing that large language models frequently cite medical references that do not fully support, or even contradict, their responses, showing significant gaps in reliability for clinical use.
Overview
CoChat Summary
TL;DR
SourceCheckup is an automated agent-based pipeline for evaluating whether LLM responses to medical questions cite sources that actually support their statements. Across seven LLMs and 800 questions, it finds that 50–90% of responses are not fully supported by their cited sources, even for GPT-4o with web search.
Key findings
- SourceCheckup showed 88.7% agreement with a consensus of three US-licensed medical experts on source verification, higher than the 86.1% average inter-doctor agreement; no statistically significant difference was found between model and doctor consensus (p = 0.21).
- Among seven evaluated LLMs, GPT-4o (RAG) had the highest citation performance, but only 55% of its responses were fully supported; Gemini Ultra 1.0 (RAG) had 34.5% response-level support, and Gemini Pro (API) had fully supported responses about 10% of the time.
- Models without web access produced valid URLs only 40–70% of the time; GPT-4o (API) produced valid URLs around 70% of the time, while Llama-2-70b and Meditron-7b produced citation URLs <5% and <1% of the time, respectively.
- On HealthSearchQA, GPT-4o (RAG) had 100% citation URL validity, 75.7% statement-level support, and 38.4% response-level support; on Reddit r/AskDocs, response-level support was 31.0%.
- SourceCleanup removed 34.7% (52/150) of unsupported statements entirely; of the remaining 98 modified statements, 85.7% (84/98) were supported after modification, for an aggregate 90.7% (136/150) removed or correctly edited.
Methods
- SourceCheckup has four modules: question generation, LLM question answering, statement and URL source parsing, and source verification. GPT-4o was used as the backbone for question generation, parsing, and source verification, with validation against US-licensed medical doctors.
- The dataset combined 400 real-world medical queries from Reddit’s r/AskDocs with 400 GPT-4o-generated questions from MayoClinic reference documents, yielding 800 questions and about 58,000 statement-source pairs; a clinician-annotated subset of 400 question/answer pairs was used.
- Seven LLMs were evaluated: GPT-4o (RAG), GPT-4o (API), Claude v2.1 (API), Mistral Medium (API), Gemini Ultra 1.0 (RAG), Gemini Pro (API), and Mixtral-8x7b (API); open-source Llama-2-70b and Meditron-7b were also tested but excluded from main results due to low citation URL production. Metrics included source URL validity, statement-level support, and response-level support.
Limitations
- The paper states that the automated pipeline can accrue errors in question generation, statement parsing, citation extraction, and source verification, leading to small fluctuations in final results.
- The paper states that source verification can be ambiguous, as shown by lack of full agreement among the three medical doctors, so individual data points may be noisy.
- The paper states that the 1-1 mapping between statements and sources does not count cases where a statement could be supported by aggregating multiple sources; however, after merging sources for previously unsupported GPT-4o (RAG) statements, 95.1% remained unsupported.
- The paper also notes that URL analysis may be affected by US-based question sources, 404 errors, and paywall access differences across institutions.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
4 Valid · 1 Not valid · 3 Not checked · 78 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“given that medical question datasets such as PubMedQA42 and MedQA43 consist of fixed question sets susceptible to memorization”
§ An automated framework for assessing how well LLMs cite relevant medical references > Methods > Question generation
What the sources say
- Not checked — no full text of the sourcecites [42] Jin, D. et al. What disease does this patient have? A large-scale open domain question answering dataset from medical exams. Appl. Sci. 11, 6421 (2021). source
- Partly supportedcites [43] Jin, Q., Dhingra, B., Liu, Z., Cohen, W. W. & Lu, X. PubMedQA: A Dataset for Biomedical Research Question Answering. (2019). sourceExperimental — accuracy not yet measured
The Abstract (p. 1) identifies PubMedQA as a dataset with a defined collection of question-answer instances, and §3.1 Data Collection (p. 3) gives the sizes of its subsets. The paper does not say those questions are susceptible to memorization. This citation identifies PubMedQA, not MedQA, so it cannot establish that part of the sentence either. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“These models were chosen as they represent current leading LLMs3,36,40,41 as of February 2024.”
§ An automated framework for assessing how well LLMs cite relevant medical references > Methods > LLM evaluations
What the sources say
- Partly supportedcites [3] Singhal, K. et al. Toward expert-level medical question answering with large language models. Nat. Med. 31, 1–8 (2025). sourceExperimental — accuracy not yet measured
The Abstract reports that Med-PaLM 2 approached or exceeded previous state-of-the-art performance on several medical benchmarks, supporting its description as a leading model. The Discussion dates its release to March 2023 and describes subsequent rapid advances, but does not establish that the citing document’s chosen models were leading LLMs as of February 2024. (LLM, conf=0.84)
- Not matchedcites [36] Tatsu. AlpacaEval Leaderboard. (2024).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [40] Eriksen Alexander, V., Sören, M. öller & Jesper, Ryg Use of GPT-4 to diagnose complex clinical cases. NEJM AI 1, AIp2300031 (2023).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“statement-level support of 75.7% (74.0–77.2 95% CI)”
§ An automated framework for assessing how well LLMs cite relevant medical references > Results > Additional validation on HealthSearchQA
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Among the 110 statement-source pairs provided by GPT-4o (RAG), the doctors confirmed that 105 statements are not supported by any source provided by GPT-4o (RAG).”
§ An automated framework for assessing how well LLMs cite relevant medical references > Results > Evaluation of source veracity in LLMs
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Second, the task of source verification can be ambiguous, as shown by the lack of full agreement among the three medical doctors in our study.”
§ An automated framework for assessing how well LLMs cite relevant medical references > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
39 more claims
Create a free account to see all 86 claims and every source verdict.
- Probably not a substantial claim. 30
- It may state two separate claims as one. 17
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ An automated framework for assessing how well LLMs cite relevant medical references > Methods > Statistics and reproducibility
“no data were excluded from the analyses.”
§ An automated framework for assessing how well LLMs cite relevant medical references > Methods > URL source parsing
“Finally, we excluded source contents that exceed the 128 K maximum token length of GPT-4o. This accounted for approximately 0.1% of all downloaded URLs.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ An automated framework for assessing how well LLMs cite relevant medical references > Results > Evaluation of source veracity in LLMs
“mainly driven by its unique ability among the models to have access to the internet via search”
§ An automated framework for assessing how well LLMs cite relevant medical references > Introduction
“Retrieval-augmented generation (RAG)-enabled GPT-4o and Gemini Ultra 1.0, which have search engine access”
Citations
Cites
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · cited 1×
“…commercially available LLMs are currently approved by the FDA for use in medical decision support settings 1 , top-performing LLMs like GPT-4o, Claude, and Med-PaLM have nonetheless demonstrated superior performance over clinicians on medical exams like the US Medical Licensing Exam (USMLE) 2 – 4 .”
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Toward expert-level medical question answering with large language models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03423-7 · Cited by 1,734 · cited 1×
“These models were chosen as they represent current leading LLMs 3,36,40,41 as of February 2024.”
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · doi:10.20944/preprints202105.0498.v1 · Cited by 718
PubMedQA: A Dataset for Biomedical Research Question Answering
Jin, Q., Dhingra, B., Liu, Z., Cohen, W., & Lu, X. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1259 · Cited by 685
Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models
Dahl, M., Magesh, V., Suzgun, M., & Ho, D. E. (2024) · Journal of Legal Analysis · doi:10.1093/jla/laae003 · Cited by 559
Self-citationRegulating ChatGPT and other Large Generative AI Models
Hacker, P., Engel, A., & Mauer, M. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594067 · Cited by 466 · cited 1×
“The need to cite the sources for medical statements goes beyond gaining clinician and patient trust -- there is also an urgent regulatory case as well 19 .”
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-T., Koh, P., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.741 · Cited by 407
AI-Generated Medical Advice—GPT and Beyond
Haupt, C. E., & Marks, M. (2023) · JAMA · doi:10.1001/jama.2023.5321 · Cited by 346 · cited 1×
“Under Section 230 of the Communications Decency Act, websites like Twitter or WebMD are not regulated by the US FDA, since they simply act as an intermediary for, rather than the author of, medical information 37 .”
Research works printed but not linked: 3
- Proposed Taxonomy for Gender Bias in Text - ACL Anthology. Httpsaclanthologyorg › Httpsaclanthologyorg ›.
- Chen, H.-T., Xu, F., Arora, S. & Choi, E. Understanding Retrieval Augmentation for Long-Form Question Answering. (2023). Proc. 2024 Conf. Lang. Model. (COLM), (2024).
- Eriksen Alexander, V., Sören, M. öller & Jesper, Ryg Use of GPT-4 to diagnose complex clinical cases. NEJM AI 1, AIp2300031 (2023).
Cited by
Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models
Dahl, M., Magesh, V., Suzgun, M., & Ho, D. E. (2024) · Journal of Legal Analysis · doi:10.1093/jla/laae003 · Cited by 559
Self-citationRetrieval augmented generation for large language models in healthcare: A systematic review
Amugongo, L. M., Mascheroni, P., Brooks, S., Doering, S., & Seidel, J. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000877 · Cited by 267
Medical large language models are vulnerable to data-poisoning attacks
Alber, D. A., Yang, Z., Alyakin, A., Yang, E., Rai, S., Valliani, A. A., Zhang, J., Rosenbaum, G. R., Amend-Thomas, A. K., Kurland, D. B., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03445-1 · Cited by 242
Ahn, S. (2024) · The Korean Journal of Physiology & Pharmacology · doi:10.4196/kjpp.2024.28.5.393 · Cited by 51
RadioRAG: Online Retrieval–Augmented Generation for Radiology Question Answering
Arasteh, S. T., Lotfinia, M., Bressem, K. K., Siepmann, R. M., Adams, L. C., Ferber, D., Kuhl, C. K., Kather, J. N., Nebelung, S., & Truhn, D. (2025) · Radiology Artificial Intelligence · doi:10.1148/ryai.240476 · Cited by 68
Metadata
- Authors
- Wu, K., Wu, E., Wei, K., Zhang, A., Casasola, A., Nguyen, T., Riantawan, S., Shi, P., Ho, D., & Zou, J.
- Affiliations
- Stanford University (US)
- University of Southern California (US)
- Loma Linda University (US)
- Corresponding author
- Zou, James
- Venue
- Nature Communications
- Published
- 2025-04-16
- Volume
- 16
- Issue
- 1
- Pages
- 3615-3615
- ISSN
- 2041-1723 (ISSN-L)
- PMC
- PMC12003634
- OpenAlex
- W4409505176
- Cited by
- 72
- References
- 31
- Citations by year
- 2026: 52 · 2025: 20
- FWCI
- 17.18
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by-nc-nd
- creativecommons.org/licenses/by-nc-nd/4.0 (tdm)
- creativecommons.org/licenses/by-nc-nd/4.0 (vor)
- Locations
- Nature Communications (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- large language models, citation evaluation, medical reference, source relevance, automated assessment, medical queries, GPT-4o, clinical reliability
- MeSH terms
- Humans, Language, Natural Language Processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: An automated framework for assessing how well LLMs cite relevant medical references. https://app.cochat.ai/papers/10.1038/s41467-025-58551-6
- The paper
- Wu, K., Wu, E., Wei, K., Zhang, A., Casasola, A., Nguyen, T., Riantawan, S., Shi, P., Ho, D., & Zou, J. (2025). An automated framework for assessing how well LLMs cite relevant medical references. Nature Communications. https://doi.org/10.1038/s41467-025-58551-6
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.