Citation Hallucination Determines Success: An Empirical Comparison of Six Medical AI Research Systems
Open accessThe DOI matches its Crossref record; 12 of the 17 references it cites were found in a registry or an index; 4 of the 4 sources its claims cite were checked for support.
Abstract
Abstract Large language model (LLM) systems can now generate complete research manuscripts, yet their reliability in clinical medicine — where citation accuracy and reporting standards carry direct consequences — has not been systematically assessed. We introduce MedResearchBench, a benchmark of three clinical epidemiology tasks built on NHANES data, and use it to evaluate six AI research systems across six quality dimensions. Evaluation combines programmatic citation verification, rule-based reporting compliance checks, and multi-model LLM judging, providing a more discriminative assessment than conventional single-judge approaches. Citation integrity emerged as the decisive quality dimension. Hallucination rates ranged from 2.9% to 36.8% across systems, and a hard-rule threshold on per-task citation scores capped four of six systems’ total scores at the penalty ceiling. Adding a multi-agent citation verification and repair pipeline to the best-performing system improved its citation integrity score from 40.0 to 90.9 and raised the weighted total from 68.9 to 81.8. Strikingly, a single-model evaluation ranked this system last (55.5), while our three-tier framework ranked it first (81.8) —a complete reversal that exposes the limitations of subjective LLM-only evaluation. These results suggest that programmatic citation verification should be a core metric in future evaluations of AI scientific writing systems, and that multi-agent quality assurance can bridge the gap between fluent text generation and trustworthy scholarship.
Overview
CoChat Summary
TL;DR
The authors introduce MedResearchBench, three NHANES-based clinical epidemiology tasks, and use a three-tier evaluation framework (programmatic citation verification, rule-based STROBE checks, and three-model LLM judging) to compare six AI research systems. Citation integrity proved the most discriminative dimension, and adding a multi-agent citation verification and repair pipeline raised the best system's citation score from 40.0 to 90.9 and its weighted total from 68.9 to 81.8.
Key findings
- Citation hallucination rates ranged from 2.9% (AI Research Army pipeline) to 36.8% (Agent Laboratory), with per-task rates reaching 90.9% for Agent Laboratory on the Metabolic 002 task (30 of 33 references failed or corrupted). The distribution was bimodal: two systems below 10%, five above 11%.
- A hard rule capping total score at 60 points when per-task D1 < 30 was triggered by four systems (Baseline GPT-4, Data-to-Paper, Agent Laboratory, AI-Researcher), leaving AI Research Army (pipeline) first at 81.8, followed by its single-prompt variant (68.9) and AI Scientist (65.9).
- The ablation showed D1 improving from 40.0 to 90.9 (+50.9, a 127% relative improvement) and weighted total from 68.9 to 81.8 (+12.9); D2 was unchanged (53.3), D5 fell slightly (72.3 → 70.7), and D6 was stable (89.4 → 87.6). On Mental 000, the single-prompt version was capped at 52.3 (D1 = 18.5, 82.6% hallucination) versus 74.5 for the pipeline (D1 = 100.0).
- Evaluation methodology reversed rankings: under v1 (single LLM judge, MiniMax-M2.7) AI-Researcher ranked first (86.7) and AI Research Army last (55.5); under v2 (three-tier framework) AI Research Army ranked first (81.8) and AI-Researcher last (58.4).
Methods
- MedResearchBench comprises three clinical epidemiology tasks on publicly available NHANES data: Cardio 000 (sodium intake and hypertension, NHANES 2017–2018, weighted logistic regression with restricted cubic splines), Mental 000 (sleep duration and depression, NHANES 2017–2020, weighted survey logistic regression with mediation), and Metabolic 002 (thyroid function and metabolic syndrome, NHANES 2007–2012, weighted multinomial logistic regression with stratified analyses). Each task supplied a standardized results package with statistical output, sample size/exclusion flow, variable definitions, and sensitivity analyses.
- Six systems were evaluated in seven configurations (Single-LLM Baseline GPT-4, AI Scientist, Data-to-Paper, Agent Laboratory, AI-Researcher, and AI Research Army in single-prompt and pipeline versions) across three tasks, using a three-tier framework: Tier 1 programmatic (D1 Citation Integrity, weight 0.25, verified against CrossRef and PubMed APIs; D2 Numerical Fidelity, 0.20), Tier 2 rule-based (D3 Structural Completeness, 0.15; D4 STROBE Reporting Compliance, 0.15), Tier 3 multi-model LLM judging (D5 Clinical Interpretation, 0.15; D6 Writing Quality, 0.10) by Gemini 3.1 Pro, Claude Sonnet 4.6, and GPT-5.4 at temperature 0.
- Given N = 7 configurations, the authors report descriptive statistics (means, ranges, SDs) and paired differences rather than formal hypothesis testing; evaluations ran March 15–April 1, 2026, with analyses in Python 3.11.
Limitations
- Stated: limited task scope (three observational NHANES epidemiology tasks, not RCTs, qualitative research, or other datasets); citation verification coverage limited to CrossRef and PubMed, risking false negatives for recent, preprint, non-English, or book-chapter references; single-time-point evaluation subject to model/API changes; the pipeline did not improve D2 Numerical Fidelity (53.3 in both versions); high inter-judge variability on D5 (SD up to 21.0); and self-evaluation bias, since the authors developed both AI Research Army and the evaluation framework, including the dimension weights and the a priori D1 < 30 threshold.
- Evident from the text: the transparency note reports that author and venue information for three of five comparison systems (Data-to-Paper, Agent Laboratory, AI-Researcher) in the authors' internal documentation were LLM hallucinations, and that the initial draft of Table 3 contained fabricated citation verification counts that did not match the raw evaluation data — errors caught only by cross-referencing the underlying JSON files.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
3 Valid · 1 Not valid · 0 Not checked · 74 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Recent studies have shown that even state-of-the-art models hallucinate references at rates of 18-90% in long-form scientific text [8, 9]”
What the sources say
- Partly supportedcites [8] sourceExperimental — accuracy not yet measured
The Results (“Extent of fabrication”; page not provided) report that 18% of GPT-4 references were fabricated, supporting the lower end of the stated range. The paper does not report a 90% rate, and its Methods describe the generated texts as short papers rather than long-form scientific texts. (LLM, conf=0.89)
- Partly supportedcites [9] sourceExperimental — accuracy not yet measured
The Main section reports that GPT-4o fabricated citations in 78–90% of cases when asked to cite recent scientific literature. The Limitations of parametric LMs section likewise reports high rates of fabricated titles, but neither passage supports the claimed 18% lower bound. The measured rates concern models without retrieval on the paper’s scientific-literature tasks, not all state-of-the-art models or long-form scientific text generally. (LLM, conf=0.88)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Four systems (Baseline, Data-to-Paper, Agent Laboratory, AI-Researcher) triggered the hard-rule cap on at least one task (per-task D1 < 30 ), rendering their manuscripts fundamentally unreliable as scientific documents despite high scores on other dimensions.”
- Also stated in: § 5.1 Citation Integrity as the Critical Bottleneck · p. 14 · plain
“Second, the hard-rule threshold (D1 < 30 ) affected four of seven systems, preventing otherwise well-written manuscripts from being considered publication-quality.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 5.1 Citation Integrity as the Critical Bottleneck · p. 14 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Adding a multi-agent citation verification and repair pipeline to the best-performing system improved its citation integrity score from 40.0 to 90.9”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“a hard-rule threshold on per-task citation scores capped four of six systems' total scores at the penalty ceiling”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“raised the weighted total from 68.9 to 81.8”
- Also stated in: § 1 Introduction · p. 2 · plain
“improving the weighted total score from 68.9 to 81.8 (+18.7%)”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · p. 2 · plain
32 more claims
Create a free account to see all 78 claims and every source verdict.
- It may state two separate claims as one. 25
- Probably not a substantial claim. 21
- The quote only partly states this claim. 3
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 4.4 Evaluation Methodology Comparison: v1 vs. v2 · p. 13
“AI-Researcher ranked first (86.7) and AI Research Army ranked last (55.5).”
§ 4.4 Evaluation Methodology Comparison: v1 vs. v2 · p. 13
“| Baseline (GPT-4) | 49.5 | 5 | 60.0 | 3 | +2 |”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 4.3 Ablation Study: Pipeline vs. Single-Prompt · p. 11
“the single-prompt version was capped at 52.3 due to D1 = 18 . 5 (82.6% hallucination rate)”
§ 4.3 Ablation Study: Pipeline vs. Single-Prompt · p. 11
“| Army (single) | 64 | 34 | 25 | 3 | 2 | 7.2% |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.2 Citation Hallucination Analysis · p. 9
“two systems achieved below 10% hallucination rates, while the remaining five exceeded 11%.”
§ 4.3 Ablation Study: Pipeline vs. Single-Prompt · p. 11
“| Army (pipeline) | 49 | 45 | 2 | 1 | 1 | 2.9% | | AI Scientist | 43 | 26 | 14 | 2 | 1 | 4.8% | | Army (single) | 64 | 34 | 25 | 3 | 2 | 7.2% |”
Citations
Cites
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2007) · The Lancet · doi:10.1016/s0140-6736(07)61602-x · Cited by 21,320
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
National health and nutrition examination survey: plan and operations, 1999-2010.
Zipf, G., Chiappa, M. M., Porter, K. S., Ostchega, Y., Lewis, B. G., & Dostal, J. (2013) · PubMed · Cited by 1,182
Detecting hallucinations in large language models using semantic entropy
Farquhar, S., Kossen, J., Kuhn, L., & Gal, Y. (2024) · Nature · doi:10.1038/s41586-024-07421-0 · Cited by 934
Fabrication and errors in the bibliographic citations generated by ChatGPT
Walters, W. H., & Wilder, E. I. (2023) · Scientific Reports · doi:10.1038/s41598-023-41032-5 · Cited by 499
The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery
Lu, C., Lu, C., Lange, R. T., Foerster, J., Clune, J., & Ha, D. (2024) · arXiv · doi:10.48550/arxiv.2408.06292 · Cited by 118
AI-Researcher: Autonomous Scientific Innovation
Tang, J., Xia, L., Li, Z., & Huang, C. (2025) · Neural Information Processing Systems · doi:10.48550/arxiv.2505.18705 · Cited by 118
Suchak, T., Aliu, A. E., Harrison, C., Zwiggelaar, R., Geifman, N., & Spick, M. (2025) · PLOS Biology · doi:10.1371/journal.pbio.3003152 · Cited by 95
Synthesizing scientific literature with retrieval-augmented language models
Asai, A., He, J., Shao, R., Shi, W., Singh, A., Chang, J. C., Lo, K., Soldaini, L., Feldman, S., D’Arcy, M., et al. (2026) · Nature · doi:10.1038/s41586-025-10072-4 · Cited by 38
Autonomous LLM-Driven Research — from Data to Human-Verifiable Research Papers
Ifargan, T., Hafner, L., Kern, M., Alcalay, O., & Kishony, R. (2025) · NEJM AI · doi:10.1056/aioa2400555 · Cited by 34
Metadata
- Authors
- Shi, X., Tian, Z., Tan, S., & Wang, X.
- Affiliations
- Shanghai University of Traditional Chinese Medicine (CN)
- Kingyee (Beijing) Technology Co., Ltd., Beijing, China
- Peking University (CN)
- Corresponding author
- Wang, Xiaolong
- Venue
- medRxiv
- Publisher
- openRxiv
- Published
- 2026-04-04
- OpenAlex
- W7149408300
- Cited by
- 0
- References
- 12
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Licence
- public-domain
- creativecommons.org/publicdomain/zero/1.0/ (vor)
- Locations
- medRxiv (acceptedVersion, public-domain)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Library and Information Sciences › Academic Writing and Publishing
- Social Sciences › Education › Academic integrity and plagiarism
- Keywords
- citation hallucination, large language models, citation verification, clinical epidemiology, NHANES, reporting compliance, multi-agent systems, citation integrity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Citation Hallucination Determines Success: An Empirical Comparison of Six Medical AI Research Systems. https://app.cochat.ai/papers/10.64898/2026.04.02.26350091
- The paper
- Shi, X., Tian, Z., Tan, S., & Wang, X. (2026). Citation Hallucination Determines Success: An Empirical Comparison of Six Medical AI Research Systems. medRxiv. https://doi.org/10.64898/2026.04.02.26350091
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.