CoChat

Citation Hallucination Determines Success: An Empirical Comparison of Six Medical AI Research Systems

Open access

The DOI matches its Crossref record; 12 of the 17 references it cites were found in a registry or an index; 4 of the 4 sources its claims cite were checked for support.

Abstract

Abstract Large language model (LLM) systems can now generate complete research manuscripts, yet their reliability in clinical medicine — where citation accuracy and reporting standards carry direct consequences — has not been systematically assessed. We introduce MedResearchBench, a benchmark of three clinical epidemiology tasks built on NHANES data, and use it to evaluate six AI research systems across six quality dimensions. Evaluation combines programmatic citation verification, rule-based reporting compliance checks, and multi-model LLM judging, providing a more discriminative assessment than conventional single-judge approaches. Citation integrity emerged as the decisive quality dimension. Hallucination rates ranged from 2.9% to 36.8% across systems, and a hard-rule threshold on per-task citation scores capped four of six systems’ total scores at the penalty ceiling. Adding a multi-agent citation verification and repair pipeline to the best-performing system improved its citation integrity score from 40.0 to 90.9 and raised the weighted total from 68.9 to 81.8. Strikingly, a single-model evaluation ranked this system last (55.5), while our three-tier framework ranked it first (81.8) —a complete reversal that exposes the limitations of subjective LLM-only evaluation. These results suggest that programmatic citation verification should be a core metric in future evaluations of AI scientific writing systems, and that multi-agent quality assurance can bridge the gap between fluent text generation and trustworthy scholarship.

Overview

CoChat Summary

TL;DR

The authors introduce MedResearchBench, three NHANES-based clinical epidemiology tasks, and use a three-tier evaluation framework (programmatic citation verification, rule-based STROBE checks, and three-model LLM judging) to compare six AI research systems. Citation integrity proved the most discriminative dimension, and adding a multi-agent citation verification and repair pipeline raised the best system's citation score from 40.0 to 90.9 and its weighted total from 68.9 to 81.8.

Key findings

  • Citation hallucination rates ranged from 2.9% (AI Research Army pipeline) to 36.8% (Agent Laboratory), with per-task rates reaching 90.9% for Agent Laboratory on the Metabolic 002 task (30 of 33 references failed or corrupted). The distribution was bimodal: two systems below 10%, five above 11%.
  • A hard rule capping total score at 60 points when per-task D1 < 30 was triggered by four systems (Baseline GPT-4, Data-to-Paper, Agent Laboratory, AI-Researcher), leaving AI Research Army (pipeline) first at 81.8, followed by its single-prompt variant (68.9) and AI Scientist (65.9).
  • The ablation showed D1 improving from 40.0 to 90.9 (+50.9, a 127% relative improvement) and weighted total from 68.9 to 81.8 (+12.9); D2 was unchanged (53.3), D5 fell slightly (72.3 → 70.7), and D6 was stable (89.4 → 87.6). On Mental 000, the single-prompt version was capped at 52.3 (D1 = 18.5, 82.6% hallucination) versus 74.5 for the pipeline (D1 = 100.0).
  • Evaluation methodology reversed rankings: under v1 (single LLM judge, MiniMax-M2.7) AI-Researcher ranked first (86.7) and AI Research Army last (55.5); under v2 (three-tier framework) AI Research Army ranked first (81.8) and AI-Researcher last (58.4).

Methods

  • MedResearchBench comprises three clinical epidemiology tasks on publicly available NHANES data: Cardio 000 (sodium intake and hypertension, NHANES 2017–2018, weighted logistic regression with restricted cubic splines), Mental 000 (sleep duration and depression, NHANES 2017–2020, weighted survey logistic regression with mediation), and Metabolic 002 (thyroid function and metabolic syndrome, NHANES 2007–2012, weighted multinomial logistic regression with stratified analyses). Each task supplied a standardized results package with statistical output, sample size/exclusion flow, variable definitions, and sensitivity analyses.
  • Six systems were evaluated in seven configurations (Single-LLM Baseline GPT-4, AI Scientist, Data-to-Paper, Agent Laboratory, AI-Researcher, and AI Research Army in single-prompt and pipeline versions) across three tasks, using a three-tier framework: Tier 1 programmatic (D1 Citation Integrity, weight 0.25, verified against CrossRef and PubMed APIs; D2 Numerical Fidelity, 0.20), Tier 2 rule-based (D3 Structural Completeness, 0.15; D4 STROBE Reporting Compliance, 0.15), Tier 3 multi-model LLM judging (D5 Clinical Interpretation, 0.15; D6 Writing Quality, 0.10) by Gemini 3.1 Pro, Claude Sonnet 4.6, and GPT-5.4 at temperature 0.
  • Given N = 7 configurations, the authors report descriptive statistics (means, ranges, SDs) and paired differences rather than formal hypothesis testing; evaluations ran March 15–April 1, 2026, with analyses in Python 3.11.

Limitations

  • Stated: limited task scope (three observational NHANES epidemiology tasks, not RCTs, qualitative research, or other datasets); citation verification coverage limited to CrossRef and PubMed, risking false negatives for recent, preprint, non-English, or book-chapter references; single-time-point evaluation subject to model/API changes; the pipeline did not improve D2 Numerical Fidelity (53.3 in both versions); high inter-judge variability on D5 (SD up to 21.0); and self-evaluation bias, since the authors developed both AI Research Army and the evaluation framework, including the dimension weights and the a priori D1 < 30 threshold.
  • Evident from the text: the transparency note reports that author and venue information for three of five comparison systems (Data-to-Paper, Agent Laboratory, AI-Researcher) in the authors' internal documentation were LLM hallucinations, and that the initial draft of Table 3 contained fabricated citation verification counts that did not match the raw evaluation data — errors caught only by cross-referencing the underlying JSON files.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.64898/2026.04.02.26350091

Cites

12 of 17 references verified0 citation errors · 5 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

3 Valid · 1 Not valid · 0 Not checked · 74 Own results and interpretations

32 more claims

Create a free account to see all 78 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 25
  • Probably not a substantial claim. 21
  • The quote only partly states this claim. 3

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

3 of 3

Citations

Cites

Show all 17 references

Metadata

Authors
Shi, X., Tian, Z., Tan, S., & Wang, X.
Affiliations
  • Shanghai University of Traditional Chinese Medicine (CN)
  • Kingyee (Beijing) Technology Co., Ltd., Beijing, China
  • Peking University (CN)
Corresponding author
Wang, Xiaolong
Venue
medRxiv
Publisher
openRxiv
Published
2026-04-04
OpenAlex
W7149408300
Cited by
0
References
12
Type
posted content
Language
en
Open access
Locations
medRxiv (acceptedVersion, public-domain)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Library and Information Sciences › Academic Writing and Publishing
  • Social Sciences › Education › Academic integrity and plagiarism
Keywords
citation hallucination, large language models, citation verification, clinical epidemiology, NHANES, reporting compliance, multi-agent systems, citation integrity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Citation Hallucination Determines Success: An Empirical Comparison of Six Medical AI Research Systems. https://app.cochat.ai/papers/10.64898/2026.04.02.26350091
The paper
Shi, X., Tian, Z., Tan, S., & Wang, X. (2026). Citation Hallucination Determines Success: An Empirical Comparison of Six Medical AI Research Systems. medRxiv. https://doi.org/10.64898/2026.04.02.26350091

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.