Cited by 85Academic integrity and plagiarismscientometrics and bibliometrics researchAcademic Publishing and Open Access

Fabricated citations: an audit across 2·5 million biomedical papers

Open access
Topaz, Maxim, Roguin, Nir, Gupta, Pallavi, Zhang, Zhihong, Peltonen, Laura-Maria·2026·The Lancet

Abstract

Scientific literature depends on the integrity of its references. Each reference implicitly asserts that a verifiable source exists and supports the claims being made. When references point to non-existent studies, readers, reviewers, and policy makers are unable to evaluate the evidence.

Integrity signals

Registry

Crossref · 10.1016/s0140-6736(26)00603-3

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

8 of 9 cited sources checked · 2 supported · 1 flagged · accuracy not yet measured

  • Premisecitation
    § Fabricated citations: an audit across 2•5 million biomedical papers

    Systematic reviews that inform clinical guidelines have included paper mill articles.

    “Paper mill articles have been included in systematic reviews informing clinical guidelines; 10”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [10] Tang, 2025, Citation contamination by paper mill articles in systematic reviews of the life sciences, JAMA Netw Open source

      The Results (Contamination Prevalence; page unavailable) identify 299 life-sciences systematic reviews that incorporated paper mill articles. The Introduction says systematic reviews inform clinical decision-making, but the paper does not establish that the contaminated reviews informed clinical guidelines specifically. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
  • Premisecitation
    § Fabricated citations: an audit across 2•5 million biomedical papers

    LLMs are known to create convincing but non-existent references, a documented failure mode.

    “Large language models (LLMs) generate plausible sounding but fictitious references, a well documented failure mode”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [2] Athaluri, 2023, Exploring the boundaries of reality: investigating the phenomenon of artificial intelligence hallucination in scientific writing through ChatGPT references, Cureus source

      The Results and Discussion report that ChatGPT generated 28 references that could not be found in an extensive search and had no functional DOI, supporting the fictitious-reference failure mode. The study examines ChatGPT, however, and does not establish that LLMs generally produce such references or assess how plausible they sound. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites [3] Walters, 2023, Fabrication and errors in the bibliographic citations generated by ChatGPT, Sci Rep source
  • Premisecitation
    § Fabricated citations: an audit across 2•5 million biomedical papers

    Prior systematic reviews report that about 25% of references in medical journals contain errors.

    “Systematic reviews have found that approximately one in four references in medical journal articles contains errors, 4”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [4] Jergas, 2015, Quotation accuracy in medical journal articles-a systematic review and meta-analysis, PeerJ source
  • Findingown data
    § Fabricated citations: an audit across 2•5 million biomedical papers

    The rate of fabricated references per 10 000 papers grew more than twelvefold, from about 4 in 2023 to 51.3 in Q4 2025 and 56.9 in early 2026.

    “The fabrication rate increased more than 12 times, from approximately four per 10 000 papers in 2023, to 51•3 per 10 000 papers in the fourth quarter of 2025, reaching 56•9 per 10 000 papers in early 2026”
    Quote in the paperSubstantial claim
  • Findingown data
    § Fabricated citations: an audit across 2•5 million biomedical papers

    Review articles showed a 57% higher fabrication rate than other article types (16.7 vs 10.6 per 10 000; p<0.0001).

    “Review articles had a fabrication rate that was 57% higher than other paper types (16•7 per 10 000 vs 10•6 per 10 000; p<0•0001; appendix p 7-8).”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 5
  • May combine two claims. 12
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Fabricated citations: an audit across 2•5 million biomedical papers

    Prior work puts the share of fabricated references among LLM-generated biomedical citations at 30-69%.

    “previous studies estimate that 30-69% of LLM-generated references in biomedical contexts are fabricated.2,3”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites [2] Athaluri, 2023, Exploring the boundaries of reality: investigating the phenomenon of artificial intelligence hallucination in scientific writing through ChatGPT references, Cureus source

      The Results report 28 references that could not be found out of 178 generated, about 16%, not 30–69%. The Discussion distinguishes those fabricated references from 41 real articles whose DOIs were erroneous; the figure 69 is a count of references with DOI problems, not a percentage of fabricated references. (LLM, conf=0.88)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
    • cites [3] Walters, 2023, Fabrication and errors in the bibliographic citations generated by ChatGPT, Sci Rep source

      The Introduction’s “Previous research” section reports that 64% of citations in a radiology study were fabricated, supporting the biomedical context. But its stated typical range is 47–69% across studies, not 30–69% for biomedical references specifically. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
  • May combine two claims.

    Premisecitation
    § Fabricated citations: an audit across 2•5 million biomedical papers

    Such references tend to be well formatted, credited to actual scientists, and dated plausibly, so standard peer review often misses them.

    “These references are often correctly formatted, attributed to real researchers, and bear plausible publication dates, making them difficult to detect by conventional peer review.4”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [4] Jergas, 2015, Quotation accuracy in medical journal articles-a systematic review and meta-analysis, PeerJ source

      The Discussion (‘What may be the reasons for quotation errors’ and ‘What can be done’; page unavailable) addresses misquotation, copied citations, and checks by reviewers. It does not say that suspect references are correctly formatted, name real researchers, bear plausible dates, or evade peer review for those reasons. Searches for fabricated references and those specific features found no supporting passage. (LLM, conf=0.75)

      Experimental — accuracy not yet measured
  • May combine two claims.

    Findingown data
    § Fabricated citations: an audit across 2•5 million biomedical papers

    One 2025 oncology paper on ureteroileal anastomotic techniques had 18 of 30 verified references (60%) fabricated; these were topic-specific, credited to real urologists, and dated 2023 or 2024.

    “A 2025 paper on ureteroileal anastomotic techniques in an open access oncology journal contained 18 (60%) fabricated references of 30 verified; each fabricated reference was tailored to the paper's narrow surgical topic, attributed to real urologists, and bore claimed publication years of 2023 or 2024.8”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [8] Ren, 2025, Comparative analysis of ureteroileal anastomotic stricture rates: Bricker versus Wallace techniques in ileal conduit urinary diversion—a single-surgeon study with BMI-matched design and long-term follow-up excluding cancer recurrence bias, Front Oncol source

      Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.

      Experimental — accuracy not yet measured
  • May combine two claims.

    Findingown data
    § Fabricated citations: an audit across 2•5 million biomedical papers

    The audit verified 97.1 million of 125.6 million references (77%) that had a PMID; the other 23%, mostly non-indexed sources such as websites, books, and grey literature, were left out.

    “Of 125•6 million references, 97•1 million (77%) carried a PMID and were verified; the remaining 23%, predominantly non-indexed references to websites, books, and grey literature, were excluded.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data
    § Fabricated citations: an audit across 2•5 million biomedical papers

    The proportion of papers with at least one fabricated reference rose from about 1 in 2828 in 2023 to 1 in 458 in 2025 and 1 in 277 in the first seven weeks of 2026.

    “In 2023, approximately one in 2828 papers contained at least one fabricated reference.By 2025, this had risen to one in 458 and in the first 7 weeks of 2026, one in 277 papers had at least one fabricated reference.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data
    § Fabricated citations: an audit across 2•5 million biomedical papers

    Among affected papers, 91% (n=2564) had one or two fabricated references, while 246 had three or more.

    “Most affected papers (91%, n=2564) contained one or two fabricated references; 246 contained three or more.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingown data
    § Fabricated citations: an audit across 2•5 million biomedical papers

    The audit found signs of paper mill activity: two authors co-occurred on 11 papers in one surgical journal in 2025, with 15 fabricated references on CRISPR diagnostics, AI-guided nanovaccines, and gut microbiome biomarkers.

    “Beyond individual papers, we identified patterns consistent with paper mill activity: the same two authors appeared across 11 papers in a single surgical journal in 2025, with 15 fabricated references covering CRISPR diagnostics, AI-guided nanovaccines, and gut microbiome biomarkers, all sharing a core coauthorship pair.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data
    § Fabricated citations: an audit across 2•5 million biomedical papers

    The fabricated references did not look obviously wrong: they were topic-specific, well formatted, credited to real researchers, and had plausible dates.

    “The fabricated references we identified were not obviously defective: topically specific, correctly formatted, attributed to real researchers, and beared plausible publication dates.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § Fabricated citations: an audit across 2•5 million biomedical papers

    If a reference points to a study that does not exist, the evidence it supposedly supports is not real.

    “When references point to nonexistent studies, the evidence they claim to support is fictional.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given
    § Fabricated citations: an audit across 2•5 million biomedical papers

    The system detects the problem but does not establish its cause.

    “Our system identifies the problem, not its cause.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § Fabricated citations: an audit across 2•5 million biomedical papers

    Automated checking of references can prevent fabricated citations from entering the published literature.

    “Routine automated verification can close this gap before fabricated references reach the published record.”
    • Also stated in: § Fabricated citations: an audit across 2•5 million biomedical papers · plain
      “Automated reference verification can close this gap.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Study design and conductown data
    § Fabricated citations: an audit across 2•5 million biomedical papers

    The system's precision was 91% (Fleiss' κ=0.71) in a masked 500-entry validation by three independent reviewers; the design measures precision, not recall.

    “Precision of our automated reference verification system was 91% (Fleiss' κ=0•71, indicating moderate agreement in about seven of every ten cases), measured in a 500-entry masked validation with three independent reviewers; this design estimates precision but not recall.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Interpretationreasoning
    § Fabricated citations: an audit across 2•5 million biomedical papers

    The steep rise beginning in mid-2024 matches the expected delay after LLMs became widely used, though paper mill activity and indexing changes may also have played a role.

    “The sharp inflection in mid-2024 coincides with the expected publication lag following widespread LLM adoption, although increased paper mill activity and changes in journal indexing practices might also have contributed.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Premisereasoning
    § Fabricated citations: an audit across 2•5 million biomedical papers

    Because LLMs became widely available in late 2022 and 2023, and journals take 100-200 days from submission to publication, LLM-assisted papers would start appearing in PubMed Central around mid-2024.

    “LLMs became broadly available in late 2022 and 2023; with submission-to-publication times of 100-200 days, 9 LLM-assisted papers would appear in PubMed Central from mid-2024 onward.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [9] Andersen, 2021, Time from submission to publication varied widely for biomedical journals: a systematic review, Curr Med Res Opin source
  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning
    § Fabricated citations: an audit across 2•5 million biomedical papers

    Dropping the 23% of references without PMIDs could bias the estimates either way: fabricated references may be more frequent in non-indexed sources, leading to undercounting, or they may be more common among references with PMIDs.

    “The exclusion of 23% of references with no PMIDs could bias estimates in either direction: fabricated references might be more common among non-indexed sources, including grey literature, websites, and books (undercounting), or they can be preferentially associated with PMIDs”
    Quote in the paperNot a substantial claim
15 of 15
  • The paper relies on this without support.

    Before this study, no systematic audit of reference integrity across the biomedical literature had been done.

    “To our knowledge, no systematic audit of reference integrity across the biomedical literature has been conducted until now.”
  • The paper relies on this without support.

    Major research integrity databases currently lack a separate category for fabricated references.

    “fabricated references do not currently exist as a discrete category in major research integrity databases”
2 of 2

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Recent research suggests that the problem of hallucinated and unreliable citations is already widespread and is becoming increasingly prevalent in the published literature and at scientific conferences.

    “Recent research suggests that this problem is already widespread and is becoming increasingly prevalent in the published literature and at scientific conferences.”

    From Detecting Hallucinated and Suspicious Citations: What Current Tools Can and Cannot Do (2026) · cites this paper as Topaz et al., 2026

    In “Fabricated citations: an audit across 2·5 million biomedical papers” (page not provided), the authors report fabricated references across 2810 published papers and a rising rate from 2023 to early 2026. The audit concerns biomedical papers, not scientific conferences; its mention of conference proceedings describes a search database, not a finding about citations at conferences. (LLM, conf=0.88)

    Experimental — accuracy not yet measured
  • Recent studies have shown that the problem of hallucinated references occurs both across large-scale publication corpora and within conference proceedings.

    “Recent studies have shown that this problem occurs both across large-scale publication corpora and within conference proceedings.”

    From Detecting Hallucinated and Suspicious Citations: What Current Tools Can and Cannot Do (2026) · cites this paper as Topaz et al., 2026

    The paper’s main text (page not supplied) reports finding 4046 fabricated references across 2810 papers in a large-scale biomedical literature audit. It mentions conference proceedings only as material indexed by Google Scholar; it does not report finding fabricated references within conference proceedings. (LLM, conf=0.88)

    Experimental — accuracy not yet measured
  • A large-scale audit published in The Lancet reported a substantial increase in fabricated citations in the biomedical literature.

    “A large-scale audit published in The Lancet reported a substantial increase in fabricated citations in the biomedical literature (Topaz et al., 2026)”
  • The Lancet study does not prove that every fabricated citation was directly created by an LLM, and it suggests that possible causes include LLM hallucinations, paper mill activity, or intentional manipulation.

    “The Lancet study does not prove that every fabricated citation was directly created by an LLM, and it suggests that possible causes include LLM hallucinations, paper mill activity, or intentional manipulation.”
  • The Lancet study found 4,046 fabricated citations in 2,810 biomedical papers.

    “The Lancet study (Topaz et al., 2026) found 4,046 fabricated citations in 2,810 biomedical papers and showed that their prevalence increased from 2023 to 2025/2026.”

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Topaz, M., Roguin, N., Gupta, P., Zhang, Z., & Peltonen, L.-M.
Affiliations
  • VNS Health (US)
  • Ben-Gurion University of the Negev (IL)
  • Columbia University (US)
  • University of Turku (FI)
Corresponding author
Topaz, Maxim
Venue
The Lancet
Publisher
Elsevier BV
Published
2026-05
Volume
407
Issue
10541
Pages
1779-1781
ISSN
  • 0099-5355 (ISSN-L)
  • 0140-6736
OpenAlex
W7160490639
Cited by
85
References
8
Citations by year
2026: 84 · 2012: 1
FWCI
324.04
Type
journal article
Language
en
Update notices
erratum (2026-07-18)
Open access
Locations
  • The Lancet (publishedVersion, other-oa)
  • PubMed (publishedVersion)
Topics
  • Social Sciences › Safety Research › Academic integrity and plagiarism
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
  • Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
Keywords
citation integrity, reference verification, biomedical literature, research integrity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review