Assessing citation integrity in biomedical publications: corpus annotation and NLP models
Open accessThe DOI matches its Crossref record; 38 of the 40 references it cites were found in a registry or an index; 3 of the 6 sources its claims cite were checked for support.
Abstract
MOTIVATION: Citations have a fundamental role in scholarly communication and assessment. Citation accuracy and transparency is crucial for the integrity of scientific evidence. In this work, we focus on quotation errors, errors in citation content that can distort the scientific evidence and that are hard to detect for humans. We construct a corpus and propose natural language processing (NLP) methods to identify such errors in biomedical publications. RESULTS: We manually annotated 100 highly-cited biomedical publications (reference articles) and citations to them. The annotation involved labeling citation context in the citing article, relevant evidence sentences in the reference article, and the accuracy of the citation. A total of 3063 citation instances were annotated (39.18% with accuracy errors). For NLP, we combined a sentence retriever with a fine-tuned claim verification model to label citations as ACCURATE, NOT_ACCURATE, or IRRELEVANT. We also explored few-shot in-context learning with generative large language models. The best performing model-which uses citation sentences as citation context, the BM25 model with MonoT5 reranker for retrieving top-20 sentences, and a fine-tuned MultiVerS model for accuracy label classification-yielded 0.59 micro-F1 and 0.52 macro-F1 score. GPT-4 in-context learning performed better in identifying accurate citations, but it lagged for erroneous citations (0.65 micro-F1, 0.45 macro-F1). Citation quotation errors are often subtle, and it is currently challenging for NLP models to identify erroneous citations. With further improvements, the models could serve to improve citation quality and accuracy. AVAILABILITY AND IMPLEMENTATION: We make the corpus and the best-performing NLP model publicly available at https://github.com/ScienceNLP-Lab/Citation-Integrity/.
Overview
CoChat Summary
TL;DR
The paper builds an annotated biomedical corpus for citation integrity, focusing on quotation/citation content errors that are hard for humans to detect, and releases it publicly. It then benchmarks retrieval and classification NLP models, finding that the best system reaches only moderate performance, especially on erroneous citations.
Key findings
- The authors manually annotated 100 highly-cited biomedical publications and their citations, producing 3063 citation instances; 39.18% were labeled with accuracy errors.
- The annotated corpus contains 3420 citation context sentences and 3791 evidence sentences; on average, there were 1.12 context and 1.24 evidence sentences per citation.
- For citation accuracy classification, the best-performing model used citation sentences as context, BM25 + MonoT5 to retrieve the top-20 evidence sentences, and a fine-tuned MultiVerS classifier; it achieved 0.59 micro-F1 and 0.52 macro-F1.
- GPT-4 in-context learning performed better on accurate citations but worse on errors, with 0.65 micro-F1 and 0.45 macro-F1; in the detailed results, GPT-4 had 0.80 F1 and 0.90 recall for ACCURATE citations, but very poor performance on NOT_ACCURATE.
Methods
- The study collected 100 highly-cited full-text biomedical research articles from PubMed Central Open Access, then sampled citing articles and annotated citation instances in three phases with five annotators.
- Annotation labeled three components: citation context in the citing article, evidence sentences in the reference article, and citation accuracy using fine-grained labels later collapsed to ACCURATE / NOT_ACCURATE / IRRELEVANT for modeling.
- Models included a PubMedBERT sentence classifier for citation context, BM25 + MonoT5 for evidence retrieval, MultiVerS for citation accuracy classification, and GPT-3.5-turbo / GPT-4 few-shot in-context learning; evaluation used precision, recall, F1, recall@k, MRR, and McNemar’s test.
Limitations
- The paper states that annotation agreement was only fair for evidence sentence annotation and citation accuracy classification; it suggests domain expertise might have improved agreement but was not feasible.
- The study did not consider whether a citation is significant/perfunctory for the citing article, and it excluded citations whose relevant evidence was in tables, figures, or Supplementary Material.
- The text makes evident that the modeling task is difficult because retrieval missed many relevant sentences: even with top-20 retrieval, almost half of annotated relevant sentences were missed, and model performance on citation errors was low.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
2 Valid · 0 Not valid · 2 Not checked · 53 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“However, citation inaccuracies and distortions are common (Wager and Middleton 2008, Jergas and Baethge 2015, Pavlovic et al. 2021).”
§ Assessing citation integrity in biomedical publications: corpus annotation and NLP models > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Wager and Middleton 2008 source
- Supportedcites Jergas and Baethge 2015 source
The Discussion reports that all included studies found substantial quotation-error rates and estimates problems with about one fourth of references in medical articles. This supports the citing sentence’s characterization of citation inaccuracies and distortions as common. (LLM, conf=0.94)
- Supportedcites Pavlovic et al. 2021 source
The Abstract reports inaccurate citations in 11% and 15% of articles in two study sets and concludes that they are common in biomedical literature. The Results identify citations of nonexistent findings and inaccurate interpretations, supporting the reference to distortions. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Retrieving top-20 sentences with BM25 + MonoT5 reranker has a recall of 0.53, which indicates significant differences between the annotated and automatically retrieved sentences.”
§ Assessing citation integrity in biomedical publications: corpus annotation and NLP models > 4 Discussion > 4.2 NLP models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We obtained high agreement on citation context annotation (κ = 0.96), as expected, as the agreement was measured at the sentence level and in most cases, citation context sentences are simply the citances.”
§ Assessing citation integrity in biomedical publications: corpus annotation and NLP models > 3 Results > 3.2 Inter-annotator agreement
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results show that even with top-20 sentence retrieval, almost half of the sentences annotated as relevant were missed.”
§ Assessing citation integrity in biomedical publications: corpus annotation and NLP models > 3 Results > 3.3 NLP results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A simplified formulation that used BM25 + MonoT5 reranker for sentence retrieval and MultiVerS for label classification only significantly improved results.”
§ Assessing citation integrity in biomedical publications: corpus annotation and NLP models > 4 Discussion > 4.2 NLP models
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
25 more claims
Create a free account to see all 57 claims and every source verdict.
- It may state two separate claims as one. 20
- Probably not a substantial claim. 11
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“When citations are missing, inaccurate, or fabricated, the resulting break in the evidence chain can obscure logical dependencies, weaken argumentation, and jeopardize academic integrity at the level of both individual papers and the broader literature [10, 22, 26].”
From CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Era (2026) · cites this paper as [22]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Given a report with citation content, tasks such as cited context identification, evidence sentence retrieval, and citation accuracy classification are commonly used to analyze citation quality Sarol et al. (2024).”
From ReportBench: Evaluating Deep Research Agents via Academic Survey Tasks (2025) · cites this paper as Sarol et al. (2024)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Research reports often include a substantial amount of citation-related content, and evaluating the precision and standardization of these citations plays a crucial role in assessing the overall quality of the report Sarol et al. (2024).”
From ReportBench: Evaluating Deep Research Agents via Academic Survey Tasks (2025) · cites this paper as Sarol et al. (2024)
Experimental — accuracy not yet measured
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 1×
“Prompting the model to reason about the steps to arrive at a conclusion is shown to further improve performance for complex reasoning tasks (Wei et al. 2022).”
BackgroundThe probabilistic relevance framework: BM25 and beyond
Robertson, S., & Zaragoza, H. (2009) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000019 · Cited by 3,374 · cited 1×
“� BM25 þ MonoT5 reranker : To identify the most semantically relevant sentences, we leverage the simple BM25 retrieval model (Robertson and Zaragoza 2009) to obtain the top 60 sentences and the MonoT5 reranker (Nogueira et al. 2020) to rerank these sentences.”
MethodDomain-specific language model pretraining for biomedical natural language processing
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3458754 · Cited by 2,170 · cited 4×
“For this classification task, we fine-tuned the PubMedBERT model (Gu et al. 2021).”
MethodInfluentialWhat do citation counts measure? A review of studies on citing behavior
Bornmann, L., & Daniel, H. (2008) · Journal of Documentation · doi:10.1108/00220410810844150 · Cited by 1,607 · cited 2×
“While most work focuses on citation counts (Bornmann and Daniel 2008), there is increasing focus on describing and classifying relationships between the citing and reference papers (ie, citation content analysis ) (Zhang et al. 2013, Tahamtan and Bornmann 2019), which aims to provide a more…”
BackgroundA review of the literature on citation impact indicators
Waltman, L. (2016) · Journal of Informetrics · doi:10.1016/j.joi.2016.02.007 · Cited by 1,356 · cited 1×
“Citation-based metrics, such as h-index and journal impact factor, are the primary mechanisms for measuring the scholarly contributions of researchers and journals (Waltman 2016).”
BackgroundAddiction Rare in Patients Treated with Narcotics
(1980) · New England Journal of Medicine · doi:10.1056/nejm198001103020221 · Cited by 1,106 · cited 1×
“A recent bibliometric analysis suggested that inaccurate citations to a letter published in the New England Journal of Medicine in 1980 (Porter and Jick 1980) may have contributed to the opioid epidemic in the United States (Leung et al. 2017).”
BackgroundHow citation distortions create unfounded authority: analysis of a citation network
Greenberg, S. A. (2009) · BMJ · doi:10.1136/bmj.b2680 · Cited by 499 · cited 3×
“Greenberg (2009) has demonstrated how speculative claims have become accepted facts in Alzheimer’s disease research through citation distortions.”
BackgroundInfluential
Research works printed but not linked: 2
- Athar, 2014, Technical report
- Smith, 1981, Citation analysis, Library Trends, 30, 83
Cited by
ReportBench: Evaluating Deep Research Agents via Academic Survey Tasks
Li, M., Zeng, Y., Cheng, Z., Ma, C., & Jia, K. (2025) · arXiv · doi:10.48550/arxiv.2508.15804 · Cited by 1
Zhang, T. M., & Abernethy, N. F. (2025) · arXiv · doi:10.48550/arxiv.2505.23824 · Cited by 1
Analysis of English Sentence Dependency Syntax Based on Python
Wang, Y. (2024) · International Conference on Power, Electrical Engineering, Electronics and Control (PEEEC) · doi:10.1109/peeec63877.2024.00222
“Word vector is a method of representing natural language as a mathematical vector matrix, which is an important method for natural language processing, modeling, feature extraction, etc. [9], laying the foundation for subsequent analysis.”
Writing Systems That Work: Dissertation Momentum, Coauthoring, and AI-Supported Revision
(2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-9340-7.ch012
Detection and interpretability analysis of quotation errors by large language models
Huang, B., Zhang, Y., Huang, S., & Zhang, C. (2026) · The Electronic Library · doi:10.1108/el-11-2025-0464
“…of quotation errors can be categorized into three types: similarity-based methods(Liu et al ., 2024), text classification methods based on small-scale pre-trained language models (PLMs) (Sarol et al ., 2024), and prompt-based approaches with large language models (LLMs)(Sarol et al ., 2024).”
Metadata
- Authors
- Sarol, M. J., Ming, S., Radhakrishna, S., Schneider, J., & Kilicoglu, H.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- School of Information Sciences, University of Illinois Urbana-Champaign , Champaign, IL 61820, United States
- Venue
- Bioinformatics
- Publisher
- Oxford University Press (OUP)
- Published
- 2024-06-26
- Volume
- 40
- Issue
- 7
- ISSN
- 1367-4803 (ISSN-L)
- 1367-4811
- PMC
- PMC11231046
- OpenAlex
- W4400031865
- Cited by
- 12
- References
- 40
- Citations by year
- 2026: 10 · 2025: 1 · 2024: 1
- FWCI
- 4.72
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Bioinformatics (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- Office of Research Integrity
- US Department of Health and Human Services (ORIIR220073)
- Topics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Social Sciences › Education › Academic integrity and plagiarism
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- citation integrity, citation errors, quotation errors, biomedical literature, citation accuracy, corpus annotation, sentence retrieval, claim verification, large language models, in-context learning, GPT-4, scientific evidence
- MeSH terms
- Humans, Natural Language Processing, Publications, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Assessing citation integrity in biomedical publications: corpus annotation and NLP models. https://app.cochat.ai/papers/10.1093/bioinformatics/btae420
- The paper
- Sarol, M. J., Ming, S., Radhakrishna, S., Schneider, J., & Kilicoglu, H. (2024). Assessing citation integrity in biomedical publications: corpus annotation and NLP models. Bioinformatics. https://doi.org/10.1093/bioinformatics/btae420
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
