CoChat

Assessing citation integrity in biomedical publications: corpus annotation and NLP models

Open access
Sarol, Maria Janina, Ming, Shufan, Radhakrishna, Shruthan, Schneider, Jodi, Kilicoglu, Halil · 2024 · Bioinformatics

The DOI matches its Crossref record; 38 of the 40 references it cites were found in a registry or an index; 3 of the 6 sources its claims cite were checked for support.

Abstract

MOTIVATION: Citations have a fundamental role in scholarly communication and assessment. Citation accuracy and transparency is crucial for the integrity of scientific evidence. In this work, we focus on quotation errors, errors in citation content that can distort the scientific evidence and that are hard to detect for humans. We construct a corpus and propose natural language processing (NLP) methods to identify such errors in biomedical publications. RESULTS: We manually annotated 100 highly-cited biomedical publications (reference articles) and citations to them. The annotation involved labeling citation context in the citing article, relevant evidence sentences in the reference article, and the accuracy of the citation. A total of 3063 citation instances were annotated (39.18% with accuracy errors). For NLP, we combined a sentence retriever with a fine-tuned claim verification model to label citations as ACCURATE, NOT_ACCURATE, or IRRELEVANT. We also explored few-shot in-context learning with generative large language models. The best performing model-which uses citation sentences as citation context, the BM25 model with MonoT5 reranker for retrieving top-20 sentences, and a fine-tuned MultiVerS model for accuracy label classification-yielded 0.59 micro-F1 and 0.52 macro-F1 score. GPT-4 in-context learning performed better in identifying accurate citations, but it lagged for erroneous citations (0.65 micro-F1, 0.45 macro-F1). Citation quotation errors are often subtle, and it is currently challenging for NLP models to identify erroneous citations. With further improvements, the models could serve to improve citation quality and accuracy. AVAILABILITY AND IMPLEMENTATION: We make the corpus and the best-performing NLP model publicly available at https://github.com/ScienceNLP-Lab/Citation-Integrity/.

Overview

CoChat Summary

TL;DR

The paper builds an annotated biomedical corpus for citation integrity, focusing on quotation/citation content errors that are hard for humans to detect, and releases it publicly. It then benchmarks retrieval and classification NLP models, finding that the best system reaches only moderate performance, especially on erroneous citations.

Key findings

  • The authors manually annotated 100 highly-cited biomedical publications and their citations, producing 3063 citation instances; 39.18% were labeled with accuracy errors.
  • The annotated corpus contains 3420 citation context sentences and 3791 evidence sentences; on average, there were 1.12 context and 1.24 evidence sentences per citation.
  • For citation accuracy classification, the best-performing model used citation sentences as context, BM25 + MonoT5 to retrieve the top-20 evidence sentences, and a fine-tuned MultiVerS classifier; it achieved 0.59 micro-F1 and 0.52 macro-F1.
  • GPT-4 in-context learning performed better on accurate citations but worse on errors, with 0.65 micro-F1 and 0.45 macro-F1; in the detailed results, GPT-4 had 0.80 F1 and 0.90 recall for ACCURATE citations, but very poor performance on NOT_ACCURATE.

Methods

  • The study collected 100 highly-cited full-text biomedical research articles from PubMed Central Open Access, then sampled citing articles and annotated citation instances in three phases with five annotators.
  • Annotation labeled three components: citation context in the citing article, evidence sentences in the reference article, and citation accuracy using fine-grained labels later collapsed to ACCURATE / NOT_ACCURATE / IRRELEVANT for modeling.
  • Models included a PubMedBERT sentence classifier for citation context, BM25 + MonoT5 for evidence retrieval, MultiVerS for citation accuracy classification, and GPT-3.5-turbo / GPT-4 few-shot in-context learning; evaluation used precision, recall, F1, recall@k, MRR, and McNemar’s test.

Limitations

  • The paper states that annotation agreement was only fair for evidence sentence annotation and citation accuracy classification; it suggests domain expertise might have improved agreement but was not feasible.
  • The study did not consider whether a citation is significant/perfunctory for the citing article, and it excluded citations whose relevant evidence was in tables, figures, or Supplementary Material.
  • The text makes evident that the modeling task is difficult because retrieval missed many relevant sentences: even with top-20 retrieval, almost half of annotated relevant sentences were missed, and model performance on citation errors was low.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1093/bioinformatics/btae420

Cites

38 of 38 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1. High-level illustration of the steps in the citation accuracy classification pipeline. Context rectangles represent citation context (in citing paper) and Evidence rectangles represent evidence segments (in reference paper).
Figure 1. High-level illustration of the steps in the citation accuracy classification pipeline. Context rectangles represent citation context (in citing paper) and Evidence rectangles represent evidence segments (in reference paper).CC BY · PubMed Central

Verification

Claims the paper makes

2 Valid · 0 Not valid · 2 Not checked · 53 Own results and interpretations

25 more claims

Create a free account to see all 57 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 20
  • Probably not a substantial claim. 11

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 38 references

Research works printed but not linked: 2

  1. Athar, 2014, Technical report
  2. Smith, 1981, Citation analysis, Library Trends, 30, 83

Cited by

See all 19

Metadata

Authors
Sarol, M. J., Ming, S., Radhakrishna, S., Schneider, J., & Kilicoglu, H.
Affiliations
  • University of Illinois Urbana-Champaign (US)
  • School of Information Sciences, University of Illinois Urbana-Champaign , Champaign, IL 61820, United States
Venue
Bioinformatics
Published
2024-06-26
Volume
40
Issue
7
ISSN
  • 1367-4803 (ISSN-L)
  • 1367-4811
OpenAlex
W4400031865
Cited by
12
References
40
Citations by year
2026: 10 · 2025: 1 · 2024: 1
FWCI
4.72
Type
journal article
Language
en
Open access
Locations
  • Bioinformatics (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
  • Office of Research Integrity
  • US Department of Health and Human Services (ORIIR220073)
Topics
  • Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
  • Social Sciences › Education › Academic integrity and plagiarism
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
citation integrity, citation errors, quotation errors, biomedical literature, citation accuracy, corpus annotation, sentence retrieval, claim verification, large language models, in-context learning, GPT-4, scientific evidence
MeSH terms
Humans, Natural Language Processing, Publications, Biomedical Research

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Assessing citation integrity in biomedical publications: corpus annotation and NLP models. https://app.cochat.ai/papers/10.1093/bioinformatics/btae420
The paper
Sarol, M. J., Ming, S., Radhakrishna, S., Schneider, J., & Kilicoglu, H. (2024). Assessing citation integrity in biomedical publications: corpus annotation and NLP models. Bioinformatics. https://doi.org/10.1093/bioinformatics/btae420

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.