Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 35 of the 36 references it cites were found in a registry or an index.

Flagging incorrect nucleotide sequence reagents in biomedical papers: To what extent does the leading publication format impede automatic error detection?

Open access

Abstract

In an idealised vision of science the scientific literature is error-free. Errors reported during peer review are supposed to be corrected prior to publication, as further research establishes new knowledge based on the body of literature. It happens, however, that errors pass through peer review, and a minority of cases errata and retractions follow. Automated screening software can be applied to detect errors in manuscripts and publications. The contribution of this paper is twofold. First, we designed the erroneous reagent checking () benchmark to assess the accuracy of fact-checkers screening biomedical publications for dubious mentions of nucleotide sequence reagents. It comes with a test collection comprised of 1679 nucleotide sequence reagents that were curated by biomedical experts. Second, we benchmarked our own screening software called Seek&Blastn with three input formats to assess the extent of performance loss when operating on various publication formats. Our findings stress the superiority of markup formats (a 79% detection rate on XML and HTML) over the prominent PDF format (a 69% detection rate at most) regarding an error flagging task. This is the first published baseline on error detection involving reagents reported in biomedical scientific publications. The benchmark is designed to facilitate the development and validation of software bricks to enhance the reliability of the peer review process.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s11192-020-03463-z

Cites

35 of 36 references verified1 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Synoptic view of the ERC benchmark introduced in this paper, designed to assess the quality of automatic fact-checking in the biomedical literature. A scientific paper is input to the error-detection software. Extracted Claimed facts are extracted from the paper: each one is a nucleotide sequence with its claimed properties (non-targeting vs. targeting with associated target). It is then matched to Computed Endorsed facts stemming from accessing an endorsed knowledge source. This comparison leads to a Computed Class { 0,6,7,8 } for each nucleotide sequence. The benchmark aims to compute quality metrics by comparing system outputs to the Ground Truth
Fig. 1 Synoptic view of the ERC benchmark introduced in this paper, designed to assess the quality of automatic fact-checking in the biomedical literature. A scientific paper is input to the error-detection software. Extracted Claimed facts are extracted from the paper: each one is a nucleotide sequence with its claimed properties (non-targeting vs. targeting with associated target). It is then matched to Computed Endorsed facts stemming from accessing an endorsed knowledge source. This comparison leads to a Computed Class { 0,6,7,8 } for each nucleotide sequence. The benchmark aims to compute quality metrics by comparing system outputs to the Ground TruthCC-BY · From the PDF
Image 2
CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 35 references

Research works printed but not linked: 1

  1. Brischoux, F., & Legagneux, P. (2009). Don’t format manuscripts. The Scientist, 23(7), 24.

Cited by

See all 6

Metadata

Authors
Labbé, C., Cabanac, G., West, R. A., Gautier, T., Favier, B., & Byrne, J. A.
Affiliations
  • Institut polytechnique de Grenoble (FR)
  • Centre National de la Recherche Scientifique (FR)
  • The University of Sydney (AU)
  • Translational Innovation in Medicine and Complexity (FR)
Corresponding author
Labbé, Cyril
Venue
Scientometrics
Published
2020-05-22
Volume
124
Issue
2
Pages
1139-1156
ISSN
  • 0138-9130 (ISSN-L)
  • 1588-2861
OpenAlex
W3026405240
Cited by
6
References
35
Citations by year
2025: 1 · 2024: 1 · 2023: 1 · 2022: 1 · 2021: 2
FWCI
0.99
Type
journal article
Language
en
Open access
Licence
Locations
  • Scientometrics (publishedVersion, cc-by)
  • HAL (submittedVersion, cc-by-sa)
  • RePEc: Research Papers in Economics (submittedVersion)
Funders
  • Office of research integrity (ORIIR180038)
  • Post-Truth Initiative, a Sydney University Research Excellence Initiative (SREI 2020)
Topics
  • Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
  • Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
  • Social Sciences › Library and Information Sciences › Publishing and Scholarly Communication
Keywords
automated fact-checking, publication formats, PDF, XML, HTML, peer review, biomedical literature, benchmarking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Flagging incorrect nucleotide sequence reagents in biomedical papers: To what extent does the leading publication format impede automatic error detection? https://app.cochat.ai/papers/10.1007/s11192-020-03463-z
The paper
Labbé, C., Cabanac, G., West, R. A., Gautier, T., Favier, B., & Byrne, J. A. (2020). Flagging incorrect nucleotide sequence reagents in biomedical papers: To what extent does the leading publication format impede automatic error detection? Scientometrics. https://doi.org/10.1007/s11192-020-03463-z

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.