The DOI matches its Crossref record; 35 of the 36 references it cites were found in a registry or an index.
Flagging incorrect nucleotide sequence reagents in biomedical papers: To what extent does the leading publication format impede automatic error detection?
Open accessAbstract
In an idealised vision of science the scientific literature is error-free. Errors reported during peer review are supposed to be corrected prior to publication, as further research establishes new knowledge based on the body of literature. It happens, however, that errors pass through peer review, and a minority of cases errata and retractions follow. Automated screening software can be applied to detect errors in manuscripts and publications. The contribution of this paper is twofold. First, we designed the erroneous reagent checking () benchmark to assess the accuracy of fact-checkers screening biomedical publications for dubious mentions of nucleotide sequence reagents. It comes with a test collection comprised of 1679 nucleotide sequence reagents that were curated by biomedical experts. Second, we benchmarked our own screening software called Seek&Blastn with three input formats to assess the extent of performance loss when operating on various publication formats. Our findings stress the superiority of markup formats (a 79% detection rate on XML and HTML) over the prominent PDF format (a 69% detection rate at most) regarding an error flagging task. This is the first published baseline on error detection involving reagents reported in biomedical scientific publications. The benchmark is designed to facilitate the development and validation of software bricks to enhance the reliability of the peer review process.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The 100 flagged C + G articles examined 13 human cancer types, where most (96/100, 96%) examined a single cancer type, typically pancreatic (35/100, 35%) or lung (22/100, 22%) cancer (Supplemental Data 3), reflecting the clinical use of cisplatin and gemcitabine (27, 28).”
From Identification of human gene research articles with wrongly identified nucleotide sequences (2022) · cites this paper as [27]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gapped BLAST and PSI-BLAST: a new generation of protein database search programs
Altschul, S. (1997) · Nucleic Acids Research · doi:10.1093/nar/25.17.3389 · Cited by 73,302 · cited 1×
“The benchmark aims to compute quality metrics by comparing system outputs to the Ground Truth and the reagent–gene homology can be assessed using the BLASTN software (Altschul et al. 1997).”
MethodThe class imbalance problem: A systematic study1
Japkowicz, N., & Stephen, S. (2002) · Intelligent Data Analysis · doi:10.3233/ida-2002-6504 · Cited by 3,295 · cited 1×
“In closing, let us stress the sensitivity of most classification approaches to class imbalance which is a well-known problem in machine learning (Japkowicz and Stephen 2002).”
BackgroundRNA interference by expression of short-interfering RNAs and hairpin RNAs in mammalian cells
Yu, J.-Y., DeRuiter, S. L., & Turner, D. L. (2002) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.092143499 · Cited by 1,039 · cited 1×
“These rules have been determined and validated on the base of laboratory practices (BF, JAB, and TG) and on common siRNA and shRNA rule design (Yu et al. 2002).”
Bibliometrics and Citation Analysis: From the Science Citation Index to Cybermetrics
Nicolaisen, J. (2010) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.21181 · Cited by 652
Bibliometrics and Citation Analysis: From the Science Citation Index to Cybermetrics
Bellis, N. D. (2009) · IRIS UNIMORE (University of Modena and Reggio Emilia) · Cited by 548
The prevalence of statistical reporting errors in psychology (1985–2013)
Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S., & Wicherts, J. M. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0664-2 · Cited by 510 · cited 2×
“These automated approaches include the detection of plagiarism (Citron and Ginsparg 2015), p-value hacking (Nuijten et al. 2016), and discrepancies between the claims presented and current knowledge on the topic (Labbé et al.”
MethodLINNAEUS: A species name identification system for biomedical literature
Gerner, M., Nenadic, G., & Bergman, C. M. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-85 · Cited by 335 · cited 1×
“Text mining also enhances information retrieval tasks by automatically identifying and indexing named entities, such as genes (Galea et al. 2018) and species (Gerner et al. 2010).”
BackgroundLopez, P. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-04346-8_62 · Cited by 242 · cited 3×
“Many programs strive to extract plain text from PDFs: Bast and Korzen (2017) tabulated 14 programs such as the well-known pdftotext,1 Grobid (Lopez 2009), and Icecite (Bast and Korzen 2013).”
BackgroundPublishers withdraw more than 120 gibberish papers
Noorden, R. V. (2014) · Nature · doi:10.1038/nature.2014.14763 · Cited by 170 · cited 1×
“Thousands of papers have been retracted from the literature (Van Noorden 2014; Ledford et al. 2017).”
BackgroundWhat a massive database of retracted papers reveals about science publishing's ‘death penalty'
(2018) · AAAS Articles DO Group · doi:10.1126/science.aav8384 · Cited by 155 · cited 1×
“With the increase in scientific publications and retractions (Brainard and You 2018), there is an urgent need to automate error screening to improve the quality of the literature.”
Background
Research works printed but not linked: 1
- Brischoux, F., & Legagneux, P. (2009). Don’t format manuscripts. The Scientist, 23(7), 24.
Cited by
Identification of human gene research articles with wrongly identified nucleotide sequences
Park, Y., West, R. A., Pathmendra, P., Favier, B., Stoeger, T., Capes-Davis, A., Cabanac, G., Labbé, C., & Byrne, J. A. (2022) · Life Science Alliance · doi:10.26508/lsa.202101203 · Cited by 37
Self-citationByrne, J. A., Park, Y., West, R. A., Capes-Davis, A., Favier, B., Cabanac, G., & Labbé, C. (2021) · Scientometrics · doi:10.1007/s11192-021-03871-9 · Cited by 21
Self-citationSchneider, J., Woods, N. D., Proescholdt, R., Fu, Y., & The RISRS Team (2021) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/ms579 · Cited by 17
Citation network analysis of retractions in molecular biology field
Feng, S., Feng, L., Han, F., Zhang, Y., Ren, Y., Wang, L., & Yuan, J. (2024) · Scientometrics · doi:10.1007/s11192-024-05101-4 · Cited by 12
“Labbe et al. (2020) proposed an erroneous reagent checking (ERC) benchmark to evaluate the accuracy of fact-checkers in screening biomedical publications for dubious mentions of nucleotide sequence reagents.”
Halffman, W., & Horbach, S. P. J. M. (2025) · Minerva · doi:10.1007/s11024-025-09618-7 · Cited by 2
Metadata
- Authors
- Labbé, C., Cabanac, G., West, R. A., Gautier, T., Favier, B., & Byrne, J. A.
- Affiliations
- Institut polytechnique de Grenoble (FR)
- Centre National de la Recherche Scientifique (FR)
- The University of Sydney (AU)
- Translational Innovation in Medicine and Complexity (FR)
- Corresponding author
- Labbé, Cyril
- ORCID
- Venue
- Scientometrics
- Published
- 2020-05-22
- Volume
- 124
- Issue
- 2
- Pages
- 1139-1156
- ISSN
- 0138-9130 (ISSN-L)
- 1588-2861
- OpenAlex
- W3026405240
- Semantic Scholar
- 88acd52ceca7079c45c816b56107c637c2954229
- Cited by
- 6
- References
- 35
- Citations by year
- 2025: 1 · 2024: 1 · 2023: 1 · 2022: 1 · 2021: 2
- FWCI
- 0.99
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Scientometrics (publishedVersion, cc-by)
- HAL (submittedVersion, cc-by-sa)
- RePEc: Research Papers in Economics (submittedVersion)
- Funders
- Office of research integrity (ORIIR180038)
- Post-Truth Initiative, a Sydney University Research Excellence Initiative (SREI 2020)
- Topics
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Social Sciences › Library and Information Sciences › Publishing and Scholarly Communication
- Keywords
- automated fact-checking, publication formats, PDF, XML, HTML, peer review, biomedical literature, benchmarking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Flagging incorrect nucleotide sequence reagents in biomedical papers: To what extent does the leading publication format impede automatic error detection? https://app.cochat.ai/papers/10.1007/s11192-020-03463-z
- The paper
- Labbé, C., Cabanac, G., West, R. A., Gautier, T., Favier, B., & Byrne, J. A. (2020). Flagging incorrect nucleotide sequence reagents in biomedical papers: To what extent does the leading publication format impede automatic error detection? Scientometrics. https://doi.org/10.1007/s11192-020-03463-z
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.