Algorithmically outsourcing the detection of statistical errors and other problems.
Open accessThe DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Published errors occur in the context of both genuine and fraudulent or fabricated research, where most studies have focussed on the detection and importance of honest errors (41, 42, 43, 44).”
From Identification of human gene research articles with wrongly identified nucleotide sequences (2022) · cites this paper as [42]
In the citing paper
“An automated check of p-values and confidence intervals found up to 5% had large errors, suggesting there are likely tens of thousands of published papers containing undetected errors. 44”
From Automated detection of over- and under-dispersion in baseline tables in randomised controlled trials. (2022) · cites this paper as [44]
Citations
Cites
1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209 · cited 1×
“There has been increasing discussion about a potential “reproducibility crisis” in science (Baker, 2016).”
BackgroundThe prevalence of statistical reporting errors in psychology (1985–2013)
Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S., & Wicherts, J. M. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0664-2 · Cited by 510 · cited 2×
“Our study also found that only 1.44% of recalculated P -values would flip the significance at P ≤ 0.05, which is similar to the 1.56% reported by Nuijten et al (2016).”
InfluentialDoes it take too long to publish research?
Powell, K. (2016) · Nature · doi:10.1038/530148a · Cited by 267 · cited 1×
“Most papers are also sent to multiple journals before they are accepted, whereby a new set of reviewers is solicited each time (Powell, 2016), meaning reviewers know there is a chance their review could be ignored entirely.”
Kovanis, M., Porcher, R., Ravaud, P., & Trinquart, L. (2016) · PLOS ONE · doi:10.1371/journal.pone.0166387 · Cited by 241 · cited 1×
“This has led some to wonder how sustainable the publishing enterprise is as the demand for peer reviewers increasingly exceeds the supply (Kovanis et al , 2016).”
Déjà vu—A study of duplicate citations in Medline
Errami, M., Hicks, J. M., Fisher, W., Trusty, D., Wren, J. D., Long, T. C., & Garner, H. R. (2008) · Bioinformatics · doi:10.1093/bioinformatics/btm574 · Cited by 95 · cited 1×
“Similarly, although algorithms can detect textual similarities (Errami et al , 2008), deciding whether such overlap constitutes plagiarism or duplicate publishing of results is best left to humans.”
Self-citation404 not found: the stability and persistence of URLs published in MEDLINE
Wren, J. D. (2004) · Bioinformatics · doi:10.1093/bioinformatics/btg465 · Cited by 90 · cited 1×
“For example, we found that 1% of clinical trial IDs did not link to a valid study (Wren, 2017), and ~ 12% of URLs are misspelled or misformatted (Wren, 2004).”
BackgroundSelf-citationReporting research antibody use: how to increase experimental reproducibility
Helsby, M. A., Fenn, J. R., & Chalmers, A. D. (2013) · F1000Research · doi:10.12688/f1000research.2-153.v2 · Cited by 29 · cited 1×
“For example, antibodies are often purported to bind to specific proteins, but reproducibility can vary widely, particularly when antibodies are produced by different groups (Helsby et al , 2013).”
Georgescu, C., & Wren, J. D. (2018) · Bioinformatics · doi:10.1093/bioinformatics/btx811 · Cited by 20 · cited 2×
“Error rates from Georgescu and Wren ( 2018 ).”
Self-citationWren, J. D. (2017) · Clinical Trials · doi:10.1177/1740774516669505 · Cited by 4 · cited 1×
“For example, we found that 1% of clinical trial IDs did not link to a valid study (Wren, 2017), and ~ 12% of URLs are misspelled or misformatted (Wren, 2004).”
Self-citationData mining of effect sizes from PubMed abstracts: a cross-study conceptual replication
Monsarrat, P., & Vergnes, J. (2018) · Bioinformatics · doi:10.1093/bioinformatics/bty153 · Cited by 4 · cited 1×
“Recently, our findings were independently replicated by another group using their own data (Monsarrat & Vergnes, 2018).”
Result
Cited by
Identification of human gene research articles with wrongly identified nucleotide sequences
Park, Y., West, R. A., Pathmendra, P., Favier, B., Stoeger, T., Capes-Davis, A., Cabanac, G., Labbé, C., & Byrne, J. A. (2022) · Life Science Alliance · doi:10.26508/lsa.202101203 · Cited by 37
“Published errors occur in the context of both genuine and fraudulent or fabricated research, where most studies have focussed on the detection and importance of honest errors (41, 42, 43, 44).”
Technology assisted research assessment: algorithmic bias and transparency issues
Thelwall, M. A., & Kousha, K. (2023) · Aslib Journal of Information Management · doi:10.1108/ajim-04-2023-0119 · Cited by 19
Barnett, A. (2022) · F1000Research · doi:10.12688/f1000research.123002.1 · Cited by 7
Beyond PRISMA 2020 and AMSTAR 2: Further Actions Are Needed to Deal With Problematic Meta‐Analyses
Baveye, P. C. (2025) · Learned Publishing · doi:10.1002/leap.1666 · Cited by 4
“These observations caused quite a stir in the field (Baker 2016; Wren 2018; Ritchie 2020).”
Automated detection of over- and under-dispersion in baseline tables in randomised controlled trialsPreprint
Barnett, A. G. (2022) · Research Square · doi:10.21203/rs.3.rs-1496802/v1 · Cited by 2
Metadata
- Authors
- Wren, J. D.
- Affiliations
- Oklahoma Medical Research Foundation (US)
- Corresponding author
- Wren, Jonathan D
- ORCID
- Wren, Jonathan D
- Venue
- The EMBO Journal
- Published
- 2018-05-24
- Volume
- 37
- Issue
- 12
- ISSN
- 0261-4189 (ISSN-L)
- 1460-2075
- PMC
- PMC6003655
- OpenAlex
- W2804663639
- Cited by
- 6
- References
- 10
- Citations by year
- 2025: 1 · 2023: 2 · 2022: 3
- FWCI
- 0.00
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- www.springer.com/tdm (tdm)
- www.springer.com/tdm (vor)
- Locations
- The EMBO Journal (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- NSF (ACI‐1345426)
- NIH (5P20GM103636, 1P30AG050911)
- Topics
- Social Sciences › Library and Information Sciences › Academic Writing and Publishing
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- statistical errors, text mining, scientific manuscripts, peer review, commission errors, omission errors, reproducibility
- MeSH terms
- Algorithms, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.