Algorithmic identification of discrepancies between published ratios and their reported confidence intervals and P-values
Open accessAbstract
Motivation: Studies, mostly from the operations/management literature, have shown that the rate of human error increases with task complexity. What is not known is how many errors make it into the published literature, given that they must slip by peer-review. By identifying paired, dependent values within text for reported calculations of varying complexity, we can identify discrepancies, quantify error rates and identify mitigating factors. Results: We extracted statistical ratios from MEDLINE abstracts (hazard ratio, odds ratio, relative risk), their 95% CIs, and their P-values. We re-calculated the ratios and P-values using the reported CIs. For comparison, we also extracted percent–ratio pairs, one of the simplest calculation tasks. Over 486 000 published values were found and analyzed for discrepancies, allowing for rounding and significant figures. Per reported item, discrepancies were less frequent in percent–ratio calculations (2.7%) than in ratio–CI and P-value calculations (5.6–7.5%), and smaller discrepancies were more frequent than large ones. Systematic discrepancies (multiple incorrect calculations of the same type) were higher for more complex tasks (14.3%) than simple ones (6.7%). Discrepancy rates decreased with increasing journal impact factor (JIF) and increasing number of authors, but with diminishing returns and JIF accounting for most of the effect. Approximately 87% of the 81 937 extracted P-values were ≤ 0.05. Conclusion: Using a simple, yet accurate, approach to identifying paired values within text, we offer the first quantitative evaluation of published error frequencies within these types of calculations. Supplementary information: Supplementary data are available at Bioinformatics online.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
S&B combines the measurement of text similarity and fact-checking of reagent identities, and can be applied independently or used in parallel with other tools such as those that detect duplicated images and/or incorrect statistical results.
“Furthermore, as S&B combines the measurement of text similarity [9] and fact-checking of reagent identities, it can either be applied independently or used in parallel with other tools, such as those that detect duplicated images [10], and/or incorrect statistical results [12, 13].”
From Semi-automated fact-checking of nucleotide sequence reagents in biomedical research publications: The Seek & Blastn tool (2019) · cites this paper as [13]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
1 retracted
Retracted Science and the Retraction Index
Fang, F. C., & Casadevall, A. (2011) · Infection and Immunity · doi:10.1128/iai.05661-11 · Cited by 405 · cited 1× · see citing works
“There have been reports of journals with higher impact factors having higher retraction rates, and it has been argued that this, in part may be a consequence of the desire to publish the most striking results (Fang and Casadevall, 2011), but this could also be due to increased scrutiny.”
BackgroundRetractedBelieve it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · see citing works
Policy: NIH plans to enhance reproducibility
Collins, F. S., & Tabak, L. A. (2014) · Nature · doi:10.1038/505612a · Cited by 1,247 · see citing works
Reproducibility in Science: Improving the Standard for Basic and Preclinical Research
Begley, C. G., & Ioannidis, J. P. A. (2015) · Circulation Research · doi:10.1161/circresaha.114.303819 · Cited by 1,035 · see citing works
The prevalence of statistical reporting errors in psychology (1985–2013)
Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S., & Wicherts, J. M. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0664-2 · Cited by 510 · see citing works
Error Detection Processes in Statistical Problem Solving
Allwood, C. M. (1984) · Cognitive Science · doi:10.1207/s15516709cog0804_5 · Cited by 154 · see citing works
404 not found: the stability and persistence of URLs published in MEDLINE
Wren, J. D. (2004) · Bioinformatics · doi:10.1093/bioinformatics/btg465 · Cited by 91 · see citing works
Self-citationError Patterns in Novice and Skilled Transcription Typing
Grudin, J. T. (1983) · Springer eBooks · doi:10.1007/978-1-4612-5470-6_6 · Cited by 74 · see citing works
Statistical Reporting Errors and Collaboration on Statistical Analyses in Psychological Science
Veldkamp, C. L. S., Nuijten, M. B., Dominguez-Alvarez, L., van Assen, M. A. L. M., & Wicherts, J. M. (2014) · PLoS ONE · doi:10.1371/journal.pone.0114876 · Cited by 69 · see citing works
Accuracy of References in Five Biomedical Informatics Journals
Aronsky, D. (2004) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1683 · Cited by 63 · see citing works
Error-Minimizing Mechanisms Boosting or Editing?
Mattson, M. E., & Baars, B. J. (1992) · Cited by 12 · see citing works
Mattson, M. E., & Baars, B. J. (1992) · Springer eBooks · doi:10.1007/978-1-4899-1164-3_11 · Cited by 10 · see citing works
Trends in the production of scientific data analysis resources
Hennessey, J., Georgescu, C., & Wren, J. D. (2014) · BMC Bioinformatics · doi:10.1186/1471-2105-15-s11-s7 · Cited by 6 · see citing works
Self-citationWren, J. D. (2017) · Clinical Trials · doi:10.1177/1740774516669505 · Cited by 4 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 2
- Tsao,Y.C. (1990) A Lexical Study of Sentences Typed by Hearing-Impaired TDD Users. In: Proceedings of the 13th International Symposium on Human Factors in Telecommunications. Turin, Italy.
- Wing,A.M. and Baddeley,A.D. (1980) Spelling Errors in Handwriting: A Corpus and Distributional Analysis. London: Academic Press.
Cited by
Sorted by citations
Labbé, C., Grima, N., Gautier, T., Favier, B., & Byrne, J. A. (2019) · PLOS ONE · doi:10.1371/journal.pone.0213266 · Cited by 55 · see citing works
“To date, research fact checkers have evaluated chemical data [11] or statistical analyses [12, 13], and the application of statistical fact-checkers has identified widespread incorrect reporting of statistical results [12, 13].”
BackgroundMethodTen simple rules on writing clean and reliable open-source scientific software
Hunter-Zinck, H., de Siqueira, A. F., Vásquez, V., Barnes, R. J., & Martinez, C. C. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1009481 · Cited by 48 · see citing works
Byrne, J. A., Grima, N., Capes-Davis, A., & Labbé, C. (2019) · Biomarker Insights · doi:10.1177/1177271919829162 · Cited by 34 · see citing works
The bias for statistical significance in sport and exercise medicine
Borg, D. N., Barnett, A. G., Caldwell, A. R., White, N. M., & Stewart, I. B. (2023) · Journal of science and medicine in sport · doi:10.1016/j.jsams.2023.03.002 · Cited by 32 · see citing works
Assessing and Improving Robustness of Psychological Research Findings in Four Steps
Nuijten, M. B. (2022) · Springer eBooks · doi:10.1007/978-3-031-04968-2_17 · Cited by 23 · see citing works
Byrne, J. A., Park, Y., West, R. A., Capes-Davis, A., Favier, B., Cabanac, G., & Labbé, C. (2021) · Scientometrics · doi:10.1007/s11192-021-03871-9 · Cited by 21 · see citing works
Misinformation: an empirical study with scientists and communicators during the COVID-19 pandemic
Parker, L., Byrne, J. A., Goldwater, M. B., & Enfield, N. J. (2021) · BMJ Open Science · doi:10.1136/bmjos-2021-100188 · Cited by 24 · see citing works
Damen, J. A., Heus, P., Lamberink, H. J., Tijdink, J. K., Bouter, L. M., Glasziou, P., Moher, D., Otte, W. M., Vinkers, C. H., & Hooft, L. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.11.020 · Cited by 18 · see citing works
Examination of CIs in health and medical journals from 1976 to 2019: an observational study
Barnett, A. G., & Wren, J. D. (2019) · BMJ Open · doi:10.1136/bmjopen-2019-032506 · Cited by 18 · see citing works
Self-citationValdez, D., Vorland, C. J., Brown, A. W., Mayo-Wilson, E., Otten, J. M., Ball, R. J., Grant, S., Levy, R., Svetina, D., & Allison, D. B. (2020) · F1000Research · doi:10.12688/f1000research.26594.1 · Cited by 13 · see citing works
The Need for Greater Rigor in Childhood Nutrition and Obesity Research
Wood, A. C., Wren, J. D., & Allison, D. B. (2019) · Archives of Pediatrics and Adolescent Medicine · doi:10.1001/jamapediatrics.2019.0015 · Cited by 12 · see citing works
Self-citationAlgorithmically outsourcing the detection of statistical errors and other problems.
Wren, J. D. (2018) · The EMBO Journal · doi:10.15252/embj.201899651 · Cited by 6 · see citing works
Self-citationvan Aert, R. C. M., Nuijten, M. B., Olsson-Collentine, A., Stoevenbelt, A. H., van den Akker, O. R., Klein, R. A., & Wicherts, J. M. (2023) · Royal Society Open Science · doi:10.1098/rsos.202326 · Cited by 6 · see citing works
Beyond the p Value Dichotomy: Alternatives for Statistical Inference—A Critical Review
Ferreira, M., Câmara, L. C., & Carvas, N. (2026) · Journal of Evaluation In Clinical Practice · doi:10.1111/jep.70373 · Cited by 4 · see citing works
Davis, C. E., Martin, J. G., & Thorpe, S. J. (2020) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2020.00002 · Cited by 3 · see citing works
Investigating the Impact of Gene Cofunctionality in Predicting Gene Mutations ofE. coli
Okwori, M., & Eslami, A. (2020) · IEEE Access · doi:10.1109/access.2020.3023662 · Cited by 3 · see citing works
Castriota, S., Frumento, P., & Suppressa, F. (2025) · Journal of Wine Economics · doi:10.1017/jwe.2025.10077 · Cited by 2 · see citing works
Hersh, W. (2020) · Health informatics · doi:10.1007/978-3-030-47686-1_2 · Cited by 2 · see citing works
Misrepresentation of Scientific Figures
Leaphart, D. M., Cronley, A. C., & Brothers, T. E. (2022) · Springer eBooks · doi:10.1007/978-3-030-99680-2_15
Preprint_vanAert_etal_Covid19Preprint
van Aert, R. C. M., Nuijten, M. B., Olsson-Collentine, A., Stoevenbelt, A. H., van den Akker, O. R., Klein, R. A., & Wicherts, J. M. (2023) · doi:10.31234/osf.io/asbfd
- Authors
- Georgescu, C., & Wren, J. D.
- Affiliations
- Oklahoma Medical Research Foundation (US)
- Corresponding author
- Wren, Jonathan D
- Venue
- Bioinformatics
- Publisher
- Oxford University Press (OUP)
- Published
- 2017-12-22
- Volume
- 34
- Issue
- 10
- Pages
- 1758-1766
- ISSN
- 1367-4803 (ISSN-L)
- 1367-4811
- PMC
- PMC5946902
- OpenAlex
- W2779283988
- Cited by
- 20
- References
- 14
- Citations by year
- 2026: 1 · 2025: 1 · 2023: 3 · 2022: 3 · 2021: 3 · 2020: 4 · 2019: 4 · 2018: 1
- FWCI
- 2.20
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Bioinformatics (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Science Foundation (ACI-1345426)
- National Institutes of Health (U54GM104938 and P20GM103636)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
- Keywords
- calculation errors, confidence intervals, p-values, hazard ratio, odds ratio, relative risk, MEDLINE abstracts, error detection algorithm, task complexity, journal impact factor, number of authors
- MeSH terms
- Algorithms, Humans, Publications, Confidence Intervals, MEDLINE
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
