No evidence of important difference in summary treatment effects between COVID-19 preprints and peer-reviewed publications: a meta-epidemiological study
Open accessAbstract
Objectives Preprints became a major source of research communication during the COVID-19 pandemic. We aimed to evaluate whether summary treatment effect estimates differ between preprint and peer-reviewed journal trials. Study Design and Setting A meta-epidemiological study. Data were derived from the COVID-NMA living systematic review (covid-nma.com) up to July 20, 2022. We identified all meta-analyses evaluating pharmacological treatments vs. standard of care or placebo for patients with COVID-19 that included at least one preprint and one peer-reviewed journal article. Difference in effect estimates between preprint and peer-reviewed journal trials were estimated by the ratio of odds ratio (ROR); ROR <1 indicated larger effects in preprint trials. Results Thirty-seven meta-analyses including 114 trials (44 preprints and 70 peer-reviewed publications) were selected. The median number of randomized controlled trials (RCTs) per meta-analysis was 2 (interquartile range [IQR], 2–4; maximum, 11), median sample size of RCTs was 199 (IQR, 99–478). Overall, there was no statistically significant difference in summary effect estimates between preprint and peer-reviewed journal trials (ROR, 0.88; 95% CI, 0.71–1.09; I 2 = 17.8%; τ 2 = 0.06). Conclusion We did not find an important difference between summary treatment effects of preprints and summary treatment effects of peer-reviewed publications. Systematic reviewers and guideline developers should assess preprint inclusion individually, accounting for risk of bias and completeness of reporting.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Initial data suggest a few differences between trial results on preprint servers and those on final publication.
“Initial data suggest a few differences between trial results on preprint servers and those on final publication.132 133 134”
From SPIRIT 2025 explanation and elaboration: updated guideline for protocols of randomised trials (2025) · cites this paper as [133]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 17 · Cited by 8
Research works printed but not linked: 2
- medRxiv.org -the preprint server for health sciences. Accessed February 3, 2022. https://www.medrxiv.org/about/FAQ/
- Davidson M, Chaimani A, Boutron I. Comparing Numerical Results Between Preprints and Peer-Reviewed Publications of COVID-19 Trials [Conference presentation abstract]. In: Ninth International Congress on Peer Review and Scientific Publication. ; 2022. Accessed July 8, 2023. https://peerreviewcongress.org/abstract/comparing-numerical-results-between- preprints-and-peer-reviewed-publications-of-covid-19-trials/
- Authors
- Davidson, M., Evrenoglou, T., Graña, C., Chaimani, A., & Boutron, I.
- Affiliations
- Inserm (FR)
- Corresponding author
- Davidson, Mauricia
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2023-10
- Volume
- 162
- Pages
- 90-97
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W4386171938
- Cited by
- 8
- References
- 17
- Citations by year
- 2026: 2 · 2025: 2 · 2024: 4
- FWCI
- 2.04
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of Clinical Epidemiology (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- HAL (submittedVersion, cc-by-nc)
- Funders
- Ministère de l'Education Nationale, de l'Enseignement Superieur et de la Recherche
- Agence Nationale de la Recherche
- Université de Paris
- Institut National de la Santé et de la Recherche Médicale
- World Health Organization
- Ministry of Health
- Assistance Publique - Hôpitaux de Paris
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Infectious Diseases › COVID-19 Clinical Research Studies
- Keywords
- COVID-19 pandemic, preprints, peer-reviewed articles, meta-epidemiology, risk of bias, reporting completeness, pharmacotherapy
- MeSH terms
- Preprints as Topic, COVID-19, Humans, Peer Review, Epidemiologic Studies, Sample Size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
