CoChat
Sign Up
Cited by 44Academic Publishing and Open AccessMeta-analysis and systematic reviewsCOVID-19 Clinical Research Studies

Consistency of covid-19 trial preprints with published reports and impact for decision making: retrospective review

Open access
Zeraatkar, Dena, Pitre, Tyler, Leung, Gareth, Cusano, Ellen, Agarwal, Arnav, Khalid, Faran, Escamilla, Zaira, Cooper, Matthew Adam +11 more·2022·BMJ Medicine
DOI 10.1136/bmjmed-2022-000309PMC PMC9951374

Published version of a preprint

Open this paper in CoChat — free account

Abstract

Objective: To assess the trustworthiness (ie, complete and consistent reporting of key methods and results between preprint and published trial reports) and impact (ie, effects of preprints on meta-analytic estimates and the certainty of evidence) of preprint trial reports during the covid-19 pandemic. Design: Retrospective review. Data sources: World Health Organization covid-19 database and the Living Overview of the Evidence (L-OVE) covid-19 platform by the Epistemonikos Foundation (up to 3 August 2021). Main outcome measures: Comparison of characteristics of covid-19 trials with and without preprints, estimates of time to publication of covid-19 preprints, and description of differences in reporting of key methods and results between preprints and their later publications. For the effects of eight treatments on mortality and mechanical ventilation, the study comprised meta-analyses including preprints and excluding preprints at one, three, and six months after the first trial addressing the treatment became available either as a preprint or publication (120 meta-analyses in total, 60 of which included preprints and 60 of which excluded preprints) and assessed the certainty of evidence using the GRADE framework. Results: Of 356 trials included in the study, 101 were only available as preprints, 181 as journal publications, and 74 as preprints first and subsequently published in journals. The median time to publication of preprints was about six months. Key methods and results showed few important differences between trial preprints and their subsequent published reports. Apart from two (3.3%) of 60 comparisons, point estimates were consistent between meta-analyses including preprints versus those excluding preprints as to whether they indicated benefit, no appreciable effect, or harm. For nine (15%) of 60 comparisons, the rating of the certainty of evidence was different when preprints were included versus being excluded—the certainty of evidence including preprints was higher in four comparisons and lower in five comparisons. Conclusion: No compelling evidence indicates that preprints provide results that are inconsistent with published papers. Preprints remain the only source of findings of many trials for several months—an unsuitable length of time in a health emergency that is not conducive to treating patients with timely evidence. The inclusion of preprints could affect the results of meta-analyses and the certainty of evidence. Evidence users should be encouraged to consider data from preprints.

AI snapshot

Written from the paper’s full text.

TL;DR

In a retrospective review of 356 covid-19 trials, the authors found no compelling evidence of important discrepancies between trial preprints and their later published reports. However, preprints often remained the only source of trial findings for months, and including them in meta-analyses sometimes changed the certainty of evidence.

Key findings

  • Of 356 eligible trials, 101 were only available as preprints, 181 only as journal publications, and 74 were first preprints and subsequently published. Median time to publication of preprints was 5.9 months; at six months half remained unpublished and at one year a third remained unpublished.
  • Among 74 preprint-publication pairs, 42 (56.8%) had one or more discrepancies in key methods or results, with a median of 1 (interquartile range 0–2) discrepancy per pair. The most common method discrepancy was allocation concealment (8 trials), but overall risk-of-bias rating changed for only one trial; the most common result discrepancy was changed outcome data (20 trials, 27.0%), often likely due to accumulating events.
  • Across 120 meta-analyses (60 including preprints, 60 excluding), point estimates were consistent as to benefit, no appreciable effect, or harm in 58 of 60 (96.7%) comparisons; 2 (3.3%) differed. Certainty of evidence differed in 9 of 60 (15%) comparisons: higher with preprints in 4 and lower with preprints in 5; 6 of these could have affected decision making.
  • Four retracted trials were identified; one had been posted as a preprint and the remainder were published in peer-reviewed journals.

Methods

  • Retrospective review using the WHO covid-19 database and the Epistemonikos Living Overview of the Evidence platform up to 3 August 2021, including randomized trials of drug treatments, antiviral antibodies and cellular treatments, prophylaxis, placebo, or standard care.
  • Pairs of reviewers independently screened, extracted data, linked preprints to subsequent publications, assessed risk of bias with RoB 2.0, and collected differences in key methods and results. Time to publication was assessed with Kaplan-Meier curves and log-rank tests; trial characteristics were compared using differences in proportions, z tests, and Mann-Whitney U tests.
  • For eight treatments and the outcomes mortality and mechanical ventilation, the authors conducted pairwise random-effects meta-analyses including versus excluding preprints at one, three, and six months after the first trial became available, plus the current timepoint, and assessed certainty of evidence using GRADE (120 meta-analyses total).

Limitations

  • The paper states that generalisability is limited to covid-19; preprints subsequently published may be the most rigorous and not representative of all trial preprints, especially unpublished ones; the WHO database does not include all preprint servers; assessment was limited to outcomes in the living systematic reviews and network meta-analyses (not adverse events); retractions were too few for confident conclusions; impact was assessed only on meta-analytic estimates, certainty of evidence, and decision making, not citations or Altmetrics; GRADE application is subjective; and the pairwise/minimally contextualised approach may differ from network meta-analyses and fully contextualised frameworks.
  • The paper also states that time to publication may be overestimated if some authors did not attempt journal publication or underestimated if preprints were posted later in the submission process, and that differences in discussion or conclusion sections were not assessed.
  • Evident from the text: the protocol was withdrawn from publication and presented in a supplement rather than published before study completion; no sample size calculation was performed; and searches were pragmatically limited after 3 August 2021.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1136/bmjmed-2022-000309

Cites

51 of 51 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 31 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    The median time to publication of preprints was about six months.

    “The median time to publication of preprints was about six months.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The WHO covid-19 database and the L-OVE covid-19 platform comprehensively capture global published and preprint literature on covid-19.
  • Findingown data

    Apart from two (3.3%) of 60 comparisons, point estimates were consistent between meta-analyses including preprints versus those excluding preprints as to whether they indicated benefit, no appreciable effect, or harm.

    “Apart from two (3.3%) of 60 comparisons, point estimates were consistent between meta-analyses including preprints versus those excluding preprints as to whether they indicated benefit, no appreciable effect, or harm.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The WHO covid-19 database and the L-OVE covid-19 platform comprehensively capture global published and preprint literature on covid-19.
  • Findingown data

    Preprints remain the only source of findings of many trials for several months.

    “Preprints remain the only source of findings of many trials for several months”
    • Also stated in: § Main findings · boosted
      “We show that preprints remain the only source of findings of many trials for several months.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The WHO covid-19 database and the L-OVE covid-19 platform comprehensively capture global published and preprint literature on covid-19.
  • Findingown data

    Key methods and results showed few important differences between trial preprints and their subsequent published reports.

    “Key methods and results showed few important differences between trial preprints and their subsequent published reports.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The WHO covid-19 database and the L-OVE covid-19 platform comprehensively capture global published and preprint literature on covid-19.
  • Findingown data

    Preprints that received government funding were published faster than preprints that did not receive government funding.

    “Preprints that received government funding, reported on inpatients, or reported on patients with severe disease were published faster than preprints that did not receive government funding, reported on outpatients, or reported on patients with mild or moderate disease.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 35
  • Classified as probably not a substantial claim. 47

  • May combine two claims.

    Findingown data

    For nine (15%) of 60 comparisons, the rating of the certainty of evidence was different when preprints were included versus being excluded-the certainty of evidence including preprints was higher in four comparisons and lower in five comparisons.

    “For nine (15%) of 60 comparisons, the rating of the certainty of evidence was different when preprints were included versus being excluded—the certainty of evidence including preprints was higher in four comparisons and lower in five comparisons.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The WHO covid-19 database and the L-OVE covid-19 platform comprehensively capture global published and preprint literature on covid-19.

  • May combine two claims.

    Findingown data

    Of 356 trials included in the study, 101 were only available as preprints, 181 as journal publications, and 74 as preprints first and subsequently published in journals.

    “Of 356 trials included in the study, 101 were only available as preprints, 181 as journal publications, and 74 as preprints first and subsequently published in journals.”
    • Also stated in: § Trial characteristics · plain
      “As of 3 August 2021, we identified 356 eligible trials, of which 101 were only available as preprints, 181 only available as journal publications, and 74 first available as preprints and subsequently published as journal articles.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The WHO covid-19 database and the L-OVE covid-19 platform comprehensively capture global published and preprint literature on covid-19.

  • May combine two claims.

    Findingown data

    One of the retracted trials was posted as a preprint and the remainder were published in peer reviewed journals.

    “One of the retracted trials was posted as a preprint29 30 and the remainder were published in peer reviewed journals.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Despite discrepancies in outcome data being common, results were similar between preprints and publications both in magnitude and precision.

    “Despite discrepancies in outcome data being common, results were similar between preprints and publications both in magnitude and precision.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Of the retracted trials, two reported on (hydroxy)chloroquine, two on favipiravir, and two on ivermectin.

    “Two trials reported on (hydroxy)chloroquine,29–32 two on favipiravir,31–34 and two on ivermectin.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    Half of all preprints remain unpublished at six months and a third at one year.

    “Half of all preprints, for example, remain unpublished at six months and a third at one year”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Among all preprints with differences in outcomes, differences in relative effects did not exceed 15%, except for one trial with very few events that included just one additional event in the publication.

    “Among all preprints with differences in outcomes, differences in relative effects did not exceed 15%, except for one trial with very few events that included just one additional event in the publication.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingcitation

    One of the retracted trials was retracted when the authors noticed an error in their analysis, and the remainder were retracted owing to concerns about data fabrication or falsification.

    “One of the trials was retracted when the authors noticed an error in their analysis35 36 and the remainder were retracted owing to concerns about data fabrication or falsification”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Nearly two thirds of trials were at high risk of bias, primarily because of their open label design.

    “Nearly two thirds of trials were at high risk of bias, primarily because of their open label design.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The WHO covid-19 database and the L-OVE covid-19 platform comprehensively capture global published and preprint literature on covid-19.

  • May combine two claims.

    Findingown data

    In total, 120 meta-analyses were performed and assessed for certainty of evidence, 60 of which included preprints and 60 of which excluded preprints.

    “In total, we performed and assessed the certainty of evidence of 120 meta-analyses, 60 of which included preprints and 60 of which excluded preprints.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Compared with published trials without preprints, trials only available as preprints and trials first available as preprints and then subsequently published were more likely to be registered.

    “Compared with published trials without preprints, trials only available as preprints and trials first available as preprints and then subsequently published were more likely to be registered”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The WHO covid-19 database and the L-OVE covid-19 platform comprehensively capture global published and preprint literature on covid-19.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Premisenone given

    Including preprints could reduce the credibility of evidence syntheses and risk serious errors if important differences appear in published reports.

    “including preprints could reduce the credibility of evidence syntheses and risk serious errors if important differences appear in published reports”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Preprints might result in the dissemination of provisional findings that contain important errors that, presumably, published papers do not.

    “preprints might result in the dissemination of provisional findings that contain important errors that, presumably, published papers do not”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [15] source

      Not checked in the free run.

    • cites [16] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Growing interest in preprints predates the covid-19 pandemic.

    “Growing interest in preprints predates the covid-19 pandemic.6 7”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [6] source

      Not checked in the free run.

    • cites [7] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationnone given

    Preprints that are subsequently published in journals might be the most rigorous and might not represent all trial preprints-particularly those that remain unpublished.

    “preprints that are subsequently published in journals might be the most rigorous and might not represent all trial preprints—particularly those that remain unpublished”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The inclusion of preprints could affect the results of meta-analyses and the certainty of evidence.

    “The inclusion of preprints could affect the results of meta-analyses and the certainty of evidence.”
    • Also stated in: § WHAT THIS STUDY ADDS · hedged
      “The inclusion of preprints could affect the results of meta-analyses and the certainty (quality) of evidence”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The WHO covid-19 database and the L-OVE covid-19 platform comprehensively capture global published and preprint literature on covid-19.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The medical community has been cautious regarding the adoption of preprints.

    “the medical community has been cautious regarding their adoption.15 16”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [15] source

      Not checked in the free run.

    • cites [16] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given

    Including preprints might increase the precision of estimates, allow timely dissemination of research, and minimise the effects of publication bias.

    “including preprints might increase the precision of estimates, allow timely dissemination of research, and minimise the effects of publication bias”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Previous studies were restricted to only publications up to August and October 2020, which is not representative of the current landscape of covid-19 research and does not include the majority of evidence being currently used to guide covid-19 care, including critical trials of the effects of corticosteroids and interleukin 6 receptor blockers.

    “These studies were, however, restricted to only publications up to August and October 2020—which is not representative of the current landscape of covid-19 research and which does not include the majority of evidence being currently used to guide covid-19 care, including critical trials of the effects of corticosteroids and interleukin 6 receptor blockers.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [1] Horby P , Lim WS , Emberson J . Effect of dexamethasone in hospitalized patients with COVID-19: preliminary report. medRxiv 2020.doi:10.1056/NEJMoa2021436 source

      Not checked in the free run.

    • cites [2] Horby P , Lim WS , Emberson JR . Dexamethasone in hospitalized patients with Covid-19 - preliminary report. N Engl J Med 2020.doi:10.1056/NEJMoa2021436 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    Preprints (and publications) might describe untrustworthy trials with fabricated or falsified data.

    “preprints (and publications) might describe untrustworthy trials with fabricated or falsified data”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [37] source

      Not checked in the free run.

    • cites [38] source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    The choice of timepoints was informed by timeframes within which guideline developers needed to issue recommendations.

    “The choice of timepoints was informed by timeframes within which guideline developers needed to issue recommendations.20”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [20] source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    To assess risk of bias, reviewers, after training and calibration exercises, used a revision of the Cochrane tool for assessing risk of bias in randomised trials (RoB 2.0).

    “To assess risk of bias, reviewers, after training and calibration exercises, used a revision of the Cochrane tool for assessing risk of bias in randomised trials (RoB 2.0)22”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [22] source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    We used a minimally contextualised approach to make judgments about imprecision.

    “We used a minimally contextualised approach to make judgments about imprecision.27”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [27] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    Peer review probably looks at only the transparency of trial reports and interpretation of results.

    “peer review probably looks at only the transparency of trial reports and interpretation of results”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Our thresholds for beneficial and harmful effects were informed by surveys of the coauthors in the parallel living SRNMAs.

    “Our thresholds for beneficial and harmful effects were informed by surveys of the coauthors in the parallel living SRNMAs.17–19”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [17] source

      Not checked in the free run.

    • cites [18] source

      Not checked in the free run.

    • cites [19] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Pairs of reviewers, after calibration exercises to ensure sufficient agreement, worked independently and in duplicate to screen titles and abstracts and full texts.

    “pairs of reviewers, after calibration exercises to ensure sufficient agreement, worked independently and in duplicate to screen titles and abstracts of search records and subsequently the full texts of records determined as potentially eligible at the title and abstract screening stage.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    A validated machine learning model facilitates efficient identification of randomised trials.

    “A validated machine learning model facilitates efficient identification of randomised trials.21”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [21] source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationnone given

    Methods to detect anomalies suggesting fabrication and falsification require subjective judgments and cannot definitively identify untrustworthy trials, but they could be useful to identify trials at high risk of such issues.

    “While such methods require subjective judgments and cannot be used to definitively identify untrustworthy trials, they could be useful to identify trials that are at high risk of such issues.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    We pragmatically limited our search because it was no longer feasible to continue to collect additional data from preprints beyond 3 August 2021.

    “we pragmatically limited our search because it was no longer feasible to continue to collect additional data from preprints beyond this timepoint.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Reviewers resolved discrepancies by discussion or, when necessary, by adjudication with a third party reviewer.

    “Reviewers resolved discrepancies by discussion or, when necessary, by adjudication with a third party reviewer.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    For each eligible trial, pairs of reviewers, after training and calibration exercises, independently extracted trial characteristics, methods, and results using a standardised, pilot tested data extraction form.

    “for each eligible trial, pairs of reviewers, after training and calibration exercises, independently extracted trial characteristics, methods, and results using a standardised, pilot tested data extraction form.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Study design and conductnone given

    For preprints with more than one version, we extracted data from the first version of the preprint, which is the least likely to have been modified in response to peer review.

    “For preprints with more than one version, we extracted data from the first version of the preprint, which is the least likely to have been modified in response to peer review.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    We used the GRADE approach to assess the certainty of evidence.

    “We used the GRADE approach to assess the certainty of evidence”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Preprint reports of trials that are subsequently published in journals might not be representative of all trial preprints.

    “although preprint reports of trials that are subsequently published in journals might not be representative of all trial preprints”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    For the current study, pairs of trained and calibrated reviewers, working independently and in duplicate and using a pilot tested data collection form, collected data on differences in key methods and results between preprint and published trial reports.

    “For the current study, pairs of trained and calibrated reviewers, working independently and in duplicate and using a pilot tested data collection form, collected data on differences in key methods and results between preprint and published trial reports.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    The results support the overall usefulness of preprints, and preprints as a venue through which the dissemination of trial findings might be accelerated.

    “Our results support the overall usefulness of preprints, and preprints as a venue through which the dissemination of trial findings might be accelerated.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    We did not perform a sample size calculation because we included all eligible trial reports identified through our living SRNMAs up to 3 August 2021.

    “We did not perform a sample size calculation because we included all eligible trial reports identified through our living SRNMAs up to 3 August 2021.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationnone given

    Preprints of trials that are subsequently published in journals could represent the most rigorous or transparently reported preprints, which are not representative of all trial preprints.

    “Preprints of trials that are subsequently published in journals could represent the most rigorous or transparently reported preprints, which are not representative of all trial preprints.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    Even trials without discrepancies between the preprint and later published report might have important limitations that reduce their trustworthiness.

    “even trials without discrepancies between the preprint and later published report might have important limitations that reduce their trustworthiness.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Study design and conductcitation

    The GRADE framework provides a transparent and systematic framework of all factors that might bear on the certainty of evidence, but its application is subjective.

    “While the GRADE framework provides a transparent and systematic framework of all factors that might bear on the certainty of evidence, its application is subjective.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [26] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    The estimate of the time to publication of preprint reports might have been over-estimated if some preprint authors did not attempt to subsequently publish in peer reviewed journals-although evidence shows that most authors of covid-19 preprints intend to publish their findings.

    “Our estimate of the time to publication of preprint reports might have been over-estimated if some preprint authors did not attempt to subsequently publish in peer reviewed journals—although evidence shows that most authors of covid-19 preprints intend to publish their findings.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [43] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Published trial reports might still contain errors.

    “published trial reports might still contain errors”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    Preprints could be less likely to be retracted because they might draw less attention and because preprint servers could be less likely than journals to have formal policies on research integrity.

    “Preprints, however, could be less likely to be retracted because they might draw less attention and because preprint servers could be less likely than journals to have formal policies on research integrity.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    Journals have expedited the publication of covid-19 research and have been publishing more prolifically on covid-19 than on other areas, which could reduce opportunity for revisions between preprints and their subsequent publications and might mean that time to and predictors of publication might differ than in other research areas.

    “Journals have expedited the publication of covid-19 research and have been publishing more prolifically on covid-19 than on other areas, which could reduce opportunity for revisions between preprints and their subsequent publications and might mean that time to and predictors of publication might differ than in other research areas.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Including preprints might affect the results of meta-analyses and the certainty of evidence, and might encourage evidence users to consider data from preprints in contexts in which decisions are being made rapidly and evidence is being produced faster than can be peer reviewed and published.

    “We show that including preprints might affect the results of meta-analyses and the certainty of evidence, and might encourage evidence users to consider data from preprints in contexts in which decisions are being made rapidly and evidence is being produced faster than can be peer reviewed and published.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The results are aligned with previous assessments before covid-19 that study interpretations and other study details do not change importantly between preprints and their later publications in high impact journals.

    “Our results are also aligned with previous assessments before covid-19 that study interpretations and other study details do not change importantly between preprints and their later publications in high impact journals.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [47] Shi, 2021, Assessment of concordance and discordance among clinical studies posted as preprints and subsequently published in high-impact journals, JAMA Netw Open source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Time to publication might also be under-estimated if preprints are made public later in the submission process.

    “Time to publication might also be under-estimated if preprints are made public later in the submission process.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Interventions informed by fewer trials may be more sensitive to including or excluding evidence from preprints.

    “Interventions informed by fewer trials may be more sensitive to including or excluding evidence from preprints.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    Differences in adverse events could exist between preprints and publications.

    “Differences in such outcomes could exist between preprints and publications.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [44] source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The assessment of differences in key results between preprints and publications was limited to the outcomes that were included in the living SRNMAs.

    “Our assessment of differences in key results between preprints and publications was limited to the outcomes that were included in our living SRNMAs.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductnone given

    The outcomes included in the living SRNMAs were identified as being important or critical to decision making by coauthors of the living SRNMA and the parallel guideline, but they do not include adverse events.

    “While these outcomes were identified as being important or critical to decision making by coauthors of the living SRNMA and the parallel guideline, they do not include adverse events.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    Posting trial reports as preprints could allow more errors to be identified before final publication.

    “posting trial reports as preprints could allow more errors to be identified before final publication”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    The strengths of this study include the comprehensive search for and inclusion of preprint and published covid-19 trial reports and rigorous data collection.

    “The strengths of this study include the comprehensive search for and inclusion of preprint and published covid-19 trial reports and rigorous data collection.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The assessment of the impact of preprints focused only on the impact of preprints on meta-analytic estimates, the certainty of evidence, and decision making and did not consider other aspects of impact, such as number of citations or Altmetrics.

    “Our assessment of the impact of preprints focused only on the impact of preprints on meta-analytic estimates, the certainty of evidence, and decision making and did not consider other aspects of impact, such as number of citations or Altmetrics.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    The study compared the characteristics of published and unpublished trial preprints and did not identify any important differences.

    “we compared the characteristics of published and unpublished trial preprints and we did not identify any important differences.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Although the study describes discrepancies in the reporting of key methods and results, it did not assess differences in the discussion or conclusion sections of trial reports.

    “although we describe discrepancies in the reporting of key methods and results, we did not assess differences in the discussion or conclusion sections of trial reports.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Despite no compelling reasons that the results will not be generalisable to network meta-analyses, more differences in judgments will be related to the certainty of evidence in a fully contextualised framework where judgments are more dependent on the magnitude and precision of estimates.

    “Despite no compelling reasons that our results will not be generalisable to network meta-analyses, more differences in judgments will be related to the certainty of evidence in a fully contextualised framework where judgments are more dependent on the magnitude and precision of estimates.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Study design and conductreasoning

    Although the WHO covid-19 database is a comprehensive source of published and preprint literature, it does not include all preprint servers-but preprint servers not covered by the search strategy include other subjects and are unlikely to include covid-19 trials.

    “Although the WHO covid-19 database is a comprehensive source of published and preprint literature, it does not include all preprint servers—but preprint servers not covered by our search strategy include other subjects and are unlikely to include covid-19 trials.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationnone given

    Scepticism might still be warranted when suspicion arises regarding falsified data.

    “Scepticism might still be warranted when suspicion arises regarding falsified data (for which we provide criteria).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Other aspects of the reporting of results (eg, baseline characteristics of patients) could be different between preprints and publications.

    “other aspects of the reporting of results (eg, baseline characteristics of patients) could be different between preprints and publications.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    While the effects of including or excluding preprints in meta-analyses might vary across interventions, the analysis looks at the interventions that have had sufficient interest and research to instigate guideline recommendations.

    “While the effects of including or excluding preprints in meta-analyses might vary across interventions, our analysis looks at the interventions that have had sufficient interest and research to instigate guideline recommendations.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductcitation

    The assessment of the contribution of trial preprint reports to meta-analytic estimates and their effect on the certainty of evidence was undertaken in the context of pairwise meta-analysis and the minimally contextualised approach for assessing the certainty of evidence, whereas parallel guideline recommendations have been based on network meta-analyses and the fully contextualised approach.

    “Our assessment of the contribution of trial preprint reports to meta-analytic estimates and their effect on the certainty of evidence was undertaken in the context of pairwise meta-analysis and the minimally contextualised approach for assessing the certainty of evidence, whereas parallel guideline recommendations have been based on network meta-analyses and the fully contextualised approach.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [20] source

      Not checked in the free run.

    • cites [27] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    One study has reviewed publication rates and citations for covid-19 research but it primarily deals with basic science research and so its findings might not apply to covid-19 clinical trials.

    “One study has reviewed publication rates and citations for covid-19 research but it primarily deals with basic science research and so its findings might not apply to covid-19 clinical trials.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [46] source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    The assessment of the impact of meta-analyses including versus excluding preprint reports on meta-analytic estimates was limited to only interventions that have been addressed by the WHO living guideline at the time of analysis.

    “We limited our assessment of the impact of meta-analyses including versus excluding preprint reports on meta-analytic estimates to only interventions that have been addressed by the WHO living guideline at the time of analysis.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [20] source

      Not checked in the free run.

64 of 64

  • The paper relies on this without support.

    The WHO covid-19 database and the L-OVE covid-19 platform comprehensively capture global published and preprint literature on covid-19.

    “a comprehensive multilingual source of global published and preprint literature on covid-19”
1 of 1

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Inpatient percentage and reported difference do not agree

    Minor

    The printed inpatient percentages for preprint-only and publication-only trials differ by −2.4 percentage points, not the reported −4.3. The publication-only percentage appears to be the source of the error: 129 of its 174 treatment trials is approximately 74.1%, rather than 72.3%.

    Reported -4.3 · recomputed -2.4

  • Direction of the drug-treatment comparison is reversed

    Minor

    The text says preprint-only trials were more likely than publication-only trials to describe drug treatments. Table 1 reports the opposite: 76.2% versus 86.7%.

  • Randomisation risk-of-bias subtotal does not match the total

    Minor

    The three trial categories list 67, 34, and 19 trials with randomisation-related high risk of bias, summing to 120 rather than the reported 122 for all trials.

    Reported 122 · recomputed 120

  • Ivermectin trial counts are taken from another treatment row

    Minor

    The narrative assigns seven trials with 1826 participants and nine with 4000 to the six-month ivermectin comparison. The corresponding ivermectin row reports four trials with 642 participants and seven with 951; the narrative counts match the interleukin 6 receptor blocker row instead.

4 of 4

Cites

8 retracted

Flagged first, then by citations

Research works printed but not linked: 2

  1. ISARIC (International Severe Acute Respiratory and Emerging Infections Consortium) . COVID-19 report: 08 June 2020 medRxiv; 2020.
  2. Horby P , Lim WS , Emberson J . Effect of dexamethasone in hospitalized patients with COVID-19 – preliminary report. medRxiv 2020.doi:10.1101/2020.06.22.20137273 · DOI · Paper page

Cited by

Sorted by citations

Related

Authors
Zeraatkar, D., Pitre, T., Leung, G., Cusano, E., Agarwal, A., Khalid, F., Escamilla, Z., Cooper, M. A., Ghadimi, M., Wang, Y., Verdugo-Paiva, F., Rada, G., Kum, E., Qasim, A., Bartoszko, J. J., Siemieniuk, R. A. C., Patel, C., Guyatt, G., & Brignardello-Petersen, R.
Affiliations
  • Harvard University (US)
  • McMaster University (CA)
  • University of Ottawa (CA)
  • University of Calgary (CA)
  • University of Toronto (CA)
  • University of Alberta (CA)
  • Beijing Chao-Yang Hospital, Capital Medical University (CN)
  • Epistemonikos Foundation, Santiago, Chile
  • Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, ON, Canada
Corresponding author
Zeraatkar, Dena
ORCID
Venue
BMJ Medicine
Publisher
BMJ
Published
2022-10
Volume
1
Issue
1
Pages
e000309
ISSN
2754-0413 (ISSN-L)
DOI
10.1136/bmjmed-2022-000309
PMC
PMC9951374
OpenAlex
W4300863266
Semantic Scholar
2f5c457a0f45280e2d603dd4970ce6b296746524
Cited by
44
References
51
Citations by year
2026: 2 · 2025: 7 · 2024: 14 · 2023: 17 · 2022: 4
FWCI
8.79
Type
journal article
Language
en
Open access
Licence
Locations
  • BMJ Medicine (publishedVersion, cc-by-nc)
  • medRxiv (acceptedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
Canadian Institutes of Health Research (Research CIHR-IRSC:057900132)
Topics
  • Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Health Sciences › Medicine › Infectious Diseases › COVID-19 Clinical Research Studies
Keywords
COVID-19 pandemic, preprints, clinical trial reporting, reporting consistency, quality of evidence, GRADE, time to publication, evidence-based decision making, mechanical ventilation, research trustworthiness

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review