Cross-sectional study of preprints and final journal publications from COVID-19 studies: discrepancies in results reporting and spin in interpretation
Open accessPublished version of a preprint
Abstract
Objective: To compare results reporting and the presence of spin in COVID-19 study preprints with their finalised journal publications. Design: Cross-sectional study. Setting: International medical literature. Participants: Preprints and final journal publications of 67 interventional and observational studies of COVID-19 treatment or prevention from the Cochrane COVID-19 Study Register published between 1 March 2020 and 30 October 2020. Main outcome measures: Study characteristics and discrepancies in (1) results reporting (number of outcomes, outcome descriptor, measure, metric, assessment time point, data reported, reported statistical significance of result, type of statistical analysis, subgroup analyses (if any), whether outcome was identified as primary or secondary) and (2) spin (reporting practices that distort the interpretation of results so they are viewed more favourably). Results: Of 67 included studies, 23 (34%) had no discrepancies in results reporting between preprints and journal publications. Fifteen (22%) studies had at least one outcome that was included in the journal publication, but not the preprint; eight (12%) had at least one outcome that was reported in the preprint only. For outcomes that were reported in both preprints and journals, common discrepancies were differences in numerical values and statistical significance, additional statistical tests and subgroup analyses and longer follow-up times for outcome assessment in journal publications. At least one instance of spin occurred in both preprints and journals in 23/67 (34%) studies, the preprint only in 5 (7%), and the journal publications only in 2 (3%). Spin was removed between the preprint and journal publication in 5/67 (7%) studies; but added in 1/67 (1%) study. Conclusions: The COVID-19 preprints and their subsequent journal publications were largely similar in reporting of study characteristics, outcomes and spin. All COVID-19 studies published as preprints and journal publications should be critically evaluated for discrepancies and spin.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Differences in adverse events could exist between preprints and publications.
“Differences in such outcomes could exist between preprints and publications.”
From Consistency of covid-19 trial preprints with published reports and impact for decision making: retrospective review (2022) · cites this paper as [44]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
1 retracted
Gautret, P., Lagier, J., Parola, P., Hoang, V. T., Meddeb, L., Mailhe, M., Doudier, B., Courjon, J., Giordanengo, V., Esteves-Vieira, V., et al. (2020) · International Journal of Antimicrobial Agents · doi:10.1016/j.ijantimicag.2020.105949 · Cited by 5,288 · see citing works
RetractedHarris, P. A., Taylor, R., Thielke, R., Payne, J., Gonzalez, N., & Conde, J. G. (2009) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2008.08.010 · Cited by 51,384 · see citing works
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845 · see citing works
Tang, W., Cao, Z., Han, M., Wang, Z., Chen, J., Sun, W., Wu, Y., Xiao, W. L., Liu, S., Chen, E., et al. (2020) · BMJ · doi:10.1136/bmj.m1849 · Cited by 1,129 · see citing works
Borba, M. G. S., Val, F. F. A., Sampaio, V. S., Alexandre, M. A. A., Melo, G. C., Brito, M., Mourão, M. P. G., Brito-Sousa, J. D., Baía-da-Silva, D., Guerra, M. V. F., et al. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.8857 · Cited by 1,038 · see citing works
The ClinicalTrials.gov Results Database — Update and Key Issues
Zarin, D. A., Tse, T., Williams, R. J., Califf, R. M., & Ide, N. C. (2011) · New England Journal of Medicine · doi:10.1056/nejmsa1012065 · Cited by 838 · see citing works
Comparison of Registered and Published Primary Outcomes in Randomized Controlled Trials
Mathieu, S. (2009) · JAMA · doi:10.1001/jama.2009.1242 · Cited by 730 · see citing works
Agarwal, A., Mukherjee, A., Kumar, G., Chatterjee, P., Bhatnagar, T., Malhotra, P., & Collaborators, P. T. (2020) · BMJ · doi:10.1136/bmj.m3939 · Cited by 718 · see citing works
‘Spin’ in published biomedical literature: A methodological systematic review
Chiu, K., Grundy, Q., & Bero, L. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2002173 · Cited by 317 · see citing works
Self-citationChorin, E., Wadhwani, L., Magnani, S., Dai, M. S., Shulman, E. H., Nadeau‐Routhier, C., Knotts, R. J., Bar-Cohen, R., Kogan, E., Barbhaiya, C. R., et al. (2020) · Heart Rhythm · doi:10.1016/j.hrthm.2020.05.014 · Cited by 295 · see citing works
Misrepresentation and distortion of research in biomedical literature
Boutron, I., & Ravaud, P. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1710755115 · Cited by 273 · see citing works
Smith, L., Amlôt, R., Lambert, H., Oliver, I., Robin, C., Yardley, L., & Rubin, G. J. (2020) · Public Health · doi:10.1016/j.puhe.2020.07.024 · Cited by 232 · see citing works
Tang, W., Cao, Z., Han, M., et al. (2020) · medRxiv · doi:10.1101/2020.04.10.20060558 · Cited by 199 · see citing works
Carneiro, C. F. D., Queiroz, V. G. S., Moulin, T. C., Carvalho, C. A. M., Haas, C. B., Rayêe, D., Henshall, D. E., De-Souza, E. A., Amorim, F. E., Boos, F. Z., et al. (2020) · Research Integrity and Peer Review · doi:10.1186/s41073-020-00101-3 · Cited by 146 · see citing works
Borba, M. G. S., Val, F., Sampaio, V. D. S., et al. (2020) · medRxiv · doi:10.1101/2020.04.07.20056424 · Cited by 125 · see citing works
Preprints Involving Medical Research—Do the Benefits Outweigh the Challenges?
Flanagin, A., Fontanarosa, P. B., & Bauchner, H. (2020) · JAMA · doi:10.1001/jama.2020.20674 · Cited by 115 · see citing works
Systematic examination of preprint platforms for use in the medical and biomedical sciences setting
Kirkham, J. J., Penfold, N. C., Murphy, F., Boutron, I., Ioannidis, J. P., Polka, J., & Moher, D. (2020) · BMJ Open · doi:10.1136/bmjopen-2020-041849 · Cited by 112 · see citing works
Self-citationHopewell, S., Collins, G. S., Boutron, I., Yu, L.-M., Cook, J., Shanyinde, M., Wharton, R., Shamseer, L., & Altman, D. G. (2014) · BMJ · doi:10.1136/bmj.g4145 · Cited by 103 · see citing works
Weber, A. G., Chau, A. S., Egeblad, M., Barnes, B. J., & Janowitz, T. (2020) · Molecular Medicine · doi:10.1186/s10020-020-00215-w · Cited by 91 · see citing works
Chorin, E., Wadhwani, L., Magnani, S., et al. (2020) · medRxiv · doi:10.1101/2020.04.27.20074583 · Cited by 84 · see citing works
Agarwal, A., Mukherjee, A., Kumar, G., et al. (2020) · medRxiv · doi:10.1101/2020.09.03.20187252 · Cited by 80 · see citing works
Añazco, D., Nicolalde, B., Espinosa, I., Camacho, J., Mushtaq, M., Gimenez, J., & Teran, E. (2021) · PeerJ · doi:10.7717/peerj.10927 · Cited by 63 · see citing works
Submissions and Downloads of Preprints in the First Year of medRxiv
Krumholz, H. M., Bloom, T., Sever, R., Rawlinson, C., Inglis, J. R., & Ross, J. S. (2020) · JAMA · doi:10.1001/jama.2020.17529 · Cited by 62 · see citing works
Lazarus, C., Haneef, R., Ravaud, P., Hopewell, S., Altman, D. G., & Boutron, I. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.04.012 · Cited by 58 · see citing works
the COVID-19 NMA Consortium, Oikonomidi, T., Boutron, I., Pierre, O., Cabanac, G., & Ravaud, P. (2020) · BMC Medicine · doi:10.1186/s12916-020-01880-8 · Cited by 54 · see citing works
Smith, L. E., Amlôt, R., Lambert, H., et al. (2020) · medRxiv · doi:10.1101/2020.06.01.20119040 · Cited by 53 · see citing works
Comparing Published Scientific Journal Articles to Their Pre-print Versions
Klein, M., Broadwell, P. M., Farb, S. E., & Grappone, T. (2016) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/2910896.2910909 · Cited by 32 · see citing works
Effect and Reach of Medical Articles Posted on Preprint Servers During the COVID-19 Pandemic
Jung, Y. E., Sun, Y., & Schluger, N. W. (2020) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2020.6629 · Cited by 27 · see citing works
Weber, A. G., Chau, A. S., Egeblad, M., et al. (2020) · medRxiv · doi:10.1101/2020.05.13.20087734 · Cited by 16 · see citing works
COVID-19 randomized controlled trials in medRxiv and PubMed
Kataoka, Y., Oide, S., Ariie, T., Tsujimoto, Y., & Furukawa, T. A. (2020) · European Journal of Internal Medicine · doi:10.1016/j.ejim.2020.09.019 · Cited by 14 · see citing works
(2020) · Nature Biotechnology · doi:10.1038/s41587-020-0536-x · Cited by 10 · see citing works
Shepherding preprints through a pandemic
Bloom, T. (2020) · BMJ · doi:10.1136/bmj.m4703 · Cited by 7 · see citing works
Nicolalde, B., Añazco, D., Mushtaq, M., et al. (2020) · Research Square · doi:10.21203/rs.3.rs-34689/v2 · Cited by 5 · see citing works
Bero, L. A., Lawrence, R., Leslie, L., et al. (2021) · medRxiv · doi:10.1101/2021.04.12.21255329 · Cited by 4 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- [29]Gautret P, Lagier J-C, Parola P, et al. Hydroxychloroquine and azithromycin as a treatment of COVID-19: results of an open-label non-randomized clinical trial. Int J Antimicrob Agents 2020; 56: 105949. 10.1016/j.ijantimicag.2020.105949 32205204 PMC7102549 · DOI
Cited by
Sorted by citations
Zeraatkar, D., Pitre, T., Leung, G., Cusano, E., Agarwal, A., Khalid, F., Escamilla, Z., Cooper, M. A., Ghadimi, M., Wang, Y., et al. (2022) · BMJ Medicine · doi:10.1136/bmjmed-2022-000309 · Cited by 44 · see citing works
Transparency and reporting characteristics of COVID-19 randomized controlled trials
Kapp, P., Esmail, L., Ghosn, L., Ravaud, P., & Boutron, I. (2022) · BMC Medicine · doi:10.1186/s12916-022-02567-y · Cited by 25 · see citing works
Davidson, M., Evrenoglou, T., Graña, C., Chaimani, A., & Boutron, I. (2024) · BMC Medical Research Methodology · doi:10.1186/s12874-023-02136-8 · Cited by 9 · see citing works
Davidson, M., Evrenoglou, T., Graña, C., Chaimani, A., & Boutron, I. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.08.011 · Cited by 8 · see citing works
- Authors
- Bero, L., Lawrence, R., Leslie, L., Chiu, K., McDonald, S., Page, M. J., Grundy, Q., Parker, L., Boughton, S., Kirkham, J. J., & Featherstone, R.
- Affiliations
- University of Colorado Anschutz (US)
- The University of Sydney (AU)
- Monash University (AU)
- Cochrane (GB)
- Manchester Academic Health Science Centre (GB)
- Corresponding author
- Bero, Lisa
- ORCID
- Venue
- BMJ Open
- Publisher
- BMJ
- Published
- 2021-07
- Volume
- 11
- Issue
- 7
- Pages
- e051821
- ISSN
- 2044-6055 (ISSN-L)
- PMC
- PMC8288242
- OpenAlex
- W3182452186
- Cited by
- 50
- References
- 34
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 8 · 2023: 19 · 2022: 12 · 2021: 6
- FWCI
- 9.81
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- BMJ Open (publishedVersion, cc-by-nc)
- medRxiv (acceptedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Research Explorer (The University of Manchester) (publishedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Research Explorer (The University of Manchester) (submittedVersion)
- The Sydney eScholarship Repository (The University of Sydney) (submittedVersion)
- The Sydney eScholarship Repository (The University of Sydney) (submittedVersion, cc-by-nc)
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Pharmacology › Pharmaceutical industry and healthcare
- Keywords
- COVID-19 pandemic, preprints, journal publishing, outcome reporting, statistical significance, subgroup analysis, follow-up duration
- MeSH terms
- Preprints as Topic, SARS-CoV-2, COVID-19 Drug Treatment, Cross-Sectional Studies, Humans, Coronavirus Infections
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
