Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index; 15 of the 32 sources its claims cite were checked for support.

Comparing quality of reporting between preprints and peer-reviewed articles in the biomedical literature

Open access

Published version of a preprint

Abstract

Background: Preprint usage is growing rapidly in the life sciences; however, questions remain on the relative quality of preprints when compared to published articles. An objective dimension of quality that is readily measurable is completeness of reporting, as transparency can improve the reader’s ability to independently interpret data and reproduce findings. Methods: In this observational study, we initially compared independent samples of articles published in bioRxiv and in PubMed-indexed journals in 2016 using a quality of reporting questionnaire. After that, we performed paired comparisons between preprints from bioRxiv to their own peer-reviewed versions in journals. Results: Peer-reviewed articles had, on average, higher quality of reporting than preprints, although the difference was small, with absolute differences of 5.0% [95% CI 1.4, 8.6] and 4.7% [95% CI 2.4, 7.0] of reported items in the independent samples and paired sample comparison, respectively. There were larger differences favoring peer-reviewed articles in subjective ratings of how clearly titles and abstracts presented the main findings and how easy it was to locate relevant reporting information. Changes in reporting from preprints to peer-reviewed versions did not correlate with the impact factor of the publication venue or with the time lag from bioRxiv to journal publication. Conclusions: Our results suggest that, on average, publication in a peer-reviewed journal is associated with improvement in quality of reporting. They also show that quality of reporting in preprints in the life sciences is within a similar range as that of peer-reviewed articles, albeit slightly lower on average, supporting the idea that preprints should be considered valid scientific contributions.

Overview

CoChat Summary

TL;DR

In this observational study, peer-reviewed biomedical articles had slightly higher completeness of reporting than bioRxiv preprints, both in independent samples and in paired preprint-to-publication comparisons. The differences were small, and changes from preprint to peer-reviewed version did not correlate with journal impact factor or time to publication.

Key findings

Peer-reviewed articles scored higher than preprints by 5.0% [95% CI 1.4, 8.6] in independent samples (bioRxiv 66.9 ± 12.2 vs. PubMed 71.9 ± 10.1) and by 4.7% [95% CI 2.4, 7.0] in paired comparisons (preprint 67.6 ± 10.8 vs. peer-reviewed 72.3 ± 10.1).

  • In the paired sample, 27% of pairs (15/56) showed a decrease in reporting score from preprint to peer-reviewed version.
  • The largest paired item difference was conflict of interest statements (46.4% vs. 83.9%, p = 1.9 × 10−5); removing this item reduced the paired difference from 4.7 to 3.3 percentage points.
  • Changes in reporting score did not correlate with journal impact factor (ρ = 0.16, p = 0.25) or time from bioRxiv to journal publication (r = 0.03, p = 0.81). Preprints later published had higher reporting scores than unpublished preprints (68.8 ± 11.8 vs. 61.1 ± 11.9, p = 0.02).
  • Subjective ratings favored peer-reviewed articles for title/abstract clarity (independent difference 0.4 [95% CI 0.2, 0.6]) and ease of extracting information (independent difference 0.7 [95% CI 0.5, 1.0]; paired difference 0.4 [95% CI 0.2, 0.6]).

Methods

Observational study with preregistered protocols: independent random samples of bioRxiv preprints and PubMed-indexed articles from 2016, matched by biological model, with 76 articles per group; paired comparison of 56 bioRxiv preprints to their own peer-reviewed journal versions.

  • Quality of reporting was assessed with an online questionnaire based on existing reporting guidelines; each article was evaluated by three evaluators, with the most prevalent answer used for objective items and the mean used for subjective items. Overall score was the percentage of applicable reporting items reported.
  • Analyses used Student’s t-tests, paired t-tests, ANOVA, Fisher’s exact or McNemar’s exact tests, and Spearman or Pearson correlations; primary outcomes used α = 0.05, with Sidak correction for secondary outcomes.

Limitations

The paper states that generalizability is limited by inclusion criteria requiring primary data and statistical comparisons, use of a single preprint platform, analysis of the first table/figure rather than necessarily the main finding, evaluators not blinded to article group origin, and limited power for individual-question and journal-feature analyses.

  • The paper notes that independent-sample differences may reflect field, region, and practice differences between bioRxiv and PubMed rather than peer review alone, and that paired changes may stem from other editorial or submission requirements, not only reviewer feedback.
  • Evident from the text: the paired sample used only the first preprint version, so changes from other feedback sources cannot be excluded; peer-review model comparisons were not performed because most articles were in single-blind journals.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s41073-020-00101-3

Cites

47 of 47 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Flowchart of article screening and inclusion in each stage of the study. Screening and inclusion criteria are described in the Methods section
Fig. 1 Flowchart of article screening and inclusion in each stage of the study. Screening and inclusion criteria are described in the Methods sectionCC BY · PubMed Central
Fig. 2 Reporting scores by source of the article. a Random samples of bioRxiv and PubMed were evaluated. Mean ± S.D.: bioRxiv = 66.9 ± 12.2, PubMed = 71.9 ± 10.1; n = 76/group. Student’s t-test, t = 2.75, p = 0.007, 95% C.I. [1.40, 8.59]. b A sample of bioRxiv articles was compared against their peer-reviewed versions published by a journal. Mean ± S.D.: Preprint = 67.6 ± 10.8, Peer-Reviewed = 72.3 ± 10.1; n = 56 pairs. Paired t-test, t = 4.15, p = 0.0001, 95% C.I. [2.44, 6.99]. On the right, absolute changes in score from preprint to peer-reviewed versions are plotted for each pair
Fig. 2 Reporting scores by source of the article. a Random samples of bioRxiv and PubMed were evaluated. Mean ± S.D.: bioRxiv = 66.9 ± 12.2, PubMed = 71.9 ± 10.1; n = 76/group. Student’s t-test, t = 2.75, p = 0.007, 95% C.I. [1.40, 8.59]. b A sample of bioRxiv articles was compared against their peer-reviewed versions published by a journal. Mean ± S.D.: Preprint = 67.6 ± 10.8, Peer-Reviewed = 72.3 ± 10.1; n = 56 pairs. Paired t-test, t = 4.15, p = 0.0001, 95% C.I. [2.44, 6.99]. On the right, absolute changes in score from preprint to peer-reviewed versions are plotted for each pairCC BY · PubMed Central
Fig. 3 Quality of reporting by article size. a Overall scores by number of figure subpanels/tables in the independent samples comparison. Spearman’s correlations: All articles, ρ = − 0.31, 95% C.I. [− 0.46, − 0.15], p = 9.5 × 10− 5; bioRxiv (shown in red), ρ = − 0.35, 95% C.I. [− 0.56, − 0.11], p = 0.002; PubMed (shown in blue), ρ = − 0.22, 95%C.I. [− 0.44, 0.0006], p = 0.05. N = 152 (76/group). b Change in score from preprint to peer-reviewed versions by change in the number of figures subpanels/tables in the paired sample. Spearman’s correlation: ρ = − 0.07, 95% C.I. [− 0.33, 0.17], p = 0.59, N = 56. One article presented a large decrease in number of figures (− 71 figures subpanels/tables), as it was published as a brief communication. c Difference between scores from peer-reviewed to preprint version by mean number of figure subpanels/tables between preprint and peer-reviewed version in the paired sample. Spearman’s correlation: ρ = 0.24, 95% C.I. [− 0.05, 0.49], p = 0.08, N = 56
Fig. 3 Quality of reporting by article size. a Overall scores by number of figure subpanels/tables in the independent samples comparison. Spearman’s correlations: All articles, ρ = − 0.31, 95% C.I. [− 0.46, − 0.15], p = 9.5 × 10− 5; bioRxiv (shown in red), ρ = − 0.35, 95% C.I. [− 0.56, − 0.11], p = 0.002; PubMed (shown in blue), ρ = − 0.22, 95%C.I. [− 0.44, 0.0006], p = 0.05. N = 152 (76/group). b Change in score from preprint to peer-reviewed versions by change in the number of figures subpanels/tables in the paired sample. Spearman’s correlation: ρ = − 0.07, 95% C.I. [− 0.33, 0.17], p = 0.59, N = 56. One article presented a large decrease in number of figures (− 71 figures subpanels/tables), as it was published as a brief communication. c Difference between scores from peer-reviewed to preprint version by mean number of figure subpanels/tables between preprint and peer-reviewed version in the paired sample. Spearman’s correlation: ρ = 0.24, 95% C.I. [− 0.05, 0.49], p = 0.08, N = 56CC BY · PubMed Central
Fig. 4 Quality of reporting by impact factor of publication venue. a Overall scores by 2016 impact factor of the publication venue in the independent samples comparison. Spearman’s correlation: ρ = − 0.11, 95% C.I. [− 0.35, 0.13], p = 0.35, n = 69. b Change in score from peer-reviewed to preprint version by impact factor of the peer-reviewed publication year in the paired sample. Spearman’s correlation: ρ = 0.16, 95% C.I. [− 0.13, 0.43], p = 0.25, n = 53. Impact factor was unavailable for 7 articles in the independent sample and 3 in the paired one
Fig. 4 Quality of reporting by impact factor of publication venue. a Overall scores by 2016 impact factor of the publication venue in the independent samples comparison. Spearman’s correlation: ρ = − 0.11, 95% C.I. [− 0.35, 0.13], p = 0.35, n = 69. b Change in score from peer-reviewed to preprint version by impact factor of the peer-reviewed publication year in the paired sample. Spearman’s correlation: ρ = 0.16, 95% C.I. [− 0.13, 0.43], p = 0.25, n = 53. Impact factor was unavailable for 7 articles in the independent sample and 3 in the paired oneCC BY · PubMed Central
Fig. 5 Quality of reporting and the peer review process. a Difference between scores from peer-reviewed to preprint version by time to publication (in months) in the paired sample. Pearson’s correlation: r = 0.03, 95% C.I. [− 0.23, 0.29], p = 0.81, n = 56 pairs. b Overall reporting scores by publication status (published or not in a peer-reviewed journal) of preprints assessed in the independent sample. Mean ± S.D.: Unpublished = 61.1 ± 11.9, n = 19; Published = 68.8 ± 11.8, n = 57. 95% C.I. [1.45, 13.93], Student’s t test: t = 2.45, p = 0.02
Fig. 5 Quality of reporting and the peer review process. a Difference between scores from peer-reviewed to preprint version by time to publication (in months) in the paired sample. Pearson’s correlation: r = 0.03, 95% C.I. [− 0.23, 0.29], p = 0.81, n = 56 pairs. b Overall reporting scores by publication status (published or not in a peer-reviewed journal) of preprints assessed in the independent sample. Mean ± S.D.: Unpublished = 61.1 ± 11.9, n = 19; Published = 68.8 ± 11.8, n = 57. 95% C.I. [1.45, 13.93], Student’s t test: t = 2.45, p = 0.02CC BY · PubMed Central
Fig. 6 Subjective assessment by article source. a Clarity of title/abstract for the independent samples comparison. Scores were given as an answer to “Do the title and abstract provide a clear idea of the article’s main findings?”. Mean ± S.D.: bioRxiv = 3.9 ± 0.6, n = 72; PubMed = 4.3 ± 0.7, n = 72. 95% C.I. [0.2–0.6]. Student’s t-test: t = 3.61, p = 0.0004. b Clarity of title/abstract for the paired sample comparison. Mean ± S.D.: Preprint = 4.0 ± 0.6, n = 56; Peer-reviewed = 4.1 ± 0.5. 95% C.I. [− 0.03, 0.4]. Paired t-test: t = 1.66, p = 0.10. Right panel shows the differences between scores in preprint and peer-reviewed versions. c Easiness to extract information for the independent samples comparison. Scores were given as an answer to “Was the required information easy to find and extract from the article?”. Mean ± S.D.: bioRxiv = 3.4 ± 0.8, n = 72; PubMed = 4.2 ± 0.6. 95% C.I. [0.5, 1.0]. Student’s t-test: t = 6.22, p = 5.1 × 10− 9. d Easiness to extract information for the paired sample comparison. Mean ± S.D.: Preprint = 3.5 ± 0.7, n = 56; Peer-reviewed = 4.0 ± 0.6, n = 56. 95% C.I. [0.2, 0.6]. Paired t-test: t = 4.12, p = 0.0001. Right panel shows changes in score from preprint to peer-reviewed versions
Fig. 6 Subjective assessment by article source. a Clarity of title/abstract for the independent samples comparison. Scores were given as an answer to “Do the title and abstract provide a clear idea of the article’s main findings?”. Mean ± S.D.: bioRxiv = 3.9 ± 0.6, n = 72; PubMed = 4.3 ± 0.7, n = 72. 95% C.I. [0.2–0.6]. Student’s t-test: t = 3.61, p = 0.0004. b Clarity of title/abstract for the paired sample comparison. Mean ± S.D.: Preprint = 4.0 ± 0.6, n = 56; Peer-reviewed = 4.1 ± 0.5. 95% C.I. [− 0.03, 0.4]. Paired t-test: t = 1.66, p = 0.10. Right panel shows the differences between scores in preprint and peer-reviewed versions. c Easiness to extract information for the independent samples comparison. Scores were given as an answer to “Was the required information easy to find and extract from the article?”. Mean ± S.D.: bioRxiv = 3.4 ± 0.8, n = 72; PubMed = 4.2 ± 0.6. 95% C.I. [0.5, 1.0]. Student’s t-test: t = 6.22, p = 5.1 × 10− 9. d Easiness to extract information for the paired sample comparison. Mean ± S.D.: Preprint = 3.5 ± 0.7, n = 56; Peer-reviewed = 4.0 ± 0.6, n = 56. 95% C.I. [0.2, 0.6]. Paired t-test: t = 4.12, p = 0.0001. Right panel shows changes in score from preprint to peer-reviewed versionsCC BY · PubMed Central
Fig. 7 Quality of reporting by subjective scores. a Overall reporting scores by title/abstract clarity in the independent samples. Pearson’s correlation: r = 0.38, 95% C.I. [0.23, 0.51], p = 3.1 × 10− 6, n = 144 (all articles); r = 0.12, 95% C.I. [− 0.11, 0.35], p = 0.29, n = 72 (bioRxiv); r = 0.55, 95% C.I. [0.36, 0.69], p = 6.3 × 10− 7, n = 72 (PubMed). b Changes in overall reporting scores by changes in title/abstract clarity in the paired sample. Pearson’s correlation: r = 0.25, 95% C.I. [− 0.01, 0.68], p = 0.06, n = 56. c Overall reporting scores by easiness to extract information in the independent samples. Pearson’s correlation: r = 0.59, 95% C.I. [0.47, 0.69], p = 8.7 × 10− 15, n = 144 (all articles); r = 0.54, 95% C.I. [0.35, 0.68], p = 1.04 × 10− 6, n = 72 (bioRxiv); r = 0.60, 95% C.I. [0.43, 0.73], p = 2.2 × 10− 8, n = 72 (PubMed). d Changes in overall reporting scores changes in easiness to extract information in the paired sample. Pearson’s correlation: r = 0.20, 95% C.I. [− 0.06, 0.44], p = 0.13, n = 56. In all panels, bioRxiv articles are in red and PubMed ones are in blue, while differences between paired articles are shown in purple
Fig. 7 Quality of reporting by subjective scores. a Overall reporting scores by title/abstract clarity in the independent samples. Pearson’s correlation: r = 0.38, 95% C.I. [0.23, 0.51], p = 3.1 × 10− 6, n = 144 (all articles); r = 0.12, 95% C.I. [− 0.11, 0.35], p = 0.29, n = 72 (bioRxiv); r = 0.55, 95% C.I. [0.36, 0.69], p = 6.3 × 10− 7, n = 72 (PubMed). b Changes in overall reporting scores by changes in title/abstract clarity in the paired sample. Pearson’s correlation: r = 0.25, 95% C.I. [− 0.01, 0.68], p = 0.06, n = 56. c Overall reporting scores by easiness to extract information in the independent samples. Pearson’s correlation: r = 0.59, 95% C.I. [0.47, 0.69], p = 8.7 × 10− 15, n = 144 (all articles); r = 0.54, 95% C.I. [0.35, 0.68], p = 1.04 × 10− 6, n = 72 (bioRxiv); r = 0.60, 95% C.I. [0.43, 0.73], p = 2.2 × 10− 8, n = 72 (PubMed). d Changes in overall reporting scores changes in easiness to extract information in the paired sample. Pearson’s correlation: r = 0.20, 95% C.I. [− 0.06, 0.44], p = 0.13, n = 56. In all panels, bioRxiv articles are in red and PubMed ones are in blue, while differences between paired articles are shown in purpleCC BY · PubMed Central
Fig. 8 Reporting quality by article formatting. a Overall reporting score by embedding of figures in all preprints assessed. Preprints assessed in both stages of the study were included only once for this analysis, with the mean of reporting scores from both assessments. Mean ± S.D.: Non-embedded = 64.6 ± 10.9, n = 59; Embedded = 70.8 ± 11.6, n = 26. 95% C.I. [1.0, 11.4]. Student’s t-test: t = 2.37, p = 0.02. b Difference between scores from peer-reviewed to preprint version by embedding of figures in the preprint version (paired sample). Mean ± S.D.: Non-embedded = 4.3 ± 8.3, n = 38; Embedded = 5.6 ± 9.0, n = 18. 95% C.I. [− 3.6, 6.1]. Student’s t-test, t = 0.55, p = 0.58
Fig. 8 Reporting quality by article formatting. a Overall reporting score by embedding of figures in all preprints assessed. Preprints assessed in both stages of the study were included only once for this analysis, with the mean of reporting scores from both assessments. Mean ± S.D.: Non-embedded = 64.6 ± 10.9, n = 59; Embedded = 70.8 ± 11.6, n = 26. 95% C.I. [1.0, 11.4]. Student’s t-test: t = 2.37, p = 0.02. b Difference between scores from peer-reviewed to preprint version by embedding of figures in the preprint version (paired sample). Mean ± S.D.: Non-embedded = 4.3 ± 8.3, n = 38; Embedded = 5.6 ± 9.0, n = 18. 95% C.I. [− 3.6, 6.1]. Student’s t-test, t = 0.55, p = 0.58CC BY · PubMed Central

Verification

Claims the paper makes

3 Valid · 1 Not valid · 12 Not checked · 151 Own results and interpretations

89 more claims

Create a free account to see all 167 claims and to verify the 17 references still open.

See all claims — free account
  • Probably not a substantial claim. 48
  • It may state two separate claims as one. 41
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

5 of 5

Citations

Cites

Show all 47 references

Research works printed but not linked: 2

  1. Anaya, J. (2016) ‘bioRxiv vs. PeerJ preprints’, Medium. Available at: https://medium.com/@OmnesRes/biorxiv-vs-peerj-preprints-f7589141c532 (Accessed: 14 Jan 2019).
  2. Inglis, J. R. and Sever, R. (2016) ‘bioRxiv: a progress report’, ASAPbio blog. Available at: http://asapbio.org/biorxiv (Accessed: 14 Jan 2019).

Cited by

See all 154

Metadata

Authors
Carneiro, C. F. D., Queiroz, V. G. S., Moulin, T. C., Carvalho, C. A. M., Haas, C. B., Rayêe, D., Henshall, D. E., De-Souza, E. A., Amorim, F. E., Boos, F. Z., Guercio, G. D., Costa, I. R., Hajdu, K. L., van Egmond, L., Modrák, M., Tan, P. B., Abdill, R. J., Burgess, S. J., Guerra, S. F. S., Bortoluzzi, V. T., & Amaral, O. B.
Affiliations
  • Universidade Federal do Rio de Janeiro (BR)
  • Universidade do Estado do Pará (BR)
  • Universidade Federal do Rio Grande do Sul (BR)
  • University of Edinburgh (GB)
  • Universidade Federal de São Paulo (BR)
  • University of Minnesota (US)
  • Czech Academy of Sciences, Institute of Microbiology (CZ)
  • University of Illinois Urbana-Champaign (US)
Corresponding author
Carneiro, Clarissa F. D.
Venue
Research Integrity and Peer Review
Published
2020-12
Volume
5
Issue
1
Pages
16-16
ISSN
2058-8615 (ISSN-L)
OpenAlex
W3110179196
Cited by
146
References
47
Citations by year
2026: 7 · 2025: 12 · 2024: 30 · 2023: 31 · 2022: 34 · 2021: 25 · 2020: 4
FWCI
23.31
Type
journal article
Language
en
Open access
Licence
Locations
  • Research Integrity and Peer Review (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Uppsala University Publications (Uppsala University) (acceptedVersion, other-oa)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • Fundação Carlos Chagas Filho de Amparo à Pesquisa do Estado do Rio de Janeiro
  • Conselho Nacional de Desenvolvimento Científico e Tecnológico
  • Programa Institucional de Bolsas de Iniciação Científica - UFRJ
Topics
  • Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Library and Information Sciences › Academic Writing and Publishing
Keywords
preprints, bioRxiv, peer-reviewed articles, reporting quality, reporting completeness, transparency, reproducibility, biomedical literature

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Comparing quality of reporting between preprints and peer-reviewed articles in the biomedical literature. https://app.cochat.ai/papers/10.1186/s41073-020-00101-3
The paper
Carneiro, C. F. D., Queiroz, V. G. S., Moulin, T. C., Carvalho, C. A. M., Haas, C. B., Rayêe, D., Henshall, D. E., De-Souza, E. A., Amorim, F. E., Boos, F. Z., Guercio, G. D., Costa, I. R., Hajdu, K. L., van Egmond, L., Modrák, M., Tan, P. B., Abdill, R. J., Burgess, S. J., Guerra, S. F. S., . . . Amaral, O. B. (2020). Comparing quality of reporting between preprints and peer-reviewed articles in the biomedical literature. Research Integrity and Peer Review. https://doi.org/10.1186/s41073-020-00101-3

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.