CoChat

Statistical Reviewers Improve Reporting in Biomedical Articles: A Randomized Trial

Open access
Cobo, Erik, Selva-O'Callagham, Albert, Ribera, Josep-Maria, Cardellach, Francesc, Dominguez, Ruth, Vilardell, Miquel · 2007 · PLoS ONE

The DOI matches its Crossref record; 44 of the 46 references it cites were found in a registry or an index.

Abstract

BACKGROUND: Although peer review is widely considered to be the most credible way of selecting manuscripts and improving the quality of accepted papers in scientific journals, there is little evidence to support its use. Our aim was to estimate the effects on manuscript quality of either adding a statistical peer reviewer or suggesting the use of checklists such as CONSORT or STARD to clinical reviewers or both. METHODOLOGY AND PRINCIPAL FINDINGS: Interventions were defined as 1) the addition of a statistical reviewer to the clinical peer review process, and 2) suggesting reporting guidelines to reviewers; with "no statistical expert" and "no checklist" as controls. The two interventions were crossed in a 2x2 balanced factorial design including original research articles consecutively selected, between May 2004 and March 2005, by the Medicina Clinica (Barc) editorial committee. We randomized manuscripts to minimize differences in terms of baseline quality and type of study (intervention, longitudinal, cross-sectional, others). Sample-size calculations indicated that 100 papers provide an 80% power to test a 55% standardized difference. We specified the main outcome as the increment in quality of papers as measured on the Goodman Scale. Two blinded evaluators rated the quality of manuscripts at initial submission and final post peer review version. Of the 327 manuscripts submitted to the journal, 131 were accepted for further review, and 129 were randomized. Of those, 14 that were lost to follow-up showed no differences in initial quality to the followed-up papers. Hence, 115 were included in the main analysis, with 16 rejected for publication after peer review. 21 (18.3%) of the 115 included papers were interventions, 46 (40.0%) were longitudinal designs, 28 (24.3%) cross-sectional and 20 (17.4%) others. The 16 (13.9%) rejected papers had a significantly lower initial score on the overall Goodman scale than accepted papers (difference 15.0, 95% CI: 4.6-24.4). The effect of suggesting a guideline to the reviewers had no effect on change in overall quality as measured by the Goodman scale (0.9, 95% CI: -0.3-+2.1). The estimated effect of adding a statistical reviewer was 5.5 (95% CI: 4.3-6.7), showing a significant improvement in quality. CONCLUSIONS AND SIGNIFICANCE: This prospective randomized study shows the positive effect of adding a statistical reviewer to the field-expert peers in improving manuscript quality. We did not find a statistically significant positive effect by suggesting reviewers use reporting guidelines.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0000332

Cites

44 of 44 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Flow chart of the review process among the four randomization groups. From 131 selected papers for possible publication, 129 were randomized and allocated into four comparison groups in a 2×2 factorial design to evaluate the effects of adding a statistical expert as a reviewer and of suggesting the use of checklists. Main analyses are based on the 115 papers which completed the review process.
Figure 1 Flow chart of the review process among the four randomization groups. From 131 selected papers for possible publication, 129 were randomized and allocated into four comparison groups in a 2×2 factorial design to evaluate the effects of adding a statistical expert as a reviewer and of suggesting the use of checklists. Main analyses are based on the 115 papers which completed the review process.CC BY · PubMed Central
Figure 2 Mean quality changes, with uncertainty confidence intervals, from initial version to final version of the full Goodman scale. The overall quality is computed by the addition of the scores obtained in each one of the 36 items of the scale (first item is a summarizing measure of quality and thus, not included). The two groups without a statistical reviewer had a final-initial difference of 4.5 and 4.7 points, equivalent to a final-initial change in one item from very poor to excellent. Indeed, the two groups with a methodological referee changed 9.1 and 11.3, more than a complete reversal in the evaluation of two items. Thus, the effect size of adding a statistical reviewer is computed as the weighted average of the changes in groups CS+CSK minus changes in C+CK.
Figure 2 Mean quality changes, with uncertainty confidence intervals, from initial version to final version of the full Goodman scale. The overall quality is computed by the addition of the scores obtained in each one of the 36 items of the scale (first item is a summarizing measure of quality and thus, not included). The two groups without a statistical reviewer had a final-initial difference of 4.5 and 4.7 points, equivalent to a final-initial change in one item from very poor to excellent. Indeed, the two groups with a methodological referee changed 9.1 and 11.3, more than a complete reversal in the evaluation of two items. Thus, the effect size of adding a statistical reviewer is computed as the weighted average of the changes in groups CS+CSK minus changes in C+CK.CC BY · PubMed Central
Figure 3 Intervention effects on overall Goodman Scale. Solid boxes indicate statistically significant effects.
Figure 3 Intervention effects on overall Goodman Scale. Solid boxes indicate statistically significant effects.CC BY · PubMed Central
Figure 4 Estimated effect of statistical review over the 36 Goodman items. Solid boxes indicate statistically significant effects. 1 Differences in improvement from baseline between groups with and without statistical reviewer
Figure 4 Estimated effect of statistical review over the 36 Goodman items. Solid boxes indicate statistically significant effects. 1 Differences in improvement from baseline between groups with and without statistical reviewerCC BY · PubMed Central
Figure 5 Estimated effect of checklist suggestion on the 36 Goodman items. Solid boxes indicate statistically significant effects. 1 Differences in improvement from baseline between groups with and without checklist suggestion
Figure 5 Estimated effect of checklist suggestion on the 36 Goodman items. Solid boxes indicate statistically significant effects. 1 Differences in improvement from baseline between groups with and without checklist suggestionCC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 44 references

Research works printed but not linked: 4

  1. 2002, The fourth international congress on biomedical peer review, JAMA, 287, 2749
  2. D Altman, 1999, Statistical peer review., 157
  3. M Mora-Ripoll, 1999, Cómo mejorar la calidad estadística de los artículos presentados a revistas biomédicas: lista de comprobación para los autores., Med Clin (Barc), 113, 138
  4. C Arnau, 2003, Effect of statistical review on manuscript quality in Medicina Clinica (Barcelona): a randomized study., Med Clin (Barc), 121, 690

Cited by

See all 112

Metadata

Authors
Cobo, E., Selva-O'Callagham, A., Ribera, J.-M., Cardellach, F., Dominguez, R., & Vilardell, M.
Affiliations
  • Universitat Politècnica de Catalunya (ES)
  • Universitat de Barcelona (ES)
  • Vall d'Hebron Hospital Universitari (ES)
Corresponding author
Cobo, Erik
Venue
PLoS ONE
Published
2007-03-28
Volume
2
Issue
3
Pages
e332
ISSN
1932-6203 (ISSN-L)
OpenAlex
W2076272198
Cited by
112
References
46
Citations by year
2026: 6 · 2025: 9 · 2024: 8 · 2023: 3 · 2022: 6 · 2021: 10 · 2020: 11 · 2019: 11 · 2018: 4 · 2017: 6 · 2016: 3 · 2015: 6 · 2014: 4 · 2013: 6 · 2012: 3
FWCI
4.05
Type
journal article
Language
en
Open access
Locations
  • PLoS ONE (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Dipòsit Digital de la Universitat de Barcelona (Universitat de Barcelona) (publishedVersion, cc-by)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Figshare (submittedVersion, cc-by)
  • RECERCAT (Consorci de Serveis Universitaris de Catalunya) (submittedVersion, cc-by)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Library and Information Sciences › Academic Writing and Publishing
  • Social Sciences › Library and Information Sciences › Publishing and Scholarly Communication
Keywords
peer review, reporting quality, reporting guidelines, CONSORT, STARD, manuscript quality, editorial process, clinical research articles, journal editorial policy, manuscript rejection, publication bias, bias in clinical trials
MeSH terms
Manuscripts as Topic, Manuscripts, Medical as Topic, Peer Review, Periodicals as Topic, Research, Peer Review, Research, Checklist

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Statistical Reviewers Improve Reporting in Biomedical Articles: A Randomized Trial. https://app.cochat.ai/papers/10.1371/journal.pone.0000332
The paper
Cobo, E., Selva-O'Callagham, A., Ribera, J.-M., Cardellach, F., Dominguez, R., & Vilardell, M. (2007). Statistical Reviewers Improve Reporting in Biomedical Articles: A Randomized Trial. PLoS ONE. https://doi.org/10.1371/journal.pone.0000332

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.