CoChat

Accuracy in detecting inadequate research reporting by early career peer reviewers using an online CONSORT-based peer-review tool (COBPeer) versus the usual peer-review process: a cross-sectional diagnostic study

Open access

The DOI matches its Crossref record; 20 of the 21 references it cites were found in a registry or an index.

Abstract

Background: The peer review process has been questioned as it may fail to allow the publication of high-quality articles. This study aimed to evaluate the accuracy in identifying inadequate reporting in RCT reports by early career researchers (ECRs) using an online CONSORT-based peer-review tool (COBPeer) versus the usual peer-review process. Methods: We performed a cross-sectional diagnostic study of 119 manuscripts, from BMC series medical journals, BMJ, BMJ Open, and Annals of Emergency Medicine reporting the results of two-arm parallel-group RCTs. One hundred and nineteen ECRs who had never reviewed an RCT manuscript were recruited from December 2017 to January 2018. Each ECR assessed one manuscript. To assess accuracy in identifying inadequate reporting, we used two tests: (1) ECRs assessing a manuscript using the COBPeer tool (after completing an online training module) and (2) the usual peer-review process. The reference standard was the assessment of the manuscript by two systematic reviewers. Inadequate reporting was defined as incomplete reporting or a switch in primary outcome and considered nine domains: the eight most important CONSORT domains and a switch in primary outcome(s). The primary outcome was the mean number of domains accurately classified (scale from 0 to 9). Results: The mean (SD) number of domains (0 to 9) accurately classified per manuscript was 6.39 (1.49) for ECRs using COBPeer versus 5.03 (1.84) for the journal’s usual peer-review process, with a mean difference [95% CI] of 1.36 [0.88–1.84] (p < 0.001). Concerning secondary outcomes, the sensitivity of ECRs using COBPeer versus the usual peer-review process in detecting incompletely reported CONSORT items was 86% [95% CI 82–89] versus 20% [16–24] and in identifying a switch in primary outcome 61% [44–77] versus 11% [3–26]. The specificity of ECRs using COBPeer versus the usual process to detect incompletely reported CONSORT domains was 61% [57–65] versus 77% [74–81] and to identify a switch in primary outcome 77% [67–86] versus 98% [92–100]. Conclusions: Trained ECRs using the COBPeer tool were more likely to detect inadequate reporting in RCTs than the usual peer review processes used by journals. Implementing a two-step peer-review process could help improve the quality of reporting. Trial registration: Clinical.Trials.govNCT03119376 (Registered April, 18, 2017).

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s12916-019-1436-0

Cites

20 of 20 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Example of the CONSORT-based peer-review tool (COBPeer)
Fig. 1 Example of the CONSORT-based peer-review tool (COBPeer)CC BY · PubMed Central
Fig. 2 Example of the CONSORT-based peer-review tool (COBPeer)
Fig. 2 Example of the CONSORT-based peer-review tool (COBPeer)CC BY · PubMed Central
Fig. 3 Proportions of items evaluated by early career reviewers and usual peer review classified as true positive (TP), false negative (FN), true negative (TN), and false positive (FP)
Fig. 3 Proportions of items evaluated by early career reviewers and usual peer review classified as true positive (TP), false negative (FN), true negative (TN), and false positive (FP)CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

2 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 20 references

Research works printed but not linked: 1

  1. Collett D. Modelling Binary Data, texts in statistical science 2nd ed. Boca Raton: Chapman & Hall/CRC; 1999. pp. 24.

Cited by

See all 74

Metadata

Authors
Chauvin, A., Ravaud, P., Moher, D., Schriger, D., Hopewell, S., Shanahan, D., Alam, S., Baron, G., Regnaux, J.-P., Crequit, P., Martinez, V., Riveros, C., Le Cleach, L., Recchioni, A., Altman, D. G., & Boutron, I.
Affiliations
  • Inserm (FR)
  • University of Ottawa (CA)
  • University of California, Los Angeles (US)
  • University of Oxford (GB)
  • Cochrane (GB)
  • Taylor and Francis (United Kingdom) (GB)
  • Assistance Publique – Hôpitaux de Paris (FR)
  • BioMed Central (United Kingdom) (GB)
Corresponding author
Chauvin, Anthony
Venue
BMC Medicine
Published
2019-11-19
Volume
17
Issue
1
Pages
205-205
ISSN
1741-7015 (ISSN-L)
OpenAlex
W2990790246
Cited by
72
References
21
Citations by year
2026: 4 · 2025: 13 · 2024: 7 · 2023: 4 · 2022: 15 · 2021: 23 · 2020: 6
FWCI
7.52
Type
journal article
Language
en
Open access
Licence
Locations
  • BMC Medicine (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • HAL (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
Cancer Research UK (16895)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
  • Social Sciences › Library and Information Sciences › Publishing and Scholarly Communication
Keywords
early-career researchers, reporting quality, CONSORT, randomized controlled trials, reporting guidelines, diagnostic accuracy, sensitivity and specificity, emergency medicine research, medical journals, publication bias
MeSH terms
Cross-Sectional Studies, Humans, Peer Review, Periodicals as Topic, Publishing, Research Report

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Accuracy in detecting inadequate research reporting by early career peer reviewers using an online CONSORT-based peer-review tool (COBPeer) versus the usual peer-review process: a cross-sectional diagnostic study. https://app.cochat.ai/papers/10.1186/s12916-019-1436-0
The paper
Chauvin, A., Ravaud, P., Moher, D., Schriger, D., Hopewell, S., Shanahan, D., Alam, S., Baron, G., Regnaux, J.-P., Crequit, P., Martinez, V., Riveros, C., Le Cleach, L., Recchioni, A., Altman, D. G., & Boutron, I. (2019). Accuracy in detecting inadequate research reporting by early career peer reviewers using an online CONSORT-based peer-review tool (COBPeer) versus the usual peer-review process: a cross-sectional diagnostic study. BMC Medicine. https://doi.org/10.1186/s12916-019-1436-0

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.