Accuracy in detecting inadequate research reporting by early career peer reviewers using an online CONSORT-based peer-review tool (COBPeer) versus the usual peer-review process: a cross-sectional diagnostic study
Open accessThe DOI matches its Crossref record; 20 of the 21 references it cites were found in a registry or an index.
Abstract
Background: The peer review process has been questioned as it may fail to allow the publication of high-quality articles. This study aimed to evaluate the accuracy in identifying inadequate reporting in RCT reports by early career researchers (ECRs) using an online CONSORT-based peer-review tool (COBPeer) versus the usual peer-review process. Methods: We performed a cross-sectional diagnostic study of 119 manuscripts, from BMC series medical journals, BMJ, BMJ Open, and Annals of Emergency Medicine reporting the results of two-arm parallel-group RCTs. One hundred and nineteen ECRs who had never reviewed an RCT manuscript were recruited from December 2017 to January 2018. Each ECR assessed one manuscript. To assess accuracy in identifying inadequate reporting, we used two tests: (1) ECRs assessing a manuscript using the COBPeer tool (after completing an online training module) and (2) the usual peer-review process. The reference standard was the assessment of the manuscript by two systematic reviewers. Inadequate reporting was defined as incomplete reporting or a switch in primary outcome and considered nine domains: the eight most important CONSORT domains and a switch in primary outcome(s). The primary outcome was the mean number of domains accurately classified (scale from 0 to 9). Results: The mean (SD) number of domains (0 to 9) accurately classified per manuscript was 6.39 (1.49) for ECRs using COBPeer versus 5.03 (1.84) for the journal’s usual peer-review process, with a mean difference [95% CI] of 1.36 [0.88–1.84] (p < 0.001). Concerning secondary outcomes, the sensitivity of ECRs using COBPeer versus the usual peer-review process in detecting incompletely reported CONSORT items was 86% [95% CI 82–89] versus 20% [16–24] and in identifying a switch in primary outcome 61% [44–77] versus 11% [3–26]. The specificity of ECRs using COBPeer versus the usual process to detect incompletely reported CONSORT domains was 61% [57–65] versus 77% [74–81] and to identify a switch in primary outcome 77% [67–86] versus 98% [92–100]. Conclusions: Trained ECRs using the COBPeer tool were more likely to detect inadequate reporting in RCTs than the usual peer review processes used by journals. Implementing a two-step peer-review process could help improve the quality of reporting. Trial registration: Clinical.Trials.govNCT03119376 (Registered April, 18, 2017).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The bullet-point structure is new to PRISMA 2020 and has been adopted to facilitate implementation of the guidance [60, 61].”
From The PRISMA 2020 statement: an updated guideline for reporting systematic reviews (2021) · cites this paper as [61]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Various strategies to increase the use of reporting guidelines and improve reporting have been proposed.”
From The PRISMA 2020 statement: An updated guideline for reporting systematic reviews (2021) · cites this paper as [61]
The cited source says what the claim attributes to it.
In the citing paper
“Various strategies to increase the use of reporting guidelines and improve reporting have been proposed.”
From The PRISMA 2020 statement: an updated guideline for reporting systematic reviews (2021) · cites this paper as [61]
In the citing paper
“La estructura de vin ˜etas es nueva en PRISMA 2020 y se ha adoptado para facilitar la implantacio ´ n de la guı ´a 56,57 .”
From Declaración PRISMA 2020: una guía actualizada para la publicación de revisiones sistemáticas (2021) · cites this paper as [57]
In the citing paper
“La estructura de vin ˜etas es nueva en PRISMA 2020 y se ha adoptado para facilitar la implantacio ´ n de la guı ´a 56,57 .”
From Declaración PRISMA 2020: una guía actualizada para la publicación de revisiones sistemáticas (2021) · cites this paper as [57]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Moher, D., Schulz, K. F., & Altman, D. G. (2001) · The Lancet · doi:10.1016/s0140-6736(00)04337-3 · Cited by 5,373
Self-citationSTARD 2015: an updated list of essential items for reporting diagnostic accuracy studies
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2015) · BMJ · doi:10.1136/bmj.h5527 · Cited by 3,660
Self-citationMoher, D., Schulz, K. F., Altman, D. G., & for the CONSORT Group* (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-134-8-200104170-00011 · Cited by 1,426
Self-citationBohannon, J. (2013) · Science · doi:10.1126/science.342.6154.60 · Cited by 934
Likelihood ratios with confidence: Sample size estimation for diagnostic test studies
Simel, D. L., Samsa, G. P., & Matchar, D. B. (1991) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(91)90128-v · Cited by 932
Boutron, I. (2010) · JAMA · doi:10.1001/jama.2010.651 · Cited by 842
Self-citationPeer review. Crude and understudied, but indispensable
Kassirer, J. P. (1994) · JAMA · doi:10.1001/jama.272.2.96 · Cited by 235
Applied analysis of variance in behavioral science
Edwards, L. K. (1993) · M. Dekker eBooks · Cited by 233
COMPare: a prospective cohort study correcting and monitoring 58 misreported trials in real time
Goldacre, B., Drysdale, H., Dale, A., Milosevic, I., Slade, E., Hartley, P., Marston, C., Powell-Smith, A., Heneghan, C., & Mahtani, K. R. (2019) · Trials · doi:10.1186/s13063-019-3173-2 · Cited by 233
What errors do peer reviewers detect, and does training improve their ability to detect them?
Schroter, S., Black, N., Evans, S., Godlee, F., Osorio, L., & Smith, R. (2008) · Journal of the Royal Society of Medicine · doi:10.1258/jrsm.2008.080062 · Cited by 226
Research works printed but not linked: 1
- Collett D. Modelling Binary Data, texts in statistical science 2nd ed. Boca Raton: Chapman & Hall/CRC; 1999. pp. 24.
Cited by
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283
Self-citationThe PRISMA 2020 statement: an updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01626-4 · Cited by 13,874
Self-citationPage, M. J., Moher, D., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n160 · Cited by 13,262
Self-citationThe PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · International Journal of Surgery · doi:10.1016/j.ijsu.2021.105906 · Cited by 11,748
Self-citationThe PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · PLoS Medicine · doi:10.1371/journal.pmed.1003583 · Cited by 5,148
“They include educators introducing reporting guidelines into graduate curricula to promote good reporting habits of early career scientists [65]; journal editors and regulators endorsing use of reporting guidelines [18]; peer reviewers evaluating adherence to reporting guidelines [61, 66]; journals requiring authors t…”
BackgroundMethodSelf-citation
Metadata
- Authors
- Chauvin, A., Ravaud, P., Moher, D., Schriger, D., Hopewell, S., Shanahan, D., Alam, S., Baron, G., Regnaux, J.-P., Crequit, P., Martinez, V., Riveros, C., Le Cleach, L., Recchioni, A., Altman, D. G., & Boutron, I.
- Affiliations
- Inserm (FR)
- University of Ottawa (CA)
- University of California, Los Angeles (US)
- University of Oxford (GB)
- Cochrane (GB)
- Taylor and Francis (United Kingdom) (GB)
- Assistance Publique – Hôpitaux de Paris (FR)
- BioMed Central (United Kingdom) (GB)
- Corresponding author
- Chauvin, Anthony
- ORCID
- Venue
- BMC Medicine
- Published
- 2019-11-19
- Volume
- 17
- Issue
- 1
- Pages
- 205-205
- ISSN
- 1741-7015 (ISSN-L)
- PMC
- PMC6864983
- OpenAlex
- W2990790246
- Cited by
- 72
- References
- 21
- Citations by year
- 2026: 4 · 2025: 13 · 2024: 7 · 2023: 4 · 2022: 15 · 2021: 23 · 2020: 6
- FWCI
- 7.52
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0/ (tdm)
- creativecommons.org/licenses/by/4.0/ (vor)
- Locations
- BMC Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- HAL (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- Cancer Research UK (16895)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Library and Information Sciences › Publishing and Scholarly Communication
- Keywords
- early-career researchers, reporting quality, CONSORT, randomized controlled trials, reporting guidelines, diagnostic accuracy, sensitivity and specificity, emergency medicine research, medical journals, publication bias
- MeSH terms
- Cross-Sectional Studies, Humans, Peer Review, Periodicals as Topic, Publishing, Research Report
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Accuracy in detecting inadequate research reporting by early career peer reviewers using an online CONSORT-based peer-review tool (COBPeer) versus the usual peer-review process: a cross-sectional diagnostic study. https://app.cochat.ai/papers/10.1186/s12916-019-1436-0
- The paper
- Chauvin, A., Ravaud, P., Moher, D., Schriger, D., Hopewell, S., Shanahan, D., Alam, S., Baron, G., Regnaux, J.-P., Crequit, P., Martinez, V., Riveros, C., Le Cleach, L., Recchioni, A., Altman, D. G., & Boutron, I. (2019). Accuracy in detecting inadequate research reporting by early career peer reviewers using an online CONSORT-based peer-review tool (COBPeer) versus the usual peer-review process: a cross-sectional diagnostic study. BMC Medicine. https://doi.org/10.1186/s12916-019-1436-0
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.