CoChat

COMPare: a prospective cohort study correcting and monitoring 58 misreported trials in real time

Open access

The DOI matches its Crossref record; 15 of the 16 references it cites were found in a registry or an index; 13 of the 26 sources its claims cite were checked for support.

Abstract

BACKGROUND: Discrepancies between pre-specified and reported outcomes are an important source of bias in trials. Despite legislation, guidelines and public commitments on correct reporting from journals, outcome misreporting continues to be prevalent. We aimed to document the extent of misreporting, establish whether it was possible to publish correction letters on all misreported trials as they were published, and monitor responses from editors and trialists to understand why outcome misreporting persists despite public commitments to address it. METHODS: We identified five high-impact journals endorsing Consolidated Standards of Reporting Trials (CONSORT) (New England Journal of Medicine, The Lancet, Journal of the American Medical Association, British Medical Journal, and Annals of Internal Medicine) and assessed all trials over a six-week period to identify every correctly and incorrectly reported outcome, comparing published reports against published protocols or registry entries, using CONSORT as the gold standard. A correction letter describing all discrepancies was submitted to the journal for all misreported trials, and detailed coding sheets were shared publicly. The proportion of letters published and delay to publication were assessed over 12 months of follow-up. Correspondence received from journals and authors was documented and themes were extracted. RESULTS: Sixty-seven trials were assessed in total. Outcome reporting was poor overall and there was wide variation between journals on pre-specified primary outcomes (mean 76% correctly reported, journal range 25-96%), secondary outcomes (mean 55%, range 31-72%), and number of undeclared additional outcomes per trial (mean 5.4, range 2.9-8.3). Fifty-eight trials had discrepancies requiring a correction letter (87%, journal range 67-100%). Twenty-three letters were published (40%) with extensive variation between journals (range 0-100%). Where letters were published, there were delays (median 99 days, range 0-257 days). Twenty-nine studies had a pre-trial protocol publicly available (43%, range 0-86%). Qualitative analysis demonstrated extensive misunderstandings among journal editors about correct outcome reporting and CONSORT. Some journals did not engage positively when provided correspondence that identified misreporting; we identified possible breaches of ethics and publishing guidelines. CONCLUSIONS: All five journals were listed as endorsing CONSORT, but all exhibited extensive breaches of this guidance, and most rejected correction letters documenting shortcomings. Readers are likely to be misled by this discrepancy. We discuss the advantages of prospective methodology research sharing all data openly and pro-actively in real time as feedback on critiqued studies. This is the first empirical study of major academic journals' willingness to publish a cohort of comparable and objective correction letters on misreported high-impact studies. Suggested improvements include changes to correspondence processes at journals, alternatives for indexed post-publication peer review, changes to CONSORT's mechanisms for enforcement, and novel strategies for research on methods and reporting.

Overview

CoChat Summary

TL;DR

In a prospective cohort of 67 trials from five high-impact journals endorsing CONSORT, 58 trials (87%) had outcome-reporting discrepancies requiring correction letters, but only 23 letters (40%) were published. Journals varied widely in outcome reporting and in willingness to publish corrections, with qualitative analysis revealing extensive editor misunderstandings about CONSORT and possible breaches of publishing/ethics guidance.

Key findings

  • Of 67 assessed trials, 58 (87%, 95% CI 78–95%) had discrepancies requiring a correction letter; 23 of 58 letters (40%, 95% CI 27–53%) were published, with median publication delay 99 days (range 0–257 days).
  • Primary outcomes were correctly reported as primary in 76.3% of cases (journal range 25–96%); secondary outcomes were correctly reported in 55.1% (range 31–72%); there was a mean of 5.4 undeclared novel outcomes per trial (range 2.9–8.3), and only 13.7% of novel outcomes were declared as novel.
  • A pre-trial protocol was publicly available for 29 studies (43%, 95% CI 31–56%; journal range 0–86%). Two journals rejected all correction letters, and qualitative analysis found extensive misunderstandings about correct outcome reporting and CONSORT, plus possible breaches of ethics and publishing guidelines.

Methods

  • Prospective cohort/mixed-methods study of all trials published between 19 October and 30 November 2015 in five journals listed as endorsing CONSORT: NEJM, The Lancet, JAMA, BMJ, and Annals of Internal Medicine; 67 trials were assessed.
  • Published reports were compared against pre-trial protocols or registry entries using CONSORT as the gold standard; correction letters were submitted for all misreported trials, and publication outcomes were followed for 12 months.
  • Quantitative summary statistics were generated for outcome reporting and letter publication; correspondence with journals and authors was documented and thematically analyzed qualitatively. Coding was checked by a second researcher and senior supervising academics, and raw data were shared publicly.

Limitations

  • Stated in the paper: all included journals were very high-impact, so lower-impact journals may perform differently; the coordinated nature of the letters may have led editors to treat them differently; workload limited the sample to five journals over six weeks; and small additional coding errors remain possible, though only two miscoded outcomes were identified out of 756 discrepancies (0.26%).
  • Evident from the text: the six-week sampling window and five-journal sample restrict generalizability, and the qualitative findings depend on the authors’ coding and interpretation of journal correspondence.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s13063-019-3173-2

Cites

15 of 15 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

12 Valid · 4 Not valid · 12 Not checked · 168 Own results and interpretations

96 more claims

Create a free account to see all 196 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 67
  • It may state two separate claims as one. 50

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

1 more claim about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 15 references

Research works printed but not linked: 1

  1. In major shift, medical journal to publish protocols along with clinical trials - Retraction Watch. In: Retraction Watch [Internet]. 2016. [cited 27 Nov 2017]. Available: http://retractionwatch.com/2016/05/13/in-major-shift-Annals-to-publish-protocols-along-with-clinical-trials/ .

Cited by

See all 233

Metadata

Authors
Goldacre, B., Drysdale, H., Dale, A., Milosevic, I., Slade, E., Hartley, P., Marston, C., Powell-Smith, A., Heneghan, C., & Mahtani, K. R.
Affiliations
  • University of Oxford (GB)
  • London School of Hygiene & Tropical Medicine (GB)
Corresponding author
Goldacre, Ben
Venue
Trials
Published
2019-02-14
Volume
20
Issue
1
Pages
118-118
ISSN
1745-6215 (ISSN-L)
OpenAlex
W2913762658
Cited by
233
References
16
Citations by year
2026: 17 · 2025: 23 · 2024: 26 · 2023: 25 · 2022: 27 · 2021: 40 · 2020: 53 · 2019: 21
FWCI
23.55
Type
journal article
Language
en
Open access
Licence
  • cc-by
  • creativecommons.org/licenses/by/4.0/ (tdm)
Locations
  • Trials (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, other-oa)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • LSHTM Research Online (London School of Hygiene and Tropical Medicine) (acceptedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
Laura and John Arnold Foundation (BZR00320)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
  • Health Sciences › Nursing › Issues, ethics and legal aspects › Pharmaceutical industry and healthcare
Keywords
CONSORT, reporting bias, research data sharing, clinical trial registries
MeSH terms
Clinical Trials as Topic, Humans, Periodicals as Topic, Prospective Studies, Publishing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: COMPare: a prospective cohort study correcting and monitoring 58 misreported trials in real time. https://app.cochat.ai/papers/10.1186/s13063-019-3173-2
The paper
Goldacre, B., Drysdale, H., Dale, A., Milosevic, I., Slade, E., Hartley, P., Marston, C., Powell-Smith, A., Heneghan, C., & Mahtani, K. R. (2019). COMPare: a prospective cohort study correcting and monitoring 58 misreported trials in real time. Trials. https://doi.org/10.1186/s13063-019-3173-2

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.