COMPare: a prospective cohort study correcting and monitoring 58 misreported trials in real time
Open accessThe DOI matches its Crossref record; 15 of the 16 references it cites were found in a registry or an index; 13 of the 26 sources its claims cite were checked for support.
Abstract
BACKGROUND: Discrepancies between pre-specified and reported outcomes are an important source of bias in trials. Despite legislation, guidelines and public commitments on correct reporting from journals, outcome misreporting continues to be prevalent. We aimed to document the extent of misreporting, establish whether it was possible to publish correction letters on all misreported trials as they were published, and monitor responses from editors and trialists to understand why outcome misreporting persists despite public commitments to address it. METHODS: We identified five high-impact journals endorsing Consolidated Standards of Reporting Trials (CONSORT) (New England Journal of Medicine, The Lancet, Journal of the American Medical Association, British Medical Journal, and Annals of Internal Medicine) and assessed all trials over a six-week period to identify every correctly and incorrectly reported outcome, comparing published reports against published protocols or registry entries, using CONSORT as the gold standard. A correction letter describing all discrepancies was submitted to the journal for all misreported trials, and detailed coding sheets were shared publicly. The proportion of letters published and delay to publication were assessed over 12 months of follow-up. Correspondence received from journals and authors was documented and themes were extracted. RESULTS: Sixty-seven trials were assessed in total. Outcome reporting was poor overall and there was wide variation between journals on pre-specified primary outcomes (mean 76% correctly reported, journal range 25-96%), secondary outcomes (mean 55%, range 31-72%), and number of undeclared additional outcomes per trial (mean 5.4, range 2.9-8.3). Fifty-eight trials had discrepancies requiring a correction letter (87%, journal range 67-100%). Twenty-three letters were published (40%) with extensive variation between journals (range 0-100%). Where letters were published, there were delays (median 99 days, range 0-257 days). Twenty-nine studies had a pre-trial protocol publicly available (43%, range 0-86%). Qualitative analysis demonstrated extensive misunderstandings among journal editors about correct outcome reporting and CONSORT. Some journals did not engage positively when provided correspondence that identified misreporting; we identified possible breaches of ethics and publishing guidelines. CONCLUSIONS: All five journals were listed as endorsing CONSORT, but all exhibited extensive breaches of this guidance, and most rejected correction letters documenting shortcomings. Readers are likely to be misled by this discrepancy. We discuss the advantages of prospective methodology research sharing all data openly and pro-actively in real time as feedback on critiqued studies. This is the first empirical study of major academic journals' willingness to publish a cohort of comparable and objective correction letters on misreported high-impact studies. Suggested improvements include changes to correspondence processes at journals, alternatives for indexed post-publication peer review, changes to CONSORT's mechanisms for enforcement, and novel strategies for research on methods and reporting.
Overview
CoChat Summary
TL;DR
In a prospective cohort of 67 trials from five high-impact journals endorsing CONSORT, 58 trials (87%) had outcome-reporting discrepancies requiring correction letters, but only 23 letters (40%) were published. Journals varied widely in outcome reporting and in willingness to publish corrections, with qualitative analysis revealing extensive editor misunderstandings about CONSORT and possible breaches of publishing/ethics guidance.
Key findings
- Of 67 assessed trials, 58 (87%, 95% CI 78–95%) had discrepancies requiring a correction letter; 23 of 58 letters (40%, 95% CI 27–53%) were published, with median publication delay 99 days (range 0–257 days).
- Primary outcomes were correctly reported as primary in 76.3% of cases (journal range 25–96%); secondary outcomes were correctly reported in 55.1% (range 31–72%); there was a mean of 5.4 undeclared novel outcomes per trial (range 2.9–8.3), and only 13.7% of novel outcomes were declared as novel.
- A pre-trial protocol was publicly available for 29 studies (43%, 95% CI 31–56%; journal range 0–86%). Two journals rejected all correction letters, and qualitative analysis found extensive misunderstandings about correct outcome reporting and CONSORT, plus possible breaches of ethics and publishing guidelines.
Methods
- Prospective cohort/mixed-methods study of all trials published between 19 October and 30 November 2015 in five journals listed as endorsing CONSORT: NEJM, The Lancet, JAMA, BMJ, and Annals of Internal Medicine; 67 trials were assessed.
- Published reports were compared against pre-trial protocols or registry entries using CONSORT as the gold standard; correction letters were submitted for all misreported trials, and publication outcomes were followed for 12 months.
- Quantitative summary statistics were generated for outcome reporting and letter publication; correspondence with journals and authors was documented and thematically analyzed qualitatively. Coding was checked by a second researcher and senior supervising academics, and raw data were shared publicly.
Limitations
- Stated in the paper: all included journals were very high-impact, so lower-impact journals may perform differently; the coordinated nature of the letters may have led editors to treat them differently; workload limited the sample to five journals over six weeks; and small additional coding errors remain possible, though only two miscoded outcomes were identified out of 756 discrepancies (0.26%).
- Evident from the text: the six-week sampling window and five-journal sample restrict generalizability, and the qualitative findings depend on the authors’ coding and interpretation of journal correspondence.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
12 Valid · 4 Not valid · 12 Not checked · 168 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“Discrepancies between pre-specified and reported outcomes are an important and widespread source of bias in clinical trials [1].”
§ COMPare: a prospective cohort study correcting and monitoring 58 misreported trials in real time > Background
What the sources say
- Supportedcites [1] Jones CW, Keil LG, Holland WC, Caughey MC, Platts-Mills TF. Comparison of registered and published outcomes in randomized controlled trials: a systematic review. BMC Med. 2015;13:282. source
The Abstract Results report a median primary-outcome discrepancy rate of 31% across included studies, supporting that discrepancies are widespread. The Results (Subgroup analyses within included studies) report that outcome changes frequently favored statistically favorable published results, supporting the concern about bias. (LLM, conf=0.90)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Where outcome misreporting is permitted, it increases the likelihood that reported differences have arisen through chance or are exaggerated [2, 3].”
§ COMPare: a prospective cohort study correcting and monitoring 58 misreported trials in real time > Background
What the sources say
- Partly supportedcites [2] Hart B, Lundh A, Bero L. Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses. BMJ. 2012;344:d7202. sourceExperimental — accuracy not yet measured
The Introduction and Meaning of study sections support the concern that selective outcome reporting can distort or exaggerate estimated treatment effects in meta-analyses of drug trials. The Discussion also reports that adding unpublished data increased efficacy estimates as often as it decreased them. The paper does not establish that misreporting makes reported differences more likely to have arisen through chance. (LLM, conf=0.83)
- Supportedcites [3] Kirkham JJ, Dwan KM, Altman DG, Gamble C, Dodd S, Smyth R, et al. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ. 2010;340:c365. source
The research methods and reporting section says statistically significant outcomes are more likely to be fully reported. Its results show that, after adjustment for suspected outcome reporting bias, some apparently significant treatment differences became nonsignificant and some effect estimates fell substantially. This supports the claim that selective outcome reporting can make differences appear more convincing or larger than they are; the paper does not establish that any particular difference arose by chance. (LLM, conf=0.86)
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“A 2015 systematic review [1] found 27 studies comparing pre-specified outcomes against those reported, in cohorts of between 1 and 198 trials (median n = 65 trials).”
§ COMPare: a prospective cohort study correcting and monitoring 58 misreported trials in real time > Background
What the sources say
- Supportedcites [1] Jones CW, Keil LG, Holland WC, Caughey MC, Platts-Mills TF. Comparison of registered and published outcomes in randomized controlled trials: a systematic review. BMC Med. 2015;13:282. source
The Results, under “Description of included studies” (page unavailable), identify 27 studies comparing registered and published primary outcomes and report a median of 65 trials per study. Table 1 in that section lists studies assessing as few as 1 trial and as many as 198. (LLM, conf=0.94)
plainn = 65- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Eight studies also assessed the impact of outcome switching on the statistical significance of the published outcome and found that outcome switching favoured the reporting of significant outcomes in half the trials.”
§ COMPare: a prospective cohort study correcting and monitoring 58 misreported trials in real time > Background
What the sources say
- Partly supportedcites [1] Jones CW, Keil LG, Holland WC, Caughey MC, Platts-Mills TF. Comparison of registered and published outcomes in randomized controlled trials: a systematic review. BMC Med. 2015;13:282. sourceExperimental — accuracy not yet measured
The Results section (‘Subgroup analyses within included studies’) says eight studies examined this question and reports a median of 50% for outcome changes favouring statistically significant published results when their impact could be assessed. The claim omits that qualifier and presents the median as though it were a finding for half of all trials; the paper says the impact could not be assessed in as many as half of the included trials. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“two additional cohort studies published in 2008 [14] and 2016 [15] report similar prevalence for discrepancies as before.”
§ COMPare: a prospective cohort study correcting and monitoring 58 misreported trials in real time > Discussion > Context of previous work
What the sources say
- Not checked — no full text of the sourcecites [14] Al-Marzouki S, Sanaa A-M, Ian R, Stephen E, Tom M. Selective reporting in clinical trials: analysis of trial protocols accepted by The Lancet. Lancet. 2008;372:201. source
- Supportedcites [15] Weston J, Dwan K, Altman D, Clarke M, Gamble C, Schroter S, et al. Feasibility study to examine discrepancy rates in pre-specified and reported outcomes in articles submitted to The BMJ. BMJ Open. 2016;6:e010075. source
The Results (Table 2) report outcome discrepancies in trials submitted to The BMJ: 19–27% of prespecified outcomes were not reported, and 11–14% of reported outcomes were not prespecified. The Discussion’s “Comparison with other studies” section gives earlier ranges of 13–31% and 10–18%, respectively, supporting the claim that this 2016 study found similar prevalence. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
96 more claims
Create a free account to see all 196 claims and every source verdict.
- Probably not a substantial claim. 67
- It may state two separate claims as one. 50
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The COMPare project demonstrated that outcome switching is pervasive but currently trackable only through laborious manual comparison”
From Artificial intelligence to enhance research integrity in evidence-based medicine: toward scalable trustworthiness assessment (2026) · cites this paper as Goldacre et al., 2019
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The COMPare Trials Project, examining 67 trials in five top medical journals, found that each trial reported only 25–96% of pre-specified primary outcomes while silently adding an average of 5.4 new outcomes per trial”
From Artificial intelligence to enhance research integrity in evidence-based medicine: toward scalable trustworthiness assessment (2026) · cites this paper as Goldacre et al., 2019
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Having access to these documents may help with trial appraisal, interpretation, and replication, as well as identification of selective outcome reporting.81 82”
From SPIRIT 2025 explanation and elaboration: updated guideline for protocols of randomised trials (2025) · cites this paper as [81]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In a more recent analysis, the COMPare study found that results were missing for 42% of outcomes prespecified in 67 trials published between October 2015 and January 2016 in the world's top five general medical journals 28 .”
From Investigating and dealing with publication bias and other reporting biases in meta-analyses of health research: a review (2021) · cites this paper as [28]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“outcome switching often requires registration-publication comparison”
From Artificial intelligence to enhance research integrity in evidence-based medicine: toward scalable trustworthiness assessment (2026) · cites this paper as Goldacre et al., 2019
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“| Missing primary outcomes | Percentage of trials with any unreported primary outcomes | 80.0% | 66.7% | 15.4% | 16.7% | 5.0% | 19.4% |”
§ COMPare: a prospective cohort study correcting and monitoring 58 misreported trials in real time > Results > Outcome reporting quality
Reported 5 · recomputed 4.545455
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ COMPare: a prospective cohort study correcting and monitoring 58 misreported trials in real time > Results > Themes in responses from journals
“The Lancet published 15 out of 20 letters, mostly with accompanying responses from trialists”
§ COMPare: a prospective cohort study correcting and monitoring 58 misreported trials in real time > Results > Narrative account of individual journals’ responses
“The Lancet published 16 out of 20 COMPare letters, mostly with author replies.”
Citations
Cites
CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials
the CONSORT Group, Schulz, K. F., Altman, D. G., & Moher, D. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-18 · Cited by 13,371
CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials
Schulz, K. F., Altman, D. G., Moher, D., & for the CONSORT Group (2010) · BMJ · doi:10.1136/bmj.c332 · Cited by 9,329
Kirkham, J. J., Dwan, K. M., Altman, D. G., Gamble, C., Dodd, S., Smyth, R., & Williamson, P. R. (2010) · BMJ · doi:10.1136/bmj.c365 · Cited by 1,133
How Open Science Helps Researchers Succeed
McKiernan, E. C., Bourne, P. E., Brown, C. T., Buck, S., Kenall, A., Lin, J., McDougall, D., Nosek, B. A., Ram, K., Soderberg, C. K., et al. (2016) · eLife · doi:10.7554/elife.16800 · Cited by 798
Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses
Hart, B., Lundh, A., & Bero, L. (2012) · BMJ · doi:10.1136/bmj.d7202 · Cited by 255
Reproducibility: A tragedy of errors
Allison, D. B., Brown, A. W., George, B. J., & Kaiser, K. A. (2016) · Nature · doi:10.1038/530027a · Cited by 255
Comparison of protocols and registry entries to published reports for randomised controlled trials
Dwan, K., Altman, D. G., Cresswell, L., Blundell, M., Gamble, C. L., & Williamson, P. R. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000031.pub2 · Cited by 243 · cited 1×
“The current Cochrane review of studies examining discrepancies between protocols or registry entries and published trial reports [16] reports high prevalence for numerous other related reporting flaws, including inconsistencies on sample size calculation, blinding, method of allocation concealment, subgroup analyses,…”
BackgroundComparison of registered and published outcomes in randomized controlled trials: a systematic review
Jones, C. W., Keil, L. G., Holland, W. C., Caughey, M. C., & Platts-Mills, T. F. (2015) · BMC Medicine · doi:10.1186/s12916-015-0520-3 · Cited by 237
Selective reporting in clinical trials: analysis of trial protocols accepted by The Lancet
Al-Marzouki, S., Roberts, I., Evans, S., & Marshall, T. (2008) · The Lancet · doi:10.1016/s0140-6736(08)61060-0 · Cited by 119
Make journals report clinical trials properly
Goldacre, B. M. (2016) · Nature · doi:10.1038/530007a · Cited by 94
Self-citation
Research works printed but not linked: 1
- In major shift, medical journal to publish protocols along with clinical trials - Retraction Watch. In: Retraction Watch [Internet]. 2016. [cited 27 Nov 2017]. Available: http://retractionwatch.com/2016/05/13/in-major-shift-Annals-to-publish-protocols-along-with-clinical-trials/ .
Cited by
Guidelines for Reporting Outcomes in Trial Reports: The CONSORT-Outcomes 2022 Extension
Butcher, N. J., Monsour, A., Mew, E. J., Chan, A.-W., Moher, D., Mayo-Wilson, E., Terwee, C. B., Chee-A-Tow, A., Baba, A., Gavin, F., et al. (2022) · JAMA · doi:10.1001/jama.2022.21022 · Cited by 719
Page, M. J., Sterne, J. A. C., Higgins, J. P. T., & Egger, M. (2021) · Research Synthesis Methods · doi:10.1002/jrsm.1468 · Cited by 294
“In a more recent analysis, the COMPare study found that results were missing for 42% of outcomes prespecified in 67 trials published between October 2015 and January 2016 in the world's top five general medical journals.(28) Other studies suggest that statistically significant results for beneficial outcomes had highe…”
Ten common statistical mistakes to watch out for when writing or reviewing a manuscript
Makin, T. R., & Orban de Xivry, J.-J. (2019) · eLife · doi:10.7554/elife.48175 · Cited by 238
Claesen, A., Gomes, S., Tuerlinckx, F., & Vanpaemel, W. (2021) · Royal Society Open Science · doi:10.1098/rsos.211037 · Cited by 197
CONSORT 2025 explanation and elaboration: updated guideline for reporting randomised trials
Hopewell, S., Chan, A.-W., Collins, G. S., Hróbjartsson, A., Moher, D., Schulz, K. F., Tunn, R., Aggarwal, R., Berkwits, M., Berlin, J. A., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081124 · Cited by 205
Metadata
- Authors
- Goldacre, B., Drysdale, H., Dale, A., Milosevic, I., Slade, E., Hartley, P., Marston, C., Powell-Smith, A., Heneghan, C., & Mahtani, K. R.
- Affiliations
- University of Oxford (GB)
- London School of Hygiene & Tropical Medicine (GB)
- Corresponding author
- Goldacre, Ben
- Venue
- Trials
- Published
- 2019-02-14
- Volume
- 20
- Issue
- 1
- Pages
- 118-118
- ISSN
- 1745-6215 (ISSN-L)
- PMC
- PMC6375128
- OpenAlex
- W2913762658
- Cited by
- 233
- References
- 16
- Citations by year
- 2026: 17 · 2025: 23 · 2024: 26 · 2023: 25 · 2022: 27 · 2021: 40 · 2020: 53 · 2019: 21
- FWCI
- 23.55
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0/ (tdm)
- Locations
- Trials (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- LSHTM Research Online (London School of Hygiene and Tropical Medicine) (acceptedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- Laura and John Arnold Foundation (BZR00320)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Health Sciences › Nursing › Issues, ethics and legal aspects › Pharmaceutical industry and healthcare
- Keywords
- CONSORT, reporting bias, research data sharing, clinical trial registries
- MeSH terms
- Clinical Trials as Topic, Humans, Periodicals as Topic, Prospective Studies, Publishing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: COMPare: a prospective cohort study correcting and monitoring 58 misreported trials in real time. https://app.cochat.ai/papers/10.1186/s13063-019-3173-2
- The paper
- Goldacre, B., Drysdale, H., Dale, A., Milosevic, I., Slade, E., Hartley, P., Marston, C., Powell-Smith, A., Heneghan, C., & Mahtani, K. R. (2019). COMPare: a prospective cohort study correcting and monitoring 58 misreported trials in real time. Trials. https://doi.org/10.1186/s13063-019-3173-2
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.