CoChat

Peer review of statistics in medical research: the other problem

Open access
Bacchetti, Peter · 2002 · BMJ
Cited by 156

The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.

Abstract

Peer review has long been criticised for failing to identify flaws in research. Here Peter Bacchetti argues that it is also guilty of the opposite: finding flaws that are not there The process of peer review before publication has long been criticised for failing to prevent the publication of statistics that are wrong, unclear, or suboptimal. 1 2 My concern here, however, is not with failing to find flaws, but with the complementary problem of finding flaws that are not really there. My impression as a collaborating and consulting statistician is that spurious criticism of sound statistics is increasingly common, mainly from subject matter reviewers with limited statistical knowledge. Of the subject matter manuscript reviews I see that raise statistical issues, perhaps half include a mistaken criticism. In grant reviews unhelpful statistical comments seem to be a near certainty, mainly due to unrealistic expectations concerning sample size planning. While funding or publication of bad research is clearly undesirable, so is preventing the funding or publication of good research. Responding to misguided comments requires considerable time and effort, and poor reviews are demoralising—a subtler but possibly more serious cost. This paper discusses the problem, its causes, and what might improve the situation. Although the main focus is on statistics, many of the causes and potential improvements apply to peer review generally. #### Summary points Peer reviewers often make unfounded statistical criticisms, particularly in difficult areas such as sample size and multiple comparisons These spurious statistical comments waste time and sap morale Reasons include overvaluation of criticism for its own sake, inappropriate statistical dogmatism, time pressure, and lack of rewards for good peer reviewing Changes in the culture of peer review could improve things, particularly honouring good performance Mistaken criticism is a general problem, but may be especially acute for statistics. The examples below illustrate …

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1136/bmj.324.7348.1271

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Cited by

See all 156

Metadata

Authors
Bacchetti, P.
Venue
BMJ
Publisher
BMJ
Published
2002-05-25
Volume
324
Issue
7348
Pages
1271.1-1273
ISSN
  • 0007-1447 (ISSN-L)
  • 0959-8138
  • 1468-5833
OpenAlex
W2017627972
Cited by
156
References
4
Type
peer review
Language
en
Open access
green

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Peer review of statistics in medical research: the other problem. https://app.cochat.ai/papers/10.1136/bmj.324.7348.1271
The paper
Bacchetti, P. (2002). Peer review of statistics in medical research: the other problem. BMJ. https://doi.org/10.1136/bmj.324.7348.1271

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.