CoChat

Widening the search for suspect data – is the flood of retractions about to become a tsunami?

Loadsman, J. A., McCulloch, T. J. · 2017 · Anaesthesia
Cited by 21

The DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.

Abstract

'Unfortunately the statistical war against fraud is now over and the cheaters have won'. 1. This pessimistic assertion is from a 1988 textbook by T. W. Körner in which he discussed the statistical methods suggested by J. B. S. Haldane 2 for recognising possibly fraudulent data. He continued his remarks thus: "The kind of tests proposed by Haldane depended on the fact that "higher order faking" required a great deal of computational work. The invention and accessibility of the computer means that the computational work involved has ceased to be a problem for the dishonest scientist" 1. However, three decades on, we are seeing a number of high-profile cases in which dishonest scientists, apparently unaware of Körner's 'advice', have been caught out faking results, exposed by the aberrant statistical distributions of their fraudulent data. As previously noted in an editorial 3 accompanying the exposure of one author's prolific body of fraudulent papers 4, methods being used now to detect fraud are similar to Philip and Haldane's 1939 analysis of the subsequently discredited genetic experiments of Franz Moewus 5. As a result of the recent cases, editors and other interested parties are now becoming far more aware of the potential for dishonest authors to submit fraudulent data. This follows a similar increase in awareness of the problem of plagiarism, and many editors are now taking a closer look at aspects like data distributions as well as textual similarity. It could be argued that journals, editors and other bodies charged with the oversight of research have been slow to learn the lessons of history, and to apply newer statistical methods to detect and analyse spurious or suspicious data, but this deficiency is now being addressed. Anaesthesia as a specialty, and particularly the journal Anaesthesia, can rightly claim with vicarious pride that one of its own, John Carlisle, is at the forefront of this effort. Carlisle's first statistical exposé, involving data from the randomised, controlled trials (RCTs) of Yoshitaka Fujii, made the research world stand up and take notice 4. After further refinement of the method 6, it was similarly applied to the RCTs of one of Fujii's regular collaborators, Yuhji Saitoh 7. Carlisle has now completed a further project of remarkable scale with arguably even more important implications – an analysis of 5087 RCTs, spanning eight journals and 16 years – published in this issue of Anaesthesia 8. The method of Carlisle's analysis has been published 6 and explained in detail elsewhere 3. Briefly, in a properly conducted and accurately reported RCT, differences in baseline characteristics between groups are, by definition, due to chance. For this reason, reporting p values for demographic and other baseline data is usually discouraged. The p value is the probability of random sampling resulting in a difference as large or larger than the observed difference so, because we already know that differences in baseline characteristics occurred by chance, it is uninformative to calculate a p value. Carlisle, however, has developed and refined a novel use for the statistical analysis of baseline data to identify instances where sampling in clinical trials may not have been random, suggesting the trial was either not properly conducted or was inaccurately reported. Essentially, Carlisle's method identifies papers in which the baseline characteristics (e.g. age, weight) exhibit either too narrow or too wide a distribution than expected by chance, resulting in an excess of p values close to either one or zero. Possible causes for anomalous distribution of baseline data include honest mistakes in the published information and fraudulently concocted data. A particularly extreme p value for baseline data distribution in a single study brings the accuracy or veracity of that study into question. Less extremely outlying p values for individual studies might or might not indicate a problem, but if such 'moderately aberrant' p values are found across a number of RCTs by the same author(s), then an overall p value for all those studies combined can be calculated and again, if particularly extreme, a serious problem with the data as a whole becomes likely. Particularly extreme p values for baseline distributions in both single-published RCTs and across entire bodies of work have been found, using the Carlisle method, for a number of authors involved in RCTs that have been retracted for fraud: Reuben, Boldt; Fujii; Saitoh (all anaesthetists); and Sato (an osteoporosis researcher) 4, 6, 9, 10. In the case of Fujii, publication of this statistical information became crucial evidence in bringing about the investigation of what is now one of the most infamous cases of scientific fraud in history. Carlisle's latest analysis of RCTs across six anaesthetic and two general medical journals has found a much higher than expected proportion of outliers. Carlisle has calculated a p value for this maldistribution

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1111/anae.13962

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 6 references

Cited by

See all 21

Metadata

Authors
Loadsman, J. A., & McCulloch, T. J.
Venue
Anaesthesia
Publisher
Wiley
Published
2017-06-04
Volume
72
Issue
8
Pages
931-935
ISSN
  • 0003-2409 (ISSN-L)
  • 1365-2044
OpenAlex
W2621405679
Cited by
21
References
6
FWCI
8.54
Type
journal article
Language
en
Open access
closed

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Widening the search for suspect data – is the flood of retractions about to become a tsunami? https://app.cochat.ai/papers/10.1111/anae.13962
The paper
Loadsman, J. A., & McCulloch, T. J. (2017). Widening the search for suspect data – is the flood of retractions about to become a tsunami? Anaesthesia. https://doi.org/10.1111/anae.13962

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.