Widening the search for suspect data – is the flood of retractions about to become a tsunami?
The DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.
Abstract
'Unfortunately the statistical war against fraud is now over and the cheaters have won'. 1. This pessimistic assertion is from a 1988 textbook by T. W. Körner in which he discussed the statistical methods suggested by J. B. S. Haldane 2 for recognising possibly fraudulent data. He continued his remarks thus: "The kind of tests proposed by Haldane depended on the fact that "higher order faking" required a great deal of computational work. The invention and accessibility of the computer means that the computational work involved has ceased to be a problem for the dishonest scientist" 1. However, three decades on, we are seeing a number of high-profile cases in which dishonest scientists, apparently unaware of Körner's 'advice', have been caught out faking results, exposed by the aberrant statistical distributions of their fraudulent data. As previously noted in an editorial 3 accompanying the exposure of one author's prolific body of fraudulent papers 4, methods being used now to detect fraud are similar to Philip and Haldane's 1939 analysis of the subsequently discredited genetic experiments of Franz Moewus 5. As a result of the recent cases, editors and other interested parties are now becoming far more aware of the potential for dishonest authors to submit fraudulent data. This follows a similar increase in awareness of the problem of plagiarism, and many editors are now taking a closer look at aspects like data distributions as well as textual similarity. It could be argued that journals, editors and other bodies charged with the oversight of research have been slow to learn the lessons of history, and to apply newer statistical methods to detect and analyse spurious or suspicious data, but this deficiency is now being addressed. Anaesthesia as a specialty, and particularly the journal Anaesthesia, can rightly claim with vicarious pride that one of its own, John Carlisle, is at the forefront of this effort. Carlisle's first statistical exposé, involving data from the randomised, controlled trials (RCTs) of Yoshitaka Fujii, made the research world stand up and take notice 4. After further refinement of the method 6, it was similarly applied to the RCTs of one of Fujii's regular collaborators, Yuhji Saitoh 7. Carlisle has now completed a further project of remarkable scale with arguably even more important implications – an analysis of 5087 RCTs, spanning eight journals and 16 years – published in this issue of Anaesthesia 8. The method of Carlisle's analysis has been published 6 and explained in detail elsewhere 3. Briefly, in a properly conducted and accurately reported RCT, differences in baseline characteristics between groups are, by definition, due to chance. For this reason, reporting p values for demographic and other baseline data is usually discouraged. The p value is the probability of random sampling resulting in a difference as large or larger than the observed difference so, because we already know that differences in baseline characteristics occurred by chance, it is uninformative to calculate a p value. Carlisle, however, has developed and refined a novel use for the statistical analysis of baseline data to identify instances where sampling in clinical trials may not have been random, suggesting the trial was either not properly conducted or was inaccurately reported. Essentially, Carlisle's method identifies papers in which the baseline characteristics (e.g. age, weight) exhibit either too narrow or too wide a distribution than expected by chance, resulting in an excess of p values close to either one or zero. Possible causes for anomalous distribution of baseline data include honest mistakes in the published information and fraudulently concocted data. A particularly extreme p value for baseline data distribution in a single study brings the accuracy or veracity of that study into question. Less extremely outlying p values for individual studies might or might not indicate a problem, but if such 'moderately aberrant' p values are found across a number of RCTs by the same author(s), then an overall p value for all those studies combined can be calculated and again, if particularly extreme, a serious problem with the data as a whole becomes likely. Particularly extreme p values for baseline distributions in both single-published RCTs and across entire bodies of work have been found, using the Carlisle method, for a number of authors involved in RCTs that have been retracted for fraud: Reuben, Boldt; Fujii; Saitoh (all anaesthetists); and Sato (an osteoporosis researcher) 4, 6, 9, 10. In the case of Fujii, publication of this statistical information became crucial evidence in bringing about the investigation of what is now one of the most infamous cases of scientific fraud in history. Carlisle's latest analysis of RCTs across six anaesthetic and two general medical journals has found a much higher than expected proportion of outliers. Carlisle has calculated a p value for this maldistribution
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The analysis of 168 randomised controlled trials to test data integrity
Carlisle, J. B. (2012) · Anaesthesia · doi:10.1111/j.1365-2044.2012.07128.x · Cited by 255
Carlisle, J. B. (2017) · Anaesthesia · doi:10.1111/anae.13938 · Cited by 229
Systematic review and statistical analysis of the integrity of 33 randomized controlled trials
Bolland, M. J., Avenell, A., Gamble, G. D., & Grey, A. (2016) · Neurology · doi:10.1212/wnl.0000000000003387 · Cited by 106 · cited 1×
“We note, however, that at least one non-anaesthesia researcher has now been successfully investigated in this way, further validating the method [10].”
Carlisle, J. B., Dexter, F., Pandit, J. J., Shafer, S. L., & Yentis, S. M. (2015) · Anaesthesia · doi:10.1111/anae.13126 · Cited by 87 · cited 2×
“The method of Carlisle’s analysis has been published [6] and explained in detail elsewhere [3].”
MethodEvidence for non‐random sampling in randomised, controlled trials by Yuhji Saitoh
Carlisle, J. B., & Loadsman, J. A. (2017) · Anaesthesia · doi:10.1111/anae.13650 · Cited by 64 · cited 3×
“In his editorial [9] accompanying publication of the other recent Carlisle analysis [7], Klein recommended all journals should follow suit in an effort to prevent publication of fraudulent material.”
BackgroundMethodInfluentialSelf-citationOn statistical methods to test if sampling in trials is genuinely random
Pandit, J. J. (2012) · Anaesthesia · doi:10.1111/j.1365-2044.2012.07114.x · Cited by 46
Cited by
Detection of data fabrication using statistical tools
Hartgerink, C. H. J., Voelkel, J. G., Wicherts, J. M., & van Assen, M. A. L. M. (2019) · doi:10.31234/osf.io/jkws4 · Cited by 16
Seeking and reporting apparent research misconduct: errors and integrity
Kharasch, E. D., & Houle, T. T. (2018) · Anaesthesia · doi:10.1111/anae.14147 · Cited by 15
Errors and Integrity in Seeking and Reporting Apparent Research Misconduct
Kharasch, E. D., & Houle, T. T. (2017) · Anesthesiology · doi:10.1097/aln.0000000000001875 · Cited by 15
How a data detective exposed suspicious medical trials.
Adam, D. (2019) · Nature · doi:10.1038/d41586-019-02241-z · Cited by 12
Metadata
- Authors
- Loadsman, J. A., & McCulloch, T. J.
- Venue
- Anaesthesia
- Publisher
- Wiley
- Published
- 2017-06-04
- Volume
- 72
- Issue
- 8
- Pages
- 931-935
- ISSN
- 0003-2409 (ISSN-L)
- 1365-2044
- OpenAlex
- W2621405679
- Cited by
- 21
- References
- 6
- FWCI
- 8.54
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.