Lies, damn lies, and statistics*
Open accessThe DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Abstract
In April 2010, this journal carried an editorial entitled 'Fraud or flawed: adverse impact of fabricated or poor quality research' [1]. In it, Moore et al. described the difficulty in telling when trial data were fabricated, and how systematic reviews could help by demonstrating aberrations that appeared when data from different studies were presented together. Moore et al. used data from one researcher in Japan to illustrate their point, referring to a previous comment on this researcher's data that was published 10 years earlier, with the beautifully understated title 'Reported data on granisetron and postoperative nausea and vomiting by Fujii et al. are incredibly nice!' [2]. The Journal received a number of responses to Moore et al.'s editorial, amongst them one from a reader who bemoaned the fact that the evidence base remained distorted by that researcher's work, and challenging anaesthetic journal editors to do something about it. This all happened just a few months after an unprecedented international collaboration between anaesthetic journal editors and publishers, that had led to retraction of almost 90 published papers (including six from this journal) by Joachim Boldt [3]. With this in mind, I counter-challenged the correspondent to perform an analysis of Fujii's work, more in-depth than that of Kranke etal.'s. If suggestive that Fujii's data should be removed from the evidence base, this analysis might be used by the same group of editors to confront Fujii and/or his institution(s). So as the correspondent started his work, his letter remained unpublished, and the international group of editors started to discuss how the case might be handled. That correspondent was John Carlisle, and in this issue of the Journal he presents his analysis, some 19 months, 18 reiterations, two consultations with the Committee on Publication Ethics (COPE) and three statisticians later [4]. This is the piece of work that was going to be presented as evidence to Fujii's institutions – but in the event, it was never used for this purpose, for as we reached the final stages of the manuscript's preparation, there were rapid developments in Japan. In response to questions raised over a completely separate and coincidental concern that arose in late 2011, Fujii's institution (Toho University, Tokyo) set up an investigation that led to the dismissal of Fujii in February of this year for lack of ethical approval (see Anaesthesia homepage: http://onlinelibrary.wiley.com/journal/10.1111/%28ISSN%291365-2044). Carlisle's analysis [4] is an extraordinary piece of work. Necessarily based on complicated calculations, its message is nevertheless easily accessible to the reader through its every graph and p value, that clearly show the degree to which Fujii's data deviate from what would be expected. In an accompanying editorial [5], Pandit provides an explanation of the methods used and their context. Every submission to Anaesthesia is screened using specific software for overlap with other published work before it is reviewed, to detect plagiarism [6]; could Carlisle's methods similarly form the basis of a screening method for detecting fabrication? I have described in these pages not so long ago the murky world of research misconduct [6]. One question I didn't cover in any depth is whose responsibility it is to look into suspect cases. This is explored by Wager, in another accompanying editorial [7]. As she points out, institutions don't always do as they should when approached by editors with concerns – though the institutions in both the Boldt case and the Fujii case should be congratulated for their investigations. As Wager describes, we all have responsibility: from readers through editors, reviewers and publishers to co-investigators, institutions and even governments. From an editorial point of view, being able to access COPE for advice and support, as well as other groups such as the World Association of Medical Editors (see http://www.wame.org/), is enormously useful, and such organisations have a valuable role in keeping up the pressure on all parties, and at all levels, reminding them of their obligations. Medicine – and, it seems, anaesthesia in particular – has seen a flurry of research misconduct scandals recently, with potentially serious implications for the wellbeing of patients who might have received treatments that were ineffective or harmful, and/or been denied those that were beneficial. A further, and possibly longer-lasting, harm is the damage to public trust that results from such cases and the publicity around them. Our specialty is still reeling from the Reuben [8] and Boldt cases [3]; the last thing we needed was another, even bigger, case. We can only hope that the rule that bad things come in threes holds true. Such cases are deeply troubling, raising difficult questions about the framework within which research is done, the motivation and support of those who are prepared to risk so much personally
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The analysis of 168 randomised controlled trials to test data integrity
Carlisle, J. B. (2012) · Anaesthesia · doi:10.1111/j.1365-2044.2012.07128.x · Cited by 255 · cited 2×
“That correspondent was John Carlisle, and in this issue of the Journal he presents his analysis, some 19 months, 18 reiterations, two consultations with the Committee on Publication Ethics (COPE) and three statisticians later [4].”
BackgroundFraud or flawed: adverse impact of fabricated or poor quality research
Moore, R. A., Derry, S., & McQuay, H. J. (2010) · Anaesthesia · doi:10.1111/j.1365-2044.2010.06295.x · Cited by 62
On statistical methods to test if sampling in trials is genuinely random
Pandit, J. J. (2012) · Anaesthesia · doi:10.1111/j.1365-2044.2012.07114.x · Cited by 46
Research works printed but not linked: 5
- 10.1213/00000539-200004000-00053 · DOI · Paper page
- Editors‐in‐Chief Statement Regarding IRB Approval of Clinical Trials by Joachim Boldt
- 10.1111/j.1365-2044.2010.06557.x · DOI · Paper page
- 10.1111/j.1365-2044.2012.07132.x · DOI
- 10.1213/ane.0b013e3181a16846 · DOI · Paper page
Cited by
Systematic review and statistical analysis of the integrity of 33 randomized controlled trials
Bolland, M. J., Avenell, A., Gamble, G. D., & Grey, A. (2016) · Neurology · doi:10.1212/wnl.0000000000003387 · Cited by 106
Carlisle, J. B., Dexter, F., Pandit, J. J., Shafer, S. L., & Yentis, S. M. (2015) · Anaesthesia · doi:10.1111/anae.13126 · Cited by 87
Self-citationBolland, M. J., Gamble, G. D., Avenell, A., & Grey, A. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.03.001 · Cited by 31
“1 Introduction The distribution of baseline variables in groups of randomised controlled trials (RCTs) has been used in assessment of research integrity and p otential research misconduct, when there have been prior concerns about the research [1-4].”
MethodBolland, M. J., Gamble, G. D., Avenell, A., Grey, A., & Lumley, T. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.05.006 · Cited by 27
Hüllemann, S., Schüpfer, G., & Mauch, J. (2017) · Der Anaesthesist · doi:10.1007/s00101-017-0333-1 · Cited by 25
Metadata
- Authors
- Yentis, S. M.
- Venue
- Anaesthesia
- Publisher
- Wiley
- Published
- 2012-03-08
- Volume
- 67
- Issue
- 5
- Pages
- 455-456
- ISSN
- 0003-2409 (ISSN-L)
- 1365-2044
- OpenAlex
- W1789442286
- Cited by
- 26
- References
- 3
- FWCI
- 11.46
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Lies, damn lies, and statistics*. https://app.cochat.ai/papers/10.1111/j.1365-2044.2012.07133.x
- The paper
- Yentis, S. M. (2012). Lies, damn lies, and statistics*. Anaesthesia. https://doi.org/10.1111/j.1365-2044.2012.07133.x
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.