CoChat

An Appraisal of the Carlisle-Stouffer-Fisher Method for Assessing Study Data Integrity and Fraud

Mascha, Edward J., Vetter, Thomas R., Pittet, Jean-Francois · 2017 · Anesthesia & Analgesia

The DOI matches its Crossref record; the references it cites are not checked yet.

Abstract

Data fabrication and scientific misconduct have been recently uncovered in the anesthesia literature, partly via the work of John Carlisle. In a recent article in Anaesthesia , Carlisle analyzed 5087 randomized clinical trials from anesthesia and general medicine journals from 2000 to 2015. He concluded that in about 6% of studies, data comparing randomized groups on baseline variables, before the given intervention, were either too similar or dissimilar compared to that expected by usual sampling variability under the null hypothesis. Carlisle used the Stouffer-Fisher method of combining P values in Table 1 (the conventional table reporting baseline patient characteristics) for each study, then calculated trial P values and assessed whether they followed a uniform distribution across studies. Extreme P values targeted studies as likely to contain data fabrication or errors. In this Statistical Grand Rounds article, we explain Carlisle’s methods, highlight perceived limitations of the proposed approach, and offer recommendations. Our main findings are (1) independence was assumed between variables in a study, which is often false and would lead to “false positive” findings; (2) an “unusual” result from a trial cannot easily be concluded to represent fraud; (3) utilized cutoff values for determining extreme P values were arbitrary; (4) trials were analyzed as if simple randomization was used, introducing bias; (5) not all P values can be accurately generated from summary statistics in a Table 1, sometimes giving incorrect conclusions; (6) small numbers of P values to assess outlier status within studies is not reliable; (7) utilized method to assess deviations from expected distributions may stack the deck; (8) P values across trials assumed to be independent; (9) P value variability not accounted for; and (10) more detailed methods needed to understand exactly what was done. It is not yet known to what extent these concerns affect the accuracy of Carlisle’s results. We recommend that Carlisle’s methods be improved before widespread use (applying them to every manuscript submitted for publication). Furthermore, lack of data integrity and fraud should ideally be assessed using multiple simultaneous statistical methods to yield more confident results. More sophisticated methods are needed for nonrandomized trials, randomized trial data reported beyond Table 1, and combating growing fraudster sophistication. We encourage all authors to more carefully scrutinize their own reporting. Finally, we believe that reporting of suspected data fraud and integrity issues should be done more discretely and directly by the involved journal to protect honest authors from the stigma of being associated with potential fraud.

Overview

Integrity signals

Registry

Crossref · 10.1213/ane.0000000000002415

Cites

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

The citation lists are queued

They are fetched later and appear here then.

References 18 · Cited by 30

Research works printed but not linked: 1

  1. Marcus, 2012, Report: Japanese anesthesiologist fabricated data in 172 studies; inquiry into PONV researcher Yoshitaka Fujii, MD, finds only three of 212 papers free of fraud., Anesthesiology News, 38(8)

Metadata

Authors
Mascha, E. J., Vetter, T. R., & Pittet, J.-F.
Affiliations
  • Cleveland Clinic (US)
  • University of Alabama at Birmingham (US)
  • Department of Anesthesiology and Perioperative Medicine, University of Alabama at Birmingham, Birmingham, Alabama.
Corresponding author
Mascha, Edward J.
Venue
Anesthesia & Analgesia
Published
2017-10
Volume
125
Issue
4
Pages
1381-1385
ISSN
0003-2999 (ISSN-L)
OpenAlex
W2744134623
Cited by
30
References
18
Citations by year
2026: 3 · 2025: 2 · 2024: 2 · 2023: 4 · 2022: 2 · 2021: 8 · 2020: 4 · 2019: 4 · 2017: 1
FWCI
5.64
Type
journal article
Language
en
Open access
closed
Locations
  • Anesthesia & Analgesia (publishedVersion)
  • PubMed (publishedVersion)
Topics
  • Social Sciences › Education › Academic integrity and plagiarism
  • Physical Sciences › Mathematics › Statistics and Probability › Benford’s Law and Fraud Detection
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Keywords
data fabrication, research integrity, p-value combination, baseline characteristics, randomized controlled trials, anesthesia research, research misconduct, outlier detection, methodological limitations
MeSH terms
Clinical Trials as Topic, Data Interpretation, Statistical, Fraud, Humans, Scientific Misconduct

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: An Appraisal of the Carlisle-Stouffer-Fisher Method for Assessing Study Data Integrity and Fraud. https://app.cochat.ai/papers/10.1213/ane.0000000000002415
The paper
Mascha, E. J., Vetter, T. R., & Pittet, J.-F. (2017). An Appraisal of the Carlisle-Stouffer-Fisher Method for Assessing Study Data Integrity and Fraud. Anesthesia & Analgesia. https://doi.org/10.1213/ane.0000000000002415

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.