An Appraisal of the Carlisle-Stouffer-Fisher Method for Assessing Study Data Integrity and Fraud
The DOI matches its Crossref record; the references it cites are not checked yet.
Abstract
Data fabrication and scientific misconduct have been recently uncovered in the anesthesia literature, partly via the work of John Carlisle. In a recent article in Anaesthesia , Carlisle analyzed 5087 randomized clinical trials from anesthesia and general medicine journals from 2000 to 2015. He concluded that in about 6% of studies, data comparing randomized groups on baseline variables, before the given intervention, were either too similar or dissimilar compared to that expected by usual sampling variability under the null hypothesis. Carlisle used the Stouffer-Fisher method of combining P values in Table 1 (the conventional table reporting baseline patient characteristics) for each study, then calculated trial P values and assessed whether they followed a uniform distribution across studies. Extreme P values targeted studies as likely to contain data fabrication or errors. In this Statistical Grand Rounds article, we explain Carlisle’s methods, highlight perceived limitations of the proposed approach, and offer recommendations. Our main findings are (1) independence was assumed between variables in a study, which is often false and would lead to “false positive” findings; (2) an “unusual” result from a trial cannot easily be concluded to represent fraud; (3) utilized cutoff values for determining extreme P values were arbitrary; (4) trials were analyzed as if simple randomization was used, introducing bias; (5) not all P values can be accurately generated from summary statistics in a Table 1, sometimes giving incorrect conclusions; (6) small numbers of P values to assess outlier status within studies is not reliable; (7) utilized method to assess deviations from expected distributions may stack the deck; (8) P values across trials assumed to be independent; (9) P value variability not accounted for; and (10) more detailed methods needed to understand exactly what was done. It is not yet known to what extent these concerns affect the accuracy of Carlisle’s results. We recommend that Carlisle’s methods be improved before widespread use (applying them to every manuscript submitted for publication). Furthermore, lack of data integrity and fraud should ideally be assessed using multiple simultaneous statistical methods to yield more confident results. More sophisticated methods are needed for nonrandomized trials, randomized trial data reported beyond Table 1, and combating growing fraudster sophistication. We encourage all authors to more carefully scrutinize their own reporting. Finally, we believe that reporting of suspected data fraud and integrity issues should be done more discretely and directly by the involved journal to protect honest authors from the stigma of being associated with potential fraud.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The Carlisle method evaluates whether baseline-variable distributions are consistent with genuine randomization”
From Artificial intelligence to enhance research integrity in evidence-based medicine: toward scalable trustworthiness assessment (2026) · cites this paper as Mascha et al., 2017
In the citing paper
“in an analysis of 5,087 RCTs, a stringent probability threshold flagged ~1.6% of unretracted trials as having highly improbable baseline patterns and, using an author-level heuristic, highlighted Boldt, Fujii, and Reuben, three authors with multiple retracted trials”
From Artificial intelligence to enhance research integrity in evidence-based medicine: toward scalable trustworthiness assessment (2026) · cites this paper as Mascha et al., 2017
In the citing paper
“A validation study using cases of fraud and nonverified controls found that applying Benford's law to RCTs is unlikely to miss cases (100% sensitivity) but may misclassify many controls as cases (poor specificity).23”
From Assessing Research Misconduct in Randomized Controlled Trials (2021) · cites this paper as [23]
In the citing paper
“It is possible to extract numbers in one article and assess whether the frequencies of the first digits and last digits are consistent with Benford's law.23,24”
From Assessing Research Misconduct in Randomized Controlled Trials (2021) · cites this paper as [23]
In the citing paper
“Benford's law describes the distribution of the digits in naturally occurring numbers.23 The first digits tend to follow a special distribution where digits 1, 2, and 3 account for more than 60% of the total probability distribution.The last digits, however, approach uniform distribution.”
From Assessing Research Misconduct in Randomized Controlled Trials (2021) · cites this paper as [23]
Citations
The citation lists are queued
They are fetched later and appear here then.
References 18 · Cited by 30
Research works printed but not linked: 1
- Marcus, 2012, Report: Japanese anesthesiologist fabricated data in 172 studies; inquiry into PONV researcher Yoshitaka Fujii, MD, finds only three of 212 papers free of fraud., Anesthesiology News, 38(8)
Metadata
- Authors
- Mascha, E. J., Vetter, T. R., & Pittet, J.-F.
- Affiliations
- Cleveland Clinic (US)
- University of Alabama at Birmingham (US)
- Department of Anesthesiology and Perioperative Medicine, University of Alabama at Birmingham, Birmingham, Alabama.
- Corresponding author
- Mascha, Edward J.
- Venue
- Anesthesia & Analgesia
- Published
- 2017-10
- Volume
- 125
- Issue
- 4
- Pages
- 1381-1385
- ISSN
- 0003-2999 (ISSN-L)
- OpenAlex
- W2744134623
- Semantic Scholar
- b3ea07aa8995e1c4b78b07e0145098763b734ec8
- Cited by
- 30
- References
- 18
- Citations by year
- 2026: 3 · 2025: 2 · 2024: 2 · 2023: 4 · 2022: 2 · 2021: 8 · 2020: 4 · 2019: 4 · 2017: 1
- FWCI
- 5.64
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Anesthesia & Analgesia (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Education › Academic integrity and plagiarism
- Physical Sciences › Mathematics › Statistics and Probability › Benford’s Law and Fraud Detection
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- data fabrication, research integrity, p-value combination, baseline characteristics, randomized controlled trials, anesthesia research, research misconduct, outlier detection, methodological limitations
- MeSH terms
- Clinical Trials as Topic, Data Interpretation, Statistical, Fraud, Humans, Scientific Misconduct
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: An Appraisal of the Carlisle-Stouffer-Fisher Method for Assessing Study Data Integrity and Fraud. https://app.cochat.ai/papers/10.1213/ane.0000000000002415
- The paper
- Mascha, E. J., Vetter, T. R., & Pittet, J.-F. (2017). An Appraisal of the Carlisle-Stouffer-Fisher Method for Assessing Study Data Integrity and Fraud. Anesthesia & Analgesia. https://doi.org/10.1213/ane.0000000000002415
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.