CoChat
Sign Up
Cited by 229Academic integrity and plagiarismBenford’s Law and Fraud DetectionAcademic Publishing and Open Access

Data fabrication and other reasons for non‐random sampling in 5087 randomised, controlled trials in anaesthetic and general medical journals

Open access
Carlisle, J. B.·2017·Anaesthesia
DOI 10.1111/anae.13938
Open this paper in CoChat — free accountOpen PDF

Abstract

Summary: Randomised, controlled trials have been retracted after publication because of data fabrication and inadequate ethical approval. Fabricated data have included baseline variables, for instance, age, height or weight. Statistical tests can determine the probability of the distribution of means, given their standard deviation and the number of participants in each group. Randomised, controlled trials have been retracted after the data distributions have been calculated as improbable. Most retracted trials have been written by anaesthetists and published by specialist anaesthetic journals. I wanted to explore whether the distribution of baseline data in trials was consistent with the expected distribution. I wanted to determine whether trials retracted after publication had distributions different to trials that have not been retracted. I wanted to determine whether data distributions in trials published in specialist anaesthetic journals have been different to distributions in non‐specialist medical journals. I analysed the distribution of 72,261 means of 29,789 variables in 5087 randomised, controlled trials published in eight journals between January 2000 and December 2015: Anaesthesia (399); Anesthesia and Analgesia (1288); Anesthesiology (541); British Journal of Anaesthesia (618); Canadian Journal of Anesthesia (384); European Journal of Anaesthesiology (404); Journal of the American Medical Association (518) and New England Journal of Medicine (935). I chose these journals as I had electronic access to the full text. Trial p values were distorted by an excess of baseline means that were similar and an excess that were dissimilar: 763/5015 (15.2%) trials that had not been retracted from publication had p values that were within 0.05 of 0 or 1 (expected 10%), that is, a 5.2% excess, p = 1.2 × 10 −7 . The p values of 31/72 (43%) trials that had been retracted after publication were within 0.05 of 0 or 1, a rate different to that for unretracted trials, p = 1.03 × 10 −10 . The difference between the distributions of these two subgroups was confirmed by comparison of their overall distributions, p = 5.3 × 10 −15 . Each journal exhibited the same abnormal distribution of baseline means. There was no difference in distributions of baseline means for 1453 trials in non‐anaesthetic journals and 3634 trials in anaesthetic journals, p = 0.30. The rate of retractions from JAMA and NEJM , 6/1453 or 1 in 242, was one‐quarter the rate from the six anaesthetic journals, 66/3634 or 1 in 55, relative risk (99% CI ) 0.23 (0.08–0.68), p = 0.00022. A probability threshold of 1 in 10,000 identified 8/72 (11%) retracted trials (7 by Fujii et al.) and 82/5015 (1.6%) unretracted trials. Some p values were so extreme that the baseline data could not be correct: for instance, for 43/5015 unretracted trials the probability was less than 1 in 10 15 (equivalent to one drop of water in 20,000 Olympic‐sized swimming pools). A probability threshold of 1 in 100 for two or more trials by the same author identified three authors of retracted trials (Boldt, Fujii and Reuben) and 21 first or corresponding authors of 65 unretracted trials. Fraud, unintentional error, correlation, stratified allocation and poor methodology might have contributed to the excess of randomised, controlled trials with similar or dissimilar means, a pattern that was common to all the surveyed journals. It is likely that this work will lead to the identification, correction and retraction of hitherto unretracted randomised, controlled trials.

Integrity signals

Registry

Crossref · 10.1111/anae.13938

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

2 of 4 cited sources checked · 1 supported · 0 flagged · accuracy not yet measured

  • Findingown data

    In 15.6% of trials, the probability of a more extreme distribution was 1 in 10.

    “In 15.6% of trials, the probability of a more extreme distribution was 1 in 10.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Variables within a trial are independent (not correlated).
    • Rests on an assumption: The trials included were genuinely randomised as reported.
  • Findingown data

    The distributions of baseline means differed between the 72 retracted trials and the 5015 unretracted trials, p = 5.3 × 10^-15.

    “More baseline variables from 72 retracted trials had very similar means or very dissimilar means than 5015 trials that are not retracted, p = 5.3 9 10 À15”
    • Also stated in: § Abstract · boosted
      “The difference between the distributions of these two subgroups was confirmed by comparison of their overall distributions, p = 5.3 9 10 À15 .”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Variables within a trial are independent (not correlated).
    • Rests on an assumption: The trials included were genuinely randomised as reported.
  • Findingown data

    Each journal had the same proportion of trials with extreme p values.

    “Each journal had the same proportion of trials with extreme p values”
    • Also stated in: § Abstract · plain
      “Each journal exhibited the same abnormal distribution of baseline means.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Variables within a trial are independent (not correlated).
    • Rests on an assumption: The trials included were genuinely randomised as reported.
  • Findingown data

    The distribution of trial p values deviated from the expected distribution, with p = 1.2 × 10^-7.

    “Consequently, the distribution of trial p values deviated from the expected distribution, p = 1.2 9 10 À7 .”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Variables within a trial are independent (not correlated).
    • Rests on an assumption: The trials included were genuinely randomised as reported.
  • Findingown data

    For 43 of 5015 unretracted trials, the probability was less than 1 in 10^15.

    “for 43/5015 unretracted trials the probability was less than 1 in 10 15”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Variables within a trial are independent (not correlated).
  • May combine two claims. 22
  • Classified as probably not a substantial claim. 6

  • May combine two claims.

    Premisecitation

    The technique has recently identified systematic problems with data in 33 randomised trials by Yoshihiro Sato, who is not an anaesthetist.

    “The technique has recently identified systematic problems with data in 33 randomised trials by Yoshihiro Sato, who is not an anaesthetist [5].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [5] 5. Bolland MJ, Avenell A, Gamble GD, Grey A. Systematic review and statistical analysis of the integrity of 33 randomized con- trolled trials. Neurology 2016; 87: 2391-402. source

      The Methods section identifies 33 randomized trials with Yoshihiro Sato as an author, and the Results report marked statistical anomalies in their data. The paper does not state whether Sato was an anaesthetist, so that part of the sentence is unsupported by this source. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The same approach has been used to investigate trials published by Yuhji Saitoh, a co-author of Dr Fujii, and formed a component of the investigation of his work.

    “The same approach has been used to investigate trials published by Yuhji Saitoh, a co-author of Dr Fujii, and form a component of the investigation of his work [4].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [4] 4. Carlisle JB, Loadsman JA. Evidence for non-random sampling in randomised, controlled trials by Yuhji Saitoh. Anaesthesia 2017; 72: 17-27. source

  • May combine two claims.

    Findingown data

    The survey analysed 5087 randomised controlled trials, comprising 72,261 means of 29,789 variables.

    “I therefore analysed 5087 trials, which included 72,261 means of 29,789 variables.”
    • Also stated in: § Abstract · plain
      “I analysed the distribution of 72,261 means of 29,789 variables in 5087 randomised, controlled trials published in eight journals between January 2000 and December 2015”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    NEJM 2007;356:911 reported mean (SE) tissue plasminogen activator concentrations of 4.5 (0.6) and 3.2 (0.4) in groups of 59 and 61; converting SE to SD gave p = 0.92, while converting the SE for the 19 other variables gave p values averaging 0.02 and a composite trial p value of 4.1 × 10^-13.

    “NEJM 2007; 356: 911 reported mean (SE) tissue plasminogen activator concentrations of 4.5 (0.6) and 3.2 (0.4) in groups of 59 and 61, respectively: conversion of the SE to SD (4.6 and 3.1) resulted in a p value of 0.92, which is not particularly near 1 (or 0).However, conversion of the 'SE' for the 19 other variables resulted in p values averaging 0.02 and a composite trial p value of 4.1 9 10 À13”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The retraction rate from JAMA and NEJM (6/1453, or 1 in 242) was one-quarter the rate from the six anaesthetic journals (66/3634, or 1 in 55), with relative risk 0.23 (99% CI 0.08-0.68), p = 0.00022.

    “The rate of retractions from JAMA and NEJM, 6/1453 or 1 in 242, was one-quarter the rate from the six anaesthetic journals, 66/ 3634 or 1 in 55, relative risk (99%CI) 0.23 (0.08-0.68), p = 0.00022.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The retraction of two trials was triggered after the author noticed duplication of baseline data: EJA 2004;21:60, which was the entire republication of EJA 2003;20:668, and Anaesthesia 2010;65:595, which presented secondary data from a previously published cohort without reference.

    “The retraction of two trials was triggered after I noticed duplication of baseline data: EJA 2004; 21: 60, which was the entire republication of EJA 2003; 20: 668; and Anaesthesia 2010; 65: 595, which presented secondary data from a previously published cohort without reference.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The distributions of baseline data differed between the 26 trials retracted for fabrication and the 44 trials retracted for inadequate ethical approval or unclear reasons, p = 3.3 × 10^-5; neither subgroup was consistent with the expected distribution, p = 1.9 × 10^-5 and p = 2.3 × 10^-5 respectively.

    “The distributions of baseline data were different in the two subgroups, p = 3.3 9 10 À5 , but neither was at all consistent with the expected distribution, p = 1.9 9 10 À5 and p = 2.3 9 10 À5 , respectively.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Variables within a trial are independent (not correlated).
    • Rests on an assumption: The trials included were genuinely randomised as reported.

  • May combine two claims.

    Findingown data

    The distribution of p values was not identical across all journals (p = 0.007), but no significant differences were found when any one journal was compared with any other.

    “Although the distribution of p values was not the same in all journals, p = 0.007, there were no significant differences when one journal was tested against any other.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Variables within a trial are independent (not correlated).
    • Rests on an assumption: The trials included were genuinely randomised as reported.

  • May combine two claims.

    Findingown data

    The distribution of 72,261 baseline means was largely consistent with random sampling, with the 5087 trial p values lying within the 99% confidence interval of the cumulative uniform distribution between 0.15 and 0.95.

    “The distribution of 72,261 baseline means was largely consistent with random sampling, with 5087 trial p values being contained within the 99% confidence interval of the cumulative uniform distribution, between p values of 0.15 and 0.95”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Variables within a trial are independent (not correlated).
    • Rests on an assumption: The trials included were genuinely randomised as reported.

  • May combine two claims.

    Findingown data

    Eighteen trials in the survey were authored by Yoshitaka Fujii, all retracted for data fabrication, as were eight trials by Scott Reuben.

    “Eighteen trials in this survey were authored by Yoshitaka Fujii, all of which were retracted for data fabrication, as were eight trials by Scott Reuben.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    A probability threshold of 1 in 100 for two or more trials by the same author identified three authors of retracted trials (Boldt, Fujii and Reuben) and 21 first or corresponding authors of 65 unretracted trials.

    “A probability threshold of 1 in 100 for two or more trials by the same author identified three authors of retracted trials (Boldt, Fujii and Reuben) and 21 first or corresponding authors of 65 unretracted trials.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Variables within a trial are independent (not correlated).

  • May combine two claims.

    Findingreasoning

    JAMA 2008;299:39 reported 31 baseline variables, 30 of which generated p values in the normal range, while the 31st variable, mean (SD) subcutaneous fat depths of 2.6 (0.8) cm and 3.5 (0.8) cm in groups of 113 and 110, generated p = 5.5 × 10^-15; other data suggest the correct means were 2.6 cm and 2.5 cm, for which p = 0.35.

    “Trial JAMA 2008; 299: 39 reported 31 baseline variables, 30 of which generated p values in the normal range, whereas the 31st variable, mean (SD) subcutaneous fat depths of 2.6 (0.8) cm and 3.5 (0.8) cm in groups of 113 and 110, respectively, generated p = 5.5 9 10 À15 .Other data in the paper suggest that the correct means were 2.6 cm and 2.5 cm, for which p = 0.35.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    NEJM 2004;370:2265 reported p = 0.03 for mean (SD) central venous pressures of 9.0 (4.7) and 8.6 (4.6) in groups of 3428 and 3423; the calculated p value is between 0.00037 and 0.0015 depending on the method used.

    “NEJM 2004; 370: 2265 reported p = 0.03 for mean (SD) central venous pressures of 9.0 (4.7) and 8.6 (4.6) in groups of 3,428 and 3,423, respectively.The calculated p value is between 0.00037 and 0.0015, depending upon the method used.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The p values of 31 of 72 (43%) retracted trials were within 0.05 of 0 or 1, a rate different from that for unretracted trials, p = 1.03 × 10^-10.

    “The p values of 31/72 (43%) trials that had been retracted after publication were within 0.05 of 0 or 1, a rate different to that for unretracted trials, p = 1.03 9 10 À10 .”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Variables within a trial are independent (not correlated).
    • Rests on an assumption: The trials included were genuinely randomised as reported.

  • May combine two claims.

    Findingown data

    A probability threshold of 1 in 10,000 identified 8 of 72 (11%) retracted trials, 7 of them by Fujii et al., and 82 of 5015 (1.6%) unretracted trials.

    “A probability threshold of 1 in 10,000 identified 8/72 (11%) retracted trials (7 by Fujii et al.) and 82/5015 (1.6%) unretracted trials.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Variables within a trial are independent (not correlated).

  • May combine two claims.

    Findingown data

    The composite distribution of all p values from five JAMA trials by an individual generated p = 0.0011 with the Anderson-Darling test and p = 0.00045 with the sum of z score statistic.

    “The composite distribution of all p values from these five trials generate p = 0.0011 with the Anderson-Darling test and p = 0.00045 with the sum of z score statistic.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Variables within a trial are independent (not correlated).

  • May combine two claims.

    Interpretationown data

    The trial with the p value closest to 0 (p = 3.6 × 10^-30) was JAMA 2002;288:2421; the authors or editors probably labelled SD incorrectly as SE, the SD calculated from the SE was too large to be plausible, and analyses assuming the SE were SD increased p to 0.90.

    “The trial with the p value closest to 0 (p = 3.6 9 10 À30 )that is, generated by similar meanswas JAMA 2002; 288: 2421.The authors or editors probably labelled SD incorrectly as SE: the SD calculated from the 'SE' was too large to be plausible and analyses assuming the 'SE' were SD increased p to 0.90.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    Supplementary data are not always exposed to the same rigour as those in the main paper, by author or editor, which might have contributed to the p value indistinguishable from 1 generated by NEJM 2013;368:1279.

    “Supplementary data are not always exposed to the same rigour as those in the main paper, by author or editor, which might have contributed to the p value indistinguishable from 1 generated by NEJM 2013; 368: 1279”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Correlated variables might explain the p value of 9.6 × 10^-4 for JAMA 2004;291:309, which reported nine baseline variables (p values averaging 0.24), six of which would be expected to be correlated because they were derived from the same exercise tests.

    “Correlated variables might explain the p value of 9.6 9 10 À4 for JAMA 2004; 291: 309 that reported nine baseline variables (p values averaging 0.24), six of which one would expect to be correlated as they were derived from the same exercise tests.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Variables within a trial are independent (not correlated).

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Manipulation of participant allocation could result in baseline mean values that are similar or dissimilar, depending on the motivation and efficacy of the method used to distort random allocation.

    “The manipulation of participant allocation could result in baseline mean values that are similar or dissimilar, depending upon the motivation and efficacy of the method used to distort random allocation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Some trials with extreme p values probably contained unintentional typographical errors, such as describing standard error as standard deviation and vice versa.

    “Some trials with extreme p values probably contained unintentional typographical errors, such as the description of standard error as standard deviation and vice versa.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Variables within a trial are independent (not correlated).

  • May combine two claims.

    Premisecitation

    Fujii features top in a list of biomedical authors with the most retractions, and two other anaesthetists appear in this list: second (Boldt) and fifteenth (Reuben).

    “Fujii features top in a list of biomedical authors with the most retractions, and, for quite separate reasons from those based on statistical data analysis, two other anaesthetists appear in this list: second (Boldt) and fifteenth (Reuben) [6].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [6] 6. Retraction Watch. The Retraction Watch Leaderboard. http:// retractionwatch.com/the-retraction-watch-leaderboard/ (acces- sed 17/01/2017).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    The association of extreme distributions with trial retraction suggests that further investigation of uncorrected unretracted trials and their authors will result in most trials being corrected and some retracted.

    “The association of extreme distributions with trial retraction suggests that further investigation of uncorrected unretracted trials and their authors will result in most trials being corrected and some retracted.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Variables within a trial are independent (not correlated).
    • Rests on an assumption: The trials included were genuinely randomised as reported.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    The excess of dissimilar means would not be explained by stratification, although correlations between variables could account for excess trials with p values at either extreme.

    “The excess of dissimilar means would not be explained by this mechanism, although correlations between variables could account for excess trials with p values at either extreme (near 0 or 1).”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Variables within a trial are independent (not correlated).

  • May combine two claims.

    Interpretationreasoning

    The correlation of three histologic scores in NEJM 2002;346:1706 probably accounts for the p value of 0.99998, as might the correlation of three osteoarthritic scores in NEJM 2010;363:1521 for the p value of 0.99997.

    “The correlation of three histologic scores in NEJM 2002; 346: 1706 probably accounts for the p value of 0.99998, as might the correlation of three osteoarthritic scores in NEJM 2010; 363: 1521 that generated a p value of 0.99997.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Variables within a trial are independent (not correlated).
25 of 25

  • The authors state this as an assumption.

    Variables within a trial are independent (not correlated).

    “I analysed variables as independent, that is, not correlated.”

  • The paper relies on this without support.

    The trials included were genuinely randomised as reported.

    “I did not study trials for which participant allocation was not described as random”
2 of 2

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

  • Table 1 understates the next category's unretracted count

    Minor

    The eight journal counts of unretracted trials in Table 1's 0.9% row sum to 79, whereas the row reports 77.

    Reported 77 · recomputed 79

  • Table 1 overstates the first category's unretracted count

    Minor

    The eight journal counts of unretracted trials in Table 1's 0.1% row sum to 99, whereas the row reports 101.

    Reported 101 · recomputed 99

2 of 2

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Carlisle, J. B.
Affiliations
Torbay Hospital (GB)
Corresponding author
Carlisle, J. B.
ORCID
Carlisle, J. B.
Venue
Anaesthesia
Publisher
Wiley
Published
2017-06-04
Volume
72
Issue
8
Pages
944-952
ISSN
  • 0003-2409 (ISSN-L)
  • 1365-2044
DOI
10.1111/anae.13938
OpenAlex
W2620666811
Cited by
229
References
5
Citations by year
2026: 10 · 2025: 12 · 2024: 31 · 2023: 34 · 2022: 34 · 2021: 30 · 2020: 15 · 2019: 21 · 2018: 21 · 2017: 20
FWCI
48.27
Type
journal article
Language
en
Open access
Licence
onlinelibrary.wiley.com/termsAndConditions#vor (vor)
Locations
  • Anaesthesia (publishedVersion)
  • PubMed (publishedVersion)
Topics
  • Social Sciences › Safety Research › Academic integrity and plagiarism
  • Physical Sciences › Mathematics › Statistics and Probability › Benford’s Law and Fraud Detection
  • Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
Keywords
data fabrication, research integrity, non-probability sampling, anesthesiology journals, general medical journals, publication ethics, research misconduct
MeSH terms
Anesthesia, Periodicals as Topic, Retraction of Publication as Topic, Scientific Misconduct, Randomized Controlled Trials as Topic

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read under definition version 3 (extract-v3) on 2026-09-28.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-09-28.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review