Susceptibility to Fraud in Systematic Reviews: Lessons from the Reuben Case
Open accessAbstract
Background: Dr. Scott Reuben allegedly fabricated data. The authors of the current article examined the impact of Reuben reports on conclusions of systematic reviews. Methods: The authors searched in ISI Web of Knowledge systematic reviews citing Reuben reports. Systematic reviews were grouped into one of three categories: I, only cited but did not include Reuben reports; II, retrieved and considered, but eventually excluded Reuben reports; III, included Reuben reports. For quantitative systematic reviews (i.e., meta-analyses), a relevant difference was defined as a significant result becoming nonsignificant (or vice versa) by excluding Reuben reports. For qualitative systematic reviews, each author decided independently whether noninclusion of Reuben reports would have changed conclusions. Results: Twenty-five systematic reviews (5 category I, 6 category II, 14 category III) cited 27 Reuben reports (published 1994-2007). Most tested analgesics in surgical patients. One of 6 quantitative category III reviews would have reached different conclusions without Reuben reports. In all 6 (30 subgroup analyses involving Reuben reports), exclusion of Reuben reports never made any difference when the number of patients from Reuben reports was less than 30% of all patients included in the analysis. Of 8 qualitative category III reviews, all authors agreed that one would certainly have reached different conclusions without Reuben reports. For another 4, the authors' judgment was not unanimous. Conclusions: Carefully performed systematic reviews proved robust against the impact of Reuben reports. Quantitative systematic reviews were vulnerable if the fraudulent data were more than 30% of the total. Qualitative systematic reviews seemed at greater risk than quantitative.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Authors of reviews that included affected trials rarely expressed caution, despite recommendations for caution in deriving conclusions from data from a very limited number of authors and centres.
“Despite recommendations for caution in deriving conclusions from data from a very limited number of authors and centres,74 75 authors of reviews that included affected trials rarely expressed caution.21 22”
From An investigation into the impact and implications of published papers from retracted research: systematic search of affected literature (2019) · cites this paper as [74]
The Discussion (page unavailable) recommends cautious interpretation when a meta-analysis relies mainly on data from one or a few authors or centers, and says some reviewers gave undue weight to particular Reuben reports. It does not establish that authors of reviews including those reports rarely expressed caution; its explicit observation that no authors expressed concerns concerns reviews that excluded the reports (Category II Systematic Reviews, page unavailable). (LLM, conf=0.78)
Experimental — accuracy not yet measuredAssessing impact in narrative reviews was often more challenging.
“assessing impact in narrative reviews was often more challenging, as others have found.74”
From An investigation into the impact and implications of published papers from retracted research: systematic search of affected literature (2019) · cites this paper as [74]
The Discussion reports a lack of unanimity when estimating the impact of Reuben reports in some qualitative systematic reviews, supporting the point that assessment could be challenging. The paper studied qualitative systematic reviews, however, not narrative reviews generally; it does not establish that the difficulty applied across that broader category. (LLM, conf=0.80)
Experimental — accuracy not yet measuredThe authors' findings are consistent with those of others who have investigated the impact of publications affected by research misconduct on subsequent publications and systematic reviews.
“Our findings are consistent with those of others who have investigated the impact of publications affected by research misconduct on subsequent publications and systematic reviews.74 76 80–83”
From An investigation into the impact and implications of published papers from retracted research: systematic search of affected literature (2019) · cites this paper as [74]
Active dissemination by self-citation in cases of prolific misconduct also occurred in the Reuben and Fujii cases.
“Such active dissemination by self-citation in cases of prolific misconduct also occurred in the Reuben and Fujii cases.74 79”
From An investigation into the impact and implications of published papers from retracted research: systematic search of affected literature (2019) · cites this paper as [74]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Schulz (1995) · doi:10.1001/jama.1995.03520290060030 · Cited by 5,634 · see citing works
Moher, D., Cook, D. J., Eastwood, S., Olkin, I., Rennie, D., & Stroup, D. F. (1999) · The Lancet · doi:10.1016/s0140-6736(99)04149-5 · Cited by 4,826 · see citing works
Moher, D., Pham, B., Jones, A., Cook, D. J., Jadad, A. R., Moher, M., Tugwell, P., & Klassen, T. P. (1998) · The Lancet · doi:10.1016/s0140-6736(98)01085-x · Cited by 3,560 · see citing works
Fanelli, D. (2009) · PLoS ONE · doi:10.1371/journal.pone.0005738 · Cited by 1,911 · see citing works
Scope and Impact of Financial Conflicts of Interest in Biomedical Research: A Systematic Review
Bekelman, J. E., Li, Y., & Gross, C. P. (2003) · JAMA · doi:10.1001/jama.289.4.454 · Cited by 1,816 · see citing works
How Quickly Do Systematic Reviews Go Out of Date? A Survival Analysis
Shojania, K. G., Sampson, M., Ansari, M. T., Ji, J., Doucette, S., & Moher, D. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-147-4-200708210-00179 · Cited by 838 · see citing works
Als-Nielsen, B., Chen, W., Gluud, C., & Kjaergard, L. L. (2003) · JAMA · doi:10.1001/jama.290.7.921 · Cited by 746 · see citing works
Impact of covert duplicate publication on meta-analysis: a case study
Tramèr, M. R., Reynolds, D. J. M., Moore, R. A., & McQuay, H. J. (1997) · BMJ · doi:10.1136/bmj.315.7109.635 · Cited by 702 · see citing works
Self-citationThe Efficacy of Preemptive Analgesia for Acute Postoperative Pain Management: A Meta-Analysis
Ong, C. K. S., Lirk, P. B., Seymour, R. A., & Jenkins, B. (2005) · Anesthesia & Analgesia · doi:10.1213/01.ane.0000144428.98767.0e · Cited by 696 · see citing works
Elia, N., Lysakowski, C., & Tramèr, M. R. (2005) · Anesthesiology · doi:10.1097/00000542-200512000-00025 · Cited by 669 · see citing works
Self-citationEffects of Nonsteroidal Antiinflammatory Drugs on Patient-controlled Analgesia Morphine Side Effects
Marret, E., Kurdi, O., Zufferey, P. J., Bonnet, F. J., & Warltier, D. C. (2005) · Anesthesiology · doi:10.1097/00000542-200506000-00027 · Cited by 570 · see citing works
Self-citationTiippana, E., Hamunen, K., Kontinen, V., & Kalso, E. A. (2007) · Anesthesia & Analgesia · doi:10.1213/01.ane.0000261517.27532.80 · Cited by 530 · see citing works
Adverse events associated with postoperative opioid analgesia: A systematic review
Wheeler, M., Oderda, G. M., Ashburn, M. A., & Lipman, A. G. (2002) · Journal of Pain · doi:10.1054/jpai.2002.123652 · Cited by 513 · see citing works
Fischer, H. B. J., Simanski, C., Sharp, C., Bonnet, F. J., Camu, F., Neugebauer, E. A. M., Rawal, N., Joshi, G. P., Schug, S. A., & Kehlet, H. (2008) · Anaesthesia · doi:10.1111/j.1365-2044.2008.05565.x · Cited by 299 · see citing works
Systematic Review and Analysis of Postdischarge Symptoms after Outpatient Surgery
Wu, C. L., Berenholtz, S. M., Pronovost, P. J., & Fleisher, L. A. (2002) · Anesthesiology · doi:10.1097/00000542-200204000-00030 · Cited by 298 · see citing works
Liu, S. S., & Wu, C. L. (2007) · Anesthesia & Analgesia · doi:10.1213/01.ane.0000278089.16848.1e · Cited by 281 · see citing works
Yank, V., Rennie, D., & Bero, L. A. (2007) · BMJ · doi:10.1136/bmj.39376.447211.be · Cited by 235 · see citing works
Pain relief from intra-articular morphine after knee surgery: a qualitative systematic review
Kalso, E. A., Tramèr, M. R., Carroll, D., McQuay, H. J., & Moore, A. R. (1997) · Pain · doi:10.1016/s0304-3959(97)03344-7 · Cited by 222 · see citing works
Self-citationWu, C. L. (2001) · Regional Anesthesia & Pain Medicine · doi:10.1053/rapm.2001.22257 · Cited by 202 · see citing works
A Systematic Review of the Peripheral Analgesic Effects of Intraarticular Morphine
Gupta, A., Bodin, L., Holmström, B., & Berggren, L. (2001) · Anesthesia & Analgesia · doi:10.1097/00000539-200109000-00042 · Cited by 195 · see citing works
Møiniche, S., Mikkelsen, S. S., Wetterslev, J., et al. (1999) · Regional Anesthesia & Pain Medicine · doi:10.1016/s1098-7339(99)90010-x · Cited by 163 · see citing works
Self-citationRømsing, J., & Møiniche, S. (2004) · Acta Anaesthesiologica Scandinavica · doi:10.1111/j.0001-5172.2004.00379.x · Cited by 160 · see citing works
Self-citationA systematic review of adjuncts for intravenous regional anesthesia for surgical procedures
Choyce, A., & Peng, P. (2002) · Canadian Journal of Anesthesia/Journal canadien d anesthésie · doi:10.1007/bf03020416 · Cited by 154 · see citing works
Straube, S., Derry, S., McQuay, H., & Moore, R. A. (2005) · Acta Anaesthesiologica Scandinavica · doi:10.1111/j.1399-6576.2005.00666.x · Cited by 152 · see citing works
Self-citationEfficacy and Adverse Effects of Prophylactic Antiemetics During Patient-Controlled Analgesia Therapy
Tramèr, M. R., & Walder, B. (1999) · Anesthesia & Analgesia · doi:10.1097/00000539-199906000-00030 · Cited by 150 · see citing works
Self-citationMcCartney, C. J. L., DUGGAN, E., & Apatu, E. (2007) · Regional Anesthesia & Pain Medicine · doi:10.1016/j.rapm.2007.02.010 · Cited by 138 · see citing works
Tramadol for osteoarthritis: a systematic review and metaanalysis.
Cepeda, M. S., Camargo, F., Zea, C., & Valencia, L. (2007) · PubMed · Cited by 137 · see citing works
Research Misconduct: The Poisoning of the Well
Smith, R. W. (2006) · Journal of the Royal Society of Medicine · doi:10.1258/jrsm.99.5.232 · Cited by 127 · see citing works
Trikalinos, N. A., Evangelou, E., & Ioannidis, J. P. A. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.11.019 · Cited by 100 · see citing works
Jirarattanaphochai, K., & Sae-Jung, S. (2008) · Journal of Neurosurgery Spine · doi:10.3171/spi/2008/9/7/022 · Cited by 97 · see citing works
Macario, A., & Lipman, A. G. (2001) · Pain Medicine · doi:10.1046/j.1526-4637.2001.01043.x · Cited by 73 · see citing works
Investigating the previous studies of a fraudulent author
Smith, R. W. (2005) · BMJ · doi:10.1136/bmj.331.7511.288 · Cited by 72 · see citing works
Rosseland, L. A. (2005) · Regional Anesthesia & Pain Medicine · doi:10.1016/j.rapm.2004.08.022 · Cited by 69 · see citing works
Rømsing, J., Møiniche, S., Mathiesen, O., & Dahl, J. B. (2005) · Acta Anaesthesiologica Scandinavica · doi:10.1111/j.1399-6576.2005.00614.x · Cited by 62 · see citing works
Self-citationSafety and Efficacy of Controlled‐Release Oxycodone: A Systematic Literature Review
Rischitelli, D. G., & Karbowicz, S. H. (2002) · Pharmacotherapy The Journal of Human Pharmacology and Drug Therapy · doi:10.1592/phco.22.11.898.33628 · Cited by 55 · see citing works
Perioperative Analgesia: What Do We Still Know?
White, P. F., Kehlet, H., & Liu, S. (2009) · Anesthesia & Analgesia · doi:10.1213/ane.0b013e3181a16835 · Cited by 50 · cited 1× · see citing works
“mise every systematic review that included these fabricated findings.(6) In this context, two questions are rele-”
BackgroundHirayama, T. T., Ishii, F., Yago, K., & Ogata, H. (2001) · YAKUGAKU ZASSHI · doi:10.1248/yakushi.121.179 · Cited by 42 · see citing works
Data Fabrication and Article Retraction
Eisenach, J. C. (2009) · Anesthesiology · doi:10.1097/aln.0b013e3181a06bf9 · Cited by 26 · see citing works
The Time to Onset and Overall Analgesic Efficacy of Rofecoxib 50 mg
Desjardins, P. J., Mehlisch, D. R., Chang, D. J., et al. (2005) · Clinical Journal of Pain · doi:10.1097/00002508-200505000-00007 · Cited by 16 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- Routine audit uncovered Reuben fraud: Missing IRB info led to discovery of fabricated data. Anesthesiology News 2009; 35:3
- Winstein KJ, Armstrong D: Top pain scientist fabricated data in studies, hospital says. The Wall Street Journal, March 11, 2009, p A12
Cited by
Sorted by citations
Why has the number of scientific retractions increased?
Steen, R. G., Casadevall, A., & Fang, F. C. (2013) · PLoS ONE · doi:10.1371/journal.pone.0068397 · Cited by 415 · see citing works
Considering bias and conflicts of interest among the included studies
Boutron, I., Page, M. J., Higgins, J. P., Altman, D. G., Lundh, A., Hróbjartsson, A., & on behalf of the Cochrane Bias Methods Group (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch7 · Cited by 258 · see citing works
Retractions in the medical literature: how many patients are put at risk by flawed research?
Steen, R. G. (2011) · Journal of Medical Ethics · doi:10.1136/jme.2011.043133 · Cited by 156 · see citing works
Bornemann-Cimenti, H., Szilagyi, I. S., & Sandner-Kiesling, A. (2016) · Science and Engineering Ethics · doi:10.1007/s11948-015-9680-y · Cited by 87 · see citing works
Fraud or flawed: adverse impact of fabricated or poor quality research
Moore, R. A., Derry, S., & McQuay, H. J. (2010) · Anaesthesia · doi:10.1111/j.1365-2044.2010.06295.x · Cited by 62 · see citing works
Self-citationAvenell, A., Stewart, F., Grey, A., Gamble, G., & Bolland, M. (2019) · BMJ Open · doi:10.1136/bmjopen-2019-031909 · Cited by 62 · see citing works
Mott, A., Fairhurst, C., & Torgerson, D. (2019) · Journal of Health Services Research & Policy · doi:10.1177/1355819618797965 · Cited by 36 · see citing works
Østengaard, L., Lundh, A., Tjørnhøj-Thomsen, T., Abdi, S., Gelle, M. H. A., Stewart, L. A., Boutron, I., & Hróbjartsson, A. (2020) · BMJ · doi:10.1136/bmj.m3764 · Cited by 35 · see citing works
- Authors
- Marret, E., Elia, N., Dahl, J. B., McQuay, H. J., Møiniche, S., Moore, R. A., Straube, S., & Tramèr, M. R.
- Affiliations
- University Hospital of Geneva (CH)
- Research Associate.
- Copenhagen University Hospital (DK)
- Professor, Nuffield Department of Anaesthetics.
- Glostrup Hospital (DK)
- John Radcliffe Hospital (GB)
- University of Göttingen (DE)
- University of Geneva (CH)
- Venue
- Anesthesiology
- Published
- 2009-12-01
- Volume
- 111
- Issue
- 6
- Pages
- 1279-1289
- ISSN
- 0003-3022 (ISSN-L)
- OpenAlex
- W2021767709
- Cited by
- 50
- References
- 39
- Citations by year
- 2025: 1 · 2024: 3 · 2023: 3 · 2022: 2 · 2021: 2 · 2020: 1 · 2019: 2 · 2018: 2 · 2016: 4 · 2015: 2 · 2014: 1 · 2013: 4 · 2012: 9
- FWCI
- 0.58
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Anesthesiology (publishedVersion)
- PubMed (publishedVersion)
- Research at the University of Copenhagen (University of Copenhagen) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Safety Research › Academic integrity and plagiarism
- Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Pharmacology › Pharmaceutical industry and healthcare
- Keywords
- scientific fraud, data fabrication, systematic reviews, meta-analysis, publication bias, subgroup analysis
- MeSH terms
- Analgesics, Humans, Postoperative Pain, Review Literature as Topic, Meta-Analysis as Topic, Scientific Misconduct, Randomized Controlled Trials as Topic, Treatment Outcome, Cyclooxygenase 2 Inhibitors
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
