How do authors of systematic reviews deal with research malpractice and misconduct in original studies? A cross-sectional analysis of systematic reviews and survey of their authors
Open accessThe DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
Abstract
Objectives: To study whether systematic reviewers apply procedures to counter-balance some common forms of research malpractice such as not publishing completed research, duplicate publications, or selective reporting of outcomes, and to see whether they identify and report misconduct. Design: Cross-sectional analysis of systematic reviews and survey of their authors. Participants: 118 systematic reviews published in four journals ( Ann Int Med, BMJ, JAMA, Lancet) , and the Cochrane Library , in 2013. Main outcomes and measures: Number (%) of reviews that applied procedures to reduce the impact of: (1) publication bias (through searching of unpublished trials), (2) selective outcome reporting (by contacting the authors of the original studies), (3) duplicate publications, (4) sponsors’ and (5) authors’ conflicts of interest, on the conclusions of the review, and (6) looked for ethical approval of the studies. Number (%) of reviewers who suspected misconduct are reported. The procedures applied were compared across journals. Results: 80 (68%) reviewers confirmed their data. 59 (50%) reviews applied three or more procedures; 11 (9%) applied none. Unpublished trials were searched in 79 (66%) reviews. Authors of original studies were contacted in 73 (62%). Duplicate publications were searched in 81 (69%). 27 reviews (23%) reported sponsors of the included studies; 6 (5%) analysed their impact on the conclusions of the review. Five reviews (4%) looked at conflicts of interest of study authors; none of them analysed their impact. Three reviews (2.5%) looked at ethical approval of the studies. Seven reviews (6%) suspected misconduct; only 2 (2%) reported it explicitly. Procedures applied differed across the journals. Conclusions: Only half of the systematic reviews applied three or more of the six procedures examined. Sponsors, conflicts of interest of authors and ethical approval remain overlooked. Research misconduct is sometimes identified, but rarely reported. Guidance on when, and how, to report suspected misconduct is needed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040296 · Cited by 23,020
Self-citationSelective publication of antidepressant trials and its influence on apparent efficacy
Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., & Rosenthal, R. (2008) · New England Journal of Medicine · doi:10.1056/nejmsa065779 · Cited by 2,435
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845
Scope and Impact of Financial Conflicts of Interest in Biomedical Research: A Systematic Review
Bekelman, J. E., Li, Y., & Gross, C. P. (2003) · JAMA · doi:10.1001/jama.289.4.454 · Cited by 1,816
Industry sponsorship and research outcome
Lundh, A., Lexchin, J., Mintzes, B., Schroll, J. B., & Bero, L. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000033.pub3 · Cited by 1,697
Martinson, B. C., Anderson, M. S., & de Vries, R. (2005) · Nature · doi:10.1038/435737a · Cited by 1,096 · cited 3×
“(4,15,26,32),J12,L8 One reviewer claimed that sponsor bias was unlikely, while three were unable to identify any sponsor bias.”
BackgroundFull publication of results initially presented in abstracts.
Scherer, R. W., Langenberg, P., & von Elm, E. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000005.pub3 · Cited by 997
Publication bias in clinical trials due to statistical significance or direction of trial results
Hopewell, S., Loudon, K., Clarke, M. J., Oxman, A. D., & Dickersin, K. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000006.pub3 · Cited by 898
Impact of covert duplicate publication on meta-analysis: a case study
Tramèr, M. R., Reynolds, D. J. M., Moore, R. A., & McQuay, H. J. (1997) · BMJ · doi:10.1136/bmj.315.7109.635 · Cited by 702
Self-citationChan, A.-W., & Altman, D. G. (2005) · BMJ · doi:10.1136/bmj.38356.424606.8f · Cited by 613
Industry sponsorship and research outcome
Lundh, A., Sismondo, S., Lexchin, J., Busuioc, O. A., & Bero, L. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000033.pub2 · Cited by 605 · cited 1×
“Despite the obvious risk of research conclusions favouring a sponsor, 33 subgroup analyses on funding were rarely performed.”
BackgroundHow the case against the MMR vaccine was fixed
Deer, B. (2011) · BMJ · doi:10.1136/bmj.c5347 · Cited by 349 · cited 10×
“We selected systematic reviews from four major medical journals and the Cochrane Library , which is considered the gold standard in terms of systematic reviewing.”
BackgroundInfluentialBes-Rastrollo, M., Schulze, M. B., Ruiz-Canela, M., & Martinez-Gonzalez, M. A. (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001578 · Cited by 343
Rising, K., Bacchetti, P., & Bero, L. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050217 · Cited by 328
Different Patterns of Duplicate Publication: An Analysis of Articles Used in Systematic Reviews
von Elm, E., Poglia, G., Walder, B., & Tramèr, M. R. (2004) · JAMA · doi:10.1001/jama.291.8.974 · Cited by 290
Self-citationShould Unpublished Data Be Included in Meta-analyses?: Current Convictions and Controversies
Cook, D. J. (1993) · JAMA · doi:10.1001/jama.1993.03500210049030 · Cited by 259 · cited 1×
“27 Other ways to address this issue include registration of the study protocols, 28 searching for unpublished trials 8 29 and contacting authors to retrieve unreported outcomes.”
BackgroundEffect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses
Hart, B., Lundh, A., & Bero, L. (2012) · BMJ · doi:10.1136/bmj.d7202 · Cited by 255 · cited 1×
“(8,14) Finally, the appendix table of one of the reviews showed that one study might have suffered from a ‘significant conflict of interest’ .”
BackgroundComparison of protocols and registry entries to published reports for randomised controlled trials
Dwan, K., Altman, D. G., Cresswell, L., Blundell, M., Gamble, C. L., & Williamson, P. R. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000031.pub2 · Cited by 243
Huss, A., Egger, M., Hug, K., Huwiler-Müntener, K., & Röösli, M. (2007) · Environmental Health Perspectives · doi:10.1289/ehp.9149 · Cited by 138 · cited 1×
“This may have led to an underestimation of its real prevalence. duplicate publications, 9 selectively reporting on outcomes or adverse effects, 10 and presenting biased results that are in favour of the sponsors ’ 11 or the authors ’ 12 interests.”
BackgroundVergnes, J.-N., Marchal-Sixou, C., Nabet, C., Maret, D., & Hamel, O. (2010) · Journal of Medical Ethics · doi:10.1136/jme.2010.039941 · Cited by 102
Elia, N., Wager, E., & Tramèr, M. R. (2014) · PLoS ONE · doi:10.1371/journal.pone.0085846 · Cited by 67
Self-citationOut of sight but not out of mind: how to search for unpublished clinical trial evidence
Chan, A.-W. (2012) · BMJ · doi:10.1136/bmj.d8013 · Cited by 62
Susceptibility to Fraud in Systematic Reviews: Lessons from the Reuben Case
Marret, E., Elia, N., Dahl, J. B., McQuay, H. J., Møiniche, S., Moore, R. A., Straube, S., & Tramèr, M. R. (2009) · Anesthesiology · doi:10.1097/aln.0b013e3181c14c3d · Cited by 50
Self-citationRetrieval of missing data for meta-analysis: A practical example
Kelley, G. A., Kelley, K. S., & Vu Tran, Z. (2004) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462304001114 · Cited by 21
Cited by
The influence of the team in conducting a systematic review
Uttley, L., & Montgomery, P. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0548-x · Cited by 54
“Alternatively, reviews could be conducted carelessly, in the chosen methods of meta-analysis or study selection (meta-bias) [12], or by failing to report research misconduct even when identified [13].”
BackgroundRandomised controlled trials evaluating endometrial scratching: assessment of methodological issues
Li, W., Suke, S., Wertaschnigg, D., Lensen, S., Wang, R., Gurrin, L., & Mol, B. W. (2019) · Human Reproduction · doi:10.1093/humrep/dez207 · Cited by 36
Meursinge Reynders, R., Ladu, L., & Di Girolamo, N. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.07.001 · Cited by 18
Metadata
- Authors
- Elia, N., von Elm, E., Chatagner, A., Pöpping, D. M., & Tramèr, M. R.
- Venue
- BMJ Open
- Publisher
- BMJ
- Published
- 2016-03
- Volume
- 6
- Issue
- 3
- Pages
- e010442
- ISSN
- 2044-6055 (ISSN-L)
- OpenAlex
- W2289116456
- Cited by
- 49
- References
- 24
- FWCI
- 3.62
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.