The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
False-positive findings in Cochrane meta-analyses with and without application of trial sequential analysis: an empirical review
Open accessAbstract
Objective: Many published meta-analyses are underpowered. We explored the role of trial sequential analysis (TSA) in assessing the reliability of conclusions in underpowered meta-analyses. Methods: We screened The Cochrane Database of Systematic Reviews and selected 100 meta-analyses with a binary outcome, a negative result and sufficient power. We defined a negative result as one where the 95% CI for the effect included 1.00, a positive result as one where the 95% CI did not include 1.00, and sufficient power as the required information size for 80% power, 5% type 1 error, relative risk reduction of 10% or number needed to treat of 100, and control event proportion and heterogeneity taken from the included studies. We re-conducted the meta-analyses, using conventional cumulative techniques, to measure how many false positives would have occurred if these meta-analyses had been updated after each new trial. For each false positive, we performed TSA, using three different approaches. Results: We screened 4736 systematic reviews to find 100 meta-analyses that fulfilled our inclusion criteria. Using conventional cumulative meta-analysis, false positives were present in seven of the meta-analyses (7%, 95% CI 3% to 14%), occurring more than once in three. The total number of false positives was 14 and TSA prevented 13 of these (93%, 95% CI 68% to 98%). In a post hoc analysis, we found that Cochrane meta-analyses that are negative are 1.67 times more likely to be updated (95% CI 0.92 to 2.68) than those that are positive. Conclusions: We found false positives in 7% (95% CI 3% to 14%) of the included meta-analyses. Owing to limitations of external validity and to the decreased likelihood of updating positive meta-analyses, the true proportion of false positives in meta-analysis is probably higher. TSA prevented 93% of the false positives (95% CI 68% to 98%).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552 · cited 1×
“Weused relative risk as the effect measure and the DerSimonian and Laird random-effects model.(26) We constructed two-sided conventional naïve monitoring boundaries for a type 1 error of 5%.”
MethodThe cochrane database of systematic reviews
Geddes, J. (1996) · Journal of Psychosomatic Research · doi:10.1016/s0022-3999(96)90063-x · Cited by 5,989
Clinical Trials: A Practical Approach.
Hill, I. D., & Pocock, S. J. (1984) · Biometrics · doi:10.2307/2531186 · Cited by 2,930
Trial sequential analysis may establish when firm evidence is reached in cumulative meta-analysis
Wetterslev, J., Thorlund, K., Brok, J., & Gluud, C. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.03.013 · Cited by 1,881
Self-citationWhat to add to nothing? Use and avoidance of continuity corrections in meta‐analysis of sparse data
J. Sweeting, M., J. Sutton, A., & C. Lambert, P. (2004) · Statistics in Medicine · doi:10.1002/sim.1761 · Cited by 1,734 · cited 1×
“We only classified a finding as a false positive if the result remained a false positive for all permutations of continuity adjustment technique and quantity.(27) Cumulative sequential analyses For the cumulative meta-analyses with false-positive findings, we assessed whether TSA conducted at the equivalent time would…”
BackgroundBrok, J., Thorlund, K., Gluud, C., & Wetterslev, J. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.10.007 · Cited by 1,113
Self-citationAntiplatelet agents for preventing pre-eclampsia and its complications
Duley, L., Henderson-Smart, D. J., Meher, S., & King, J. F. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004659.pub2 · Cited by 1,016
Vaccines for preventing pneumococcal infection in adults
Moberley, S., Holden, J., Tatham, D. P., & Andrews, R. M. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000422.pub3 · Cited by 960
Estimating required information size by quantifying diversity in random-effects model meta-analyses
Wetterslev, J., Thorlund, K., Brok, J., & Gluud, C. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-86 · Cited by 938
Self-citationBrok, J., Thorlund, K., Wetterslev, J., & Gluud, C. (2009) · International Journal of Epidemiology · doi:10.1093/ije/dyn188 · Cited by 917
Self-citationCan trial sequential monitoring boundaries reduce spurious inferences from meta-analyses?
Thorlund, K., Devereaux, P. J., Wetterslev, J., Guyatt, G., Ioannidis, J. P. A., Thabane, L., Gluud, L.-L., Als-Nielsen, B., & Gluud, C. (2009) · International Journal of Epidemiology · doi:10.1093/ije/dyn179 · Cited by 833
Self-citationThe impact of study size on meta-analyses: examination of underpowered studies in Cochrane reviews
Turner, R. M., Bird, S. M., & Higgins, J. P. T. (2013) · PLoS ONE · doi:10.1371/journal.pone.0059202 · Cited by 672
Interim analysis: The alpha spending function approach
Demets, D. L., & Lan, K. K. G. (1994) · Statistics in Medicine · doi:10.1002/sim.4780131308 · Cited by 564
Pogue, J. M., & Yusuf, S. (1997) · Controlled Clinical Trials · doi:10.1016/s0197-2456(97)00051-2 · Cited by 415
Turner, R. M., Jackson, D., Wei, Y., Thompson, S. G., & Higgins, J. P. T. (2015) · Statistics in Medicine · doi:10.1002/sim.6381 · Cited by 410
Thorlund, K., Imberger, G., Walsh, M., Chu, R., Gluud, C., Wetterslev, J., Guyatt, G., Devereaux, P. J., & Thabane, L. (2011) · PLoS ONE · doi:10.1371/journal.pone.0025491 · Cited by 319
Self-citationFetal electrocardiogram (ECG) for fetal monitoring during labour
Neilson, J. P. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000116.pub5 · Cited by 279
Sequential methods for random‐effects meta‐analysis
Higgins, J. P. T., Whitehead, A., & Simmonds, M. (2011) · Statistics in Medicine · doi:10.1002/sim.4088 · Cited by 258
Ducharme, F. M., Ni Chroinin, M., Greenstone, I., & Lasserson, T. J. (2010) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd005533.pub2 · Cited by 185
Statistics: The Problem of Examining Accumulating Data More Than Once
McPherson, K. (1974) · New England Journal of Medicine · doi:10.1056/nejm197402282900907 · Cited by 183
Pereira, T. V., & Ioannidis, J. P. A. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.12.012 · Cited by 160 · cited 1×
“This increased risk has been demonstrated by theoretical arguments,(8) 9 evidence from simulation studies,(2) 3 10–12 and evidence from empirical work.(13) Given that so many published Cochrane meta-analyses are underpowered and subject to continued updating, this Strengths and limitations of this study”
BackgroundHMG CoA reductase inhibitors (statins) for dialysis patients
Navaneethan, S. D., Nigwekar, S. U., Perkovic, V., Johnson, D. W., Craig, J. C., & Strippoli, G. F. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004289.pub3 · Cited by 158
Attention should be given to multiplicity issues in systematic reviews
Bender, R., Bunce, C., Clarke, M. J., Gates, S., Lange, S., Pace, N. L., & Thorlund, K. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.03.004 · Cited by 149
Self-citationThe knowledge system underpinning healthcare is not fit for purpose and must change
Roberts, I., Ker, K., Edwards, P. J., Beecher, D., Manno, D., & Sydenham, E. (2015) · BMJ · doi:10.1136/bmj.h2463 · Cited by 135
Early stopping of randomized clinical trials for overt efficacy is problematic.
Bassler, D., Montori, V. M., Briel, M., Glasziou, P., & Guyatt, G. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.07.016 · Cited by 130
HMG CoA reductase inhibitors (statins) for dialysis patients
Palmer, S. C., Navaneethan, S. D., Craig, J. C., Johnson, D. W., Perkovic, V., Nigwekar, S. U., Hegbrant, J., & Strippoli, G. F. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004289.pub5 · Cited by 114
Updating meta-analyses leads to larger type I errors than publication bias
Borm, G. F., & Donders, A. R. T. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.08.010 · Cited by 112
Evidence synthesis as the key to more coherent and efficient research
Sutton, A. J., Cooper, N. J., & Jones, D. R. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-29 · Cited by 105
Calcium antagonists for acute ischemic stroke
Horn, J., & Limburg, M. (2000) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001928 · Cited by 102
Uncertainty of the time of first significance in random effects cumulative meta-analysis
Berkey, C. S., Mosteller, F. C., Lau, J., & Antman, E. M. (1996) · Controlled Clinical Trials · doi:10.1016/s0197-2456(96)00014-1 · Cited by 80
Calcium antagonists for acute ischemic stroke
Zhang, J., Yang, J., Zhang, C., Jiang, X., Zhou, H., & Liu, M. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001928.pub2 · Cited by 73
A prospectively planned cumulative meta-analysis applied to a series of concurrent clinical trials
Whitehead, A. (1997) · Statistics in Medicine · Cited by 66
Hu, M., Cappelleri, J. C., & Lan, K. K. G. (2007) · Clinical Trials · doi:10.1177/1740774507081219 · Cited by 61
Sequential meta-analysis: an efficient decision-making tool
van der Tweel, I., & Bollen, C. W. (2010) · Clinical Trials · doi:10.1177/1740774509360994 · Cited by 52
Perez, M. I., Musini, V. M., & Wright, J. M. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006743.pub2 · Cited by 41
Fetal electrocardiogram (ECG) for fetal monitoring during labour
Neilson, J. P. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000116.pub3 · Cited by 28
Cited by
Trial Sequential Analysis in systematic reviews with meta-analysis
Wetterslev, J., Jakobsen, J. C., & Gluud, C. (2017) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0315-7 · Cited by 1,189
Self-citationCastellini, G., Bruschettini, M., Gianola, S., Gluud, C., & Moja, L. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0770-1 · Cited by 161
Self-citationA framework for prospective, adaptive meta-analysis (FAME) of aggregate data from randomised trials.
Tierney, J. F., Fisher, D. J., Vale, C. L., Burdett, S., Rydzewska, L. H., Rogozińska, E., Godolphin, P. J., White, I. R., & Parmar, M. K. B. (2021) · PLOS Medicine · doi:10.1371/journal.pmed.1003629 · Cited by 31
Valkenburg, C., Van der Weijden, F., & Slot, D. E. (2019) · International Journal of Dental Hygiene · doi:10.1111/idh.12364 · Cited by 15
Metadata
- Authors
- Imberger, G., Thorlund, K., Gluud, C., & Wetterslev, J.
- Venue
- BMJ Open
- Publisher
- BMJ
- Published
- 2016-08
- Volume
- 6
- Issue
- 8
- Pages
- e011890
- ISSN
- 2044-6055 (ISSN-L)
- OpenAlex
- W2514621555
- Cited by
- 277
- References
- 36
- FWCI
- 19.25
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.