Inflation of type I error rate in two statistical tests for the detection of publication bias in meta‐analyses with binary outcomes
The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
The use of meta‐analysis to combine results of several trials is still increasing in the medical field. The validity of a meta‐analysis may be affected by various sources of bias (for example, publication bias, language bias). Therefore, an analysis of bias should be an integral part of any systematic review. Statistical tests and graphical methods have been developed for this purpose. In this paper, two statistical tests for the detection of bias in meta‐analysis were investigated in a simulation study. Binary outcome data, which are very common in medical applications, were considered and relative effect measures (odds ratios, relative risk) were used for pooling. Sample sizes were generated according to findings in a survey of eight German medical journals. Simulation results indicate an inflation of type I error rates for both tests when the data are sparse. Results get worse with increasing treatment effect and number of trials combined. Valid statistical tests for the detection of bias in meta‐analysis with sparse data need to be developed. Copyright © 2002 John Wiley & Sons, Ltd.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Beta blockade during and after myocardial infarction: An overview of the randomized trials
Yusuf, S., Peto, R., Lewis, J., Collins, R., & Sleight, P. (1985) · Progress in Cardiovascular Diseases · doi:10.1016/s0033-0620(85)80003-7 · Cited by 3,205
The Existence of Publication Bias and Risk Factors for Its Occurrence
Dickersin, K. (1990) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1990.03440100097014 · Cited by 1,417
Empirical assessment of effect of publication bias on meta-analyses
Sutton, A. J. (2000) · BMJ · doi:10.1136/bmj.320.7249.1574 · Cited by 1,398
Language bias in randomised controlled trials published in English and German
Egger, M., Zellweger-Zähner, T., Schneider, M., Junker, C., Lengeler, C., & Antes, G. (1997) · The Lancet · doi:10.1016/s0140-6736(97)02419-7 · Cited by 1,028
Self-citationSterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964
Publication bias: the case for an international registry of clinical trials.
Simes, R. J. (1986) · Journal of Clinical Oncology · doi:10.1200/jco.1986.4.10.1529 · Cited by 747
Meta-analysis: Potentials and promise
Egger, M., & Smith, G. D. (1997) · BMJ · doi:10.1136/bmj.315.7119.1371 · Cited by 646
Bayesian approaches to random-effects meta-analysis: a comparative study
Smith, T. C., Spiegelhalter, D. J., & Thomas, A. (1995) · Statistics in Medicine · doi:10.1002/sim.4780142408 · Cited by 568
The Science of Reviewing Researcha
OXMAN, A. D., & GUYATT, G. H. (1993) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.1993.tb26342.x · Cited by 472
Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995) · The American Statistician · doi:10.1080/00031305.1995.10476125 · Cited by 453
Cited by
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553
Deeks, J. J., Macaskill, P., & Irwig, L. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.01.016 · Cited by 2,910
Comparison of two methods to detect publication bias in meta-analysis.
Peters, J. L. (2006) · JAMA · doi:10.1001/jama.295.6.676 · Cited by 2,278
A modified test for small‐study effects in meta‐analyses of controlled trials with binary endpoints
Harbord, R. M., Egger, M., & Sterne, J. A. C. (2006) · Statistics in Medicine · doi:10.1002/sim.2380 · Cited by 2,267
The case of the misleading funnel plot
Lau, J., Ioannidis, J. P. A., Terrin, N., Schmid, C. H., & Olkin, I. (2006) · BMJ · doi:10.1136/bmj.333.7568.597 · Cited by 1,766
Dissemination and publication of research findings: an updated review of related biases
Song, F., Parekh, Hooper, L., Loke, Y., Ryder, J., Sutton, A., Hing, C., Kwok, C., Pang, C., & Harvey, I. (2010) · Health Technology Assessment · doi:10.3310/hta14080 · Cited by 1,039
Wang, Y.-H., Li, J.-Q., Shi, J.-F., Que, J.-Y., Liu, J.-J., Lappin, J. M., Leung, J., Ravindran, A. V., Chen, W.-Q., Qiao, Y.-L., et al. (2020) · Molecular Psychiatry · doi:10.1038/s41380-019-0595-x · Cited by 885
Adjusting for publication bias in the presence of heterogeneity
Terrin, N., Schmid, C. H., Lau, J., & Olkin, I. (2003) · Statistics in Medicine · doi:10.1002/sim.1461 · Cited by 809
Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy
(2023) · doi:10.1002/9781119756194 · Cited by 757
Interpretation of tests of heterogeneity and bias in meta-analysis
Ioannidis, J. P. A. (2008) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2008.00986.x · Cited by 696
Recent developments in meta‐analysis
Sutton, A. J., & Higgins, J. P. T. (2008) · Statistics in Medicine · doi:10.1002/sim.2934 · Cited by 590
Terrin, N., Schmid, C. H., & Lau, J. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.01.006 · Cited by 492
Arcsine test for publication bias in meta‐analyses with binary outcomes
Rücker, G., Schwarzer, G., & Carpenter, J. (2008) · Statistics in Medicine · doi:10.1002/sim.2971 · Cited by 444
Self-citationMoreno, S. G., Sutton, A. J., Ades, A., Stanley, T. D., Abrams, K. R., Peters, J. L., & Cooper, N. J. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-2 · Cited by 373
Treatment-effect estimates adjusted for small-study effects via a limit meta-analysis
Rucker, G., Schwarzer, G., Carpenter, J. R., Binder, H., & Schumacher, M. (2011) · Biostatistics · doi:10.1093/biostatistics/kxq046 · Cited by 286
Self-citationStatistical methods for dealing with publication bias in meta‐analysis
Jin, Z., Zhou, X., & He, J. (2015) · Statistics in Medicine · doi:10.1002/sim.6342 · Cited by 252
Debray, T. P. A., Moons, K. G. M., & Riley, R. D. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1266 · Cited by 212
Musiat, P., Johnson, C., Atkinson, M., Wilksch, S., & Wade, T. (2022) · Psychological Medicine · doi:10.1017/s0033291721004621 · Cited by 166
Past major depression and smoking cessation outcome: A systematic review and meta-analysis update
Hitsman, B., Papandonatos, G. D., McChargue, D. E., DeMott, A., Herrera, M. J., Spring, B., Borrelli, B., & Niaura, R. (2013) · Addiction · doi:10.1111/add.12009 · Cited by 162
“ORs versus SEs [38], making such plots potentially misleading.”
BackgroundA historical review of publication bias
Marks‐Anglin, A., & Chen, Y. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1452 · Cited by 165
Detecting and adjusting for small‐study effects in meta‐analysis
Rücker, G., Carpenter, J. R., & Schwarzer, G. (2011) · Biometrical Journal · doi:10.1002/bimj.201000151 · Cited by 132
“Beyond publication bias and related biases, we point out that small-study effects may occur for a number of other potential reasons, for example clinical heterogeneity between patients in large and small trials, a mathematical artifact (Schwarzer et al., 2002) or mere coincidence (Rothstein et al.”
BackgroundMethodSelf-citation
Metadata
- Authors
- Schwarzer, G., Antes, G., & Schumacher, M.
- Venue
- Statistics in Medicine
- Publisher
- Wiley
- Published
- 2002-08-09
- Volume
- 21
- Issue
- 17
- Pages
- 2465-2477
- ISSN
- 0277-6715 (ISSN-L)
- 1097-0258
- OpenAlex
- W2089585551
- Cited by
- 80
- References
- 10
- FWCI
- 3.59
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.