Publication Bias in Meta-Analysis: Confidence Intervals for Rosenthal’s Fail-Safe Number
Open accessAbstract
The purpose of the present paper is to assess the efficacy of confidence intervals for Rosenthal’s fail-safe number. Although Rosenthal’s estimator is highly used by researchers, its statistical properties are largely unexplored. First of all, we developed statistical theory which allowed us to produce confidence intervals for Rosenthal’s fail-safe number. This was produced by discerning whether the number of studies analysed in a meta-analysis is fixed or random. Each case produces different variance estimators. For a given number of studies and a given distribution, we provided five variance estimators. Confidence intervals are examined with a normal approximation and a nonparametric bootstrap. The accuracy of the different confidence interval estimates was then tested by methods of simulation under different distributional assumptions. The half normal distribution variance estimator has the best probability coverage. Finally, we provide a table of lower confidence intervals for Rosenthal’s estimator.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Rosenthal's fail-safe N test is a technique for computing the number of missing studies that would be necessary to nullify the found effect [38].
“Rosenthal’s fail-safe N test is a technique for computing the number of missing studies that would be necessary to nullify the found effect [38].”
From Innovative Technology–Based Interventions to Reduce Stigma Toward People With Mental Illness: Systematic Review and Meta-analysis (2022) · cites this paper as [38]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,406 · see citing works
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500 · see citing works
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230 · see citing works
Bootstrap methods: another look at the jackknife
Efron, B. (1979) · The Annals of Statistics · doi:10.1214/aos/1176344552 · Cited by 17,602 · see citing works
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099 · see citing works
Wasserman, S. S., Hedges, L. V., & Olkin, I. (1988) · Journal of Educational Statistics · doi:10.2307/1164953 · Cited by 10,347 · see citing works
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021) · doi:10.1002/9781119558378 · Cited by 9,799 · see citing works
Continuous Univariate Distributions.
Omori, Y., Johnson, N. L., Kotz, S., & Balakrishnan, N. (1995) · Journal of the American Statistical Association · doi:10.2307/2291547 · Cited by 9,085 · see citing works
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · see citing works
Bootstrap Methods and their Application
Davison, A. C., & Hinkley, D. V. (1997) · doi:10.1017/cbo9780511802843 · Cited by 7,350 · see citing works
Casella, G., & Berger, R. L. (1993) · Chapman & Hall/CRC eBooks · doi:10.1201/9781003456285 · Cited by 5,835 · see citing works
Bootstrap Methods and Their Application
Loughin, T. M., Davison, A. C., & HINKLEY, D. V. (1999) · Journal of the American Statistical Association · doi:10.2307/2669707 · Cited by 5,478 · see citing works
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553 · see citing works
Fixed-and random-effects models in meta-analysis.
Hedges, L. V., & Vevea, J. L. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.4.486 · Cited by 2,977 · see citing works
A Nonparametric “Trim and Fill” Method of Accounting for Publication Bias in Meta-Analysis
Duval, S., & Tweedie, R. (2000) · Journal of the American Statistical Association · doi:10.1080/01621459.2000.10473905 · Cited by 2,955 · see citing works
A class of distributions which includes the normal ones
Azzalini, A. (1985) · Padua Research Archive (University of Padova) · Cited by 2,612 · see citing works
Publication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368 · see citing works
A Fail-Safe N for Effect Size in Meta-Analysis
Orwin, R. G. (1983) · Journal of Educational Statistics · doi:10.2307/1164923 · Cited by 2,064 · cited 1× · see citing works
“Orwin’s [38] approach is very similar to Rosenthal’s [11] approach without considering the normal variates but taking Cohen’s d [22] to compute a fail-safe number.”
MethodP-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500 · see citing works
Publication bias in meta-analysis its causes and consequences
Thornton, A. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(99)00161-4 · Cited by 1,291 · cited 1× · see citing works
“The first includes the funnel plot (and other visualisation methods such as the normal quantile plot) and regression/correlation-based tests, while the second includes the fail-safe (also called file-drawer) number, the trim and fill method, and selection model approaches [10, 14, 18].”
MethodRosenberg, M. S. (2005) · Evolution · doi:10.1111/j.0014-3820.2005.tb01004.x · Cited by 1,285 · cited 1× · see citing works
“Rosenberg’s fail-safe number is very similar to Rosenthal’s and Orwin fail-safe number[37].”
ResultThe American Soldier: Adjustment During Army Life.
Riley, J. W., Stouffer, S. A., Suchman, E. A., Devinney, L. C., Star, S. A., & Williams, R. M. (1949) · American Sociological Review · doi:10.2307/2087216 · Cited by 1,240 · see citing works
Methodological issues and advances in biological meta-analysis
Nakagawa, S., & Santos, E. S. A. (2012) · Evolutionary Ecology · doi:10.1007/s10682-012-9555-5 · Cited by 1,032 · see citing works
Combining results of independent studies.
Rosenthal, R. (1978) · Psychological Bulletin · doi:10.1037/0033-2909.85.1.185 · Cited by 910 · see citing works
Newberry, S. J. (2012) · JAMA · doi:10.1001/jama.2012.3507 · Cited by 870 · see citing works
Publication bias—a problem in interpreting medical data
Begg, C. B., & Berlin, J. A. (1988) · Journal of the Royal Statistical Society. Series A (Statistics in Society) · doi:10.2307/2982993 · Cited by 814 · see citing works
An exploratory test for an excess of significant findings
Ioannidis, J. P., & Trikalinos, T. A. (2007) · Clinical Trials · doi:10.1177/1740774507079441 · Cited by 751 · see citing works
A general parametric approach to the meta‐analysis of randomized clinical trials
Whitehead, A., & Whitehead, J. (1991) · Statistics in Medicine · doi:10.1002/sim.4780101105 · Cited by 750 · see citing works
Reward, punishment, and cooperation: A meta-analysis.
Balliet, D. P., Mulder, L. B., & Lange, P. A. M. V. (2011) · Psychological Bulletin · doi:10.1037/a0023489 · Cited by 714 · see citing works
A Nonparametric "Trim and Fill" Method of Accounting for Publication Bias in Meta-Analysis
Duval, S., & Tweedie, R. L. (2000) · Journal of the American Statistical Association · doi:10.2307/2669529 · Cited by 694 · see citing works
A comparison of statistical methods for meta‐analysis
Brockwell, S. E., & Gordon, I. R. (2001) · Statistics in Medicine · doi:10.1002/sim.650 · Cited by 573 · see citing works
Handbook of the Poisson Distribution.
Bassett, E. E., & Haight, F. A. (1968) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2343850 · Cited by 552 · see citing works
Ferguson, C. J., & Brannick, M. Τ. (2011) · Psychological Methods · doi:10.1037/a0024445 · Cited by 493 · see citing works
The Folded Normal Distribution
Leone, F. C., Nelson, L. S., & Nottingham, R. B. (1961) · Technometrics · doi:10.1080/00401706.1961.10489974 · Cited by 415 · see citing works
Cooper, H. M. (1979) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.37.1.131 · Cited by 413 · see citing works
Eine neue Herleitung des Exponentialgesetzes in der Wahrscheinlichkeitsrechnung
Lindeberg, J. W. (1922) · Mathematische Zeitschrift · doi:10.1007/bf01494395 · Cited by 410 · see citing works
Selection Models and the File Drawer Problem
Iyengar, S., & Greenhouse, J. B. (1988) · Statistical Science · doi:10.1214/ss/1177013012 · Cited by 399 · see citing works
Statistical versus traditional procedures for summarizing research findings.
Cooper, H. M., & Rosenthal, R. (1980) · Psychological Bulletin · doi:10.1037/0033-2909.87.3.442 · Cited by 353 · see citing works
Zakzanis, K. K. (2001) · Archives of Clinical Neuropsychology · doi:10.1093/arclin/16.7.653 · Cited by 346 · see citing works
Publication Bias (The "File-Drawer Problem") in Scientific Inference
Scargle, J. D. (1999) · arXiv · doi:10.48550/arxiv.physics/9909033 · Cited by 313 · see citing works
Modelling publication bias in meta-analysis: a review
Sutton, A. J., Song, F., Gilbody, S. M., & Abrams, K. R. (2000) · Statistical Methods in Medical Research · doi:10.1177/096228020000900503 · Cited by 288 · see citing works
Debunking Myths and Urban Legends About Meta-Analysis
Aguinis, H., Pierce, C. A., Bosco, F. A., Dalton, D. R., & Dalton, C. M. (2010) · Organizational Research Methods · doi:10.1177/1094428110375720 · Cited by 261 · see citing works
Handbook of the Normal Distribution
Aroian, L. A. (1983) · Technometrics · doi:10.1080/00401706.1983.10487836 · Cited by 259 · see citing works
Cumulative Meta-Analysis: A Consideration of Indicators of Sufficiency and Stability
Mullen, B. R., Muellerleile, P. A., & Bryant, B. (2001) · Personality and Social Psychology Bulletin · doi:10.1177/01461672012711006 · Cited by 234 · see citing works
The Nontruncated Marginal of a Truncated Bivariate Normal Distribution
Arnold, B. C., Beaver, R. J., Groeneveld, R. A., & Meeker, W. Q. (1993) · Psychometrika · doi:10.1007/bf02294652 · Cited by 195 · see citing works
Finite mixture modelling using the skew normal distribution
Lin, T., Lee, J. C., & Yen, S. Y. (2007) · Cited by 195 · see citing works
PUBLICATION BIAS: A CASE STUDY OF FOUR TEST VENDORS
McDaniel, M. A., Rothstein, H. R., & Whetzel, D. L. (2006) · Personnel Psychology · doi:10.1111/j.1744-6570.2006.00059.x · Cited by 143 · see citing works
Avoiding Bias in Publication Bias Research: The Value of “Null” Findings
Kepes, S., Banks, G. C., & Oh, I. (2012) · Journal of Business and Psychology · doi:10.1007/s10869-012-9279-0 · Cited by 117 · see citing works
Models for estimating the number of unpublished studies.
Gleser, L., & Olkin, I. (1996) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19961215)15:23 · Cited by 116 · see citing works
Combining significance levels.
Becker, B. J. (1994) · Cited by 103 · see citing works
The Usefulness of the "Fail-Safe" Statistic in Meta-Analysis
Carson, K. P., Schriesheim, C., & Kinicki, A. (1990) · Educational and Psychological Measurement · doi:10.1177/0013164490502001 · Cited by 96 · see citing works
The Folded Normal Distribution: Two Methods of Estimating Parameters from Moments
Elandt, R. C. (1961) · Technometrics · doi:10.1080/00401706.1961.10489975 · Cited by 93 · see citing works
Mavridis, D., Sutton, A. J., Cipriani, A., & Salanti, G. (2012) · Statistics in Medicine · doi:10.1002/sim.5494 · Cited by 83 · see citing works
Modelling publication bias in meta-analysis: a review
Sutton, A. J., Song, F., Gilbody, S. M., et al. (2000) · Statistical Methods in Medical Research · doi:10.1191/096228000701555244 · Cited by 58 · see citing works
On Knowing What We Do Not Know
Kromrey, J. D., & Rendina‐Gobioff, G. (2006) · Educational and Psychological Measurement · doi:10.1177/0013164405278585 · Cited by 51 · see citing works
Fail-Safe Ns for One- Versus Two-Tailed Tests Lead to Different Conclusions About Publication Bias
Hsu, L. (2002) · Understanding Statistics · doi:10.1207/s15328031us0102_02 · Cited by 24 · see citing works
An updated bibliography on the jackknife method
Frangos, C. C. (1987) · Communication in Statistics- Theory and Methods · doi:10.1080/03610928708829454 · Cited by 17 · see citing works
Self-citationA Generalized Publication Bias ModelPreprint
Schönemann, P., & Scargle, J. (2008) · arXiv (Cornell University) · doi:10.6129/cjp.2008.5001.02 · Cited by 14 · see citing works
Discussion of the Paper by Begg and Berlin
(1988) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.1988.tb00052.x · Cited by 9 · see citing works
Bayesian Estimation of the Number of Unseen Studies in a Meta-Analysis
Eberly, L. E., & Casella, G. (1999) · Journal of Official Statistics · Cited by 7 · see citing works
Tsagris, M., Elmatzoglou, I., & Frangos, C. C. (2011) · Communication in Statistics- Theory and Methods · doi:10.1080/03610926.2010.521281 · Cited by 4 · see citing works
Self-citationMeta-analysis: formulating, evaluating, combining, and reporting
Normand, S.-L. T. (1999) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19990215)18:3
Flagged first, then by citations
Research works printed but not linked: 3
- [4]Rothman K. J. Greenland S. Lash T. L. Modern Epidemiology 2008 Philadelphia, Pa, USA Lippincott Williams & Wilkins
- [8]Hedges L. V. Olkin I. Statistical Methods for Meta-Analysis 1985 New York, NY, USA Academic Press MR798597
- [23]Becker B. J. Rothstein H. R. Sutton A. J. Borenstein M. The failsafe N or file-drawer number Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments 2005 Chichester, UK John Wiley & Sons 111 126
Cited by
Sorted by citations
Emotion Regulation and Substance Use: A Meta-Analysis
Weiss, N. H., Kiefer, R., Goncharenko, S., Raudales, A. M., Forkus, S. R., Schick, M. R., & Contractor, A. A. (2022) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2021.109131 · Cited by 235 · see citing works
Rodríguez-Rivas, M. E., Cangas, A. J., Cariola, L. A., Varela, J. J., & Valdebenito, S. (2022) · JMIR Serious Games · doi:10.2196/35099 · Cited by 68 · see citing works
Lu, D., He, Y., & Tan, Y. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.719425 · Cited by 66 · see citing works
Altobelli, E., Angeletti, P. M., Marziliano, C., Mastrodomenico, M., Giuliani, A. R., & Petrocelli, R. (2021) · Nutrients · doi:10.3390/nu13020404 · Cited by 67 · see citing works
Lai, L., Liu, Y., McCracken, L. M., Li, Y., & Ren, Z. (2023) · Behaviour Research and Therapy · doi:10.1016/j.brat.2023.104308 · Cited by 60 · see citing works
Influence of artificial intelligence tools on higher order thinking skills: a meta-analysis
Ilgun Dibek, M., Sahin Kursad, M., & Erdogan, T. (2025) · Interactive Learning Environments · doi:10.1080/10494820.2024.2402028 · Cited by 45 · see citing works
- Authors
- Fragkos, K. C., Tsagris, M., & Frangos, C. C.
- Affiliations
- University College London (GB)
- American University of the Middle East (KW)
- Technological Educational Institute of Athens (GR)
- Corresponding author
- Fragkos, Konstantinos C.
- Venue
- International Scholarly Research Notices
- Publisher
- Hindawi Limited
- Published
- 2014-12-03
- Volume
- 2014
- Pages
- 1-17
- ISSN
- 2356-7872 (ISSN-L)
- PMC
- PMC4897051
- OpenAlex
- W2135241513
- Cited by
- 159
- References
- 63
- Citations by year
- 2026: 20 · 2025: 31 · 2024: 19 · 2023: 14 · 2022: 18 · 2021: 19 · 2020: 10 · 2019: 10 · 2018: 5 · 2017: 5 · 2016: 6 · 2015: 2
- FWCI
- 4.23
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- International Scholarly Research Notices (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Keywords
- publication bias, meta-analysis, confidence intervals, variance estimators, half-normal distribution, nonparametric bootstrap, coverage probability, simulation study
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
