Beyond Publication Bias
Open accessThe DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index.
Abstract
This review considers several meta-regression and graphical methods that can differentiate genuine empirical effect from publication bias. Publication selection exists when editors, reviewers, or researchers have a preference for statistically significant results. Because all areas of empirical research are susceptible to publication selection, any average or tally of significant/insignificant studies is likely to be biased and potentially misleading. Meta-regression analysis can see through the murk of random sampling error and selected misspecification bias to identify the underlying statistical structures that characterize genuine empirical effect. Meta-significance testing and precision-effect testing (PET) are offered as a means to identify empirical effect beyond publication bias and are applied to four areas of empirical economics research – minimum wage effects, union-productivity effects, price elasticities, and tests of the natural rate hypothesis.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763
A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity
White, H. (1980) · Econometrica · doi:10.2307/1912934 · Cited by 25,861
Operating characteristics of a rank correlation test for publication bias
Begg, C. B., & Mazumdar, M. (1994) · Biometrics · doi:10.2307/2533446 · Cited by 16,955
Statistical Methods for Meta-Analysis
Wasserman, S., Hedges, L. V., & Olkin, I. (1988) · Journal of Educational Statistics · doi:10.2307/1164953 · Cited by 10,326
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
The Logic of Scientific Discovery
Popper, K. R., & Weiss, G. (1959) · Physics Today · doi:10.1063/1.3060577 · Cited by 2,717
Publication and related bias in meta-analysis
Sterne, J. A. C., Gavaghan, D., & Egger, M. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(00)00242-0 · Cited by 2,282
Econometric Theory and Methods
Davidson, R., & MacKinnon, J. G. (2003) · Cited by 1,768
Davidson, J. E. H., Hendry, D. F., Srba, F., & Yeo, S. (1978) · The Economic Journal · doi:10.2307/2231972 · Cited by 1,628
Cited by
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553
A meta-analysis of estimates of urban agglomeration economies
Melo, P. C., Graham, D. J., & Noland, R. B. (2009) · Regional Science and Urban Economics · doi:10.1016/j.regsciurbeco.2008.12.002 · Cited by 821
META‐REGRESSION ANALYSIS: A QUANTITATIVE METHOD OF LITERATURE SURVEYS
Stanley, T. D., & Jarrell, S. B. (2005) · Journal of Economic Surveys · doi:10.1111/j.0950-0804.2005.00249.x · Cited by 805
Self-citationMETA‐REGRESSION ANALYSIS: A QUANTITATIVE METHOD OF LITERATURE SURVEYS
Stanley, T. D., & Jarrell, S. B. (1989) · Journal of Economic Surveys · doi:10.1111/j.1467-6419.1989.tb00064.x · Cited by 805
Self-citationReflections on the Natural Rate Hypothesis
Stiglitz, J. (1997) · Journal of Economic Perspectives · doi:10.1257/jep.11.1.3 · Cited by 664
The Power of Bias in Economics Research
Ioannidis, J. P. A., Stanley, T. D., & Doucouliagos, H. (2017) · The Economic Journal · doi:10.1111/ecoj.12461 · Cited by 634
Self-citationEDUCATION AND ENTREPRENEURSHIP SELECTION AND PERFORMANCE: A REVIEW OF THE EMPIRICAL LITERATURE
Van Der Sluis, J., Van Praag, M., & Vijverberg, W. (2008) · Journal of Economic Surveys · doi:10.1111/j.1467-6419.2008.00550.x · Cited by 616
Stanley, T. D. (2008) · Oxford Bulletin of Economics and Statistics · doi:10.1111/j.1468-0084.2007.00487.x · Cited by 605
Self-citationRecent developments in meta‐analysis
Sutton, A. J., & Higgins, J. P. T. (2008) · Statistics in Medicine · doi:10.1002/sim.2934 · Cited by 590
Democracy and Economic Growth: A Meta‐Analysis
Doucouliagos, H., & Ulubaşoğlu, M. A. (2008) · American Journal of Political Science · doi:10.1111/j.1540-5907.2007.00299.x · Cited by 584
Financial performance of socially responsible investing (SRI): what have we learned? A meta‐analysis
Revelli, C., & Viviani, J. (2015) · Business Ethics: A European Review · doi:10.1111/beer.12076 · Cited by 539
The effect of FDI on environmental emissions: Evidence from a meta-analysis
Demena, B. A., & Afesorgbor, S. K. (2020) · Energy Policy · doi:10.1016/j.enpol.2019.111192 · Cited by 538
Estimating vertical spillovers from FDI: Why results vary and what the true effect is
Havranek, T., & Irsova, Z. (2011) · Journal of International Economics · doi:10.1016/j.jinteco.2011.07.004 · Cited by 535
The Use (and Abuse) of Meta-Analysis in Environmental and Natural Resource Economics: An Assessment
Nelson, J. P., & Kennedy, P. E. (2009) · Environmental and Resource Economics · doi:10.1007/s10640-008-9253-5 · Cited by 530
THE AID EFFECTIVENESS LITERATURE: THE SAD RESULTS OF 40 YEARS OF RESEARCH
Doucouliagos, H., & Paldam, M. (2009) · Journal of Economic Surveys · doi:10.1111/j.1467-6419.2008.00568.x · Cited by 527
WHAT HAVE WE LEARNED FROM THREE DECADES OF RESEARCH ON THE PRODUCTIVITY OF PUBLIC CAPITAL?
Bom, P. R. D., & Ligthart, J. E. (2014) · Journal of Economic Surveys · doi:10.1111/joes.12037 · Cited by 525
Publication Selection Bias in Minimum‐Wage Research? A Meta‐Regression Analysis
Doucouliagos, H., & Stanley, T. D. (2009) · British Journal of Industrial Relations · doi:10.1111/j.1467-8543.2009.00723.x · Cited by 507
Self-citationRose, A. K., & Stanley, T. D. (2005) · Journal of Economic Surveys · doi:10.1111/j.0950-0804.2005.00251.x · Cited by 463
Self-citationBel, G., Fageda, X., & Warner, M. E. (2010) · Journal of Policy Analysis and Management · doi:10.1002/pam.20509 · Cited by 461
Transparency, Reproducibility, and the Credibility of Economics Research
Christensen, G., & Miguel, E. (2018) · Journal of Economic Literature · doi:10.1257/jel.20171350 · Cited by 446
Natural Resources and Economic Growth: A Meta-Analysis
Havranek, T., Horvath, R., & Zeynalov, A. (2016) · World Development · doi:10.1016/j.worlddev.2016.07.016 · Cited by 436
Star Wars: The Empirics Strike Back
Brodeur, A., Lé, M., Sangnier, M., & Zylberberg, Y. (2016) · American Economic Journal: Applied Economics · doi:10.1257/app.20150044 · Cited by 430
Steenbergen-Hu, S., Makel, M. C., & Olszewski-Kubilius, P. (2016) · Review of Educational Research · doi:10.3102/0034654316675417 · Cited by 429
Does Compact Development Make People Drive Less?
Stevens, M. R. (2017) · Journal of the American Planning Association · doi:10.1080/01944363.2016.1240044 · Cited by 405
Methods for testing publication bias in ecological and evolutionary meta‐analyses
Nakagawa, S., Lagisz, M., Jennions, M. D., Koricheva, J., Noble, D. W. A., Parker, T. H., Sánchez‐Tójar, A., Yang, Y., & O'Dea, R. E. (2022) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13724 · Cited by 399
Carter, E. C., & McCullough, M. E. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00823 · Cited by 381
The effectiveness of R&D subsidies: A meta-regression analysis of the evaluation literature
Dimos, C., & Pugh, G. (2016) · Research Policy · doi:10.1016/j.respol.2016.01.002 · Cited by 381
Weingarten, E., Chen, Q., McAdams, M., Yi, J., Hepler, J., & Albarracín, D. (2016) · Psychological Bulletin · doi:10.1037/bul0000030 · Cited by 380
van Essen, M., Otten, J., & Carberry, E. J. (2015) · Journal of Management · doi:10.1177/0149206311429378 · Cited by 372
MEASURING INTERTEMPORAL SUBSTITUTION: THE IMPORTANCE OF METHOD CHOICES AND SELECTIVE REPORTING
Havránek, T. (2015) · Journal of the European Economic Association · doi:10.1111/jeea.12133 · Cited by 365
Aid effectiveness on growth: A meta study
Doucouliagos, H., & Paldam, M. (2008) · European Journal of Political Economy · doi:10.1016/j.ejpoleco.2007.06.002 · Cited by 363
Determinants of Horizontal Spillovers from FDI: Evidence from a Large Meta-Analysis
Iršová, Z., & Havránek, T. (2013) · World Development · doi:10.1016/j.worlddev.2012.07.001 · Cited by 361
Natural disasters and economic growth: A meta-analysis
Klomp, J., & Valckx, K. (2014) · Global Environmental Change · doi:10.1016/j.gloenvcha.2014.02.006 · Cited by 354
A meta-analysis of consumer willingness to pay for farm animal welfare
Lagerkvist, C. J., & Hess, S. (2011) · European Review of Agricultural Economics · doi:10.1093/erae/jbq043 · Cited by 349
Income Elasticities and Global Values of a Statistical Life
Viscusi, W. K., & Masterman, C. J. (2017) · Journal of Benefit-Cost Analysis · doi:10.1017/bca.2017.12 · Cited by 345
Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics
Brodeur, A., Cook, N., & Heyes, A. (2020) · American Economic Review · doi:10.1257/aer.20190687 · Cited by 333
Education and Economic Growth: A Meta-Regression Analysis
Benos, N., & Zotou, S. (2014) · World Development · doi:10.1016/j.worlddev.2014.06.034 · Cited by 324
Multinationals’ profit response to tax differentials: Effect size and shifting channels
Heckemeyer, J. H., & Overesch, M. (2017) · Canadian Journal of Economics/Revue canadienne d'économique · doi:10.1111/caje.12283 · Cited by 322
The Synthesis of Regression Slopes in Meta-Analysis
Becker, B. J., & Wu, M.-J. (2007) · Statistical Science · doi:10.1214/07-sts243 · Cited by 322
FINANCIAL DEVELOPMENT AND ECONOMIC GROWTH: A META‐ANALYSIS
Valickova, P., Havranek, T., & Horvath, R. (2015) · Journal of Economic Surveys · doi:10.1111/joes.12068 · Cited by 315
What meta-analyses reveal about the replicability of psychological research.
Stanley, T. D., Carter, E. C., & Doucouliagos, H. (2018) · Psychological Bulletin · doi:10.1037/bul0000169 · Cited by 307
Self-citationMETHODS, TRENDS AND CONTROVERSIES IN CONTEMPORARY BENEFIT TRANSFER
Johnston, R. J., & Rosenberger, R. S. (2010) · Journal of Economic Surveys · doi:10.1111/j.1467-6419.2009.00592.x · Cited by 299
The Role of the Beneficiary in Willingness to Pay for Socially Responsible Products: A Meta-analysis
Tully, S. M., & Winer, R. S. (2014) · Journal of Retailing · doi:10.1016/j.jretai.2014.03.004 · Cited by 294
Swedberg, R. (2020) · The Production of Knowledge · doi:10.1017/9781108762519.002 · Cited by 293
FDI AND TAXATION: A META-STUDY
Feld, L. P., & Heckemeyer, J. H. (2011) · Journal of Economic Surveys · doi:10.1111/j.1467-6419.2010.00674.x · Cited by 284
ARE ALL ECONOMIC FACTS GREATLY EXAGGERATED? THEORY COMPETITION AND SELECTIVITY
Doucouliagos, C., & Stanley, T. D. (2013) · Journal of Economic Surveys · doi:10.1111/j.1467-6419.2011.00706.x · Cited by 276
Self-citationBranger, F., & Quirion, P. (2014) · Ecological Economics · doi:10.1016/j.ecolecon.2013.12.010 · Cited by 273
Cross-country heterogeneity in intertemporal substitution
Havranek, T., Horvath, R., Irsova, Z., & Rusnak, M. (2015) · Journal of International Economics · doi:10.1016/j.jinteco.2015.01.012 · Cited by 273
Cohen, M. A., & Tubb, A. (2018) · Journal of the Association of Environmental and Resource Economists · doi:10.1086/695613 · Cited by 272
Military expenditure and economic growth: A meta-analysis
Alptekin, A., & Levine, P. (2012) · European Journal of Political Economy · doi:10.1016/j.ejpoleco.2012.07.002 · Cited by 271
DOES EDUCATION REDUCE INCOME INEQUALITY? A META‐REGRESSION ANALYSIS
Abdullah, A., Doucouliagos, H., & Manning, E. (2015) · Journal of Economic Surveys · doi:10.1111/joes.12056 · Cited by 267
Remittances and economic growth: A meta-analysis
Cazachevici, A., Havranek, T., & Horvath, R. (2020) · World Development · doi:10.1016/j.worlddev.2020.105021 · Cited by 262
Demand for gasoline is more price-inelastic than commonly thought
Havranek, T., Irsova, Z., & Janda, K. (2012) · Energy Economics · doi:10.1016/j.eneco.2011.09.003 · Cited by 164
Could It Be Better to Discard 90% of the Data? A Statistical Paradox
Stanley, T. D., Jarrell, S. B., & Doucouliagos, H. (2010) · The American Statistician · doi:10.1198/tast.2009.08205 · Cited by 170
Self-citationA historical review of publication bias
Marks‐Anglin, A., & Chen, Y. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1452 · Cited by 165
Estimating the Armington elasticity: The importance of study design and publication bias
Bajzik, J., Havranek, T., Irsova, Z., & Schwarz, J. (2020) · Journal of International Economics · doi:10.1016/j.jinteco.2020.103383 · Cited by 150
“As we have noted, if publication selection is present, the reported estimates and standard errors are correlated (Stanley, 2005; Havranek, 2010; Havranek & Irsova, 2011; Havranek et al., 2018b): σij = σ0 + δ · SE(σij) + µij , (1) where σij denotes i-th estimate of the Armington elasticity with the…”
BackgroundA kinked meta‐regression model for publication bias correction
Bom, P. R. D., & Rachinger, H. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1352 · Cited by 133
How puzzling is the forward premium puzzle? A meta-analysis
Zigraiova, D., Havranek, T., Irsova, Z., & Novak, J. (2021) · European Economic Review · doi:10.1016/j.euroecorev.2021.103714 · Cited by 71
Corporate ownership and managerial turnover in China and Eastern Europe: A comparative meta-analysis
Iwasaki, I., Ma, X., & Mizobata, S. (2020) · Journal of Economics and Business · doi:10.1016/j.jeconbus.2020.105928 · Cited by 51
“…values is equal is accepted only for the domestic outside ownership variable in the case of studies of China and 17 The analytical approach, whereby the mean of the most precise 10% of estimates is regarded as the approximate value of the true effect, was originally proposed by Stanley (2005).”
BackgroundMueller, K. F., Meerpohl, J. J., Briel, M., Antes, G., von Elm, E., Lang, B., Motschall, E., Schwarzer, G., & Bassler, D. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.04.015 · Cited by 48
Bogg, T., & Lasecki, L. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.01589 · Cited by 41
“Valentine et al. (2010) advised that retrospective meta-analytic power analyses are not informative when they are based on the observed estimates of the modeling procedure (e.g., values from a random-effects model).”
BackgroundMethodDownie, A. T., Wu, N. C., Cramp, R. L., & Franklin, C. E. (2023) · Global Change Biology · doi:10.1111/gcb.16848 · Cited by 34
Metadata
- Authors
- Stanley, T. D.
- Venue
- Journal of Economic Surveys
- Publisher
- Wiley
- Published
- 2005-06-22
- Volume
- 19
- Issue
- 3
- Pages
- 309-345
- ISSN
- 0950-0804 (ISSN-L)
- OpenAlex
- W1864180066
- Cited by
- 742
- References
- 47
- FWCI
- 21.01
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.