Identification of and Correction for Publication Bias
Open accessThe DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
Abstract
Some empirical results are more likely to be published than others. Selective publication leads to biased estimates and distorted inference. We propose two approaches for identifying the conditional probability of publication as a function of a study’s results, the first based on systematic replication studies and the second on meta-studies. For known conditional publication probabilities, we propose bias-corrected estimators and confidence sets. We apply our methods to recent replication studies in experimental economics and psychology, and to a meta-study on the effect of the minimum wage. When replication and meta-study data are available, we find similar results from both.(JEL C13, C90, I23, J23, J38, L82)
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“the selection model, developed by Andrews and Kasy (2019), which tests for publication selection bias using the conditional probability of publication as a function of a study's results”
From Economic development and the finance–growth nexus: a meta-analytic approach (2024) · cites this paper as Andrews and Kasy (2019)
In the citing paper
“Such connections between sample sizes and effect sizes would violate the assumptions needed to interpret asymmetric funnel plots and Egger's test as publication bias (it would also violate the assumptions underlying the selection models suggested by, e.g., Hedges, 1992 and Andrews & Kasy, 2019).”
From Targeted school‐based interventions for improving reading and mathematics for students with or at risk of academic difficulties in Grades K‐6: A systematic review (2021) · cites this paper as Andrews & Kasy, 2019
Citations
Cites
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
Journal of The American Statistical Association
Fienberg, S. E. (2005) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1317 · Cited by 6,125
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Moffitt, R. A. (2009) · American Economic Review · doi:10.1257/aer.99.2.660 · Cited by 4,694
A Nonparametric “Trim and Fill” Method of Accounting for Publication Bias in Meta-Analysis
Duval, S., & Tweedie, R. (2000) · Journal of the American Statistical Association · doi:10.1080/01621459.2000.10473905 · Cited by 2,955
Publication bias in the social sciences: Unlocking the file drawer
Franco, A., Malhotra, N., & Simonovits, G. (2014) · Science · doi:10.1126/science.1255484 · Cited by 1,542
Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0399-z · Cited by 1,521
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500
Why Most Discovered True Associations Are Inflated
Ioannidis, J. P. A. (2008) · Epidemiology · doi:10.1097/ede.0b013e31818131e7 · Cited by 1,495
Evaluating replicability of laboratory experiments in economics
Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., et al. (2016) · Science · doi:10.1126/science.aaf0918 · Cited by 1,314
Stanley, T. D., & Doucouliagos, H. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1095 · Cited by 916
Small Telescopes: Detectability and the Evaluation of Replication Results
Simonsohn, U. (2015) · Psychological Science · doi:10.1177/0956797614567341 · Cited by 742
The Power of Bias in Economics Research
Ioannidis, J. P. A., Stanley, T. D., & Doucouliagos, H. (2017) · The Economic Journal · doi:10.1111/ecoj.12461 · Cited by 634
Stanley, T. D. (2008) · Oxford Bulletin of Economics and Statistics · doi:10.1111/j.1468-0084.2007.00487.x · Cited by 605
Publication Selection Bias in Minimum‐Wage Research? A Meta‐Regression Analysis
Doucouliagos, H., & Stanley, T. D. (2009) · British Journal of Industrial Relations · doi:10.1111/j.1467-8543.2009.00723.x · Cited by 507
Comment on “Estimating the reproducibility of psychological science.”
Gilbert, D. T., King, G., Pettigrew, S., & Wilson, T. D. (2016) · Science · doi:10.1126/science.aad7243 · Cited by 474
Exactly Median-Unbiased Estimation of First Order Autoregressive/Unit Root Models
Andrews, D. W. K. (1993) · Econometrica · doi:10.2307/2951781 · Cited by 470
Star Wars: The Empirics Strike Back
Brodeur, A., Lé, M., Sangnier, M., & Zylberberg, Y. (2016) · American Economic Journal: Applied Economics · doi:10.1257/app.20150044 · Cited by 430
MEASURING INTERTEMPORAL SUBSTITUTION: THE IMPORTANCE OF METHOD CHOICES AND SELECTIVE REPORTING
Havránek, T. (2015) · Journal of the European Economic Association · doi:10.1111/jeea.12133 · Cited by 365
Median Unbiased Estimation of Coefficient Variance in a Time-Varying Parameter Model
Stock, J. H., & Watson, M. W. (1998) · Journal of the American Statistical Association · doi:10.1080/01621459.1998.10474116 · Cited by 364
Modeling Publication Selection Effects in Meta-Analysis
Hedges, L. V. (1992) · Statistical Science · doi:10.1214/ss/1177011364 · Cited by 359
Promises and Perils of Pre-Analysis Plans
Olken, B. A. (2015) · Journal of Economic Perspectives · doi:10.1257/jep.29.3.61 · Cited by 299
Are all Economic Hypotheses False?
De Long, J. B., & Lang, K. (1992) · Journal of Political Economy · doi:10.1086/261860 · Cited by 295
Patil, P., Peng, R. D., & Leek, J. T. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616646366 · Cited by 243
Interpreting observational studies: why empirical calibration is needed to correct p‐values
Schuemie, M. J., Ryan, P. B., DuMouchel, W., Suchard, M. A., & Madigan, D. (2014) · Statistics in Medicine · doi:10.1002/sim.5925 · Cited by 234
THE MEANING OF FAILED REPLICATIONS: A REVIEW AND PROPOSAL
Clemens, M. A. (2017) · Journal of Economic Surveys · doi:10.1111/joes.12139 · Cited by 213
p-Curve and p-Hacking in Observational Research
Bruns, S. B., & Ioannidis, J. P. A. (2016) · PLOS ONE · doi:10.1371/journal.pone.0149144 · Cited by 210
Response to Comment on “Estimating the reproducibility of psychological science”
Anderson, C. J., Bahník, Š., Barnett-Cowan, M., Bosco, F. A., Chandler, J., Chartier, C. R., Cheung, F., Christopherson, C. D., Cordes, A., Cremata, E. J., et al. (2016) · Science · doi:10.1126/science.aad9163 · Cited by 191
Predicting Experimental Results: Who Knows What?
DellaVigna, S., & Pope, D. (2018) · Journal of Political Economy · doi:10.1086/699976 · Cited by 183
On the Reproducibility of Psychological Science
Johnson, V. E., Payne, R. D., Wang, T., Asher, A., & Mandal, S. (2016) · Journal of the American Statistical Association · doi:10.1080/01621459.2016.1240079 · Cited by 164
Gelman, A. (2017) · Personality and Social Psychology Bulletin · doi:10.1177/0146167217729162 · Cited by 164
Finding the power to reduce publication bias
Stanley, T. D., Doucouliagos, H., & Ioannidis, J. P. A. (2017) · Statistics in Medicine · doi:10.1002/sim.7228 · Cited by 131
Predicting Experimental Results: Who Knows What?
DellaVigna, S., & Pope, D. G. (2016) · National Bureau of Economic Research · doi:10.3386/w22566 · Cited by 124
How to make replication the norm
Gertler, P., Galiani, S., & Romero, M. (2018) · Nature · doi:10.1038/d41586-018-02108-9 · Cited by 81
Conservative Tests under Satisficing Models of Publication Bias
McCrary, J., Christensen, G., & Fanelli, D. (2016) · PLoS ONE · doi:10.1371/journal.pone.0149590 · Cited by 49
Meta‐Regression Models and Observational Research
Bruns, S. B. (2017) · Oxford Bulletin of Economics and Statistics · doi:10.1111/obes.12172 · Cited by 31
Cited by
How Much Should We Trust Staggered Difference-in-Differences Estimates?
Baker, A. C., Larcker, D. F., & Wang, C. C. Y. (2022) · Journal of Financial Economics · doi:10.1016/j.jfineco.2022.01.004 · Cited by 3,149
Pretest with Caution: Event-Study Estimates after Testing for Parallel Trends
Roth, J. (2022) · American Economic Review: Insights · doi:10.1257/aeri.20210236 · Cited by 769
How Much Should We Trust Staggered Difference-In-Differences Estimates?
Baker, A., Larcker, D. F., & Wang, C. C. Y. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3794018 · Cited by 670
A unified welfare analysis of government policies.
Hendren, N., & Sprung-Keyser, B. (2020) · The Quarterly Journal of Economics · doi:10.1093/qje/qjaa006 · Cited by 571
RCTs to Scale: Comprehensive Evidence From Two Nudge Units
DellaVigna, S., & Linos, E. (2022) · Econometrica · doi:10.3982/ecta18709 · Cited by 480
Metadata
- Authors
- Andrews, I., & Kasy, M.
- Affiliations
- Harvard University (US)
- Venue
- American Economic Review
- Publisher
- American Economic Association
- Published
- 2019-08-01
- Volume
- 109
- Issue
- 8
- Pages
- 2766-2794
- ISSN
- 0002-8282 (ISSN-L)
- OpenAlex
- W2598238536
- Cited by
- 388
- References
- 38
- Citations by year
- 2026: 44 · 2025: 47 · 2024: 51 · 2023: 62 · 2022: 56 · 2021: 53 · 2020: 50 · 2019: 14
- FWCI
- 28.94
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- American Economic Review (publishedVersion)
- Institutional Repository (IHS Vienna) (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Funders
- Styrelsen för Internationellt Utvecklingssamarbete
- Department for International Development
- United Nations University World Institute for Development Economics Research
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- publication bias, selective publication, replication study, meta-study, bias-corrected estimators, confidence regions, experimental economics, minimum wage
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.