Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics

Brodeur, Abel; Cook, Nikolai; Heyes, Anthony

doi:10.1257/aer.20190687

Cited by 333Advanced Causal Inference TechniquesMeta-analysis and systematic reviewsscientometrics and bibliometrics research

Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics

Brodeur, Abel, Cook, Nikolai, Heyes, Anthony·2020·American Economic Review

DOI 10.1257/aer.20190687

Open this paper in CoChat — free account

Abstract

The credibility revolution in economics has promoted causal identification using randomized control trials (RCT), difference-in-differences (DID), instrumental variables (IV) and regression discontinuity design (RDD). Applying multiple approaches to over 21,000 hypothesis tests published in 25 leading economics journals, we find that the extent of p-hacking and publication bias varies greatly by method. IV (and to a lesser extent DID) are particularly problematic. We find no evidence that (i) papers published in the Top 5 journals are different to others; (ii) the journal “revise and resubmit” process mitigates the problem; (iii) things are improving through time. (JEL A14, C12, C52)

Integrity signals

Registry

Crossref · 10.1257/aer.20190687

Cites

41 of 41 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Brodeur et al. (2020) find substantial evidence of p-hacking in the quasi-experimental literature, particularly in reporting of results from DiD designs.
“Brodeur et al. (2020) find sub-”
From Can Policies Stall the Fertility Fall? A Systematic Review of the (Quasi‐) Experimental Literature (2021) · cites this paper as Brodeur et al. 2020

Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505 · see citing works
False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567 · see citing works
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · see citing works
Optimal Bandwidth Choice for the Regression Discontinuity Estimator
Imbens, G. W., & Kalyanaraman, K. (2011) · The Review of Economic Studies · doi:10.1093/restud/rdr043 · Cited by 2,135 · see citing works
Understanding and misunderstanding randomized controlled trials
Deaton, A., & Cartwright, N. (2018) · Social Science & Medicine · doi:10.1016/j.socscimed.2017.12.005 · Cited by 1,841 · see citing works
The American Economic Review
(2009) · American Economic Review · doi:10.1257/aer.99.1.i · Cited by 1,764 · see citing works
The Credibility Revolution in Empirical Economics: How Better Research Design is Taking the Con out of Econometrics
Angrist, J. D., & Pischke, J.-S. (2010) · Journal of Economic Perspectives · doi:10.1257/jep.24.2.3 · Cited by 1,566 · see citing works
Weak Instruments in Instrumental Variables Regression: Theory and Practice
Andrews, I., Stock, J. H., & Sun, L. (2019) · Annual Review of Economics · doi:10.1146/annurev-economics-080218-025643 · Cited by 849 · see citing works
The Power of Bias in Economics Research
Ioannidis, J. P. A., Stanley, T. D., & Doucouliagos, H. (2017) · The Economic Journal · doi:10.1111/ecoj.12461 · Cited by 634 · see citing works
Meta‐Regression Methods for Detecting and Estimating Empirical Effects in the Presence of Publication Selection*
Stanley, T. D. (2008) · Oxford Bulletin of Economics and Statistics · doi:10.1111/j.1468-0084.2007.00487.x · Cited by 605 · see citing works
Promoting Transparency in Social Science Research
Miguel, E. A., Camerer, C. F., Casey, K., Cohen, J., Esterling, K., Gerber, A. S., Glennerster, R., GREEN, D. P., Humphreys, M., Imbens, G. W., et al. (2014) · Science · doi:10.1126/science.1245317 · Cited by 544 · see citing works
Reshaping Institutions: Evidence on Aid Impacts Using a Pre-Analysis Plan
Casey, K., Glennerster, R., & Miguel, E. (2012) · The Quarterly Journal of Economics · doi:10.1093/qje/qje027 · Cited by 524 · see citing works
Beyond Publication Bias
Stanley, T. (2005) · Journal of Economic Surveys · doi:10.1111/j.0950-0804.2005.00250.x · Cited by 489 · see citing works
Better LATE Than Nothing: Some Comments on Deaton (2009) and Heckman and Urzua (2009)
Imbens, G. W. (2010) · Journal of Economic Literature · doi:10.1257/jel.48.2.399 · Cited by 466 · see citing works
Transparency, Reproducibility, and the Credibility of Economics Research
Christensen, G., & Miguel, E. A. (2018) · Journal of Economic Literature · doi:10.1257/jel.20171350 · Cited by 448 · see citing works
Star Wars: The Empirics Strike Back
Brodeur, A., Lé, M., Sangnier, M., & Zylberberg, Y. (2016) · American Economic Journal: Applied Economics · doi:10.1257/app.20150044 · Cited by 430 · see citing works
Self-citation
A review of estimates of the schooling/earnings relationship, with tests for publication bias
Ashenfelter, O., Harmon, C. P., & Oosterbeek, H. (1999) · Labour Economics · doi:10.1016/s0927-5371(99)00041-x · Cited by 413 · see citing works
Identification of and correction for publication bias
Andrews, I., & Kasy, M. (2019) · American Economic Review · doi:10.1257/aer.20180310 · Cited by 388 · see citing works
MEASURING INTERTEMPORAL SUBSTITUTION: THE IMPORTANCE OF METHOD CHOICES AND SELECTIVE REPORTING
Havránek, T. (2015) · Journal of the European Economic Association · doi:10.1111/jeea.12133 · Cited by 369 · see citing works
Nine Facts about Top Journals in Economics
Card, D., & DellaVigna, S. (2013) · Journal of Economic Literature · doi:10.1257/jel.51.1.144 · Cited by 333 · see citing works
Reporting the Fragility of Regression Estimates
Leamer, E. E., & Leonard, H. B. “. (1983) · The Review of Economics and Statistics · doi:10.2307/1924497 · Cited by 328 · see citing works
Are all Economic Hypotheses False?
Long, J. B. D., & Lang, K. (1992) · Journal of Political Economy · doi:10.1086/261860 · Cited by 303 · see citing works
ARE ALL ECONOMIC FACTS GREATLY EXAGGERATED? THEORY COMPETITION AND SELECTIVITY
Doucouliagos, H., & Stanley, T. D. (2011) · Journal of Economic Surveys · doi:10.1111/j.1467-6419.2011.00706.x · Cited by 276 · see citing works
Do Statistical Reporting Standards Affect What Is Published? Publication Bias in Two Leading Political Science Journals
Gerber, A. S., & Malhotra, N. (2008) · Quarterly Journal of Political Science · doi:10.1561/100.00008024 · Cited by 260 · see citing works
Consistency without Inference: Instrumental Variables in Practical Application
Young, A. (2022) · European Economic Review · doi:10.1016/j.euroecorev.2022.104112 · Cited by 215 · see citing works
The Loss Function Has Been Mislaid: The Rhetoric of Significance Tests
McCLOSKEY, D. N., Ames, E. S., Reiter, S., Feige, E. L., Leamer, E. E., Mayer, T., Lovell, M. C., & Denton, F. T. (1985) · American Economic Review · Cited by 201 · see citing works
Publication Bias in Empirical Sociological Research
Gerber, A. S., & Malhotra, N. (2008) · Sociological Methods & Research · doi:10.1177/0049124108318973 · Cited by 175 · see citing works
Statistical Nonsignificance in Empirical Economics
Abadie, A. (2020) · American Economic Review Insights · doi:10.1257/aeri.20190252 · Cited by 142 · see citing works
Do consumers really follow a rule of thumb? Three thousand estimates from 144 studies say "probably not
Havranek, T., & Sokolova, A. (2020) · Review of Economic Dynamics · doi:10.1016/j.red.2019.05.004 · Cited by 141 · see citing works
What Do Editors Maximize? Evidence from Four Economics Journals
Card, D., & DellaVigna, S. (2020) · The Review of Economics and Statistics · doi:10.1162/rest_a_00839 · Cited by 123 · see citing works
Strategic Disclosure of Research Results: The Cost of Proving Your Honesty
Henry, E. (2009) · The Economic Journal · doi:10.1111/j.1468-0297.2009.02265.x · Cited by 89 · see citing works
Publication Bias and Editorial Statement on Negative Findings
Pérez, C. B., & Brodeur, A. (2019) · The Economic Journal · doi:10.1093/ej/ueaa011 · Cited by 83 · see citing works
Self-citation
Publication Bias under Aggregation Frictions: Theory, Evidence, and a New Correction MethodPreprint
Furukawa, C. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3362053 · Cited by 78 · see citing works
IS PEER REVIEW IN DECLINE?
Ellison, G. (2010) · Economic Inquiry · doi:10.1111/j.1465-7295.2010.00261.x · Cited by 75 · see citing works
The Empirical Economist's Toolkit: From Models to Methods
Panhans, M. T., & Singleton, J. D. (2015) · History of Political Economy · doi:10.1215/00182702-4166299 · Cited by 67 · see citing works
Should the Randomistas (Continue to) Rule?
Ravallion, M. D. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.003.0003 · Cited by 62 · see citing works
Specification Searching and Significance Inflation Across Time, Methods and Disciplines
Vivalt, E. (2019) · Oxford Bulletin of Economics and Statistics · doi:10.1111/obes.12289 · Cited by 58 · see citing works
Reporting errors and biases in published empirical findings: Evidence from innovation research
Bruns, S. B., Asanov, I., Bode, R., Dunger, M., Funk, C., Hassan, S. M., Hauschildt, J., Heinisch, D. P., Kempa, K., König, J., et al. (2019) · Research Policy · doi:10.1016/j.respol.2019.05.005 · Cited by 47 · see citing works
Measuring the Income Elasticity of Water Demand: The Importance of Publication and Endogeneity Biases
Havránek, T., Iršová, Z., & Vlach, T. (2018) · Land Economics · doi:10.3368/le.94.2.259 · Cited by 45 · see citing works
Theory and Measurement
Biddle, J. E., & Hamermesh, D. S. (2017) · History of Political Economy · doi:10.1215/00182702-4166251 · Cited by 27 · see citing works
Data and Code for: Methods Matter: P-Hacking and Publication Bias in Causal Analysis in Economics
Brodeur, A., Cook, N., & Heyes, A. (2015) · ICPSR Data Holdings · doi:10.3886/e120246 · Cited by 2 · see citing works
Self-citation

Flagged first, then by citations

Sorted by citations

RCTs to Scale: Comprehensive Evidence From Two Nudge Units
DellaVigna, S., & Linos, E. (2022) · Econometrica · doi:10.3982/ecta18709 · Cited by 510 · see citing works
Financial Education Affects Financial Knowledge and Downstream Behaviors
Kaiser, T., Lusardi, A., Menkhoff, L., & Urban, C. (2022) · Journal of Financial Economics · doi:10.1016/j.jfineco.2021.09.022 · Cited by 508 · see citing works
The Short-Term Economic Consequences of Covid-19: Exposure to Disease, Remote Work and Government Response
Beland, L., Brodeur, A., & Wright, T. J. (2020) · PLoS ONE · doi:10.1371/journal.pone.0270341 · Cited by 321 · see citing works
Self-citation
Can Policies Stall the Fertility Fall? A Systematic Review of the (Quasi‐) Experimental Literature
Bergsvik, J., Fauske, A., & Hart, R. K. (2021) · Population and Development Review · doi:10.1111/padr.12431 · Cited by 259 · see citing works
Causality redux: The evolution of empirical methods in accounting research and the growth of quasi-experiments
Armstrong, C., Kepler, J. D., Samuels, D., & Taylor, D. J. (2022) · Journal of Accounting and Economics · doi:10.1016/j.jacceco.2022.101521 · Cited by 238 · see citing works
The COVID-19 pandemic and the 2020 US presidential election
Baccini, L., Brodeur, A., & Weymouth, S. J. (2021) · Journal of Population Economics · doi:10.1007/s00148-020-00820-3 · Cited by 189 · see citing works
Self-citation
Nonreplicable publications are cited more than replicable ones
Serra-Garcia, M., & Gneezy, U. (2021) · Science Advances · doi:10.1126/sciadv.abd1705 · Cited by 177 · see citing works
Estimating the Armington elasticity: The importance of study design and publication bias
Bajzik, J., Havranek, T., Irsova, Z., & Schwarz, J. (2020) · Journal of International Economics · doi:10.1016/j.jinteco.2020.103383 · Cited by 150 · see citing works
Technology and jobs: A systematic literature review
Hötte, K., Somers, M. A., & Theodorakopoulos, A. (2023) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2023.122750 · Cited by 185 · see citing works
Meta-analysis of Empirical Estimates of Loss Aversion
Brown, A. L., Imai, T., Vieider, F. M., & Camerer, C. F. (2024) · Journal of Economic Literature · doi:10.1257/jel.20221698 · Cited by 164 · see citing works
Evidence of questionable research practices in clinical prediction models
White, N. M., Parsons, R., Collins, P. G. S., & Barnett, A. G. (2023) · BMC Medicine · doi:10.1186/s12916-023-03048-6 · Cited by 149 · see citing works
Measuring capital-labor substitution: The importance of method choices and publication bias
Gechert, S., Havránek, T., Iršová, Z., & Kolcunová, D. (2021) · Review of Economic Dynamics · doi:10.1016/j.red.2021.05.003 · Cited by 131 · see citing works
Big little lies: a compendium and simulation of p -hacking strategies
Stefan, A. M., & Schönbrodt, F. D. (2023) · Royal Society Open Science · doi:10.1098/rsos.220346 · Cited by 149 · see citing works
Methodological Variation in Empirical Corporate Finance
Mitton, T. (2021) · Review of Financial Studies · doi:10.1093/rfs/hhab030 · Cited by 95 · see citing works
The Distribution of School Spending Impacts
Jackson, C. K., & Mackevicius, C. (2021) · doi:10.3386/w28517 · Cited by 89 · see citing works
Individual discount rates: a meta-analysis of experimental evidence
Matousek, J., Havránek, T., & Iršová, Z. (2021) · Experimental Economics · doi:10.1007/s10683-021-09716-9 · Cited by 86 · see citing works
Political Language in Economics
Jelveh, Z., Kogut, B., & Naidu, S. (2014) · The Economic Journal · doi:10.1093/ej/ueae026 · Cited by 88 · see citing works
Unemployment and health: A meta‐analysis
Picchio, M., & Ubaldi, M. (2023) · Journal of Economic Surveys · doi:10.1111/joes.12588 · Cited by 85 · see citing works
Does Democracy Matter?
Gerring, J., Knutsen, C. H., & Berge, J. (2022) · Annual Review of Political Science · doi:10.1146/annurev-polisci-060820-060910 · Cited by 76 · see citing works
What Impacts Can We Expect from School Spending Policy? Evidence from Evaluations in the United States
Jackson, C. K., & Mackevicius, C. L. (2023) · American Economic Journal Applied Economics · doi:10.1257/app.20220279 · Cited by 83 · see citing works
Heterogeneity in effect size estimates
Holzmeister, F., Johannesson, M., BOHM, R. M., Dreber, A., Huber, J., & Kirchler, M. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2403490121 · Cited by 82 · see citing works
Carbon taxes and emission trading systems: Which one is more effective in reducing carbon emissions?—A meta-analysis
Ahmad, M., Li, X. F., & Wu, Q. (2024) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2024.143761 · Cited by 72 · see citing works
How puzzling is the forward premium puzzle? A meta-analysis
Zigraiova, D., Havranek, T., Irsova, Z., & Novak, J. (2021) · European Economic Review · doi:10.1016/j.euroecorev.2021.103714 · Cited by 71 · see citing works
The Populist Backlash Against Globalization: A Meta-Analysis of the Causal Evidence
Scheiring, G., Serrano‐Alarcón, M., Moise, A. D., McNamara, C. L., & Stückler, D. (2024) · British Journal of Political Science · doi:10.1017/s0007123424000024 · Cited by 76 · see citing works
Meta‐analysis of social science research: A practitioner's guide
Iršová, Z., Doucouliagos, H., Havránek, T., & Stanley, T. D. (2023) · Journal of Economic Surveys · doi:10.1111/joes.12595 · Cited by 69 · see citing works
Detecting p‐Hacking
Elliott, G., Kudrin, N., & Wüthrich, K. (2022) · Econometrica · doi:10.3982/ecta18583 · Cited by 61 · see citing works
Regression discontinuity designs in agricultural and environmental economics
Wuepper, D., & Finger, R. (2023) · European Review of Agricultural Economics · doi:10.1093/erae/jbac023 · Cited by 55 · see citing works
Examining the generalizability of research findings from archival data
Delios, A., Clemente, E. G., Wu, T., Tan, H., Wang, Y., Gordon, M., Viganola, D., Chen, Z., Dreber, A., Johannesson, M., et al. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2120377119 · Cited by 56 · see citing works
Publication and Attenuation Biases in Measuring Skill Substitution
Havránek, T., Iršová, Z., Laslopová, Ľ., & Zeynalova, O. (2022) · The Review of Economics and Statistics · doi:10.1162/rest_a_01227 · Cited by 49 · see citing works
Do economists replicate?
Peters, J., Fiala, N., & Neubauer, F. (2023) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2023.05.009 · Cited by 55 · see citing works
Questionable Research Practices and Open Science in Quantitative Criminology
Chin, J. M., Pickett, J. T., Vazire, S., & Holcombe, A. O. (2021) · Journal of Quantitative Criminology · doi:10.1007/s10940-021-09525-6 · Cited by 59 · see citing works
Quantitative Political Science Research Is Greatly Underpowered
Arel‐Bundock, V., Briggs, R. C., Doucouliagos, H., Aviña, M. M., & Stanley, T. D. (2024) · The Journal of Politics · doi:10.1086/734279 · Cited by 49 · see citing works
Factorial Designs, Model Selection, and (Incorrect) Inference in Randomized Experiments
Muralidharan, K., Romero, M., & Wüthrich, K. (2020) · The Review of Economics and Statistics · doi:10.1162/rest_a_01317 · Cited by 96 · see citing works
Non-standard errors in asset pricing: Mind your sorts
Soebhag, A., van Vliet, B., & Verwijmeren, P. (2024) · Journal of Empirical Finance · doi:10.1016/j.jempfin.2024.101517 · Cited by 47 · see citing works
The Null Result Penalty
Chopra, F., Haaland, I., Roth, C., & Stegmann, A. (2023) · The Economic Journal · doi:10.1093/ej/uead060 · Cited by 48 · see citing works
Unpacking p-Hacking and Publication Bias
Brodeur, A., Carrell, S., Figlio, D., & Lusher, L. R. (2023) · American Economic Review · doi:10.1257/aer.20210795 · Cited by 47 · see citing works
Self-citation
Replications in agricultural economics
Finger, R., Grebitus, C., & Henningsen, A. (2023) · Applied Economic Perspectives and Policy · doi:10.1002/aepp.13386 · Cited by 43 · see citing works
Rain, Rain, Go Away: 195 Potential Exclusion-Restriction Violations for Studies Using Weather as an Instrumental VariablePreprint
Mellon, J. (2021) · doi:10.31235/osf.io/9qj4f · Cited by 48 · see citing works
Preference-based instrumental variables in health research rely on important and underreported assumptions: a systematic review
Widding‐Havneraas, T., Chaulagain, A., Lyhmann, I., Zachrisson, H. D., Elwert, F., Markussen, S., McDaid, D., & Mykletun, A. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.06.006 · Cited by 40 · see citing works
Causal Machine Learning and its use for public policy
Lechner, M. (2023) · Zeitschrift für schweizerische Statistik und Volkswirtschaft/Schweizerische Zeitschrift für Volkswirtschaft und Statistik/Swiss journal of economics and statistics · doi:10.1186/s41937-023-00113-y · Cited by 48 · see citing works
Unemployment Scarring Effects: An Overview and Meta-analysis of Empirical Studies
Filomena, M. (2023) · Italian Economic Journal · doi:10.1007/s40797-023-00228-4 · Cited by 40 · see citing works
A framework for evaluating reproducibility and replicability in economics
Dreber, A., & Johannesson, M. (2024) · Economic Inquiry · doi:10.1111/ecin.13244 · Cited by 38 · see citing works
Rain, rain, go away: 194 potential exclusion‐restriction violations for studies using weather as an instrumental variable
Mellon, J. (2024) · American Journal of Political Science · doi:10.1111/ajps.12894 · Cited by 38 · see citing works
Discontinuous Distribution of Test Statistics Around Significance Thresholds in Empirical Accounting Studies
Chang, X., Gao, H., & Li, W. (2024) · Journal of Accounting Research · doi:10.1111/1475-679x.12579 · Cited by 35 · see citing works
Nudging for Tax Compliance: A Meta-Analysis
Antinyan, A., & Asatryan, Z. (2024) · The Economic Journal · doi:10.1093/ej/ueae088 · Cited by 35 · see citing works
The Significance of Data-Sharing Policy
Askarov, Z., Doucouliagos, A., Doucouliagos, H., & Stanley, T. D. (2022) · Journal of the European Economic Association · doi:10.1093/jeea/jvac053 · Cited by 33 · see citing works
Big Little Lies: A Compendium and Simulation of p-Hacking StrategiesPreprint
Stefan, A. M., & Schönbrodt, F. D. (2022) · doi:10.31234/osf.io/xy2dk · Cited by 35 · see citing works
When Does Monetary Policy Sway House Prices? A Meta-Analysis
Ehrenbergerová, D., Bajzík, J., & Havránek, T. (2022) · IMF Economic Review · doi:10.1057/s41308-022-00185-5 · Cited by 33 · see citing works
False Discovery in A/B Testing
Berman, R., & den Bulte, C. V. (2020) · Management Science · doi:10.1287/mnsc.2021.4207 · Cited by 34 · see citing works
Relative Risk Aversion: A Meta‐Analysis
Elminejad, A., Havránek, T., & Iršová, Z. (2025) · Journal of Economic Surveys · doi:10.1111/joes.12689 · Cited by 32 · see citing works
Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics: Comment
Kranz, S., & Pütz, P. (2022) · American Economic Review · doi:10.1257/aer.20210121 · Cited by 31 · see citing works
Is Investment in Preprimary Education Too Low? Lessons from (Quasi) Experimental Evidence across Countries
Holla, A., Bendini, M., Dinarte, L., & Trako, I. (2021) · World Bank, Washington, DC eBooks · doi:10.1596/1813-9450-9723 · Cited by 31 · see citing works
Student Employment and Education: A Meta-Analysis
Kroupová, K., Havránek, T., & Iršová, Z. (2024) · Economics of Education Review · doi:10.1016/j.econedurev.2024.102539 · Cited by 30 · see citing works
On the reliability of published findings using the regression discontinuity design in political science
Stommes, D., Aronow, P. M., & Sävje, F. (2023) · Research & Politics · doi:10.1177/20531680231166457 · Cited by 30 · see citing works
The p-Value Debate and Statistical (Mal)practice – Implications for the Agricultural and Food Economics Community
Heckelei, T., Hüttel, S., Rommel, J., & Rommel, J. (2023) · German Journal of Agricultural Economics · doi:10.30430/gjae.2023.0231 · Cited by 38 · see citing works
Rain, Rain, Go Away: 137 Potential Exclusion-Restriction Violations for Studies Using Weather as an Instrumental VariablePreprint
Mellon, J. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3715610 · Cited by 28 · see citing works
Experimental and behavioral economics to inform agri-environmental programs and policies
Palm‐Forster, L. H., & Messer, K. D. (2021) · Handbook of agricultural economics · doi:10.1016/bs.hesagr.2021.10.006 · Cited by 26 · see citing works
Mass Reproducibility and Replicability: A New HopePreprint
Brodeur, A., Mikola, D., & Cook, N. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4790780 · Cited by 26 · see citing works
Self-citation
The effectiveness of cash and cash plus interventions on livelihoods outcomes: Evidence from a systematic review and meta-analysisPreprint
Leight, J., Hirvonen, K., & Zafar, S. (2024) · doi:10.31219/osf.io/dnc2r · Cited by 25 · see citing works
Intertemporal substitution in labor supply: A meta-analysis
Elminejad, A., Havránek, T., Horváth, R., & Iršová, Z. (2023) · Review of Economic Dynamics · doi:10.1016/j.red.2023.10.001 · Cited by 23 · see citing works
Promoting Reproducibility and Replicability in Political Science
Brodeur, A., Esterling, K., Peters, J., Bueno, N., DESPOSATO, S. W., Dreber, A., Genovese, F., GREEN, D. P., Hepplewhite, M., de la Guardia, F. H., et al. (2024) · Research & Politics · doi:10.1177/20531680241233439 · Cited by 27 · see citing works
Self-citation
The Potency of Quality Incentives: Evidence from the Indonesian Dairy Value Chain
Treurniet, M. (2021) · American Journal of Agricultural Economics · doi:10.1111/ajae.12176 · Cited by 23 · see citing works
A Framework for Evaluating Reproducibility and Replicability in EconomicsPreprint
Dreber, A., & Johannesson, M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4458153 · Cited by 23 · see citing works
Ulysses' pact or U lysses' raft: Using pre‐analysis plans in experimental and nonexperimental research
Janzen, S., & Michler, J. D. (2021) · Applied Economic Perspectives and Policy · doi:10.1002/aepp.13133 · Cited by 22 · see citing works
P ‐hacking in meta‐analyses: A formalization and new meta‐analytic methods
Mathur, M. B. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1701 · Cited by 24 · see citing works
Replicability in Brain Imaging
Kelly, R. E., & Hoptman, M. J. (2022) · Brain Sciences · doi:10.3390/brainsci12030397 · Cited by 19 · see citing works
P-Hacking, Data Type and Data-Sharing Policy
Brodeur, A., Cook, N., & Neisser, C. (2022) · The Economic Journal · doi:10.1093/ej/uead104 · Cited by 21 · see citing works
Self-citation
Meta-Analysis of Empirical Estimates of Loss-AversionPreprint
Brown, A. L., Imai, T., Vieider, F. M., & Camerer, C. (2020) · Social Science Research Network · doi:10.31222/osf.io/hnefr · Cited by 161 · see citing works
Persistence studies: a new kind of economic history?
Cioni, M., Federico, G., & Vasta, M. (2022) · Review of Regional Research · doi:10.1007/s10037-022-00167-0 · Cited by 20 · see citing works
Detecting p-hackingPreprint
Elliott, G., Kudrin, N., & Wüthrich, K. (2019) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.1906.06711 · Cited by 20 · see citing works
What meta‐research has taught us about research and changes to research practices
Ioannidis, J. P. A. (2024) · Journal of Economic Surveys · doi:10.1111/joes.12666 · Cited by 18 · see citing works
Deindustrialisation and the post-socialist mortality crisis
Scheiring, G., Azarova, A., Irdam, D., Doniec, K., McKee, M., Stückler, D., & KING, L. P. (2022) · Cambridge Journal of Economics · doi:10.1093/cje/beac072 · Cited by 17 · see citing works
The case for mixed methods research: Embracing qualitative research to understand the (informal) economy
Cartwright, E. J., & Igudia, E. (2023) · Review of Development Economics · doi:10.1111/rode.13069 · Cited by 17 · see citing works
Is economics self‐correcting? Replications in the American Economic Review
Peters, J., Fiala, N., & Neubauer, F. (2024) · Economic Inquiry · doi:10.1111/ecin.13222 · Cited by 17 · see citing works
Towards improved and more transparent ethics in randomised controlled trials in development social science
Evans, D. K. (2023) · Journal of Development Effectiveness · doi:10.1080/19439342.2023.2196978 · Cited by 17 · see citing works
Estimating the Impact of Weather on Agriculture
Michler, J. D., Josephson, A., Kilic, T., & Murray, S. (2021) · World Bank, Washington, DC eBooks · doi:10.1596/1813-9450-9867 · Cited by 17 · see citing works
Do Pre-Registration and Pre-analysis Plans Reduce p-Hacking and Publication Bias?Preprint
Brodeur, A., Cook, N., Hartley, J., & Heyes, A. (2022) · doi:10.31222/osf.io/uxf39 · Cited by 17 · see citing works
Self-citation
How Does Data Access Shape Science? The Impact of Federal Statistical Research Data Centers on Economics Research
Nagaraj, A., & Tranchero, M. (2023) · National Bureau of Economic Research · doi:10.3386/w31372 · Cited by 17 · see citing works
Financial crime and punishment: A meta‐analysis
de Batz, L., & Evžen Kočenda, A. (2023) · Journal of Economic Surveys · doi:10.1111/joes.12580 · Cited by 17 · see citing works
Selective reporting of placebo tests in top economics journals
Dreber, A., Johannesson, M., & Yang, Y. (2024) · Economic Inquiry · doi:10.1111/ecin.13217 · Cited by 16 · see citing works
U.S. School Finance: Resources and Outcomes
Handel, D., & Hanushek, E. A. (2022) · National Bureau of Economic Research · doi:10.3386/w30769 · Cited by 17 · see citing works
Selective and (mis)leading economics journals: Meta‐research evidence
Askarov, Z., Doucouliagos, A., Doucouliagos, H., & Stanley, T. D. (2023) · Journal of Economic Surveys · doi:10.1111/joes.12598 · Cited by 15 · see citing works
Revisiting the Income Inequality-Crime Puzzle
Pazzona, M. (2024) · World Development · doi:10.1016/j.worlddev.2023.106520 · Cited by 15 · see citing works
Reconsidering macroeconomic policy prescriptions with meta-analysis
Gechert, S. (2022) · Industrial and Corporate Change · doi:10.1093/icc/dtac005 · Cited by 15 · see citing works
When Shooting for the Stars Becomes Aiming for Asterisks: P -Hacking in Family Business Research
Brinkerink, J. (2021) · Entrepreneurship Theory and Practice · doi:10.1177/10422587211050354 · Cited by 14 · see citing works
Tempest in a teacup: An analysis of p-Hacking in organizational research
Gupta, A., & Bosco, F. A. (2023) · PLoS ONE · doi:10.1371/journal.pone.0281938 · Cited by 15 · see citing works
Methods Used in Economic Research: An Empirical Study of Trends and Levels
Paldam, M. (2021) · Economics · doi:10.1515/econ-2021-0003 · Cited by 14 · see citing works
Questionable Research Practices and Open Science in Quantitative CriminologyPreprint
Chin, J. M., Pickett, J. T., Vazire, S., & Holcombe, A. O. (2021) · doi:10.31235/osf.io/bwm7s · Cited by 12 · see citing works
Income growth and inequality reduction through rural industrial integration: Evidence from small-scale farmers in China
Jian-hua, Z., Zijin, Z., & Yanxi, S. (2025) · Journal of Asian Economics · doi:10.1016/j.asieco.2025.102009 · Cited by 13 · see citing works
Reproducibility and robustness of economics and political science research
Brodeur, A., Mikola, D., Cook, N., Fiala, L., Brailey, T., Briggs, R. C., de Gendre, A., Dupraz, Y., Gabani, J., Gauriot, R., et al. (2026) · Nature · doi:10.1038/s41586-026-10251-x · Cited by 13 · see citing works
Self-citation
Causal Analysis of Policy Effects on Fertility
Hart, R. K., Bergsvik, J., Fauske, A., & Kim, W. (2024) · Springer eBooks · doi:10.1007/978-3-319-57365-6_451-1 · Cited by 13 · see citing works
The cost of operational complexity: A causal assessment of pre-fire mitigation and wildfire suppression
Young, J. D., Belval, E. J., Gannon, B. M., Wei, Y., O’Connor, C. D., Dunn, C. J., Pietruszka, B. M., Calkin, D. E., & Thompson, M. P. (2024) · Forest Policy and Economics · doi:10.1016/j.forpol.2024.103351 · Cited by 13 · see citing works
Does agricultural cooperative membership improve farm productive performance? A meta‐regression analysis
Ayouba, K., Duvivier, C., & Bussière, C. (2024) · Agricultural Economics · doi:10.1111/agec.12862 · Cited by 13 · see citing works
The evolving ethics of analysis, publication, and transparency in applied economics
Lybbert, T. J., & Buccola, S. T. (2020) · Applied Economic Perspectives and Policy · doi:10.1002/aepp.13131 · Cited by 13 · see citing works
Hedge Fund Performance: A Quantitative Survey
Yang, F., Havránek, T., Iršová, Z., & Novák, J. (2022) · Journal of Economic Surveys · doi:10.1111/joes.12574 · Cited by 13 · see citing works
The Returns to Education: A Meta‐Study
Clark, G., & Nielsen, C. A. A. (2026) · Kyklos · doi:10.1111/kykl.70041 · Cited by 12 · see citing works
Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics: Reply
Brodeur, A., Cook, N., & Heyes, A. (2022) · American Economic Review · doi:10.1257/aer.20220277 · Cited by 12 · see citing works
Self-citation
Ownership Concentration and Firm Value: New Evidence from Owner Stakes in IPOs
Larraín, B., Roosenboom, P., Sertsios, G., & Francisco Urzúa, I. (2023) · Management Science · doi:10.1287/mnsc.2021.01039 · Cited by 11 · see citing works
Registered reports and associated benefits for agricultural economics
Arpinon, T., & Lefebvre, M. (2024) · Q Open · doi:10.1093/qopen/qoae011 · Cited by 11 · see citing works
Conventional wisdom, meta‐analysis, and research revision in economics
Gechert, S., Mey, B., Opatrný, M., Havránek, T., Stanley, T. D., Bom, P. R. D., Doucouliagos, H., Heimberger, P., Iršová, Z., & Rachinger, H. J. (2024) · Journal of Economic Surveys · doi:10.1111/joes.12630 · Cited by 11 · see citing works

Authors: Brodeur, A., Cook, N., & Heyes, A.
Affiliations: University of Ottawa (CA)
ORCID: Brodeur, Abel
Cook, Nikolai
Heyes, Anthony
Venue: American Economic Review
Publisher: American Economic Association
Published: 2020-11-01
Volume: 110
Issue: 11
Pages: 3634-3660
ISSN: 0002-8282 (ISSN-L)
DOI: 10.1257/aer.20190687
OpenAlex: W3047016911
Cited by: 333
References: 41
Citations by year: 2026: 44 · 2025: 50 · 2024: 91 · 2023: 49 · 2022: 47 · 2021: 41 · 2020: 7 · 2019: 2 · 2018: 1 · 2014: 1
FWCI: 25.39
Type: journal article
Language: en
Open access: closed
Locations: American Economic Review (publishedVersion)
Open Research Exeter (University of Exeter) (publishedVersion)
RePEc: Research Papers in Economics (submittedVersion)
Figshare (submittedVersion)
University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
Topics: Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
Keywords: p-hacking, publication bias, credibility revolution, causal inference, instrumental variables, difference-in-differences, regression discontinuity design, randomized controlled trials, hypothesis testing, economics journals

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Related

Record

About these checks