Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say 'Usually Not'
Open accessAbstract
We attempt to replicate 67 papers published in 13 well-regarded economics journals using author-provided replication files that include both data and code. Some journals in our sample require data and code replication files, and other journals do not require such files. Aside from 6 papers that use confidential data, we obtain data and code replication files for 29 of 35 papers (83%) that are required to provide such files as a condition of publication, compared to 11 of 26 papers (42%) that are not required to provide data and code replication files. We successfully replicate the key qualitative result of 22 of 67 papers (33%) without contacting the authors. Excluding the 6 papers that use confidential data and the 2 papers that use software we do not possess, we replicate 29 of 59 papers (49%) with assistance from the authors. Because we are able to replicate less than half of the papers in our sample even with help from the authors, we assert that economics research is usually not replicable. We conclude with recommendations on improving replication of economics research.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Chang & Li reported that they could only reproduce the key findings of 22 of 59 (approx. 37%) published economics papers when they had access to study data.
“Similarly, Chang & Li [20] reported that they could only reproduce the key findings of 22 of 59 (approx. 37%) published economics papers (when they had access to study data).”
From Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal <i>Cognition</i> (2018) · cites this paper as [20]
The Conclusion and Recommendations reports 22 of 59 papers (37%), matching the cited figure. But the Introduction distinguishes that denominator from the 38 papers for which data and code files were available; among those, the rate was 22 of 38 (58%). The claim’s qualification that the 37% rate applied when they had access to study data is therefore not supported. (LLM, conf=0.88)
Experimental — accuracy not yet measuredA smattering of studies have sought to investigate analytic reproducibility in several scientific disciplines, and most have encountered substantial reproducibility problems.
“A smattering of studies have sought to investigate this issue in several scientific disciplines, and most have encountered substantial reproducibility problems [19–21].”
From Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal <i>Cognition</i> (2018) · cites this paper as [20]
This is comparable to the success rates observed in other assessments of analytic reproducibility in political science [19] and economics [20], but is substantially worse than a recent study of clinical RCTs [22].
“This is comparable to the success rates observed in other assessments of analytic reproducibility in political science [19] and economics [20], but is substantially worse than a recent study of clinical RCTs [22].”
From Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal <i>Cognition</i> (2018) · cites this paper as [20]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
1 retracted
When contact changes minds: An experiment on transmission of support for gay equality
LaCour, M. J., & GREEN, D. P. (2014) · Science · doi:10.1126/science.1256151 · Cited by 113 · see citing works
RetractedSims, C. A. (1980) · Econometrica · doi:10.2307/1912017 · Cited by 12,841 · see citing works
Economic Growth in a Cross Section of Countries
Barro, R. J. (1991) · The Quarterly Journal of Economics · doi:10.2307/2937943 · Cited by 9,449 · see citing works
Discretion versus policy rules in practice
Taylor, J. B. (1993) · Carnegie-Rochester Conference Series on Public Policy · doi:10.1016/0167-2231(93)90009-l · Cited by 9,065 · see citing works
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · see citing works
Postwar U.S. Business Cycles: An Empirical Investigation
Hodrick, R. J., & Prescott, E. C. (1997) · Journal of money credit and banking · doi:10.2307/2953682 · Cited by 7,496 · see citing works
Time to Build and Aggregate Fluctuations
Kydland, F., & Prescott, E. (1982) · Econometrica · doi:10.2307/1913386 · Cited by 6,093 · see citing works
Not All Oil Price Shocks Are Alike: Disentangling Demand and Supply Shocks in the Crude Oil Market
Kilian, L. (2009) · American Economic Review · doi:10.1257/aer.99.3.1053 · Cited by 4,728 · see citing works
Tests for parameter instability and structural change with unknown change point.
Andrews, D. W. K. (1993) · Econometrica · doi:10.2307/2951764 · Cited by 4,334 · see citing works
Credit spreads and business cycle fluctuations
Gilchrist, S., & Zakrajšek, E. (2012) · American Economic Review · doi:10.1257/aer.102.4.1692 · Cited by 2,213 · see citing works
Card, D., & Krueger, A. B. (2000) · American Economic Review · doi:10.1257/aer.90.5.1397 · Cited by 2,208 · see citing works
Hansen, P. R., Lunde, A., & Nason, J. M. (2011) · Econometrica · doi:10.3982/ecta5771 · Cited by 2,153 · see citing works
The Macroeconomic Effects of Tax Changes: Estimates Based on a New Measure of Fiscal Shocks
Romer, C., & Romer, D. (2010) · American Economic Review · doi:10.1257/aer.100.3.763 · Cited by 2,139 · see citing works
Measuring the Output Responses to Fiscal Policy
Auerbach, A. J., & Gorodnichenko, Y. (2012) · American Economic Journal Economic Policy · doi:10.1257/pol.4.2.1 · Cited by 1,918 · see citing works
Identifying Government Spending Shocks: It's all in the Timing*
Ramey, V. (2011) · The Quarterly Journal of Economics · doi:10.1093/qje/qjq008 · Cited by 1,708 · see citing works
The Aggregate Demand for Treasury Debt
Krishnamurthy, A., & Vissing‐Jørgensen, A. (2012) · Journal of Political Economy · doi:10.1086/666526 · Cited by 1,625 · see citing works
The Granular Origins of Aggregate Fluctuations
Gabaix, X. (2011) · Econometrica · doi:10.3982/ecta8769 · Cited by 1,388 · see citing works
Does high public debt consistently stifle economic growth? A critique of Reinhart and Rogoff
Herndon, T., Ash, M., & Pollin, R. (2013) · Cambridge Journal of Economics · doi:10.1093/cje/bet075 · Cited by 1,215 · see citing works
Macroeconomic Effects of Financial Shocks
Jermann, U. J., & Quadrini, V. (2012) · American Economic Review · doi:10.1257/aer.102.1.238 · Cited by 1,209 · see citing works
Does Competition Among Public Schools Benefit Students and Taxpayers?
Hoxby, C. (2000) · American Economic Review · doi:10.1257/aer.90.5.1209 · Cited by 1,105 · see citing works
The Dynamic Effects of Personal and Corporate Income Tax Changes in the United States
Mertens, K., & Ravn, M. O. (2013) · American Economic Review · doi:10.1257/aer.103.4.1212 · Cited by 1,061 · see citing works
Using Electoral Cycles in Police Hiring to Estimate the Effect of Police on Crime: Comment
McCrary, J. (2002) · American Economic Review · doi:10.1257/00028280260344768 · Cited by 867 · see citing works
Macroeconomic Effects From Government Purchases and Taxes *
Barro, R. J., & Redlick, C. (2011) · The Quarterly Journal of Economics · doi:10.1093/qje/qjq002 · Cited by 840 · see citing works
Durably reducing transphobia: A field experiment on door-to-door canvassing
Broockman, D., & Kalla, J. (2016) · Science · doi:10.1126/science.aad9713 · Cited by 798 · see citing works
Using Electoral Cycles in Police Hiring to Estimate the Effect of Police on CrimePreprint
Levitt, S. D. (1995) · SSRN Electronic Journal · Cited by 580 · see citing works
What's News in Business Cycles
Schmitt‐Grohé, S., & Uribe, M. (2012) · Econometrica · doi:10.3982/ecta8050 · Cited by 558 · see citing works
Time-Varying Effects of Oil Supply Shocks on the US Economy
Baumeister, C. J. S., & Peersman, G. (2013) · American Economic Journal Macroeconomics · doi:10.1257/mac.5.4.1 · Cited by 530 · see citing works
Replication in Empirical Economics: The Journal of Money, Credit and Banking Project
Dewald, W. G., Thursby, J. G., & Anderson, R. G. (2016) · American Economic Review · Cited by 491 · see citing works
The Macroeconomic Effects of Large‐scale Asset Purchase Programmes
Chen, H., Cúrdia, V., & Ferrero, A. (2012) · The Economic Journal · doi:10.1111/j.1468-0297.2012.02549.x · Cited by 469 · see citing works
Heutel, G. (2011) · Review of Economic Dynamics · doi:10.1016/j.red.2011.05.002 · Cited by 467 · see citing works
Reinhart, C. M., & Rogoff, K. S. (2010) · American Economic Review · doi:10.1257/aer.100.2.573 · Cited by 438 · see citing works
What's News in Business Cycles
Schmitt‐Grohé, S., & Uribe, M. (2008) · National Bureau of Economic Research · doi:10.3386/w14215 · Cited by 415 · see citing works
Follow the Leader: Theory and Evidence on Political Participation
Shachar, R., & Nalebuff, B. (1999) · American Economic Review · doi:10.1257/aer.89.3.525 · Cited by 396 · see citing works
Using Electoral Cycles in Police Hiring to Estimate the Effects of Police on Crime: Reply
Levitt, S. D. (2002) · American Economic Review · doi:10.1257/00028280260344777 · Cited by 392 · see citing works
Neumark, D., & Wascher, W. L. (2000) · American Economic Review · doi:10.1257/aer.90.5.1362 · Cited by 356 · see citing works
Viewpoint: Replication in economics
Hamermesh, D. S. (2007) · Canadian Journal of Economics/Revue canadienne d économique · doi:10.1111/j.1365-2966.2007.00428.x · Cited by 281 · see citing works
Identifying Government Spending Shocks: It's All in the Timing
Ramey, V. (2009) · National Bureau of Economic Research · doi:10.3386/w15464 · Cited by 281 · see citing works
Replication research's disturbing trend
Evanschitzky, H., Baumgarth, C., Hubbard, R., & Armstrong, J. S. (2007) · Journal of Business Research · doi:10.1016/j.jbusres.2006.12.003 · Cited by 271 · see citing works
Morley, J., & Piger, J. M. (2010) · The Review of Economics and Statistics · doi:10.1162/rest_a_00169 · Cited by 270 · see citing works
Fiscal Stimulus with Spending Reversals
Corsetti, G., Meier, A., & Müller, G. J. (2011) · The Review of Economics and Statistics · doi:10.1162/rest_a_00233 · Cited by 246 · see citing works
Nakov, A. A., & Pescatori, A. (2009) · The Economic Journal · doi:10.1111/j.1468-0297.2009.02302.x · Cited by 228 · see citing works
Verifying the Solution from a Nonlinear Solver: A Case Study
McCullough, B. D., & Vinod, H. D. (2003) · American Economic Review · doi:10.1257/000282803322157133 · Cited by 220 · see citing works
Using Stock Returns to Identify Government Spending Shocks
Fisher, J. D. M., & Peters, R. H. (2010) · The Economic Journal · doi:10.1111/j.1468-0297.2010.02355.x · Cited by 218 · see citing works
Understanding the aggregate effects of anticipated and unanticipated tax policy shocks
Mertens, K., & Ravn, M. O. (2010) · Review of Economic Dynamics · doi:10.1016/j.red.2010.07.004 · Cited by 208 · see citing works
A Real-Time Data Set for Macroeconomists: Does the Data Vintage Matter?
Croushore, D., & Stark, T. (2003) · The Review of Economics and Statistics · doi:10.1162/003465303322369759 · Cited by 203 · see citing works
Does Competition Among Public Schools Benefit Students and Taxpayers?
Hoxby, C. (1994) · National Bureau of Economic Research · doi:10.3386/w4979 · Cited by 190 · see citing works
On the Welfare Cost of Inflation and the Recent Behavior of Money Demand
Ireland, P. N. (2009) · American Economic Review · doi:10.1257/aer.99.3.1040 · Cited by 186 · see citing works
Averaging forecasts from VARs with uncertain instabilities
Clark, T. E., & McCracken, M. W. (2009) · Journal of Applied Econometrics · doi:10.1002/jae.1127 · Cited by 181 · see citing works
Measuring Tax Multipliers: The Narrative Method in Fiscal VARs
Favero, C. A., & Giavazzi, F. (2012) · American Economic Journal Economic Policy · doi:10.1257/pol.4.2.69 · Cited by 175 · see citing works
McCullough, B. D., McGeary, K. A., & Harrison, T. D. (2006) · Journal of Money, Credit, and Banking · doi:10.1353/mcb.2006.0061 · Cited by 173 · see citing works
Does Competition Among Public Schools Benefit Students and Taxpayers? Comment
Rothstein, J. (2007) · American Economic Review · doi:10.1257/aer.97.5.2026 · Cited by 169 · see citing works
Read All about It!! What Happens Following a Technology Shock?
Alexopoulos, M. (2011) · American Economic Review · doi:10.1257/aer.101.4.1144 · Cited by 169 · see citing works
Relative Goods' Prices, Pure Inflation, and The Phillips Correlation
Reis, R. A. M. R., & Watson, M. W. (2010) · American Economic Journal Macroeconomics · doi:10.1257/mac.2.3.128 · Cited by 167 · see citing works
Using Electoral Cycles in Police Hiring to Estimate the Effect of Policeon Crime
Levitt, S. D. (1995) · National Bureau of Economic Research · doi:10.3386/w4991 · Cited by 159 · see citing works
Do economics journal archives promote replicable research?
McCullough, B. D., McGeary, K. A., & Harrison, T. D. (2008) · Canadian Journal of Economics/Revue canadienne d économique · doi:10.1111/j.1540-5982.2008.00509.x · Cited by 140 · see citing works
Monetary Policy Rules and Macroeconomic Stability: Some New Evidence
Mavroeidis, S. (2010) · American Economic Review · doi:10.1257/aer.100.1.491 · Cited by 119 · see citing works
Oil price shocks and the optimality of monetary policy
Kormilitsina, A. (2010) · Review of Economic Dynamics · doi:10.1016/j.red.2010.11.001 · Cited by 113 · see citing works
The Granular Origins of Aggregate FluctuationsPreprint
Gabaix, X. (2008) · SSRN Electronic Journal · doi:10.2139/ssrn.1111765 · Cited by 112 · see citing works
Interest Rate Risk and Other Determinants of Post-WWII US Government Debt/GDP Dynamics
Hall, G., & Sargent, T. J. (2011) · American Economic Journal Macroeconomics · doi:10.1257/mac.3.3.192 · Cited by 98 · see citing works
The Myth of Financial Innovation and the Great Moderation
Haan, W. J. D., & Sterk, V. (2010) · The Economic Journal · doi:10.1111/j.1468-0297.2010.02400.x · Cited by 89 · see citing works
Identifying the Sources of Instabilities in Macroeconomic Fluctuations
Inoue, A., & Rossi, B. (2010) · The Review of Economics and Statistics · doi:10.1162/rest_a_00130 · Cited by 85 · see citing works
Does Competition Among Public Schools Benefit Students and Taxpayers? Reply
Hoxby, C. (2007) · American Economic Review · doi:10.1257/aer.97.5.2038 · Cited by 81 · see citing works
Business cycles with a common trend in neutral and investment-specific productivity
Schmitt‐Grohé, S., & Uribe, M. (2010) · Review of Economic Dynamics · doi:10.1016/j.red.2010.07.001 · Cited by 77 · see citing works
Do Expectations Matter? The Great Moderation Revisited
Canova, F., & Gambetti, L. (2010) · American Economic Journal Macroeconomics · doi:10.1257/mac.2.3.183 · Cited by 76 · see citing works
The Model Confidence SetPreprint
Hansen, P. R., Lunde, A., & Nason, J. M. (2004) · SSRN Electronic Journal · doi:10.2139/ssrn.522382 · Cited by 66 · see citing works
Time-Varying Effects of Oil Supply Shocks on the US EconomyPreprint
Baumeister, C. J. S., & Peersman, G. (2008) · SSRN Electronic Journal · doi:10.2139/ssrn.1093702 · Cited by 62 · see citing works
A Real-Time Data Set for Macroeconomists: Does the Data Vintage Matter?Preprint
Croushore, D., & Stark, T. (2000) · SSRN Electronic Journal · doi:10.2139/ssrn.203012 · Cited by 60 · see citing works
A Century of Inflation Forecasts
D’Agostino, A., & Surico, P. (2011) · The Review of Economics and Statistics · doi:10.1162/rest_a_00235 · Cited by 52 · see citing works
The Macroeconomic Effects of Large-Scale Asset Purchase ProgramsPreprint
Chen, H., Cúrdia, V., & Ferrero, A. (2011) · SSRN Electronic Journal · doi:10.2139/ssrn.1976319 · Cited by 52 · see citing works
Report on the American Economic Review Data Availability Compliance Project
Glandon, P. (2010) · Digital Kenyon (Kenyon College) · Cited by 51 · see citing works
Does Competition Among Public Schools Benefit Students and Taxpayers? A Comment on Hoxby (2000)
Rothstein, J. M. (2005) · National Bureau of Economic Research · doi:10.3386/w11215 · Cited by 51 · see citing works
On the Welfare Cost of Inflation and the Recent Behavior of Money Demand
Ireland, P. (2008) · National Bureau of Economic Research · doi:10.3386/w14098 · Cited by 45 · see citing works
Corrigendum: Measuring the Output Responses to Fiscal Policy
Auerbach, A. J., & Gorodnichenko, Y. (2013) · American Economic Journal Economic Policy · doi:10.1257/pol.5.3.320 · Cited by 41 · see citing works
Chang, A. C., & Li, P. (2018) · Economic Inquiry · doi:10.1111/ecin.12567 · Cited by 31 · see citing works
Self-citationDo Economics Journal Archives Promote Replicable Research?Preprint
McCullough, B. D., McGeary, K. A., & Harrison, T. D. (2011) · SSRN Electronic Journal · doi:10.2139/ssrn.931231 · Cited by 28 · see citing works
Volumes of evidence: examining technical change in the last century through a new lens
Alexopoulos, M., & Cohen, J. S. (2011) · Canadian Journal of Economics/Revue canadienne d économique · doi:10.1111/j.1540-5982.2011.01639.x · Cited by 27 · see citing works
Replication Research in Marketing Revisited: A Note on a Disturbing TrendPreprint
Evanschitzky, H., Baumgarth, C., Hubbard, R., & Armstrong, J. S. (2008) · SSRN Electronic Journal · doi:10.2139/ssrn.1153144 · Cited by 22 · see citing works
Factor analysis of permanent and transitory dynamics of the US economy and the stock market
Şenyüz, Z. (2010) · Journal of Applied Econometrics · doi:10.1002/jae.1193 · Cited by 20 · see citing works
Business Cycles and Gender Diversification: An Analysis of Establishment-Level Gender Dissimilarity
Bansak, C., Graham, M., & Zebedee, A. A. (2012) · American Economic Review · doi:10.1257/aer.102.3.561 · Cited by 19 · see citing works
Relative Goods' Prices, Pure Inflation, and the Phillips Correlation
Reis, R. A. M. R., & Watson, M. W. (2007) · National Bureau of Economic Research · doi:10.3386/w13615 · Cited by 18 · see citing works
Business Cycles With A Common Trend in Neutral and Investment-Specific Productivity
Schmitt‐Grohé, S., & Uribe, M. (2010) · National Bureau of Economic Research · doi:10.3386/w16071 · Cited by 15 · see citing works
From the Peaks to the Valleys: Cross-State Evidence on Income Volatility over the Business Cycle
Carey, C. M., & Shore, S. H. (2012) · The Review of Economics and Statistics · doi:10.1162/rest_a_00299 · Cited by 12 · see citing works
Do Expectations Matter? The Great Moderation RevisitedPreprint
Canova, F., & Luca Gambetti, Z. (2008) · SSRN Electronic Journal · doi:10.2139/ssrn.1143308 · Cited by 9 · see citing works
Verifying the Solution from a Nonlinear Solver: A Case Study: Reply
McCullough, B. D., & Vinod, H. D. (2004) · American Economic Review · doi:10.1257/000282804322970887 · Cited by 7 · see citing works
Šimundić, A.-M., Topić, E., & Čvoriščec, D. (2008) · Biochemia Medica · doi:10.11613/bm.2008.013 · Cited by 6 · see citing works
Report of the Editor: American Economic Review
Bernanke, B. (2004) · American Economic Review · doi:10.1257/0002828041302352 · Cited by 5 · see citing works
Flagged first, then by citations
6 of the 92 references couldn’t be matched to a known paper.
Research works printed but not linked: 3
- Bernanke, Ben S., "Editorial Statement," American Economic Review 94:1 (2004), 404.
- Card, David, and Alan B. Krueger, "Minimum Wages and Employment: A Case Study of the Fast-Food Industry in New Jersey and Pennsylvania," American Economic Review 84:4 (1994), 772-793.
- Thompson Reuters' Web of Science Journal Citation Reports: Social Science Edition. https://isiknowledge.com/ (2014).
Cited by
Sorted by citations
THE SHORT-TERM IMPACT OF UNCONDITIONAL CASH TRANSFERS TO THE POOR: EXPERIMENTAL EVIDENCE FROM KENYA
Haushofer, J., & Shapiro, J. P. (2016) · Quarterly Journal of Economics · doi:10.1093/qje/qjw025 · Cited by 947 · see citing works
Contextual sensitivity in scientific reproducibility
Van Bavel, J. J., Mende-Siedlecki, P., Brady, W. J., & Reinero, D. A. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1521897113 · Cited by 398 · see citing works
Swedberg, R. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.002 · Cited by 408 · see citing works
An empirical analysis of journal policy effectiveness for computational reproducibility
Stodden, V., Seiler, J., & Ma, Z. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708290115 · Cited by 369 · see citing works
Hardwicke, T. E., Mathur, M. B., MacDonald, K., Nilsonne, G., Banks, G. C., Kidwell, M. C., Hofelich Mohr, A., Clayton, E., Yoon, E. J., Henry Tessler, M., et al. (2018) · Royal Society Open Science · doi:10.1098/rsos.180448 · Cited by 336 · see citing works
Publication bias and the canonization of false facts
Nissen, S. B., Magidson, T., Gross, K., & Bergstrom, C. T. (2016) · eLife · doi:10.7554/elife.21451 · Cited by 266 · see citing works
“Excellence R Us”: university research and the fetishisation of excellence
Moore, S. A., Neylon, C., Eve, M. P., O'Donnell, D. P., & Pattinson, D. (2017) · Palgrave Communications · doi:10.1057/palcomms.2016.105 · Cited by 71 · see citing works
Anomalies and False Rejections
Chordia, T., Goyal, A., & Alessio Saretto, B. (2020) · Review of Financial Studies · doi:10.1093/rfs/hhaa018 · Cited by 200 · see citing works
Hardwicke, T. E., Wallach, J. D., Kidwell, M. C., Bendixen, T., Crüwell, S., & Ioannidis, J. P. A. (2020) · Royal Society Open Science · doi:10.1098/rsos.190806 · Cited by 217 · see citing works
Bergh, D. D., Sharp, B. M., Aguinis, H., & Li, M. (2017) · Strategic Organization · doi:10.1177/1476127017701076 · Cited by 183 · see citing works
Estimating the Reproducibility of Experimental Philosophy
Cova, F., Strickland, B., Abatista, A., Allard, A., Andow, J., Attie, M., Beebe, J. R., Berniûnas, R., Boudesseul, J., Colombo, M., et al. (2018) · Review of Philosophy and Psychology · doi:10.1007/s13164-018-0400-9 · Cited by 184 · see citing works
Replicating AnomaliesPreprint
Hou, K., Chen, X., & Zhang, L. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2961979 · Cited by 176 · see citing works
Ahn, R., Woodbridge, A., Abraham, A., Saba, S., Korenstein, D., Madden, E., Boscardin, W., & Keyhani, S. (2017) · British medical journal · doi:10.1136/bmj.i6770 · Cited by 156 · see citing works
A large-scale study on research code quality and execution
Trišović, A., Lau, M. K., Pasquier, T., & Crosas, M. (2022) · Scientific Data · doi:10.1038/s41597-022-01143-6 · Cited by 136 · see citing works
(2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519 · Cited by 135 · see citing works
Schweinsberg, M., Feldman, M., Staub, N., van den Akker, O. R., van Aert, R. C. M., van Assen, M. A. L. M., Liu, Y., Althoff, T., Heer, J., Kale, A. M., et al. (2021) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2021.02.003 · Cited by 129 · see citing works
Hardwicke, T. E., Bohn, M., MacDonald, K., Hembacher, E., Nuijten, M. B., Peloquin, B. N., deMayo, B. E., Long, B., Yoon, E. J., & Frank, M. C. (2021) · Royal Society Open Science · doi:10.1098/rsos.201494 · Cited by 108 · see citing works
Metaresearch for Evaluating Reproducibility in Ecology and Evolution
Fidler, F. M., Chee, Y. E., Wintle, B. C., Burgman, M. A., Mccarthy, M. A., & Gordon, A. (2016) · BioScience · doi:10.1093/biosci/biw159 · Cited by 98 · see citing works
Z-curve 2.0: Estimating Replication Rates and Discovery Rates
Bartos̆, F., & Schimmack, U. (2022) · Meta-Psychology · doi:10.15626/mp.2021.2720 · Cited by 92 · see citing works
When open data is a Trojan Horse: The weaponization of transparency in science and governance
Levy, K. E. C., & Johns, D. M. (2016) · Big Data & Society · doi:10.1177/2053951715621568 · Cited by 76 · see citing works
A Call for Openness in Research Reporting: How to Turn Covert Practices Into Helpful Tools
Schwab, A., & Starbuck, W. (2017) · Academy of Management Learning and Education · doi:10.5465/amle.2016.0039 · Cited by 59 · see citing works
Hendriks, F., Kienhues, D., & Bromme, R. (2020) · Public Understanding of Science · doi:10.1177/0963662520902383 · Cited by 73 · see citing works
How to make replication the norm
Gertler, P., Galiani, S., & Romero, M. (2018) · Nature · doi:10.1038/d41586-018-02108-9 · Cited by 81 · see citing works
Furniture for Quantitative Scientists
Tyson, S. B., & Brignone, E. (2017) · The R Journal · doi:10.32614/rj-2017-037 · Cited by 70 · see citing works
A Preanalysis Plan to Replicate Sixty Economics Research Papers That Worked Half of the Time
Chang, A. C., & Li, P. (2017) · American Economic Review · doi:10.1257/aer.p20171034 · Cited by 70 · see citing works
Self-citationReplication and Reproducibility in Cross-Cultural Psychology
Milfont, T. L., & Klein, R. A. (2018) · Journal of Cross-Cultural Psychology · doi:10.1177/0022022117744892 · Cited by 67 · see citing works
Making ERP research more transparent: Guidelines for preregistration
Paul, M., Govaart, G., & Schettino, A. (2021) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2021.02.016 · Cited by 66 · see citing works
Systematizing Confidence in Open Research and Evidence (SCORE)Preprint
Alipourfard, N., Arendt, B., Benjamin, D. M., Benkler, N., Bishop, M. M., Burstein, M. H., Bush, M., Caverlee, J., Chen, Y., Clark, C., et al. (2021) · doi:10.31235/osf.io/46mnb · Cited by 69 · see citing works
Artner, R., Verliefde, T., Steegen, S., Gomes, S. L. B. T., Traets, F., Tuerlinckx, F., & Vanpaemel, W. (2020) · Psychological Methods · doi:10.1037/met0000365 · Cited by 63 · see citing works
Hensel, P. G. (2021) · European Management Journal · doi:10.1016/j.emj.2021.01.002 · Cited by 64 · see citing works
Laurinavichyute, A., Yadav, H., & Vasishth, S. (2022) · Journal of Memory and Language · doi:10.1016/j.jml.2022.104332 · Cited by 60 · see citing works
Examining the generalizability of research findings from archival data
Delios, A., Clemente, E. G., Wu, T., Tan, H., Wang, Y., Gordon, M., Viganola, D., Chen, Z., Dreber, A., Johannesson, M., et al. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2120377119 · Cited by 56 · see citing works
Page, M. J., Moher, D., Fidler, F. M., Higgins, J. P. T., Brennan, S., Haddaway, N., Hamilton, D. G., Kanukula, R., Karunananthan, S., Maxwell, L. J., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01670-0 · Cited by 55 · see citing works
Hardwicke, T. E., Mathur, M. B., MacDonald, K. E., Nilsonne, G., Banks, G. C., Kidwell, M. C., Mohr, A. H., Clayton, E., Yoon, E. J., Tessler, M. H., et al. (2018) · doi:10.31222/osf.io/39cfb · Cited by 60 · see citing works
Context may be King, but generalizability is the Emperor!
Cheng, Z., Dimoka, A., & Pavlou, P. A. (2016) · Journal of Information Technology · doi:10.1057/s41265-016-0005-7 · Cited by 50 · see citing works
Transparency, Reproducibility, and the Credibility of Economics ResearchPreprint
Christensen, G., & Miguel, E. A. (2017) · doi:10.31222/osf.io/9a3rw · Cited by 57 · see citing works
Open Science: What, Why, and HowPreprint
Spellman, B., Gilbert, E. A., & Corker, K. S. (2017) · doi:10.31234/osf.io/ak6jr · Cited by 50 · see citing works
Krafczyk, M. S., Shi, A., Bhaskar, A., Marinov, D., & Stodden, V. (2021) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2020.0069 · Cited by 46 · see citing works
Discounting model selection with area‐based measures: A case for numerical integration
Gilroy, S. P., & Hantula, D. A. (2018) · Journal of the Experimental Analysis of Behavior · doi:10.1002/jeab.318 · Cited by 44 · see citing works
A Proposal to Organize and Promote Replications
Coffman, L. C., Niederle, M., & Wilson, A. J. (2017) · American Economic Review · doi:10.1257/aer.p20171122 · Cited by 42 · see citing works
Forecasting crude oil prices with DSGE models
Rubaszek, M. (2020) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2020.07.004 · Cited by 40 · see citing works
The Practicalities of Running Randomized Evaluations
Glennerster, R. (2017) · Handbook of economic field experiments · doi:10.1016/bs.hefe.2016.10.002 · Cited by 35 · see citing works
Improving norms in research culture to incentivize transparency and rigor
Mellor, D. T. (2021) · Educational Psychologist · doi:10.1080/00461520.2021.1902329 · Cited by 36 · see citing works
Transparent, Open, and Reproducible Prevention Science
Grant, S., Wendt, K. E., Leadbeater, B. J. R., Supplee, L. H., Mayo‐Wilson, E., Gardner, F. E. M., & Bradshaw, C. P. (2022) · Prevention Science · doi:10.1007/s11121-022-01336-w · Cited by 36 · see citing works
Estimating the Reproducibility of Experimental PhilosophyPreprint
Cova, F., Strickland, B., Abatista, A., Allard, A., Andow, J., Attie, M., Beebe, J. R., Berniûnas, R., Boudesseul, J., Colombo, M., et al. (2018) · doi:10.31234/osf.io/sxdah · Cited by 37 · see citing works
Design and Analysis of Replication Studies
Bonett, D. G. (2020) · Organizational Research Methods · doi:10.1177/1094428120911088 · Cited by 34 · see citing works
Identification of and correction for publication biasPreprint
Kasy, M., & Andrews, I. (2018) · doi:10.31222/osf.io/49yst · Cited by 42 · see citing works
Chang, A. C., & Li, P. (2018) · Economic Inquiry · doi:10.1111/ecin.12567 · Cited by 31 · see citing works
Self-citationTransparency, Reproducibility, and the Credibility of Economics Research
Christensen, G., & Miguel, E. A. (2016) · National Bureau of Economic Research · doi:10.3386/w22989 · Cited by 30 · see citing works
Pandemic Shocks and Household Spending*
Finck, D., & Tillmann, P. (2021) · Oxford Bulletin of Economics and Statistics · doi:10.1111/obes.12471 · Cited by 27 · see citing works
Big Data and the Little Big Bang: An Epistemological (R)evolution
Balazka, D., & Rodighiero, D. (2020) · Frontiers in Big Data · doi:10.3389/fdata.2020.00031 · Cited by 29 · see citing works
Hensel, P. G. (2018) · European Management Journal · doi:10.1016/j.emj.2018.03.004 · Cited by 29 · see citing works
Pre-registration and Results-Free Review in Observational and Qualitative Research
Jacobs, A. M. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.009 · Cited by 30 · see citing works
Radford, J. P., Pilny, A., Reichelmann, A. V., Keegan, B., Welles, B. F., Hoye, J., Ognyanova, K., Meleis, W. M., & Lazer, D. M. J. (2016) · Social Psychology Quarterly · doi:10.1177/0190272516675866 · Cited by 27 · see citing works
Stockhammer, E., Dammerer, Q., & Kapur, S. (2020) · Cambridge Journal of Economics · doi:10.1093/cje/beaa054 · Cited by 27 · see citing works
The Safety of Autonomous Vehicles: Lessons from Philosophy of Science
Hicks, D. (2018) · IEEE Technology and Society Magazine · doi:10.1109/mts.2018.2795123 · Cited by 27 · see citing works
The Research Reproducibility Crisis and Economics of Science
Maniadis, Z., & Tufano, F. (2017) · The Economic Journal · doi:10.1111/ecoj.12526 · Cited by 27 · see citing works
Muradchanian, J., Hoekstra, R., Kiers, H. A. L., & van Ravenzwaaij, D. (2021) · Royal Society Open Science · doi:10.1098/rsos.201697 · Cited by 33 · see citing works
Spellman, B. A., Gilbert, E. A., & Corker, K. S. (2018) · Wiley eBooks · doi:10.1002/9781119170174.epcn519 · Cited by 84 · see citing works
Cobey, K. D., Fehlmann, C. A., Franco, M. C., Ayala, A. P., Sikora, L., Rice, D. B., Xu, C., Ioannidis, J. P. A., Lalu, M. M., Ménard, A., et al. (2023) · eLife · doi:10.7554/elife.78518 · Cited by 24 · see citing works
Vicens, J., Perelló, J., & Duch, J. (2018) · PLoS ONE · doi:10.1371/journal.pone.0207219 · Cited by 25 · see citing works
Marshall, E. C., & Underwood, A. (2019) · The Journal of Economic Education · doi:10.1080/00220485.2018.1551100 · Cited by 24 · see citing works
Open Science is for Aging Research, Too
Isaacowitz, D. M., & Lind, M. (2019) · Innovation in Aging · doi:10.1093/geroni/igz028 · Cited by 25 · see citing works
On the Transportability of Laboratory Results
Bader, F., Baumeister, B., Berger, R. D., & Keuschnigg, M. (2019) · Sociological Methods & Research · doi:10.1177/0049124119826151 · Cited by 23 · see citing works
Magness, P. (2018) · Social Science Quarterly · doi:10.1111/ssqu.12672 · Cited by 21 · see citing works
Naaman, K., Grant, S., Kianersi, S., Supplee, L. H., Henschel, B., & Mayo-Wilson, E. (2023) · Royal Society Open Science · doi:10.1098/rsos.221093 · Cited by 25 · see citing works
Is the Political Slant of Psychology Research Related to Scientific Replicability?
Reinero, D. A., Wills, J. A., Brady, W. J., Mende‐Siedlecki, P., Crawford, J. T., & Bavel, J. J. V. (2020) · Perspectives on Psychological Science · doi:10.1177/1745691620924463 · Cited by 19 · see citing works
Machery, É. (2019) · Review of Philosophy and Psychology · doi:10.1007/s13164-019-00440-1 · Cited by 18 · see citing works
Markowitz, A. J. (2017) · AERA Open · doi:10.1177/2332858417712717 · Cited by 20 · see citing works
The Trouble with Instruments: The Need for Pre-Treatment Balance in Shock-IV DesignsPreprint
Atanasov, V. A., & Black, B. S. (2015) · SSRN Electronic Journal · doi:10.2139/ssrn.2417689 · Cited by 17 · see citing works
Western Skeptic vs Indian Realist. Cross-Cultural Differences in Zebra Case Intuitions
Sękowski, K., Ziółkowski, A., & Tarnowski, M. (2021) · Review of Philosophy and Psychology · doi:10.1007/s13164-021-00586-x · Cited by 16 · see citing works
Seven Reasons Why: A User’s Guide to Transparency and Reproducibility
Filho, D. B. F., Lins, R. G. P., Domingos, A. R. M., Janz, N., & Silva, L. (2019) · Brazilian Political Science Review · doi:10.1590/1981-3821201900020001 · Cited by 16 · see citing works
Machery, É. (2021) · European Journal for Philosophy of Science · doi:10.1007/s13194-021-00354-9 · Cited by 15 · see citing works
Kedron, P., Holler, J., & Bardin, S. (2023) · Annals of the American Association of Geographers · doi:10.1080/24694452.2023.2276115 · Cited by 13 · see citing works
Making Research Data Accessible
Kapiszewski, D., & Karcher, S. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.008 · Cited by 15 · see citing works
Evans, N., Marušić, A., Foeger, N., Löfstrôm, E., van Hoof, M., Vrijhoef-Welten, S., Inguaggiato, G., Dierickx, K., Bouter, L. M., & Widdershoven, G. A. M. (2021) · Research Ideas and Outcomes · doi:10.3897/rio.7.e68258 · Cited by 14 · see citing works
Vilhuber, L. (2021) · AEA Papers and Proceedings · doi:10.1257/pandp.111.808 · Cited by 14 · see citing works
Dawning of a new age? Economics journals’ data policies on the test bench
Vlaeminck, S. (2021) · LIBER Quarterly The Journal of the Association of European Research Libraries · doi:10.53377/lq.10940 · Cited by 13 · see citing works
Drongstrup, D. (2019) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24151 · Cited by 13 · see citing works
The politics of the globalization backlash: Sources and implicationsPreprint
Frieden, J. (2018) · RePEc: Research Papers in Economics · Cited by 12 · see citing works
Geukes, K., Schönbrodt, F. D., Utesch, T., Geukes, S., & Back, M. D. (2016) · Zeitschrift für Sportpsychologie · doi:10.1026/1612-5010/a000167 · Cited by 12 · see citing works
Reproducibility: A Primer on Semantics and Implications for Research
Pellizzari, E. D., Lohr, K. N., Blatecky, A. R., & Creel, D. V. (2017) · doi:10.3768/rtipress.2017.bk.0020.1708 · Cited by 12 · see citing works
Residential location and education in the United States
Hanushek, E. A., & Yılmaz, K. (2022) · Edward Elgar Publishing eBooks · doi:10.4337/9781789908497.00012 · Cited by 12 · see citing works
Hensel, P. G. (2021) · Accountability in Research · doi:10.1080/08989621.2021.1981870 · Cited by 13 · see citing works
A Systematic Literature Review of Undergraduate Data Science Education Research
Dogucu, M., Demirci, S., Bendekgey, H., Ricci, F. Z., & Medina, C. M. (2025) · Journal of Statistics and Data Science Education · doi:10.1080/26939169.2025.2486656 · Cited by 11 · see citing works
Forschungstransparenz als hohes wissenschaftliches Gut stärken
Schönbrodt, F. D., Maier, M., Heene, M., & Bühner, M. (2018) · Psychologische Rundschau · doi:10.1026/0033-3042/a000386 · Cited by 11 · see citing works
Eliciting group judgements about replicability: a technical implementation of the IDEA Protocol
Pearson, R., Fraser, H. S., Bush, M., Mody, F., Widjaja, I., Head, A., Wilkinson, D. P., Sinnott, R., Wintle, B. C., Burgman, M. A., et al. (2021) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2021.055 · Cited by 8 · see citing works
Wang, G. G., Doty, D. H., & Yang, S. (2021) · Advances in Developing Human Resources · doi:10.1177/15234223211054457 · Cited by 11 · see citing works
Using Meta-Analysis in the Social Sciences to Improve Environmental Policy
Maki, A., Cohen, M. A., & Vandenbergh, M. P. (2017) · World sustainability series · doi:10.1007/978-3-319-67122-2_2 · Cited by 11 · see citing works
To P or not to P? The Usefulness of P‐values in Quantitative Political Science Research
Troeger, V. E. (2019) · Swiss Political Science Review · doi:10.1111/spsr.12377 · Cited by 11 · see citing works
Improving Transparency in Observational Social Science Research: A Pre-Analysis Plan ApproachPreprint
Burlig, F. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2731920 · Cited by 11 · see citing works
Tavlas, G. S., Gibson, H. D., Hall, S. G., & Petroulas, P. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.4197872 · Cited by 12 · see citing works
To Grasp Cognition in Action, Combine Behavioral Experiments with Protocol Analysis
Reypens, C., & Levine, S. S. (2017) · Emerald Group Publishing Limited eBooks · doi:10.1108/s2397-52102017006 · Cited by 10 · see citing works
Can You Repeat That? Exploring the Definition of a Successful Model Replication in Health Economics
McManus, E., Turner, D., & Sach, T. (2019) · PharmacoEconomics · doi:10.1007/s40273-019-00836-y · Cited by 10 · see citing works
Design-Based Approaches to Causal Replication Studies
Wong, V. C., Anglin, K., & Steiner, P. M. (2021) · Prevention Science · doi:10.1007/s11121-021-01234-7 · Cited by 11 · see citing works
Replication in labor economics
Reed, W. R. (2017) · IZA World of Labor · doi:10.15185/izawol.413 · Cited by 8 · see citing works
Adapting the case method in an economics capstone research course
Depro, B. M., & Rouse, K. (2022) · International Review of Economics Education · doi:10.1016/j.iree.2022.100249 · Cited by 8 · see citing works
Open Science als wissenschaftspolitische Problemlösungsformel?
Franzen, M. (2016) · Springer eBooks · doi:10.1007/978-3-658-05455-7_23 · Cited by 8 · see citing works
A replication recipe: list your ingredients before you start cooking
Chang, A. C. (2018) · Economics · doi:10.5018/economics-ejournal.ja.2018-39 · Cited by 8 · see citing works
Self-citationReplication Redux: The Reproducibility Crisis and the Case of Deworming
Ozier, O. W. (2020) · The World Bank Research Observer · doi:10.1093/wbro/lkaa005 · Cited by 8 · see citing works
- Authors
- Chang, A. C., & Li, P.
- Affiliations
- Federal Reserve (US)
- Office of the Comptroller of the Currency (US)
- Venue
- Finance and Economics Discussion Series
- Publisher
- Elsevier BV
- Published
- 2015
- Volume
- 2015.0
- Issue
- 83
- Pages
- 1-26
- ISSN
- 1936-2854 (ISSN-L)
- OpenAlex
- W2195775529
- Cited by
- 217
- References
- 92
- Citations by year
- 2026: 2 · 2025: 6 · 2024: 6 · 2023: 8 · 2022: 13 · 2021: 32 · 2020: 44 · 2019: 16 · 2018: 27 · 2017: 37 · 2016: 21 · 2015: 5
- FWCI
- 80.38
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- public-domain
- Locations
- Finance and Economics Discussion Series (publishedVersion, public-domain)
- SSRN Electronic Journal (acceptedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Keywords
- research reproducibility, open data policy, economics journals
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
