WHAT'S TO KNOW ABOUT THE CREDIBILITY OF EMPIRICAL ECONOMICS?
The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
The scientific credibility of economics is itself a scientific question that can be addressed with both theoretical speculations and empirical data. In this review, we examine the major parameters that are expected to affect the credibility of empirical economics: sample size, magnitude of pursued effects, number and pre‐selection of tested relationships, flexibility and lack of standardization in designs, definitions, outcomes and analyses, financial and other interests and prejudices, and the multiplicity and fragmentation of efforts. We summarize and discuss the empirical evidence on the lack of a robust reproducibility culture in economics and business research, the prevalence of potential publication and other selective reporting biases, and other failures and biases in the market of scientific information. Overall, the credibility of the economics literature is likely to be modest or even low.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Self-citationMeasuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
Fanelli, D. (2009) · PLoS ONE · doi:10.1371/journal.pone.0005738 · Cited by 1,911 · cited 2×
“Most errors are inadvertent or due to suboptimal research practices and deficient quality control, but falsification may also be occasionally involved (Fanelli 2009).”
BackgroundAngrist, J. D., & Pischke, J.-S. (2010) · Journal of Economic Perspectives · doi:10.1257/jep.24.2.3 · Cited by 1,566
Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546 · cited 1×
“Small-study effects suggestive of possible selective reporting biases remain common in randomized trials in other disciplines (Dwan et al., 2008).4 Similarly, we do not know how vulnerable these studies are to the winner’s curse, i.e. inflated effect sizes when effects are discovered by relatively…”
BackgroundSelf-citationWhy Most Discovered True Associations Are Inflated
Ioannidis, J. P. A. (2008) · Epidemiology · doi:10.1097/ede.0b013e31818131e7 · Cited by 1,495 · cited 1×
“…possible selective reporting biases remain common in randomized trials in other disciplines (Dwan et al., 2008).4 Similarly, we do not know how vulnerable these studies are to the winner’s curse, i.e. inflated effect sizes when effects are discovered by relatively small studies (Ioannidis, 2008).”
BackgroundSelf-citationNegative results are disappearing from most disciplines and countries
Fanelli, D. (2012) · Scientometrics · doi:10.1007/s11192-011-0494-7 · Cited by 1,248
An exploratory test for an excess of significant findings
Ioannidis, J. P., & Trikalinos, T. A. (2007) · Clinical Trials · doi:10.1177/1740774507079441 · Cited by 751 · cited 1×
“Selective reporting biases cumulatively create literature where there are just too many nominally significant research findings, a situation that can be probed with an excess significance test (Ioannidis and Trikalinos, 2007).”
BackgroundSelf-citation“Positive” results increase down the Hierarchy of the Sciences
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010068 · Cited by 734 · cited 3×
“After controlling for differences between pure and applied disciplines and between papers testing one or several hypotheses, the odds of reporting in favour of the tested hypothesis were five times higher among papers in Economics and Business compared to Space Science (Fanelli, 2010).”
BackgroundResultInfluentialWhy Science Is Not Necessarily Self-Correcting
Ioannidis, J. P. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612464056 · Cited by 563 · cited 4×
“The credibility of research and the market for evidence have been critically questioned (see Ioannidis, 2005, 2012a; Young et al., 2008).”
BackgroundInfluentialSelf-citation
Cited by
What does research reproducibility mean?
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016) · Science Translational Medicine · doi:10.1126/scitranslmed.aaf5027 · Cited by 1,293
Self-citationHow to Make More Published Research True
Ioannidis, J. P. A. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001747 · Cited by 760
“Adoption of more appropriate statistical methods [38], standardized definitions and analyses and more stringent thresholds for claiming discoveries or ‘‘successes’’ [39] may decrease false-positive rates in fields that have to-date been too lenient (like epidemiology [40], psychology [41,42], or economics [43]).”
MethodSelf-citationThe Power of Bias in Economics Research
Ioannidis, J. P. A., Stanley, T. D., & Doucouliagos, H. (2017) · The Economic Journal · doi:10.1111/ecoj.12461 · Cited by 634
Self-citationMachine Translation: Mining Text for Social Theory
Evans, J. A., & Aceves, P. (2016) · Annual Review of Sociology · doi:10.1146/annurev-soc-081715-074206 · Cited by 333
Using prediction markets to estimate the reproducibility of scientific research
Dreber, A., Pfeiffer, T., Almenberg, J., Isaksson, S., Wilson, B., Chen, Y., Nosek, B. A., & Johannesson, M. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1516179112 · Cited by 331
Machine Learning for Sociology
Molina, M., & Garip, F. (2019) · Annual Review of Sociology · doi:10.1146/annurev-soc-073117-041106 · Cited by 295
Adopt or innovate: Understanding technological responses to cap-and-trade
Calel, R. (2020) · American Economic Journal: Economic Policy · doi:10.1257/pol.20180135 · Cited by 166
False (and Missed) Discoveries in Financial Economics
HARVEY, C. R., & LIU, Y. (2020) · The Journal of Finance · doi:10.1111/jofi.12951 · Cited by 118
Significance testing in empirical finance: a critical review and assessment
Kim, J. H., & Ji, P. I. (2015) · Journal of Empirical Finance · doi:10.1016/j.jempfin.2015.08.006 · Cited by 86
de Winter, J. C., & Dodou, D. (2015) · PeerJ · doi:10.7717/peerj.733 · Cited by 78
Gordon, M., Viganola, D., Dreber, A., Johannesson, M., & Pfeiffer, T. (2021) · PLOS ONE · doi:10.1371/journal.pone.0248780 · Cited by 44
“The process for selecting the finding to be replicated from a publication was as follows: (1) select the most central finding in the paper (among the between-subject treatment comparisons) based on to what extent the findings were emphasized in the published versions; (2) if there was more than one equally central fin…”
ResultThe Military Origins of Soviet Industrialization
Kontorovich, V. (2015) · Comparative Economic Studies · doi:10.1057/ces.2015.8 · Cited by 13
Metadata
- Authors
- Ioannidis, J., & Doucouliagos, C.
- Venue
- Journal of Economic Surveys
- Publisher
- Wiley
- Published
- 2013-04-12
- Volume
- 27
- Issue
- 5
- Pages
- 997-1004
- ISSN
- 0950-0804 (ISSN-L)
- 1467-6419
- OpenAlex
- W2154826683
- Cited by
- 118
- References
- 12
- FWCI
- 8.25
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.