The DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.
Reporting the Fragility of Regression Estimates
Abstract
E MPIRICAL results reported in economics journals are selected from a large set of estimated models. Journals, through their editorial policies, engage in some selection, which in turn stimulates extensive model searching and prescreening by prospective authors. Since this process is well known to professional readers, the reported results are widely regarded to overstate the precision of the estimates, and probably to distort them as well. As a consequence, statistical analyses are either greatly discounted or completely ignored. This unfortunate equilibrium in the market for information is a result of the current econometric technology, which generates inferences only if a precisely defined model were available, and which can be used to explore the sensitivity of inferences only to discrete changes in assumptions. The reporting of a complete sensitivity analysis is ruled out therefore first, because the econometric theory which takes models as given would be rendered explicitly inadequate if the sensitivity analysis were reported, and, second, because the econometric technology, if used to explore sensitivity issues, would generate vast numbers of estimated models that journals are rightfully reluctant to print. It is the purpose of this article to discuss an alternative econometric technology that could increase the value of our profession's limited data resources. The basic assumption underlying this technology is that no econometric model can be taken as given. Because there are many models which could serve as a basis for a data analysis, there are many conflicting inferences which could be drawn from a given data set. If this fact of life is acknowledged, it deflects econometric theory from the traditional task of identifying the unique inferences implied by a specific model to the task of determining the range of inferences generated by a range of models. We propose that researchers be given the task of identifying interesting families of alternative models and be expected to summarize the range of inferences which are implied by each of the families. When a range of inferences is small enough to be useful and when the corresponding family of models is broad enough to be believable, we may conclude that these data yield useful information. When the range of inferences is too wide to be useful, and when the corresponding family of models is so narrow that it cannot credibly be reduced, then we must conclude that inferences from these data are too fragile to be useful. This contrasts greatly with the reporting schemes currently used by individuals. As a profession, however, we do suspend judgment on econometric results until they hold up to inspections by other researchers using other models. The advocacy process we use to accumulate professional opinion is therefore aimed in the same direction as our proposals, but the path we recommend is much more direct and the outcome is much more clearly stated. A simple introduction to this alternative econometric technology is given in section I of this paper. In writing this section we have attempted to communicate the main ideas as concisely as possible. As a consequence, there is no reference to any sophisticated statistical theory and especially no mention of the Reverend Thomas Bayes. For a more complete statement as well as theological fanfare, consult Leamer (1978). The proper test of our proposals is whether they are useful in practice. We believe that researchers will find them to be efficient tools for discovering the information in data sets and for communicating findings to the consuming public. In an effort to make clear the value of these techniques we present two examples in section II. These methods are not without their own problems, the most serious of which is their concentration on the point estimation problem and their neglect of hypothesis testing or interval estimation. The basic approach to studying and reporting the fragility of estimates which we describe in this paper can be readily extended to studying and reporting the fragility of t-values, though computational difficulties do arise. Received for publication June 15, 1981. Revision accepted for publication August 2, 1982. * University of California, Los Angeles, and Harvard University, respectively. Research supported by NSF Grant SOC78-09477. Comments of the referees have helped to improve both the content and the exposition. Thomas Wolff is thanked for able research assistance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Estimation with Quadratic Loss
James, W., & Stein, C. (1992) · Springer Series in Statistics · doi:10.1007/978-1-4612-0919-5_30 · Cited by 2,252
Specification Searches: Ad Hoc Inference with Non-Experimental Data.
Richard, J.-F., & Leamer, E. E. (1981) · Economica · doi:10.2307/2553491 · Cited by 1,114
Self-citationRidge Regression: Biased Estimation for Nonorthogonal Problems
Hoerl, A. E., & Kennard, R. W. (2000) · Technometrics · doi:10.1080/00401706.2000.10485983 · Cited by 808
Identification and Estimation of Money Demand
Cooley, T. F., & LeRoy, S. F. (1981) · American Economic Review · Cited by 260
The Effects of Learning by Doing on Nuclear Plant Operating Reliability
Joskow, P. L., & Rozanski, G. A. (1979) · The Review of Economics and Statistics · doi:10.2307/1924583 · Cited by 107
Matrix Weighted Averages and Posterior Bounds
Chamberlain, G., & Leamer, E. E. (1976) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1976.tb01569.x · Cited by 105
Self-citation
Cited by
(1991) · RePEc: Research Papers in Economics · Cited by 6,037
Durlauf, S. N., Johnson, P. A., & Temple, J. R. W. (2005) · Handbook of Economic Growth · doi:10.1016/s1574-0684(05)01008-7 · Cited by 994
Hou, K., Xue, C., & Zhang, L. (2020) · The Review of Financial Studies · doi:10.1093/rfs/hhy131 · Cited by 989
META‐REGRESSION ANALYSIS: A QUANTITATIVE METHOD OF LITERATURE SURVEYS
Stanley, T. D., & Jarrell, S. B. (2005) · Journal of Economic Surveys · doi:10.1111/j.0950-0804.2005.00249.x · Cited by 805
META‐REGRESSION ANALYSIS: A QUANTITATIVE METHOD OF LITERATURE SURVEYS
Stanley, T. D., & Jarrell, S. B. (1989) · Journal of Economic Surveys · doi:10.1111/j.1467-6419.1989.tb00064.x · Cited by 805
Bauer, P. W., Berger, A. N., Ferrier, G. D., & Humphrey, D. B. (1998) · Journal of Economics and Business · doi:10.1016/s0148-6195(97)00072-6 · Cited by 803
The Power of Bias in Economics Research
Ioannidis, J. P. A., Stanley, T. D., & Doucouliagos, H. (2017) · The Economic Journal · doi:10.1111/ecoj.12461 · Cited by 634
What have we learned from a decade of empirical research on growth? Growth Empirics and Reality
Brock, W. A. (2001) · The World Bank Economic Review · doi:10.1093/wber/15.2.229 · Cited by 529
Social capital, innovation and growth: Evidence from Europe
Akçomak, İ. S., & ter Weel, B. (2009) · European Economic Review · doi:10.1016/j.euroecorev.2008.10.001 · Cited by 467
An Empirical Evaluation of Explanations for State Repression
HILL, D. W., & JONES, Z. M. (2014) · American Political Science Review · doi:10.1017/s0003055414000306 · Cited by 450
Transparency, Reproducibility, and the Credibility of Economics Research
Christensen, G., & Miguel, E. (2018) · Journal of Economic Literature · doi:10.1257/jel.20171350 · Cited by 446
Star Wars: The Empirics Strike Back
Brodeur, A., Lé, M., Sangnier, M., & Zylberberg, Y. (2016) · American Economic Journal: Applied Economics · doi:10.1257/app.20150044 · Cited by 430
THREE ECONOMETRIC METHODOLOGIES: A CRITICAL APPRAISAL1
Pagan, A. (1987) · Journal of Economic Surveys · doi:10.1111/j.1467-6419.1987.tb00022.x · Cited by 366
Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics
Brodeur, A., Cook, N., & Heyes, A. (2020) · American Economic Review · doi:10.1257/aer.20190687 · Cited by 333
What will take the con out of econometrics
(1986) · International Journal of Forecasting · doi:10.1016/0169-2070(86)90113-5 · Cited by 296
Are all Economic Hypotheses False?
De Long, J. B., & Lang, K. (1992) · Journal of Political Economy · doi:10.1086/261860 · Cited by 295
Swedberg, R. (2020) · The Production of Knowledge · doi:10.1017/9781108762519.002 · Cited by 293
On the determinants of local government performance: A two-stage nonparametric approach
Balaguer-Coll, M. T., Prior, D., & Tortosa-Ausina, E. (2007) · European Economic Review · doi:10.1016/j.euroecorev.2006.01.007 · Cited by 291
Shogren, J. F., Margolis, M., Koo, C., & List, J. A. (2001) · Journal of Economic Behavior & Organization · doi:10.1016/s0167-2681(01)00165-2 · Cited by 286
Patel, C. J., Burford, B., & Ioannidis, J. P. A. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.05.029 · Cited by 274
Are regional trading arrangements trade creating?
Ghosh, S., & Yamarik, S. (2004) · Journal of International Economics · doi:10.1016/s0022-1996(03)00058-8 · Cited by 260
Empirical Asset Pricing via Machine Learning: The Role of Research Design Choices
Lalwani, V., Meshram, V., & Jindal, V. (2026) · European Financial Management · doi:10.1111/eufm.70033
Metadata
- Authors
- Leamer, E., & Leonard, H.
- Venue
- The Review of Economics and Statistics
- Publisher
- JSTOR
- Published
- 1983-05
- Volume
- 65
- Issue
- 2
- Pages
- 306
- ISSN
- 0034-6535 (ISSN-L)
- OpenAlex
- W2013118805
- Cited by
- 322
- References
- 6
- FWCI
- 8.27
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.