Model Uncertainty and Robustness: A Computational Framework for Multimodel Analysis
The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Abstract
Model uncertainty is pervasive in social science. A key question is how robust empirical results are to sensible changes in model specification. We present a new approach and applied statistical software for computational multimodel analysis. Our approach proceeds in two steps: First, we estimate the modeling distribution of estimates across all combinations of possible controls as well as specified functional form issues, variable definitions, standard error calculations, and estimation commands. This allows analysts to present their core, preferred estimate in the context of a distribution of plausible estimates. Second, we develop a model influence analysis showing how each model ingredient affects the coefficient of interest. This shows which model assumptions, if any, are critical to obtaining an empirical result. We demonstrate the architecture and interpretation of multimodel analysis using data on the union wage premium, gender dynamics in mortgage lending, and tax flight migration among U.S. states. These illustrate how initial results can be strongly robust to alternative model specifications or remarkably dependent on a knife-edge specification.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Bayesian model selection in social research
Raftery, A. E. (1995) · Sociological Methodology · doi:10.2307/271063 · Cited by 6,823
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Silva, J. M. C. S., & Tenreyro, S. (2006) · The Review of Economics and Statistics · doi:10.1162/rest.88.4.641 · Cited by 6,408
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“After building familiarity with the core approach, we proceed to the influence analysis of step 2: The decomposition of the modeling distribution, showing what elements of the model have greatest influence on the conclusions.”
MethodHoeting, J. A., Madigan, D., Raftery, A. E., & Volinsky, C. T. (1999) · Statistical Science · doi:10.1214/ss/1009212519 · Cited by 4,118
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979
Angrist, J. D., & Krueger, A. B. (2001) · Journal of Economic Perspectives · doi:10.1257/jep.15.4.69 · Cited by 2,355
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · cited 1×
“The company recently reported that in its efforts at replication, two-thirds of the published findings it studied could not be supported (Prinz et al. 2011).”
BackgroundNonparametric estimates of standard error: The jackknife, the bootstrap and other methods
EFRON, B. (1981) · Biometrika · doi:10.1093/biomet/68.3.589 · Cited by 1,399
Endogenous Selection Bias: The Problem of Conditioning on a Collider Variable
Elwert, F., & Winship, C. (2014) · Annual Review of Sociology · doi:10.1146/annurev-soc-071913-043455 · Cited by 1,168 · cited 1×
“Controlling for race (with the dummy variable ‘‘black’’) also increases the effect size of gender by 1.91, a full 83 percent higher than the mean estimate.”
BackgroundInfluentialPanel Models in Sociological Research: Theory into Practice
Halaby, C. N. (2004) · Annual Review of Sociology · doi:10.1146/annurev.soc.30.012703.110629 · Cited by 946
The Phantom Menace: Omitted Variable Bias in Econometric Research
Clarke, K. A. (2005) · Conflict Management and Peace Science · doi:10.1080/07388940500339183 · Cited by 484 · cited 1×
“Two control variables clearly stand out as most influential: marital status and race.”
BackgroundTaxation and International Migration of Superstars: Evidence from the European Football Market
Kleven, H. J., Landais, C., & Saez, E. (2013) · American Economic Review · doi:10.1257/aer.103.5.1892 · Cited by 381
Vague Theory and Model Uncertainty in Macrosociology
Western, B. (1996) · Sociological Methodology · doi:10.2307/271022 · Cited by 133
The Impact of Genetic Research on our Understanding of Normal Cognitive Ageing: 1995 to 2009
Payton, A. (2009) · Neuropsychology Review · doi:10.1007/s11065-009-9116-z · Cited by 95 · cited 1×
“However, industry laboratories that try to replicate published biomedical research often find the results are not robust and are unable to reproduce the findings.”
BackgroundResearcher Incentives and Empirical Methods
GLAESER, E. L. (2008) · The Foundations of Positive and Normative Economics · doi:10.1093/acprof:oso/9780195328318.003.0013 · Cited by 15
Cited by
Entrepreneurship and well-being: Past, present, and future
Wiklund, J., Nikolaev, B., Shir, N., Foo, M.-D., & Bradley, S. (2019) · Journal of Business Venturing · doi:10.1016/j.jbusvent.2019.01.002 · Cited by 675
Simonsohn, U., Simmons, J. P., & Nelson, L. D. (2020) · Nature Human Behaviour · doi:10.1038/s41562-020-0912-z · Cited by 651
What Is Your Estimand? Defining the Target Quantity Connects Statistical Evidence to Theory
Lundberg, I., Johnson, R., & Stewart, B. M. (2021) · American Sociological Review · doi:10.1177/00031224211004187 · Cited by 364
Machine Learning for Sociology
Molina, M., & Garip, F. (2019) · Annual Review of Sociology · doi:10.1146/annurev-soc-073117-041106 · Cited by 295
Swedberg, R. (2020) · The Production of Knowledge · doi:10.1017/9781108762519.002 · Cited by 293
Metadata
- Authors
- Young, C., & Holsteen, K.
- Venue
- Sociological Methods & Research
- Publisher
- SAGE Publications
- Published
- 2016-07-09
- Volume
- 46
- Issue
- 1
- Pages
- 3-40
- ISSN
- 0049-1241 (ISSN-L)
- 1552-8294
- OpenAlex
- W2269963894
- Cited by
- 239
- References
- 18
- FWCI
- 7.01
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.