The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Model Uncertainty and Robustness: A Computational Framework for Multimodel Analysis
Abstract
Model uncertainty is pervasive in social science. A key question is how robust empirical results are to sensible changes in model specification. We present a new approach and applied statistical software for computational multimodel analysis. Our approach proceeds in two steps: First, we estimate the modeling distribution of estimates across all combinations of possible controls as well as specified functional form issues, variable definitions, standard error calculations, and estimation commands. This allows analysts to present their core, preferred estimate in the context of a distribution of plausible estimates. Second, we develop a model influence analysis showing how each model ingredient affects the coefficient of interest. This shows which model assumptions, if any, are critical to obtaining an empirical result. We demonstrate the architecture and interpretation of multimodel analysis using data on the union wage premium, gender dynamics in mortgage lending, and tax flight migration among U.S. states. These illustrate how initial results can be strongly robust to alternative model specifications or remarkably dependent on a knife-edge specification.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“After building familiarity with the core approach, we proceed to the influence analysis of step 2: The decomposition of the modeling distribution, showing what elements of the model have greatest influence on the conclusions.”
MethodHoeting, J. A., Madigan, D., Raftery, A. E., & Volinsky, C. T. (1999) · Statistical Science · doi:10.1214/ss/1009212519 · Cited by 4,118
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979
Angrist, J. D., & Krueger, A. B. (2001) · Journal of Economic Perspectives · doi:10.1257/jep.15.4.69 · Cited by 2,355
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · cited 1×
“The company recently reported that in its efforts at replication, two-thirds of the published findings it studied could not be supported (Prinz et al. 2011).”
BackgroundNonparametric estimates of standard error: The jackknife, the bootstrap and other methods
EFRON, B. (1981) · Biometrika · doi:10.1093/biomet/68.3.589 · Cited by 1,399
Endogenous Selection Bias: The Problem of Conditioning on a Collider Variable
Elwert, F., & Winship, C. (2014) · Annual Review of Sociology · doi:10.1146/annurev-soc-071913-043455 · Cited by 1,168 · cited 1×
“Controlling for race (with the dummy variable ‘‘black’’) also increases the effect size of gender by 1.91, a full 83 percent higher than the mean estimate.”
BackgroundInfluentialPanel Models in Sociological Research: Theory into Practice
Halaby, C. N. (2004) · Annual Review of Sociology · doi:10.1146/annurev.soc.30.012703.110629 · Cited by 946
Cited by
Entrepreneurship and well-being: Past, present, and future
Wiklund, J., Nikolaev, B., Shir, N., Foo, M.-D., & Bradley, S. (2019) · Journal of Business Venturing · doi:10.1016/j.jbusvent.2019.01.002 · Cited by 675
Simonsohn, U., Simmons, J. P., & Nelson, L. D. (2020) · Nature Human Behaviour · doi:10.1038/s41562-020-0912-z · Cited by 651
What Is Your Estimand? Defining the Target Quantity Connects Statistical Evidence to Theory
Lundberg, I., Johnson, R., & Stewart, B. M. (2021) · American Sociological Review · doi:10.1177/00031224211004187 · Cited by 364
Machine Learning for Sociology
Molina, M., & Garip, F. (2019) · Annual Review of Sociology · doi:10.1146/annurev-soc-073117-041106 · Cited by 295
Swedberg, R. (2020) · The Production of Knowledge · doi:10.1017/9781108762519.002 · Cited by 293
Metadata
- Authors
- Young, C., & Holsteen, K.
- Venue
- Sociological Methods & Research
- Publisher
- SAGE Publications
- Published
- 2016-07-09
- Volume
- 46
- Issue
- 1
- Pages
- 3-40
- ISSN
- 0049-1241 (ISSN-L)
- 1552-8294
- OpenAlex
- W2269963894
- Cited by
- 239
- References
- 15
- FWCI
- 7.01
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Model Uncertainty and Robustness: A Computational Framework for Multimodel Analysis. https://app.cochat.ai/papers/10.1177/0049124115610347
- The paper
- Young, C., & Holsteen, K. (2017). Model Uncertainty and Robustness: A Computational Framework for Multimodel Analysis. Sociological Methods & Research. https://doi.org/10.1177/0049124115610347
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.