Making sense of sensitivity: extending omitted variable bias
Open accessAbstract
Summary We extend the omitted variable bias framework with a suite of tools for sensitivity analysis in regression models that does not require assumptions on the functional form of the treatment assignment mechanism nor on the distribution of the unobserved confounders, naturally handles multiple confounders, possibly acting non-linearly, exploits expert knowledge to bound sensitivity parameters and can be easily computed by using only standard regression results. In particular, we introduce two novel sensitivity measures suited for routine reporting. The robustness value describes the minimum strength of association that unobserved confounding would need to have, both with the treatment and with the outcome, to change the research conclusions. The partial R2 of the treatment with the outcome shows how strongly confounders explaining all the residual outcome variation would have to be associated with the treatment to eliminate the estimated effect. Next, we offer graphical tools for elaborating on problematic confounders, examining the sensitivity of point estimates and t-values, as well as ‘extreme scenarios’. Finally, we describe problems with a common ‘benchmarking’ practice and introduce a novel procedure to bound the strength of confounders formally on the basis of a comparison with observed covariates. We apply these methods to a running example that estimates the effect of exposure to violence on attitudes toward peace.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Sensitivity to omitted variable bias is tested using the Cinelli and Hazlett (2020) methodology.
“sensitivity to omitted variable bias (OVB) is tested using the Cinelli and Hazlett (2020) methodology.”
From Extension and management pathways for enhanced farm sustainability: evidence from Irish dairy farms (2023) · cites this paper as Cinelli and Hazlett (2020)
Cinelli and Hazlett (2020) extends the OVB framework to analyse the sensitivity to unobserved confounders.
“Cinelli and Hazlett (2020) extends the OVB framework to analyse the sensitivity to unobserved confounders.”
From Extension and management pathways for enhanced farm sustainability: evidence from Irish dairy farms (2023) · cites this paper as Cinelli and Hazlett (2020)
Econometric tools such as the Cinelli and Hazlett (2020) sensitivity analysis can help applied researchers expand knowledge on extension or technology effect when detailed information about farmers' extension participation, technology adoption behaviours and sustainability performance is available over time.
“econometric tools, such as the Cinelli and Hazlett (2020) sensitivity analysis, can help applied researchers to expand the level of knowledge on extension or technology effect, especially when detailed information about farmers' extension participation, technology adoption behaviours and sustainability performance is available over time.”
From Extension and management pathways for enhanced farm sustainability: evidence from Irish dairy farms (2023) · cites this paper as Cinelli and Hazlett, 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241 · see citing works
Causality: Models, Reasoning, and Inference
Pearl, J. (2009) · doi:10.1017/cbo9780511803161 · Cited by 11,653 · see citing works
Sensitivity analysis in observational research: Introducing the E-value
VanderWeele, T. J., & Ding, P. (2017) · Annals of Internal Medicine · doi:10.7326/m16-2607 · Cited by 6,350 · see citing works
Unobservable Selection and Coefficient Stability: Theory and Evidence
Oster, E. (2019) · Journal of Business & Economic Statistics · doi:10.1080/07350015.2016.1227711 · Cited by 5,367 · see citing works
Mostly Harmless Econometrics: An Empiricist's Companion
Angrist, J. D., & Pischke, J.-S. (2008) · doi:10.2307/j.ctvcm4j72 · Cited by 5,051 · see citing works
Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction
Imbens, G. W., & Rubin, D. B. (2015) · doi:10.1017/cbo9781139025751 · Cited by 4,210 · see citing works
Rosenbaum, P. R. (2002) · Springer series in statistics · doi:10.1007/978-1-4757-3692-2 · Cited by 2,414 · see citing works
Let's Take the Con out of Econometrics
Leamer, E. E. (2026) · Journal of Business Cycle Research · doi:10.1007/s41549-026-00137-2 · Cited by 2,201 · see citing works
Identification, Inference and Sensitivity Analysis for Causal Mediation Effects
Imai, K., Keele, L., & Yamamoto, T. (2010) · Statistical Science · doi:10.1214/10-sts321 · Cited by 1,628 · see citing works
Rosenbaum, P., & Rubin, D. (1983) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1983.tb01242.x · Cited by 1,264 · see citing works
Design of Observational Studies
Rosenbaum, P. R. (2010) · Springer Series in Statistics · doi:10.1007/978-1-4419-1213-8 · Cited by 1,150 · see citing works
Impact of a Confounding Variable on a Regression Coefficient
Frank, K. A. (2000) · Sociological Methods & Research · doi:10.1177/0049124100029002001 · Cited by 1,042 · see citing works
Natural Experiments in the Social Sciences
Dunning, T. (2012) · Cambridge University Press eBooks · doi:10.1017/cbo9781139084444 · Cited by 963 · see citing works
Identification, Inference, and Sensitivity Analysis for Causal Mediation Effects
Imai, K., & Keele, L. J. (2008) · Cited by 887 · see citing works
The Consquences of Adjustment for a Concomitant Variable That Has Been Affected by the Treatment
Rosenbaum, P. R. (1984) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2981697 · Cited by 849 · see citing works
Smoking and Lung Cancer: Recent Evidence and a Discussion of Some Questions
Cornfield, J., Haenszel, W. M., Hammond, E., Lilienfeld, A. M., Shimkin, M. B., & Wynder, E. L. (1959) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/22.1.173 · Cited by 842 · see citing works
Partial Time Regressions as Compared with Individual Trends
Frisch, R., & Waugh, F. V. (1933) · Econometrica · doi:10.2307/1907330 · Cited by 629 · see citing works
Sensitivity to Exogeneity Assumptions in Program Evaluation
Imbens, G. W. (2003) · American Economic Review · doi:10.1257/000282803321946921 · Cited by 529 · see citing works
VanderWeele, T. J., & Arah, O. A. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181f74493 · Cited by 455 · see citing works
Seasonal Adjustment of Economic Time Series and Multiple Regression Analysis
Lovell, M. C. (1963) · Journal of the American Statistical Association · doi:10.1080/01621459.1963.10480682 · Cited by 448 · see citing works
Encyclopedia of Statistics in Behavioral Science
Guha, M. (2006) · Reference Reviews · doi:10.1108/09504120610664123 · Cited by 419 · see citing works
Frank, K. A., Maroulis, S. J., Duong, M. Q., & Kelcey, B. M. (2013) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373713493129 · Cited by 406 · see citing works
Association, Causation, And Marginal Structural Models
Robins, J. M. (1999) · Synthese · doi:10.1023/a:1005285815569 · Cited by 404 · see citing works
An Evaluation of Instrumental Variable Strategies for Estimating the Effects of Catholic Schooling
Altonji, J. G., Elder, T. E., & Taber, C. R. (2005) · The Journal of Human Resources · doi:10.3368/jhr.xl.4.791 · Cited by 380 · see citing works
Darfur: A New History of a Long War
Flint, J., & de Waal, A. (2008) · Zed Books Ltd · doi:10.5040/9781350219489 · Cited by 324 · see citing works
An Evaluation of Instrumental Variable Strategies for Estimating the Effects of Catholic Schools
Altonji, J. G., Elder, T. E., & Taber, C. R. (2002) · National Bureau of Economic Research · doi:10.3386/w9358 · Cited by 265 · see citing works
Unobservable Selection and Coefficient Stability: Theory and Validation
Oster, E. F. (2013) · National Bureau of Economic Research · doi:10.3386/w19054 · Cited by 252 · see citing works
Sensemakr: Sensitivity Analysis Tools for OLS in R and Stata
Cinelli, C., Ferwerda, J., & Hazlett, C. J. (2024) · Observational Studies · doi:10.1353/obs.2024.a946583 · Cited by 225 · see citing works
Self-citationBrumback, B., Hernán, M. Á., Haneuse, S., & Robins, J. M. (2004) · Statistics in Medicine · doi:10.1002/sim.1657 · Cited by 212 · see citing works
Invited commentary: understanding bias amplification.
Pearl, J. (2011) · American Journal of Epidemiology · doi:10.1093/aje/kwr352 · Cited by 197 · see citing works
Observation and Experiment: An Introduction to Causal Inference
Rosenbaum, P. (2017) · doi:10.4159/9780674982697 · Cited by 170 · see citing works
Sensemakr: Sensitivity Analysis Tools for OLS in R and StataPreprint
Cinelli, C., Ferwerda, J., & Hazlett, C. J. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3588978 · Cited by 153 · see citing works
Self-citationTo Adjust or Not to Adjust? Sensitivity Analysis of M-Bias and Butterfly-Bias
Ding, P., & Miratrix, L. (2014) · Journal of Causal Inference · doi:10.1515/jci-2013-0021 · Cited by 138 · see citing works
Concepts of Relative Importance in Recent Scientific Literature
Kruskal, W., & Majors, R. (1989) · The American Statistician · doi:10.1080/00031305.1989.10475596 · Cited by 115 · see citing works
The sensitivity of linear regression coefficients’ confidence limits to the omission of a confounder
Hosman, C., Hansen, B. B., & Holland, P. W. (2010) · The Annals of Applied Statistics · doi:10.1214/09-aoas315 · Cited by 115 · see citing works
A flexible, interpretable framework for assessing sensitivity to unmeasured confounding
Dorie, V., Harada, M., Carnegie, N. B., & Hill, J. (2016) · Statistics in Medicine · doi:10.1002/sim.6973 · Cited by 104 · see citing works
A Simple Proof of the FWL Theorem
Lovell, M. C. (2008) · The Journal of Economic Education · doi:10.3200/jece.39.1.88-91 · Cited by 97 · see citing works
Assessing Sensitivity to Unmeasured Confounding Using a Simulated Potential Confounder
Carnegie, N. B., Harada, M., & Hill, J. (2016) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2015.1078862 · Cited by 94 · see citing works
The Mechanics of Omitted Variable Bias: Bias Amplification and Cancellation of Offsetting Biases
Steiner, P. M., & Kim, Y. (2016) · Journal of Causal Inference · doi:10.1515/jci-2016-0009 · Cited by 88 · see citing works
Undergraduate Econometrics Instruction: Through Our Classes, Darkly
Angrist, J. D., & Pischke, J. (2017) · The Journal of Economic Perspectives · doi:10.1257/jep.31.2.125 · Cited by 86 · see citing works
Flexible Sensitivity Analysis for Observational Studies Without Observable Implications
Franks, A. M., D’Amour, A., & Feller, A. (2019) · Journal of the American Statistical Association · doi:10.1080/01621459.2019.1604369 · Cited by 81 · see citing works
Does NBPTS Certification Affect the Number of Colleagues a Teacher Helps With Instructional Matters?
Frank, K. A., Sykes, G., Anagnostopoulos, D., Cannata, M. A., Chard, L., Krause, A. E., & McCrory, R. (2008) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373707313781 · Cited by 71 · see citing works
A Selection Bias Approach to Sensitivity Analysis for Causal Effects
Blackwell, M. (2013) · Political Analysis · doi:10.1093/pan/mpt006 · Cited by 63 · see citing works
Bias Amplification and Bias Unmasking
Middleton, J. A., Scott, M. A., Diakow, R., & Hill, J. (2016) · Political Analysis · doi:10.1093/pan/mpw015 · Cited by 61 · see citing works
Weighting-Based Sensitivity Analysis in Causal Mediation Studies
Hong, G., Qin, X., & Yang, F. (2018) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998617749561 · Cited by 52 · see citing works
10. Indices of Robustness for Sample Representation
Frank, K. A., & Min, K.-S. (2007) · Sociological Methodology · doi:10.1111/j.1467-9531.2007.00186.x · Cited by 39 · see citing works
S -values: Conventional context-minimal measures of the sturdiness of regression coefficients
Leamer, E. E. (2016) · Journal of Econometrics · doi:10.1016/j.jeconom.2015.10.013 · Cited by 39 · see citing works
Sensitivity Analysis of Linear Structural Causal Models
Cinelli, C., Kumor, D. R., Chen, B., Pearl, J., & Bareinboim, E. (2019) · International Conference on Machine Learning · Cited by 30 · see citing works
Self-citationUndergraduate Econometrics Instruction: Through Our Classes, Darkly
Angrist, J. D., & Pischke, J. (2017) · National Bureau of Economic Research · doi:10.3386/w23144 · Cited by 10 · see citing works
Flagged first, then by citations
2 of the 51 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- Imbens, G. W. and Rubin, D. B. (2015) Causal Inference in Statistics, Social, and Biomedical Sciences. New York: Cambridge University Press.
- Pearl, J. (2009) Causality. New York: Cambridge University Press.
Cited by
Sorted by citations
A Crash Course in Good and Bad Controls
Cinelli, C., Forney, A., & Pearl, J. (2020) · Sociological Methods & Research · doi:10.1177/00491241221099552 · Cited by 728 · see citing works
“In this case, progress can be made with sensitivity analyses, by, for instance, positing plausible bounds on the the strength of the direct effect of Z on Y , or on the strength of the effects of the latent variables (Cinelli et al., 2019; Cinelli and Hazlett, 2020a).”
Self-citationClimate policies that achieved major emission reductions: Global evidence from two decades
Stechemesser, A., Koch, N., Mark, E., Dilger, E., Klösel, P., Menicacci, L., Nachtigall, D., Pretis, F., Ritter, N., Schwarz, M., et al. (2024) · Science · doi:10.1126/science.adl6547 · Cited by 350 · see citing works
Meaningful associations in the adolescent brain cognitive development study
Dick, A. S., Lopez, D. A., Watts, A. L., Heeringa, S. G., Reuter, C., Bartsch, H., Fan, C. C., Kennedy, D. N., Palmer, C. E., Marshall, A. T., et al. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.118262 · Cited by 234 · see citing works
Placebo Tests for Causal Inference
Eggers, A. C., Tuñón, G., & Dafoe, A. (2023) · American Journal of Political Science · doi:10.1111/ajps.12818 · Cited by 213 · see citing works
Sensemakr: Sensitivity Analysis Tools for OLS in R and Stata
Cinelli, C., Ferwerda, J., & Hazlett, C. J. (2024) · Observational Studies · doi:10.1353/obs.2024.a946583 · Cited by 225 · see citing works
Self-citationWildfire Exposure Increases Pro-Environment Voting within Democratic but Not Republican Areas
Hazlett, C., & Mildenberger, M. (2020) · American Political Science Review · doi:10.1017/s0003055420000441 · Cited by 140 · see citing works
“Some studies find that experiencing hazards increases intention to engage in mitigation and adaptation policies (Demski et al. 2017; Spence et al. 2011) and climate risk perceptions (Lujala, Lein, and Rød 2015).”
BackgroundSelf-citationThe VIF Score. What is it Good For? Absolutely Nothing
Kalnins, A. T., & Hill, K. P. (2023) · Organizational Research Methods · doi:10.1177/10944281231216381 · Cited by 185 · see citing works
“Similarly, Cinelli and Hazlett (2020) examine treatment effects and the necessary variance that they must explain for researchers to be assured of valid causal inference.”
Omitted variable bias: A threat to estimating causal relationships
Wilms, R., Mäthner, E., Winnen, L., & Lanwehr, R. (2021) · Methods in Psychology · doi:10.1016/j.metip.2021.100075 · Cited by 159 · see citing works
Using the president’s tweets to understand political diversion in the age of social media
Lewandowsky, S., Jetter, M., & Ecker, U. K. H. (2020) · Nature Communications · doi:10.1038/s41467-020-19644-6 · Cited by 139 · see citing works
Common methodological mistakes
Wulff, J. N., Sajons, G. B., Pogrebna, G., Lonati, S., Bastardoz, N., Banks, G. C., & Antonakis, J. (2023) · The Leadership Quarterly · doi:10.1016/j.leaqua.2023.101677 · Cited by 120 · see citing works
Nuclear power and renewable energy are both associated with national decarbonization
Fell, H. G., Gilbert, A. Q., Jenkins, J., & Mildenberger, M. (2022) · Nature Energy · doi:10.1038/s41560-021-00964-w · Cited by 111 · see citing works
Economic insecurity and the demand for populism in Europe
Guiso, L., Herrera, H., Morelli, M., & Sonno, T. (2024) · Economica · doi:10.1111/ecca.12513 · Cited by 116 · see citing works
Recent Developments in Causal Inference and Machine Learning
Brand, J. E., Zhou, X., & Xie, Y. F. (2023) · Annual Review of Sociology · doi:10.1146/annurev-soc-030420-015345 · Cited by 124 · see citing works
“Cinelli & Hazlett (2020) provide additional measures and graphical tools for assessing sensitivity to unobserved confounding.”
BackgroundFamily Matters: How Immigrant Histories Can Promote Inclusion
Williamson, S., Adida, C. L., Lo, A., Izama, M. P., Prather, L., & Werfel, S. H. (2020) · American Political Science Review · doi:10.1017/s0003055420001057 · Cited by 94 · see citing works
kinkyreg: Instrument-free inference for linear regression models with endogenous regressors
Kripfganz, S., & Kiviet, J. F. (2021) · The Stata Journal Promoting communications on statistics and Stata · doi:10.1177/1536867x211045575 · Cited by 92 · see citing works
“These two procedures can be applied with the community-contributed commands plausexog and imperfectiv by Clarke and Matta (2018), respectively.”
MethodInfluentialTopiwala, A., Ebmeier, K. P., Maullin-Sapey, T., & Nichols, T. E. (2022) · NeuroImage Clinical · doi:10.1016/j.nicl.2022.103066 · Cited by 91 · see citing works
“Partial R2 and robustness values were calculated using R’s sensemakr package, which estimates the necessary strength of an unobserved confounder required to fully account for alcohol effects on MRI marker (Cinelli and Hazlett, 2020).”
MethodCausal Inference in the Social Sciences
Imbens, G. W. (2023) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-033121-114601 · Cited by 95 · see citing works
Causal Inference With Observational Data and Unobserved Confounding Variables
Byrnes, J. E. K., & Dee, L. E. (2025) · Ecology Letters · doi:10.1111/ele.70023 · Cited by 90 · see citing works
“Further, in concert with the approaches presented here, we suggest using sensitivity tests (Altonji, Elder, and Taber 2005; Cinelli and Hazlett 2020; Oster 2019; Rosenbaum 2002) or designs that differ in assumptions to assess robustness of results [see Dee et al. (2023) for an ecological example].”
Congressional Representation: Accountability from the Constituent's Perspective
Ansolabehere, S. D., & Kuriwaki, S. (2021) · American Journal of Political Science · doi:10.1111/ajps.12607 · Cited by 89 · see citing works
Hot Dry Days Increase Perceived Experience With Global Warming
Marlon, J. R., Wang, X., Mildenberger, M., Bergquist, P., Swain, S., Hayhoe, K., Howe, P. D., Maibach, E. W., & Leiserowitz, A. (2019) · Global Environmental Change · doi:10.1016/j.gloenvcha.2021.102247 · Cited by 90 · see citing works
Sheltering Populists? House Prices and the Support for Populist Parties
Ansell, B. W., Hjorth, F., Nyrup, J., & Larsen, M. V. (2021) · The Journal of Politics · doi:10.1086/718354 · Cited by 77 · see citing works
“We also estimated how likely it is that our effects are driven by a missing control variable (Cinelli and Hazlett 2020).”
BackgroundCirone, A. E., & Pepinsky, T. B. (2021) · Annual Review of Political Science · doi:10.1146/annurev-polisci-051120-104325 · Cited by 67 · see citing works
Causal inference on human behaviour
Bailey, D. H., Jung, A. J., Beltz, A. M., Eronen, M. I., Gische, C., Hamaker, E. L., Kording, K. P., Lebel, C., Lindquist, M. A., Moeller, J., et al. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-01939-z · Cited by 56 · see citing works
“As can be seen in various formulas that quantify the bias due to multiple omitted confounders 22 , the cumulative impact of those omitted confounders (including their higher-order interactive effects) could indeed rarely be ignorable.”
Wealth of Tongues: Why Peripheral Regions Vote for the Radical Right in Germany
Ziblatt, D. F., Hilbig, H., & Bischof, D. (2023) · American Political Science Review · doi:10.1017/s0003055423000862 · Cited by 56 · see citing works
Temperature variability and long-run economic development
Linsenmeier, M. (2023) · Journal of Environmental Economics and Management · doi:10.1016/j.jeem.2023.102840 · Cited by 55 · see citing works
Yan, D., Sun, W., Li, P., Liu, C., & Li, Y. (2024) · Cities · doi:10.1016/j.cities.2024.105518 · Cited by 53 · see citing works
Tao, Q., Akhter‐Khan, S. C., Ang, T. F. A., DeCarli, C. S., Alosco, M. L., Mez, J. B., Killiany, R., Devine, S., Rokach, A., Itchapurapu, I. S., et al. (2022) · EClinicalMedicine · doi:10.1016/j.eclinm.2022.101643 · Cited by 52 · see citing works
Green finance drives renewable energy development: empirical evidence from 53 countries worldwide
Hou, H., Wang, Y., & Zhang, M. (2023) · Environmental Science and Pollution Research · doi:10.1007/s11356-023-28111-w · Cited by 55 · see citing works
Semester Course Load and Student Performance
Huntington‐Klein, N., & Gill, A. M. (2020) · Research in Higher Education · doi:10.1007/s11162-020-09614-8 · Cited by 58 · see citing works
Sensitivity Analyses for Unmeasured Confounders
McGowan, L. D. (2022) · Current Epidemiology Reports · doi:10.1007/s40471-022-00308-6 · Cited by 51 · see citing works
Li, L., Jin, G., Guo, Y., Zhang, Y., & Jing, R. (2022) · Journal of Affective Disorders · doi:10.1016/j.jad.2022.12.001 · Cited by 54 · see citing works
Cook, S. J., & Fortunato, D. (2022) · American Political Science Review · doi:10.1017/s0003055422000624 · Cited by 49 · see citing works
Refugees welcome? Inter-group interaction and host community attitude formation
Betts, A., Stierna, M. F., Omata, N., & Sterck, O. C. (2022) · World Development · doi:10.1016/j.worlddev.2022.106088 · Cited by 49 · see citing works
Bammens, Y. P. M., & Hünermund, P. (2023) · Research Policy · doi:10.1016/j.respol.2023.104826 · Cited by 49 · see citing works
The impact of radical right parties on family benefits
Ennser‐Jedenastik, L. (2021) · West European Politics · doi:10.1080/01402382.2021.1936944 · Cited by 48 · see citing works
(Un)Natural Disasters: Electoral Cycles in Disaster Relief
Cooperman, A. (2022) · Comparative Political Studies · doi:10.1177/00104140211047410 · Cited by 41 · see citing works
Brown, J. P., Hunnicutt, J. N., Ali, M. S., Bhaskaran, K., Cole, A. L., Langan, S., Nitsch, D., Rentsch, C. T., Galwey, N. W., Wing, K., et al. (2024) · BMJ · doi:10.1136/bmj-2023-076365 · Cited by 47 · see citing works
Obesity and renal cell carcinoma: Biological mechanisms and perspectives
Venkatesh, N., Martini, A., McQuade, J. L., Msaouel, P., & Hahn, A. W. (2023) · Seminars in Cancer Biology · doi:10.1016/j.semcancer.2023.06.001 · Cited by 46 · see citing works
Hypothetical case replacement can be used to quantify the robustness of trial results
Frank, K. A., Lin, Q., Maroulis, S. J., Mueller, A. S., Xu, R., Rosenberg, J. M., Hayter, C. S., Mahmoud, R. A., Kolak, M., Dietz, T. M., et al. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.01.025 · Cited by 45 · see citing works
Arbolino, R., Lantz, T. L., & Napolitano, O. (2022) · Regional Studies · doi:10.1080/00343404.2022.2069745 · Cited by 45 · see citing works
CEO personality traits, strategic flexibility, and firm dynamics
Aabo, T., Pantzalis, C., Park, J. C., Trigeorgis, L. G., & Wulff, J. N. (2023) · Journal of Corporate Finance · doi:10.1016/j.jcorpfin.2023.102524 · Cited by 45 · see citing works
Organizations, Resistance, and Democracy: How Civil Society Organizations Impact Democratization
Pinckney, J. C., Butcher, C. R., & Braithwaite, J. M. (2022) · International Studies Quarterly · doi:10.1093/isq/sqab094 · Cited by 44 · see citing works
The effects of algorithmic content selection on user engagement with news on Twitter
Dujeancourt, E., & Garz, M. (2023) · The Information Society · doi:10.1080/01972243.2023.2230471 · Cited by 54 · see citing works
An omitted variable bias framework for sensitivity analysis of instrumental variables
Cinelli, C., & Hazlett, C. J. (2025) · Biometrika · doi:10.1093/biomet/asaf004 · Cited by 45 · see citing works
Self-citationBrown, J. R., Enos, R. D., Feigenbaum, J., & Mazumder, S. (2021) · Science Advances · doi:10.1126/sciadv.abe8432 · Cited by 44 · see citing works
Angry or Weary? How Violence Impacts Attitudes toward Peace among Darfurian Refugees
Hazlett, C. J. (2019) · Journal of Conflict Resolution · doi:10.1177/0022002719879217 · Cited by 46 · see citing works
“…more productively discuss “how strong confounding would need to be to substantively alter our conclusions” through a sensitivity analysis following Cinelli and Hazlett (2018).10 Table 2 shows the regression results for the OLS-long model described above with Peace Index as the outcome,…”
BackgroundSelf-citationQuantifying the robustness of causal inferences: Sensitivity analysis for pragmatic social science
Frank, K. A., Lin, Q., Xu, R., Maroulis, S. J., & Mueller, A. S. (2022) · Social Science Research · doi:10.1016/j.ssresearch.2022.102815 · Cited by 45 · see citing works
Zor, S. (2022) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2022.135339 · Cited by 43 · see citing works
Digital credit and the gender gap in financial inclusion: Empirical evidence from Kenya
Johnen, C., & Mußhoff, O. (2022) · Journal of International Development · doi:10.1002/jid.3687 · Cited by 42 · see citing works
Jensen, T. W., Ersbøll, A. K., Folke, F., Wolthers, S. A., Andersen, M. P., Blomberg, S. N. F., Andersen, L. B., Lippert, F. K., Torp-Pedersen, C. T., & Christensen, H. C. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.3338 · Cited by 42 · see citing works
“Born for a Storm”: Hard-Right Social Media and Civil Unrest
Karell, D., Linke, A. M., Holland, E. C., & Hendrickson, E. (2023) · American Sociological Review · doi:10.1177/00031224231156190 · Cited by 48 · see citing works
Mobile money adoption in Kenya: The role of mobile money agents
Johnen, C., Parlasca, M. C., & Mußhoff, O. (2023) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2023.122503 · Cited by 46 · see citing works
Uncovering Sociological Effect Heterogeneity Using Tree-Based Machine Learning
Brand, J. E., Xu, J., Koch, B., & Geraldo, P. (2021) · Sociological Methodology · doi:10.1177/0081175021993503 · Cited by 53 · see citing works
“Alternative approaches for sensitivity analyses are also possible (e.g., Cinelli and Hazlett 2020), but they follow the same general logic.”
BackgroundMellon, J. (2021) · doi:10.31235/osf.io/9qj4f · Cited by 48 · see citing works
Lei, H., Wang, X., Chiu, M. M., Du, M., & Xie, T. (2022) · School Psychology International · doi:10.1177/01430343221122453 · Cited by 56 · see citing works
Paradox of Middleware: Innovation Enabler and Creativity Constraint
Miric, M., Ozalp, H., & Yilmaz, E. D. (2019) · Strategic Management Journal · doi:10.1002/smj.3457 · Cited by 42 · see citing works
Year-round utilization of sea ice-associated carbon in Arctic ecosystems
Koch, C. W., Brown, T. A., Amiraux, R., Ruiz-Gonzalez, C., MacCorquodale, M., Yunda‐Guarin, G., Kohlbach, D., Loseto, L. L., Rosenberg, B., Hussey, N. E., et al. (2023) · Nature Communications · doi:10.1038/s41467-023-37612-8 · Cited by 39 · see citing works
Emigration and radical right populism
Dancygier, R. M., Dehdari, S. H., Laitin, D. D., Marbach, M., & Vernby, K. (2024) · American Journal of Political Science · doi:10.1111/ajps.12852 · Cited by 39 · see citing works
Nachtigall, F., Wagner, F., Berrill, P., & Creutzig, F. (2025) · Transportation Research Part D Transport and Environment · doi:10.1016/j.trd.2025.104593 · Cited by 39 · see citing works
“We calculate the partial R 2 of the treatment with the outcome, which, as shown by Cinelli and Hazlett (2020), corresponds to the degree to which confounders explaining all the residual outcome variation would have to be associated with the treatment to eliminate the effect.”
Don't put all your eggs in one basket: Testing an integrative model of household food waste
Bretter, C., Unsworth, K. L., Russell, S., Quested, T. E., Doriza, A., & Kaptan, G. (2022) · Resources Conservation and Recycling · doi:10.1016/j.resconrec.2022.106442 · Cited by 42 · see citing works
Robustness of Empirical Evidence for the Democratic Peace: A Nonparametric Sensitivity Analysis
Imai, K., & Lo, J. (2021) · International Organization · doi:10.1017/s0020818321000126 · Cited by 38 · see citing works
“See, for example, Blackwell 2013; Cinelli and Hazlett 2020; Imai, Keele, and Yamamoto 2010; Imbens 2003.”
BackgroundGoodrich, J. A., Wang, H., Jia, Q., Stratakis, N., Zhao, Y., Maître, L., Bustamante, M., Vafeiadi, M., Aung, M. T., Andrušaitytė, S., et al. (2024) · Environment International · doi:10.1016/j.envint.2024.108930 · Cited by 38 · see citing works
“…robustness value represents the minimum strength of association (as a percent of variance explained) between the unknown confounder with the residual variance in the exposure and the residual variance in the outcome which is needed to reduce the observed effect to zero (Cinelli and Hazlett, 2020).”
Public data and corporate employment: Evidence from the launch of Chinese public data platform
Li, X., Liu, Z., & Ye, Y. (2024) · Economic Analysis and Policy · doi:10.1016/j.eap.2024.08.023 · Cited by 40 · see citing works
Sensitivity Analysis via the Proportion of Unmeasured Confounding
Bonvini, M., & Kennedy, E. H. (2020) · Journal of the American Statistical Association · doi:10.1080/01621459.2020.1864382 · Cited by 38 · see citing works
“In addition, in Section 6.1 of the supplementary materials, we apply the sensitivity analysis designed for linear models proposed in Cinelli and Hazlett (2020).”
MethodInfluentialResidential Segregation by Race and Ethnicity and the Changing Geography of Neighborhood Poverty
Hess, C. (2020) · Spatial Demography · doi:10.1007/s40980-020-00066-3 · Cited by 38 · see citing works
Mellon, J. (2024) · American Journal of Political Science · doi:10.1111/ajps.12894 · Cited by 37 · see citing works
Lopez, D. A., Foxe, J. J., Mao, Y., Thompson, W. K., Martin, H., & Freedman, E. G. (2021) · Frontiers in Public Health · doi:10.3389/fpubh.2021.657422 · Cited by 37 · see citing works
“The analysis used a robustness value (RV) to summarize the resilience of the point estimates taking into account the partial R-squared of the treatment with the outcome (54).”
MethodWang, M., He, B., Zhang, J., & Jin, Y. (2021) · Sustainability · doi:10.3390/su132313172 · Cited by 37 · see citing works
On the Nuisance of Control Variables in Causal Regression Analysis
Hünermund, P., & Louw, B. (2023) · Organizational Research Methods · doi:10.1177/10944281231219274 · Cited by 196 · see citing works
“That is because Z 1 is correlated with Z 2 and will thus partially pick up an effect of Z 2 on Y too (Cinelli & Hazlett, 2020).”
Lee, J. S., & Chiu, M. M. (2023) · Annual Review of Applied Linguistics · doi:10.1017/s0267190523000090 · Cited by 36 · see citing works
Bias mitigation in empirical peace and conflict studies: A short primer on posttreatment variables
Dworschak, C. (2023) · Journal of Peace Research · doi:10.1177/00223433221145531 · Cited by 35 · see citing works
Multiple-bias Sensitivity Analysis Using Bounds
Smith, L. H., Mathur, M. B., & VanderWeele, T. J. (2021) · Epidemiology · doi:10.1097/ede.0000000000001380 · Cited by 35 · see citing works
Coherence within and across Categories: The Dynamic Viability of Product Categories on Kickstarter
Soublière, J., Lo, J. Y.-C., & Rhee, E. Y. (2023) · Academy of Management Journal · doi:10.5465/amj.2022.0105 · Cited by 34 · see citing works
Booker, J. A., Fıvush, R., & Graci, M. E. (2021) · Journal of Personality · doi:10.1111/jopy.12668 · Cited by 40 · see citing works
The Virality of Hate Speech on Social Media
Maarouf, A., Pröllochs, N., & Feuerriegel, S. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3641025 · Cited by 35 · see citing works
“Relevance: Hate speech presents a widespread threat to the well-being of individuals [34, 46], and it further propels the ongoing segregation and radicalization in many societies [2, 8].”
BackgroundClimate and health benefits of a transition from gas to electric cooking
Gould, C. F., Bejarano, M. L., Cuesta, B. D. L., Jack, D. W., Schlesinger, S. B., Valarezo, A., & Burke, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2301061120 · Cited by 32 · see citing works
“877 [44] Carlos Cinelli and Chad Hazlett.”
Inferno: A Guide to Field Experiments in Online Display Advertising
Johnson, G. A. (2020) · Journal of Economics & Management Strategy · doi:10.1111/jems.12513 · Cited by 33 · see citing works
Sharp Sensitivity Analysis for Inverse Propensity Weighting via Quantile Balancing
Dorn, J., & Guo, K. (2022) · Journal of the American Statistical Association · doi:10.1080/01621459.2022.2069572 · Cited by 32 · see citing works
“Caveats to this approach and more sophisticated empirical calibration strategies are discussed in [21, 59, 11].”
BackgroundMethodInfluentialBass, L., Cameron, C. M., & Kastellec, J. (2020) · Political Science Research and Methods · doi:10.1017/psrm.2022.21 · Cited by 31 · see citing works
“Second, we implement a sensitivity analysis that probes the possibility that omitted variable bias is driving the key results (Cinelli and Hazlett, 2020).”
MethodIn the Eye of the Storm: Hurricanes, Climate Migration, and Climate Attitudes
Arias, S., & Blair, C. (2024) · American Political Science Review · doi:10.1017/s0003055424000352 · Cited by 26 · see citing works
Hu, Q., Shen, W., Yan, J., Kong, W., Li, W., & Zhang, Z. (2024) · Land Use Policy · doi:10.1016/j.landusepol.2024.107212 · Cited by 30 · see citing works
Sensitivity Analysis of Linear Structural Causal Models
Cinelli, C., Kumor, D. R., Chen, B., Pearl, J., & Bareinboim, E. (2019) · International Conference on Machine Learning · Cited by 30 · see citing works
Self-citationWang, C., & Lu, Y. (2020) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2020.120194 · Cited by 30 · see citing works
Shapiro, D. D., Soeung, M., Perelli, L., Dondossola, E., Surasi, D. S., Tripathi, D. N., Bertocchio, J., Xia, R., Starbuck, M. W., Alstine, M. L. V., et al. (2021) · Cancers · doi:10.3390/cancers13236022 · Cited by 30 · see citing works
“, RMC diagnosis), we performed a robust sensitivity analysis, as described by Cinelli and Hazlett, that does not require any assumptions on the functional form of the observed association or on the distribution and number of unobserved confounders [21].”
BackgroundMethodCinelli, C., LaPierre, N., Hill, B. L., Sankararaman, S., & Eskin, E. (2022) · Nature Communications · doi:10.1038/s41467-022-28553-9 · Cited by 29 · see citing works
Self-citationWhat drives partisan conflict and consensus on welfare state issues?
Ennser‐Jedenastik, L. (2020) · Journal of Public Policy · doi:10.1017/s0143814x20000240 · Cited by 29 · see citing works
Causal Fairness Analysis: A Causal Toolkit for Fair Machine Learning
Plečko, D., & Bareinboim, E. (2024) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000106 · Cited by 29 · see citing works
Held, U., Burgstaller, J. M., Deforth, M. E., Steurer, J., Pichierri, G., & Wertli, M. M. (2022) · Scientific Reports · doi:10.1038/s41598-022-06797-1 · Cited by 29 · see citing works
Cultural tightness emancipation and venture profitability: An international experience lens
Pidduck, R. J., Hechavarria, D. M., & Patel, A. (2023) · Journal of Business Research · doi:10.1016/j.jbusres.2023.114363 · Cited by 29 · see citing works
Exact parametric causal mediation analysis for a binary outcome with a binary mediator
Doretti, M., Raggi, M., & Stanghellini, E. (2021) · Statistical Methods & Applications · doi:10.1007/s10260-021-00562-w · Cited by 26 · see citing works
Liepe, K. J., van der Maaten, E., van der Maaten‐Theunissen, M., & Liesebach, M. (2022) · Frontiers in Forests and Global Change · doi:10.3389/ffgc.2022.804857 · Cited by 28 · see citing works
Steiner, N. D., Schimpf, C. H., & Wuttke, A. (2022) · Politische Vierteljahresschrift · doi:10.1007/s11615-022-00416-4 · Cited by 28 · see citing works
COVID-19 fatalities and internal conflict: Does government economic support matter?
Farzanegan, M. R., & Gholipour, H. F. (2023) · European Journal of Political Economy · doi:10.1016/j.ejpoleco.2023.102368 · Cited by 28 · see citing works
Mellon, J. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3715610 · Cited by 28 · see citing works
Adomako, S., Medase, S. K., & Zhang, S. X. (2024) · Research Policy · doi:10.1016/j.respol.2024.105061 · Cited by 28 · see citing works
When local business faded away: the uneven impact of Airbnb on the geography of economic activities
Hidalgo, A., Riccaboni, M., & Velázquez, F. J. (2023) · Cambridge Journal of Regions Economy and Society · doi:10.1093/cjres/rsad009 · Cited by 26 · see citing works
Scaling-up ecological understanding with remote sensing and causal inference
Cleemput, E. V., Adler, P. B., Suding, K. N., Rebelo, A. J., Poulter, B., & Dee, L. E. (2024) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2024.09.006 · Cited by 26 · see citing works
How do firms respond to climate change? Evidence based on ESG performance
Xue, M., Lu, M., Du, A. M., & Zheng, B. (2025) · International Review of Economics & Finance · doi:10.1016/j.iref.2025.103863 · Cited by 26 · see citing works
A Causal Framework for Observational Studies of Discrimination
Gaebler, J., Cai, W., Basse, G., Shroff, R., Goel, S., & Hill, J. (2022) · Statistics and Public Policy · doi:10.1080/2330443x.2021.2024778 · Cited by 28 · see citing works
“The plots indicate the maximum amount the ŝateM may change under the Cinelli and Hazlett [2020] model of confounding, parameterized by two partial R2 values.”
MethodInstrumental goal activation increases online petition support across languages.
Markowitz, D. M. (2022) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000333 · Cited by 20 · see citing works
- Authors
- Cinelli, C., & Hazlett, C.
- Affiliations
- University of California, Los Angeles (US)
- Corresponding author
- Hazlett, Chad
- Venue
- Journal of the Royal Statistical Society Series B: Statistical Methodology
- Publisher
- Oxford University Press (OUP)
- Published
- 2019-12-17
- Volume
- 82
- Issue
- 1
- Pages
- 39-67
- ISSN
- 1369-7412 (ISSN-L)
- 1467-9868
- OpenAlex
- W2996483157
- Cited by
- 1,088
- References
- 51
- Citations by year
- 2026: 275 · 2025: 248 · 2024: 194 · 2023: 141 · 2022: 107 · 2021: 80 · 2020: 34 · 2019: 8 · 2015: 1
- FWCI
- 44.97
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model (vor)
- Locations
- Journal of the Royal Statistical Society Series B (Statistical Methodology) (publishedVersion)
- Funders
- University of California, Los Angeles
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- omitted variable bias, sensitivity analysis, unmeasured confounding, benchmarking, expert knowledge, treatment effect estimation, violence exposure, attitudes toward peace
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
