Design Sensitivity and Efficiency in Observational Studies
The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Abstract
An observational study attempts to draw inferences about the effects caused by a treatment when subjects are not randomly assigned to treatment or control as they would be in a randomized trial. After adjustments have been made for imbalances in measured covariates, the key source of uncertainty in an observational study is the possibility that subjects were not comparable prior to treatment in terms of some unmeasured covariate, so that differing outcomes in treated and control groups are not effects caused by the treatment. A sensitivity analysis asks about the magnitude of the departure from random assignment needed to alter the qualitative conclusions of the study, and the power of a sensitivity analysis and the design sensitivity anticipate the outcome of a sensitivity analysis under an assumed model for treatment effect. Lacking theoretical guidance, we tend to select statistical methods for use in observational studies based on their efficiency in randomized experiments. This turns out to be a mistake. A highly efficient method for detecting small treatment effects in randomized experiments need not, and often does not, have the highest power in a sensitivity analysis or the largest design sensitivity. That is, the best procedure assuming the observational study is a randomized experiment need not be the best procedure under more realistic assumptions. Small effects are sensitive to small biases, and methods targeted at detecting small effects in the absence of bias may not give due emphasis to evidence that the effect is stable and not small, and hence not easily attributed to small or moderate biases. The issue is illustrated in a practical example, is explored informally using a graphical heuristic, is studied asymptotically by determining the design sensitivity for a signed rank statistic with general scores, and is evaluated in a simulation. Among robust procedures with similar Pitman efficiencies for several distributions, there are large differences in design sensitivity and hence substantial differences in the power of a sensitivity analysis.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Estimating causal effects of treatments in randomized and nonrandomized studies
Rubin, D. B. (1974) · Journal of Educational Psychology · doi:10.1037/h0037350 · Cited by 9,436
Assessing the sensitivity of regression results to unmeasured confounders in observational studies
Lin, D. Y., Psaty, B. M., & Kronmal, R. A. (1998) · Biometrics · doi:10.2307/2533848 · Cited by 742 · cited 1×
“…analysis in observational studies, see Cornfield et al. (1959), Rosenbaum and Rubin (1983), Yanagawa (1984), Gastwirth (1992), Marcus (1997), Lin, Psaty, and Kronmal (1998), Robins, Rotnitzky, and Scharfstein (1999), Copas and Eguchi (2001), Imbens (2003), Diprete and Gangl (2004), Wang and…”
BackgroundTests for no treatment effect in randomized clinical trials
GAIL, M. H., TAN, W. Y., & PIANTADOSI, S. (1988) · Biometrika · doi:10.1093/biomet/75.1.57 · Cited by 143 · cited 1×
“For instance, Gail, Tan, and Piantadosi (1988) use the randomization distribution of likelihood-based tests.”
Method
Cited by
Sensitivity analysis in observational research: Introducing the E-value
VanderWeele, T. J., & Ding, P. (2017) · Annals of Internal Medicine · doi:10.7326/m16-2607 · Cited by 6,350
“, that some third factor related to both the treatment and the outcome might explain their association, with no true causal effect(12-22).”
BackgroundMethodInfluentialCapacity shortfalls hinder the performance of marine protected areas globally
Gill, D. A., Mascia, M. B., Ahmadia, G. N., Glew, L., Lester, S. E., Barnes, M., Craigie, I., Darling, E. S., Free, C. M., Geldmann, J., et al. (2017) · Nature · doi:10.1038/nature21708 · Cited by 941
Valid post-selection inference
Berk, R., Brown, L., Buja, A., Zhang, K., & Zhao, L. (2013) · The Annals of Statistics · doi:10.1214/12-aos1077 · Cited by 599
A flexible, interpretable framework for assessing sensitivity to unmeasured confounding
Dorie, V., Harada, M., Carnegie, N. B., & Hill, J. (2016) · Statistics in Medicine · doi:10.1002/sim.6973 · Cited by 104
Hill, L., Scaggs, S. J. A., & Bales, W. D. (2017) · Journal of Offender Rehabilitation · doi:10.1080/10509674.2016.1257535 · Cited by 15
Metadata
- Authors
- Rosenbaum, P. R.
- Venue
- Journal of the American Statistical Association
- Publisher
- Informa UK Limited
- Published
- 2010-06
- Volume
- 105
- Issue
- 490
- Pages
- 692-702
- ISSN
- 0162-1459 (ISSN-L)
- 1537-274X
- OpenAlex
- W1988110689
- Cited by
- 83
- References
- 3
- FWCI
- 5.12
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Design Sensitivity and Efficiency in Observational Studies. https://app.cochat.ai/papers/10.1198/jasa.2010.tm09570
- The paper
- Rosenbaum, P. R. (2010). Design Sensitivity and Efficiency in Observational Studies. Journal of the American Statistical Association. https://doi.org/10.1198/jasa.2010.tm09570
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.