CoChat
Sign Up
PreprintCited by 96Advanced Causal Inference TechniquesEvaluation and Performance AssessmentStatistical Methods in Clinical Trials

Efficient Estimation for Staggered Rollout Designs

Open access
Roth, Jonathan, Sant'Anna, Pedro H. C.·2021·arXiv
DOI 10.48550/arxiv.2102.01291arXiv 2102.01291
Open this paper in CoChat — free accountOpen PDF

Abstract

We study estimation of causal effects in staggered rollout designs, i.e. settings where there is staggered treatment adoption and the timing of treatment is as-good-as randomly assigned. We derive the most efficient estimator in a class of estimators that nests several popular generalized difference-in-differences methods. A feasible plug-in version of the efficient estimator is asymptotically unbiased with efficiency (weakly) dominating that of existing approaches. We provide both $t$-based and permutation-test-based methods for inference. In an application to a training program for police officers, confidence intervals for the proposed estimator are as much as eight times shorter than for existing approaches.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2102.01291

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Roth (2021) notes that leveraging earlier pre-treatment periods increases the bias of a DID estimator, since one makes comparisons from earlier periods.

    “In such instances, Roth (2021) notes that leveraging earlier pre-treatment periods increases the bias of a DID estimator, since one makes comparisons from earlier periods.”

    From Two-Way Fixed Effects and Differences-in-Differences with Heterogeneous Treatment Effects: A Survey (2022) · cites this paper as Roth (2021)

    The Introduction (p. 3) says that, under (quasi-)random treatment timing, adjusting with pre-treatment outcomes is unbiased for any weight; it discusses the choice of weight as affecting variance, not increasing bias. The paper therefore makes a contrary point in the setting it studies, rather than saying that comparisons with earlier periods increase DiD bias. (LLM, conf=0.88)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Roth and Sant'Anna (2021) show that if one assumes treatment timing is as good as random, one can obtain more efficient estimates than using staggered DiD methods.

    “Roth and Sant'Anna (2021) show that if one is willing to assume treatment timing is as good as random, one can obtain more efficient estimates than using the staggered DiD methods discussed in Section 3.3.”

    From What's trending in difference-in-differences? A synthesis of the recent econometrics literature (2023) · cites this paper as Roth and Sant'Anna (2021)

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 51 · Cited by 96

Research works printed but not linked: 54

  1. Abadie, Alberto, Susan Athey, Guido W. Imbens, and Jeffrey M. Wooldridge , 'SamplingBased versus Design-Based Uncertainty in Regression Analysis,' Econometrica , 2020, 88 (1), 265-296.
  2. Abbring, Jaap H. and Gerard J. van den Berg , 'The Nonparametric Identification of Treatment Effects in Duration Models,' Econometrica , 2003, 71 (5), 1491-1517.
  3. Angrist, Joshua D. and Jorn-Steffen Pischke , Mostly Harmless Econometrics: An Empiricist's Companion , Princeton: Princeton University Press, 2009.
  4. Athey, Susan and Guido W. Imbens , 'Design-based analysis in Difference-In-Differences settings with staggered adoption,' Journal of Econometrics , 2022, 226 (1), 62-79.
  5. Bai, Yuehao, Joseph P. Romano, and Azeem M. Shaikh , 'Inference in Experiments With Matched Pairs,' Journal of the American Statistical Association , 2022, 117 (540), 1726-1737.
  6. Bailey, Martha J. and Andrew Goodman-Bacon , 'The War on Poverty's Experiment in Public Medicine: Community Health Centers and the Mortality of Older Americans,' American Economic Review , 2015, 105 (3), 1067-1104.
  7. Basse, Guillaume W, Yi Ding, and Panos Toulis , 'Minimax designs for causal effects in temporal experiments with treatment habituation,' Biometrika , 2023, 110 (1), 155-168.
  8. Borusyak, Kirill and Peter Hull , 'Non-Random Exposure to Exogenous Shocks: Theory and Applications,' September 2020. and Xavier Jaravel , 'Revisiting Event Study Designs,' SSRN Scholarly Paper ID 2826228, Social Science Research Network, Rochester, NY 2018. Working Paper.
  9. Brown, Celia A. and Richard J. Lilford , 'The stepped wedge trial design: A systematic review,' BMC Medical Research Methodology , 2006, 6 , 1-9.
  10. Bugni, Federico A., Ivan A. Canay, and Azeem M. Shaikh , 'Inference Under CovariateAdaptive Randomization,' Journal of the American Statistical Association , 2018, 113 (524), 17841796.
  11. Callaway, Brantly and Pedro H. C. Sant'Anna , 'Difference-in-Differences with multiple time periods,' Journal of Econometrics , 2021, 225 (2), 200-230.
  12. Chung, Eun Yi and Joseph P. Romano , 'Multivariate and multiple permutation tests,' Journal of Econometrics , 2016, 193 (1), 76-91.
  13. Chung, EunYi and Joseph P. Romano , 'Exact and asymptotically robust permutation tests,' The Annals of Statistics , 2013, 41 (2), 484-507.
  14. de Chaisemartin, Clément and Xavier D'Haultfoeuille , 'Two-Way Fixed Effects Estimators with Heterogeneous Treatment Effects,' American Economic Review , September 2020, 110 (9), 2964-2996. and , 'Difference-in-Differences Estimators of Intertemporal Treatment Effects,' arXiv:2007.04267 [econ] , May 2021. arXiv: 2007.04267. · arXiv
  15. Deshpande, Manasi and Yue Li , 'Who Is Screened Out? Application Costs and the Targeting of Disability Programs,' American Economic Journal: Economic Policy , 2019, 11 (4), 213-248.
  16. DiCiccio, Cyrus J. and Joseph P. Romano , 'Robust Permutation Tests For Correlation And Regression Coefficients,' Journal of the American Statistical Association , 2017, 112 (519), 12111220.
  17. Ding, Peng and Fan Li , 'A bracketing relationship between difference-in-differences and laggeddependent-variable adjustment,' Political Analysis , 2019, 27 (4), 605-615.
  18. Fadlon, Itzik and Torben Heien Nielsen , 'Family Labor Supply Responses to Severe Health Shocks: Evidence from Danish Administrative Records,' American Economic Journal: Applied Economics , 2021, 13 (3), 1-30.
  19. Freedman, David A. , 'On Regression Adjustments in Experiments with Several Treatments,' The Annals of Applied Statistics , 2008, 2 (1), 176-196.
  20. , 'On regression adjustments to experimental data,' Advances in Applied Mathematics , 2008, 40 (2), 180-193.
  21. Frison, L. and S. J. Pocock , 'Repeated measures in clinical trials: analysis using mean summary statistics and its implications for design,' Statistics in Medicine , 1992, 11 (13), 1685-1704.
  22. Funatogawa, Takashi, Ikuko Funatogawa, and Yu Shyr , 'Analysis of covariance with pretreatment measurements in randomized trials under the cases that covariances and post-treatment variances differ between groups,' Biometrical Journal , 2011, 53 (3), 512-524.
  23. Glynn, Adam N. and Kevin M. Quinn , 'An Introduction to the Augmented Inverse Propensity Weighted Estimator,' Political Analysis , 2010, 18 (1), 36-56.
  24. Goodman-Bacon, Andrew , 'Difference-in-differences with variation in treatment timing,' Journal of Econometrics , 2021, 225 (2), 254-277.
  25. Hahn, Jinyong , 'On the Role of the Propensity Score in Efficient Semiparametric Estimation of Average Treatment Effects,' Econometrica , 1998, 66 (2), 315-331.
  26. Imbens, Guido W. , 'Nonparametric Estimation of Average Treatment Effects Under Exogeneity: A Review,' The Review of Economics and Statistics , 2004, 86 (1), 4-29. and Donald B. Rubin , Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction , 1 edition ed., New York: Cambridge University Press, 2015.
  27. Janssen, Arnold , 'Studentized permutation tests for non-i.i.d. hypotheses and the generalized Behrens-Fisher problem,' Statistics & Probability Letters , 1997, 36 (1), 9-21.
  28. Ji, Xinyao, Gunther Fink, Paul Jacob Robyn, and Dylan S. Small , 'Randomization inference for stepped-wedge cluster-randomized trials: An application to community-based health insurance,' Annals of Applied Statistics , 2017, 11 (1), 1-20.
  29. Lechner, Michael , 'The Estimation of Causal Effects by Difference-in-Difference Methods,' Foundations and Trends in Econometrics , 2010, 4 (3), 165-224.
  30. Lei, Lihua and Peng Ding , 'Regression adjustment in completely randomized experiments with a diverging number of covariates,' Biometrika , 2021, 108 (4), 815-828.
  31. Li, Xinran and Peng Ding , 'General Forms of Finite Population Central Limit Theorems with Applications to Causal Inference,' Journal of the American Statistical Association , 2017, 112 (520), 1759-1769.
  32. Lin, Winston , 'Agnostic notes on regression adjustments to experimental data: Reexamining Freedman's critique,' Annals of Applied Statistics , 2013, 7 (1), 295-318.
  33. Lindner, Stephan and K John Mcconnell , 'Heterogeneous treatment effects and bias in the analysis of the stepped wedge design,' Health Services and Outcomes Research Methodology , 2021, 21 , 419-438.
  34. MacKinnon, James G and Matthew D Webb , 'Randomization inference for difference-indifferences with few treated clusters,' Journal of Econometrics , 2020, 218 (2), 435-450.
  35. Malani, Anup and Julian Reif , 'Interpreting pre-trends as anticipation: Impact on estimated treatment effects from tort reform,' Journal of Public Economics , 2015, 124 , 1-17.
  36. Manski, Charles F. and John V. Pepper , 'How Do Right-to-Carry Laws Affect Crime Rates? Coping with Ambiguity Using Bounded-Variation Assumptions,' Review of Economics and Statistics , 2018, 100 (2), 232-244.
  37. McKenzie, David , 'Beyond baseline and follow-up: The case for more T in experiments,' Journal of Development Economics , 2012, 99 (2), 210-221.
  38. Negi, Akanksha and Jeffrey M. Wooldridge , 'Revisiting regression adjustment in experiments with heterogeneous treatment effects,' Econometric Reviews , 2021, 40 (5), 504-534.
  39. Nekoei, Arash and David Seim , 'How Do Inheritances Shape Wealth Inequality? Theory and Evidence from Sweden,' The Review of Economic Studies , 2023, 90 (1), 463-498.
  40. Neyman, Jerzy , 'On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9,' Statistical Science , 1923, 5 (4), 465-472.
  41. Owens, Emily, David Weisburd, Karen L. Amendola, and Geoffrey P. Alpert , 'Can You Build a Better Cop?,' Criminology & Public Policy , 2018, 17 (1), 41-87.
  42. Parker, Jonathan A, Nicholas S Souleles, David S Johnson, and Robert McClelland , 'Consumer Spending and the Economic Stimulus Payments of 2008,' American Economic Review , 2013, 103 (6), 2530-2553.
  43. Rambachan, Ashesh and Jonathan Roth , 'Design-Based Uncertainty for Quasi-Experiments,' arXiv:2008.00602 [econ, stat] , August 2020. arXiv: 2008.00602. · arXiv
  44. Roth, Jonathan , 'Pretest with Caution: Event-Study Estimates after Testing for Parallel Trends,' American Economic Review: Insights , 2022, 4 (3), 305-322. and Pedro H. C. Sant'Anna , 'Efficient Estimation for Staggered Rollout Designs,' arXiv:2102.01291 [econ, math, stat] , June 2021. arXiv: 2102.01291. and , 'When Is Parallel Trends Sensitive to Functional Form?,' Econometrica , 2023, 91 (2), 737-747.
  45. Sant'Anna, Pedro H. C. and Jun Zhao , 'Doubly robust difference-in-differences estimators,' Journal of Econometrics , 2020, 219 (1), 101-122.
  46. Sekhon, Jasjeet S. and Yotam Shem-Tov , 'Inference on a New Class of Sample Average Treatment Effects,' Journal of the American Statistical Association , 2021, (534), 798-804.
  47. Shaikh, Azeem M. and Panos Toulis , 'Randomization Tests in Observational Studies With Staggered Adoption of Treatment,' Journal of the American Statistical Association , 2021, 116 (536), 1835-1848.
  48. Sun, Liyang and Sarah Abraham , 'Estimating dynamic treatment effects in event studies with heterogeneous treatment effects,' Journal of Econometrics , 2021, 225 (2), 175-199.
  49. Wan, Fei , 'Analyzing pre-post designs using the analysis of covariance models with and without the interaction term in a heterogeneous study population,' Statistical Methods in Medical Research , 2020, 29 (1), 189-204.
  50. Wood, George, Tom R. Tyler, and Andrew V. Papachristos , 'Procedural justice training reduces police use of force and complaints against officers,' Proceedings of the National Academy of Sciences , 2020, 117 (18), 9815-9821.
  51. Wu, Jason and Peng Ding , 'Randomization Tests for Weak Null Hypotheses in Randomized Experiments,' Journal of the American Statistical Association , 2021, 116 (536), 1898-1913.
  52. Xiong, Ruoxuan, Susan Athey, Mohsen Bayati, and Guido Imbens , 'Optimal Experimental Design for Staggered Rollouts,' arXiv:1911.03764 [econ, stat] , November 2019. arXiv: 1911.03764. · arXiv
  53. Yang, Li and Anastasios A Tsiatis , 'Efficiency Study of Estimators for a Treatment Effect in a Pretest-Posttest Trial,' The American Statistician , 2001, 55 (4), 314-321.
  54. Zhao, Anqi and Peng Ding , 'Covariate-adjusted Fisher randomization tests for the average treatment effect,' Journal of Econometrics , 2021, 225 (2), 278-294.

Authors
Roth, J., & Sant'Anna, P. H. C.
Affiliations
Brown University (US)
Venue
arXiv
Publisher
arXiv
Published
2021
Volume
1
Issue
4
Pages
669-709
ISSN
2832-9368 (ISSN-L)
DOI
10.48550/arxiv.2102.01291
arXiv
2102.01291
OpenAlex
W3127623389
Cited by
96
References
51
Citations by year
2026: 15 · 2025: 24 · 2024: 26 · 2023: 10 · 2022: 14 · 2021: 5 · 2020: 1 · 2018: 1
FWCI
15.23
Type
journal article
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • Journal of Political Economy Microeconomics (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Evaluation and Performance Assessment
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
Keywords
staggered treatment adoption, causal effect estimation, efficient estimators, plug-in estimator, asymptotic unbiasedness, permutation tests, police training programs, treatment timing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review