Panel experiments and dynamic causal effects: A finite population perspective
Open accessThe DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index.
Abstract
In panel experiments, we randomly assign units to different interventions, measuring their outcomes, and repeating the procedure in several periods. Using the potential outcomes framework, we define finite population dynamic causal effects that capture the relative effectiveness of alternative treatment paths. For a rich class of dynamic causal effects, we provide a nonparametric estimator that is unbiased over the randomization distribution and derive its finite population limiting distribution as either the sample size or the duration of the experiment increases. We develop two methods for inference: a conservative test for weak null hypotheses and an exact randomization test for sharp null hypotheses. We further analyze the finite population probability limit of linear fixed effects estimators. These commonly‐used estimators do not recover a causally interpretable estimand if there are dynamic causal effects and serial correlation in the assignments, highlighting the value of our proposed estimator.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“We note that, as described in Bojinov, Rambachan and Shephard (2021), the no carryover assumption may be restrictive in many settings”
From What's trending in difference-in-differences? A synthesis of the recent econometrics literature (2023) · cites this paper as Bojinov, Rambachan and Shephard (2021)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Estimating causal effects of treatments in randomized and nonrandomized studies
Rubin, D. B. (1974) · Journal of Educational Psychology · doi:10.1037/h0037350 · Cited by 9,436 · cited 2×
“Combining this assumption with a further requirement that every unit receives the same version of the treatment is often referred to as the “Stable Unit Treatment Value Assumption,” in cross-sectional applications (Rubin, 1980) or the “Temporal Stable Unit Treatment Value Assumption” in time series experiments (Bojino…”
MethodDifference-in-differences with variation in treatment timing
Goodman-Bacon, A. (2021) · Journal of Econometrics · doi:10.1016/j.jeconom.2021.03.014 · Cited by 8,046 · cited 1×
“Boryusak and Jaravel (2017), Athey and Imbens (2018), and Goodman-Bacon (2018) also provided a decomposition of the two-way fixed effects estimand in staggered designs.”
Statistics and Causal Inference
Holland, P. W. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478354 · Cited by 5,293 · cited 1×
“In this section, we develop a nonparametric Horvitz and Thompson (1952) type estimator of the i, tth lag-p dynamic causal effects and derive its properties.”
MethodImpulse response analysis in nonlinear multivariate models
Koop, G., Pesaran, M. H., & Potter, S. M. (1996) · Journal of Econometrics · doi:10.1016/0304-4076(95)01753-4 · Cited by 5,049
Two-Way Fixed Effects Estimators with Heterogeneous Treatment Effects
de Chaisemartin, C., & D’Haultfœuille, X. (2020) · American Economic Review · doi:10.1257/aer.20181169 · Cited by 5,031
A generalization of sampling without replacement from a finite universe
Horvitz, D. G., & Thompson, D. J. (1952) · Journal of the American Statistical Association · doi:10.1080/01621459.1952.10483446 · Cited by 4,293 · cited 1×
“3 Nonparametric estimation and inference We now develop a nonparametric Horvitz and Thompson (1952) type estimator of the i, t-th lag-p dynamic causal effects.”
MethodCausal Inference for Statistics, Social, and Biomedical Sciences: An Introduction
Imbens, G. W., & Rubin, D. B. (2015) · doi:10.1017/cbo9781139025751 · Cited by 4,210
A Generalization of Sampling Without Replacement From a Finite Universe
Horvitz, D. G., & Thompson, D. J. (1952) · Journal of the American Statistical Association · doi:10.2307/2280784 · Cited by 3,011
Martingale Limit Theory and its Application.
Loynes, R. M., Hall, P., & Heyde, C. C. (1984) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2981599 · Cited by 2,685
The Production of Human Capital and the Life Cycle of Earnings
Ben-Porath, Y. (1967) · Journal of Political Economy · doi:10.1086/259291 · Cited by 2,657
A contextual-bandit approach to personalized news article recommendation
Li, L., Chu, W., Langford, J., & Schapire, R. E. (2010) · Proceedings of the 19th international conference on World wide web · doi:10.1145/1772690.1772758 · Cited by 2,562 · cited 2×
“Such experiments are widely used by technology companies to quickly discern user preferences in recommendation algorithms (Li et al. (2010), Li, Karatzoglou, and Gentile (2016)) and by academics interested in improving their power against a particular hypothesis (van der Laan Mark (2008)).”
MethodRobins, J. (1986) · Mathematical Modelling · doi:10.1016/0270-0255(86)90088-6 · Cited by 2,451 · cited 4×
“…are no treatment spillovers across units (Cox (1958)).3 3The idea of defining potential outcomes as a function of assignment paths first appears in Robins (1986) and has been further developed in subsequent work such as Robins (1994), Robins, Greenland, and Hu (1999), Murphy et al. (2001),…”
BackgroundInfluentialAsymptotically efficient adaptive allocation rules
Lai, T. L., & Robbins, H. (1985) · Advances in Applied Mathematics · doi:10.1016/0196-8858(85)90002-8 · Cited by 2,406
Some aspects of the sequential design of experiments
Robbins, H. (1952) · Bulletin of the American Mathematical Society · doi:10.1090/s0002-9904-1952-09620-8 · Cited by 2,184
On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9
Splawa-Neyman, J., Dabrowska, D. M., & Speed, T. P. (1990) · Statistical Science · doi:10.1214/ss/1177012031 · Cited by 1,627
Estimating the Technology of Cognitive and Noncognitive Skill Formation
(2010) · Econometrica · doi:10.3982/ecta6551 · Cited by 1,603
Experimental methods: Between-subject and within-subject design
Charness, G., Gneezy, U., & Kuhn, M. A. (2012) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2011.08.009 · Cited by 1,469
Estimating the Returns to Schooling: Some Econometric Problems
Griliches, Z. (1977) · Econometrica · doi:10.2307/1913285 · Cited by 1,390
When Should You Adjust Standard Errors for Clustering?
Abadie, A., Athey, S., Imbens, G., & Wooldridge, J. (2017) · National Bureau of Economic Research · doi:10.3386/w24003 · Cited by 1,360 · cited 7×
“, Abadie et al. (2017), Athey and Imbens (2018), Athey et al. (2018), Imai and Kim (2019), Arkhangelsky and Imbens (2019), Imai and Kim (2020), de Chaisemartin and D’Haultfoeuille (2020)).”
BackgroundInfluential(1958) · Medical Entomology and Zoology · Cited by 1,059
Optimal Dynamic Treatment Regimes
Murphy, S. A. (2003) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/1467-9868.00389 · Cited by 1,007
Design-based analysis in Difference-In-Differences settings with staggered adoption
Athey, S., & Imbens, G. W. (2022) · Journal of Econometrics · doi:10.1016/j.jeconom.2020.10.012 · Cited by 917 · cited 5×
“Similarly, Athey and Imbens (2018), Athey et al. (2018) and Arkhangelsky and Imbens (2019) also introduce a potential outcome model for panel data, but assume away carryover effects.”
BackgroundInfluentialCausal Inference for Statistics, Social, and Biomedical Sciences: An Introduction
Imbens, G. W., & Rubin, D. B. (2015) · Thammasat University Digital Collections · Cited by 838
Difference-in-Differences with Variation in Treatment Timing
Goodman-Bacon, A. (2018) · doi:10.3386/w25018 · Cited by 757
The n-of-1 clinical trial: the ultimate strategy for individualizing medicine?
Lillie, E. O., Patay, B., Diamant, J., Issell, B., Topol, E. J., & Schork, N. J. (2011) · Personalized Medicine · doi:10.2217/pme.11.7 · Cited by 706
On the Use of Two-Way Fixed Effects Regression Models for Causal Inference with Panel Data
Imai, K., & Kim, I. S. (2021) · Political Analysis · doi:10.1017/pan.2020.33 · Cited by 623 · cited 1×
“For example, Sobel (2006), Athey and Imbens (2018), and Imai and Kim (2019a) study β̂TWFE in the causal literature on panel data models.”
BackgroundCorrecting for non-compliance in randomized trials using structural nested mean models
Robins, J. M. (1994) · Communications in Statistics - Theory and Methods · doi:10.1080/03610929408831393 · Cited by 543
Matrix Completion Methods for Causal Panel Data Models
Athey, S., Bayati, M., Doudchenko, N., Imbens, G., & Khosravi, K. (2018) · doi:10.3386/w25132 · Cited by 497
Revisiting Event Study Designs
Borusyak, K., & Jaravel, X. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2826228 · Cited by 463
On regression adjustments to experimental data
Freedman, D. A. (2008) · Advances in Applied Mathematics · doi:10.1016/j.aam.2006.12.003 · Cited by 417
Marginal Mean Models for Dynamic Regimes
Murphy, S. A., van der Laan, M. J., Robins, J. M., & Conduct Problems Prevention Research Group (2001) · Journal of the American Statistical Association · doi:10.1198/016214501753382327 · Cited by 393 · cited 3×
“The idea of defining potential outcomes as a function of assignment paths first appears in Robins (1986) and has been further developed in subsequent work such as Robins (1994), Robins et al. (1999), Murphy et al.”
BackgroundInfluentialWhen Should We Use Unit Fixed Effects Regression Models for Causal Inference with Longitudinal Data?
Imai, K., & Kim, I. S. (2019) · American Journal of Political Science · doi:10.1111/ajps.12417 · Cited by 369 · cited 1×
“1 is related to yet crucially different from results in Imai and Kim (2019), which show that the unit fixed effects estimator recover a weighted average of unit-specific contemporaneous causal effects if there are no carryover effects.”
ResultWooldridge, J. M. (2005) · Review of Economics and Statistics · doi:10.1162/0034653053970320 · Cited by 363 · cited 1×
“We adapt notation used in Wooldridge (2005) for analyzing panel fixed effects models.”
MethodAronow, P. M., & Samii, C. (2017) · The Annals of Applied Statistics · doi:10.1214/16-aoas1005 · Cited by 347 · cited 1×
“To derive the asymptotic distribution of causal estimators in such settings, researchers typically require the assignment mechanism to be independent (Chin (2018)) or at least have only limited dependence structure across units (Aronow and Samii (2017)).”
BackgroundCollaborative Filtering Bandits
Li, S., Karatzoglou, A., & Gentile, C. (2016) · Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval · doi:10.1145/2911451.2911548 · Cited by 287
Sampling‐Based versus Design‐Based Uncertainty in Regression Analysis
Abadie, A., Athey, S., Imbens, G. W., & Wooldridge, J. M. (2020) · Econometrica · doi:10.3982/ecta12675 · Cited by 285
Average treatment effects in the presence of unknown interference
Sävje, F., Aronow, P. M., & Hudgens, M. G. (2021) · The Annals of Statistics · doi:10.1214/20-aos1973 · Cited by 272
The Randomization Theory of Experimental Inference*
Kempthorne, O. (1955) · Journal of the American Statistical Association · doi:10.1080/01621459.1955.10501979 · Cited by 258 · cited 1×
“Our approach builds on the potential outcomes formulation of causal inference (Neyman, 1923; Kempthorne, 1955; Cox, 1958; Rubin, 1974) and takes a purely design-based perspective on uncertainty, allowing us to be agnostic to the outcomes model and avoid assumptions about hypothetical super-populations.”
MethodEarnings and Consumption Dynamics: A Nonlinear Panel Data Framework
Arellano, M., Blundell, R., & Bonhomme, S. (2017) · Econometrica · doi:10.3982/ecta13795 · Cited by 254
Robins, J. M., Greenland, S., & Hu, F.-C. (1999) · Journal of the American Statistical Association · doi:10.1080/01621459.1999.10474168 · Cited by 242
Confidence intervals for policy evaluation in adaptive experiments
Hadad, V., Hirshberg, D. A., Zhan, R., Wager, S., & Athey, S. C. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2014602118 · Cited by 242 · cited 3×
“There has been a growing interest in drawing causal inferences based on the collected data in such adaptive experimental designs (Hadad et al. (2021), Zhang, Janson, and Murphy (2020)).”
Time Series Experiments and Causal Estimands: Exact Randomization Tests and Trading
Bojinov, I., & Shephard, N. (2019) · Journal of the American Statistical Association · doi:10.1080/01621459.2018.1527225 · Cited by 203 · cited 5×
“8In a time series experiment with N = 1, Bojinov and Shephard (2019) introduced defining causal effects that depend on the observed assignment path because most potential outcomes are unobserved since there is only one experimental unit in their setting.”
BackgroundMethodInfluentialSelf-citationEstimating Dynamic Treatment Effects in Event Studies with Heterogeneous Treatment Effects
(2018) · doi:10.48550/arxiv.1804.05785 · Cited by 189
Modelling Income Processes with Lots of Heterogeneity
BROWNING, M., EJRNAES, M., & ALVAREZ, J. (2010) · Review of Economic Studies · doi:10.1111/j.1467-937x.2010.00612.x · Cited by 167
General Forms of Finite Population Central Limit Theorems with Applications to Causal Inference
Li, X., & Ding, P. (2017) · Journal of the American Statistical Association · doi:10.1080/01621459.2017.1295865 · Cited by 167 · cited 1×
“As discussed in Li and Ding (2017), this assumption can often be replaced by a finite-population analogue of the Lindeberg condition in analyses of cross-sectional, randomized experiments.”
BackgroundHow to Make Causal Inferences with Time-Series Cross-Sectional Data Under Selection on Observables
Blackwell, M., & Glynn, A. (2018) · American Political Science Review · doi:10.1017/s0003055418000357 · Cited by 155
Assessing time-varying causal effect moderation in mobile health
Boruvka, A., Almirall, D., Witkiewitz, K., & Murphy, S. A. (2018) · Journal of the American Statistical Association · doi:10.1080/01621459.2017.1305274 · Cited by 152 · cited 1×
“For example, in the mental healthcare digital experiments of Boruvka et al. (2018), it is compelling to use sampling-based arguments as the experimental units are drawn from a larger group of patients for whom we wish to make inference on as, if successful, the technology will be broadly rolled out.”
BackgroundZhang, K. W., Janson, L., & Murphy, S. A. (2020) · PubMed · doi:10.48550/arxiv.2002.03217 · Cited by 144
Arellano, M., & Bonhomme, S. (2011) · Annual Review of Economics · doi:10.1146/annurev-economics-111809-125139 · Cited by 135 · cited 1×
“, see Arellano and Bonhomme (2016), Arellano, Blundell, and Bonhomme (2017), and the review in Arellano and Bonhomme (2012)).”
BackgroundEstimating Dynamic Treatment Effects in Event Studies With Heterogeneous Treatment Effects
Sun, L., & Abraham, S. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3158747 · Cited by 118
The Dozen Things Experimental Economists Should Do (More of)
Czibor, E., Jimenez‐Gomez, D., & List, J. A. (2019) · Southern Economic Journal · doi:10.1002/soej.12392 · Cited by 116
Andreoni, J., & Samuelson, L. (2005) · Journal of Economic Theory · doi:10.1016/j.jet.2004.09.002 · Cited by 113
Nonlinear panel data estimation via quantile regressions
Arellano, M., & Bonhomme, S. (2016) · Econometrics Journal · doi:10.1111/ectj.12062 · Cited by 97
Heckman, J. J., Humphries, J. E., & Veramendi, G. (2016) · Journal of Econometrics · doi:10.1016/j.jeconom.2015.12.001 · Cited by 80 · cited 1×
“Heckman, Humphries, and Veramendi (2016), Hull (2018), and Han (2019) considered a potential outcome model similar to ours but again rely on superpopulation arguments to perform inference.”
BackgroundEstimating Treatment Effects in Mover DesignsPreprint
Hull, P. (2018) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.1804.06721 · Cited by 51
Simulating power of economic experiments: the powerBBK package
Bellemare, C., Bissonnette, L., & Kröger, S. (2016) · Journal of the Economic Science Association · doi:10.1007/s40881-016-0028-4 · Cited by 50
Central limit theorems via Stein's method for randomized experiments under interferencePreprint
Chin, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.03105 · Cited by 40
The Construction and Analysis of Adaptive Group Sequential Designs
Laan, V. D. P. P., & Mark, J. W. (2008) · Cited by 37
Statistical Power of within and Between-Subjects Designs in Economic ExperimentsPreprint
Bellemare, C., Bissonnette, L., & Kröger, S. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3149007 · Cited by 27
Double-Robust Identification for Causal Panel Data Models
Arkhangelsky, D., & Imbens, G. W. (2021) · National Bureau of Economic Research · doi:10.3386/w28364 · Cited by 20
Design and Analysis of Switchback ExperimentsPreprint
Bojinov, I., Simchi‐Levi, D., & Zhao, J. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3684168 · Cited by 15 · cited 1×
“(2001), Boruvka et al. (2018), and Blackwell and Glynn (2018).”
Self-citationStatistical Power of within and Between-Subjects Designs in Economic ExperimentsPreprint
Bellemare, C., Bissonnette, L., & Kröger, S. (2014) · SSRN Electronic Journal · doi:10.2139/ssrn.2517886 · Cited by 13
When do common time series estimands have nonparametric causal meaning?Preprint
Rambachan, A., & Shephard, N. (2019) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.1903.01637 · Cited by 12
Self-citationRambachan, A., & Shephard, N. (2019) · RePEc: Research Papers in Economics · Cited by 10
Self-citationIdentification in Nonparametric Models for Dynamic Treatment EffectsPreprint
Han, S. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3182800 · Cited by 9
Design and Analysis of Switchback ExperimentsPreprint
Bojinov, I., Simchi‐Levi, D., & Zhao, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.00148 · Cited by 7 · cited 1×
“(2001), Boruvka et al. (2018), and Blackwell and Glynn (2018).”
Self-citationTime Series Experiments and Causal Estimands
Bojinov, I., & Shephard, N. (2019) · Cited by 1
Self-citationDoubly Robust Identification for Causal Panel Data ModelsPreprint
Arkhangelsky, D., & Imbens, G. W. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.09412 · Cited by 1
Research works printed but not linked: 7
- Bojinov, 2020, Harvard Business Review, 98, 48
- Cunha, 2006, Handbook of the Economics of Education, 1, 697
- Han, 2019, Journal of Econometrics
- Imai, 2020, Political Analysis
- Kempthorne, 1955, Journal of the American Statistical Association, 50, 946
- Li, 2016, Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, 539
- Neyman, 1923, Statistical Science, 5, 465
Cited by
Difference-in-Differences with multiple time periods
Callaway, B., & Sant’Anna, P. H. C. (2021) · Journal of Econometrics · doi:10.1016/j.jeconom.2020.12.001 · Cited by 9,126
What's trending in difference-in-differences? A synthesis of the recent econometrics literature
Roth, J., Sant’Anna, P. H. C., Bilinski, A., & Poe, J. (2023) · Journal of Econometrics · doi:10.1016/j.jeconom.2023.03.008 · Cited by 2,027
“We note that, as described in Bojinov et al. (2021), the no carryover assumption may be restrictive in many settings — for example, if the treatment is a raise in the minimum wage and the outcome is employment, then the no carryover assumption requires that employment in period t depends only on…”
Difference-in-Differences Estimators of Intertemporal Treatment Effects
de Chaisemartin, C., & D’Haultfœuille, X. (2026) · Review of Economics and Statistics · doi:10.1162/rest_a_01414 · Cited by 740
Two-way fixed effects and differences-in-differences with heterogeneous treatment effects: a survey
de Chaisemartin, C., & D’Haultfœuille, X. (2023) · The Econometrics Journal · doi:10.1093/ectj/utac017 · Cited by 522
Two-way fixed effects and differences-in-differences estimators with several treatments
de Chaisemartin, C., & D’Haultfœuille, X. (2023) · Journal of Econometrics · doi:10.1016/j.jeconom.2023.105480 · Cited by 188
Metadata
- Authors
- Bojinov, I., Rambachan, A., & Shephard, N.
- Affiliations
- Harvard University (US)
- Venue
- Quantitative Economics
- Publisher
- The Econometric Society
- Published
- 2021
- Volume
- 12
- Issue
- 4
- Pages
- 1171-1196
- ISSN
- 1759-7323 (ISSN-L)
- OpenAlex
- W3012766575
- Cited by
- 60
- References
- 66
- Citations by year
- 2026: 5 · 2025: 8 · 2024: 8 · 2023: 10 · 2022: 19 · 2021: 6
- FWCI
- 8.04
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Quantitative Economics (publishedVersion, cc-by-nc)
- arXiv (Cornell University) (submittedVersion)
- Econstor (Econstor) (submittedVersion, cc-by-nc)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1745303)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Spatial and Panel Data Analysis
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Keywords
- finite population inference, potential outcomes, nonparametric estimators, permutation tests
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.