Randomization Tests Under an Approximate Symmetry Assumption
The DOI matches its Crossref record; 23 of the 24 references it cites were found in a registry or an index.
Abstract
This paper develops a theory of randomization tests under an approximate symmetry as-sumption. Randomization tests provide a general means of constructing tests that control size in finite samples whenever the distribution of the observed data exhibits symmetry under the null hypothesis. Here, by exhibits symmetry we mean that the distribution remains invariant under a group of transformations. In this paper, we provide conditions under which the same construction can be used to construct tests that asymptotically control the probability of a false rejection whenever the distribution of the observed data exhibits approximate symmetry in the sense that the limiting distribution of a function of the data exhibits symmetry under the null hypothesis. An important application of this idea is in settings where the data may be grouped into a fixed number of “clusters ” with a large number of observations within each cluster. In such settings, we show that the distribution of the observed data satisfies our ap-proximate symmetry requirement under weak assumptions. In particular, our results allow for the clusters to be heterogeneous and also have dependence not only within each cluster, but also across clusters. This approach enjoys several advantages over other approaches in these settings. Among other things, it leads to a test that is asymptotically similar, which, as shown in a simulation study, translates into improved power at many alternatives. Finally, we use our results to revisit the analysis of Angrist and Lavy (2009), who examine the impact of a cash award on exam performance for low-achievement students in Israel.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Canay, Romano and Shaikh (2017), Ibragimov and Müller (2016), Hagemann (2021), and Chernozhukov, Wüthrich and Zhu (2021) respectively propose permutation-based, t -test based, adjusted permutation-based, and conformal inference-based procedures that allow one to relax distributional assumptions about common shocks and accommodate richer forms of heterogeneity.”
From What's trending in difference-in-differences? A synthesis of the recent econometrics literature (2023) · cites this paper as Canay, Romano and Shaikh (2017)
In the citing paper
“The key restriction is that one is comfortable limiting the time-series dependence of the cluster-specific-shocks, and strengthening the parallel trends assumption to hold in many pre- and post-treatment time periods.”
From What's trending in difference-in-differences? A synthesis of the recent econometrics literature (2023) · cites this paper as Canay et al. (2017)
Citations
Cites
How Much Should We Trust Differences-in-Differences Estimates?
Bertrand, M., Duflo, E., & Mullainathan, S. (2004) · The Quarterly Journal of Economics · doi:10.1162/003355304772839588 · Cited by 10,641 · cited 1×
“CCE: This test is the one proposed by Bertrand et al. (2004).”
MethodTesting Statistical Hypotheses
(2021) · Wiley Series in Probability and Statistics · doi:10.1002/9781119243830.ch12 · Cited by 5,106
Bootstrap-Based Improvements for Inference With Clustered Errors
Cameron, A. C., Gelbach, J. B., & Miller, D. L. (2008) · Review of Economics and Statistics · doi:10.1162/rest.90.3.414 · Cited by 4,417 · cited 2×
“CGM: This test is the one proposed by Cameron et al. (2008) based on the wild bootstrap.”
Inference with “Difference in Differences” with a Small Number of Policy Changes
Conley, T. G., & Taber, C. R. (2011) · Review of Economics and Statistics · doi:10.1162/rest_a_00049 · Cited by 633 · cited 8×
“We first consider an asymptotic framework similar to the one in Conley and Taber (2011), where |J 1 | = q is fixed, |J 0 | → ∞, and min{|T 0 |, |T 1”
BackgroundMethodInfluentialA NEW ASYMPTOTIC THEORY FOR HETEROSKEDASTICITY-AUTOCORRELATION ROBUST TESTS
Kiefer, N. M., & Vogelsang, T. J. (2005) · Econometric Theory · doi:10.1017/s0266466605050565 · Cited by 419 · cited 1×
“Bester et al. (2011) and Ibragimov and Müller (2010) also find that their tests outperform the test proposed by Kiefer and Vogelsang (2002, 2005), in which the 1 − α2 quantile of the N(0, 1) distribution is replaced with an alternative critical value that does not require q →∞.”
BackgroundThe Large-Sample Power of Tests Based on Permutations of Observations
Hoeffding, W. (1952) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729436 · Cited by 365 · cited 2×
“Earlier work on the asymptotic behavior of randomization tests includes Hoeffding (1952), Romano (1989) and Romano (1990).”
BackgroundBias reduction in standard errors for linear regression with multi-stage samples
McCaffrey, D. F., & Bell, R. M. (2003) · Quality Engineering · Cited by 318
t-Statistic Based Correlation and Heterogeneity Robust Inference
Ibragimov, R., & Müller, U. K. (2010) · Journal of Business & Economic Statistics · doi:10.1198/jbes.2009.08046 · Cited by 299 · cited 13×
“With the exception of the t-test approach developed in Ibragimov and Müller (2010), the competing approaches we discuss in Section 5.1 below do not share this feature.”
BackgroundMethodResultInfluentialThe Effects of High Stakes High School Achievement Awards: Evidence from a Randomized Trial
Angrist, J., & Lavy, V. (2009) · American Economic Review · doi:10.1257/aer.99.4.1384 · Cited by 270 · cited 10×
“The bias correction is highlighted by Angrist and Pischke (2008, page 320) and is used by Angrist and Lavy (2009) without the degrees of freedom adjustment.”
InfluentialInference with dependent data using cluster covariance estimators
Bester, C. A., Conley, T. G., & Hansen, C. B. (2011) · Journal of Econometrics · doi:10.1016/j.jeconom.2011.01.007 · Cited by 258 · cited 7×
“By replacing the time index t with a vector index, as in Bester et al. (2011), we can accommodate more complicated dependence structures, such as those found in spatially dependent data or in panel data.”
BackgroundMethodResultInfluential
Cited by
What's trending in difference-in-differences? A synthesis of the recent econometrics literature
Roth, J., Sant’Anna, P. H. C., Bilinski, A., & Poe, J. (2023) · Journal of Econometrics · doi:10.1016/j.jeconom.2023.03.008 · Cited by 2,027
“When in fact the number of time periods is small, as frequently occurs in DiD applications, one can still use some of these methods, but the underlying assumptions are stronger — see, e.g., Remark 4.5 and Section 4.2 of Canay et al. (2017).”
InfluentialCluster-robust inference: A guide to empirical practice
MacKinnon, J. G., Nielsen, M. Ø., & Webb, M. D. (2023) · Journal of Econometrics · doi:10.1016/j.jeconom.2022.04.001 · Cited by 407
“The results of Canay et al. (2021) and MacKinnon and Webb (2020b) strongly suggest that homogeneity across clusters is more important for RI than for the”
BackgroundRandomization inference for difference-in-differences with few treated clusters
MacKinnon, J. G., & Webb, M. D. (2020) · Journal of Econometrics · doi:10.1016/j.jeconom.2020.04.024 · Cited by 167
Inference in Differences-in-Differences with Few Treated Groups and Heteroskedasticity
Ferman, B., & Pinto, C. (2019) · The Review of Economics and Statistics · doi:10.1162/rest_a_00759 · Cited by 178
Practical and Effective Approaches to Dealing With Clustered Data
Esarey, J., & Menger, A. (2019) · Political Science Research and Methods · doi:10.1017/psrm.2017.42 · Cited by 141
Metadata
- Authors
- Canay, I. A., Romano, J. P., & Shaikh, A. M.
- Venue
- Econometrica
- Publisher
- The Econometric Society
- Published
- 2017
- Volume
- 85
- Issue
- 3
- Pages
- 1013-1030
- ISSN
- 0012-9682 (ISSN-L)
- OpenAlex
- W2188869357
- Cited by
- 125
- References
- 24
- Citations by year
- 2026: 17 · 2025: 12 · 2024: 8 · 2023: 12 · 2022: 12 · 2021: 15 · 2020: 13 · 2019: 12 · 2018: 7 · 2017: 11
- FWCI
- 8.49
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Econometrica (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Spatial and Panel Data Analysis
- Keywords
- permutation tests, approximate symmetries, finite-sample inference, cluster randomization, heterogeneous clusters, statistical power, exam performance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.