Randomization Tests Under an Approximate Symmetry Assumption
The DOI matches its Crossref record; 23 of the 24 references it cites were found in a registry or an index.
Abstract
This paper develops a theory of randomization tests under an approximate symmetry as-sumption. Randomization tests provide a general means of constructing tests that control size in finite samples whenever the distribution of the observed data exhibits symmetry under the null hypothesis. Here, by exhibits symmetry we mean that the distribution remains invariant under a group of transformations. In this paper, we provide conditions under which the same construction can be used to construct tests that asymptotically control the probability of a false rejection whenever the distribution of the observed data exhibits approximate symmetry in the sense that the limiting distribution of a function of the data exhibits symmetry under the null hypothesis. An important application of this idea is in settings where the data may be grouped into a fixed number of “clusters ” with a large number of observations within each cluster. In such settings, we show that the distribution of the observed data satisfies our ap-proximate symmetry requirement under weak assumptions. In particular, our results allow for the clusters to be heterogeneous and also have dependence not only within each cluster, but also across clusters. This approach enjoys several advantages over other approaches in these settings. Among other things, it leads to a test that is asymptotically similar, which, as shown in a simulation study, translates into improved power at many alternatives. Finally, we use our results to revisit the analysis of Angrist and Lavy (2009), who examine the impact of a cash award on exam performance for low-achievement students in Israel.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Canay, Romano and Shaikh (2017), Ibragimov and Müller (2016), Hagemann (2021), and Chernozhukov, Wüthrich and Zhu (2021) respectively propose permutation-based, t -test based, adjusted permutation-based, and conformal inference-based procedures that allow one to relax distributional assumptions about common shocks and accommodate richer forms of heterogeneity.”
From What's trending in difference-in-differences? A synthesis of the recent econometrics literature (2023) · cites this paper as Canay, Romano and Shaikh (2017)
In the citing paper
“The key restriction is that one is comfortable limiting the time-series dependence of the cluster-specific-shocks, and strengthening the parallel trends assumption to hold in many pre- and post-treatment time periods.”
From What's trending in difference-in-differences? A synthesis of the recent econometrics literature (2023) · cites this paper as Canay et al. (2017)
Citations
Cites
How Much Should We Trust Differences-in-Differences Estimates?
Bertrand, M., Duflo, E., & Mullainathan, S. (2004) · The Quarterly Journal of Economics · doi:10.1162/003355304772839588 · Cited by 10,641 · cited 1×
“CCE: This test is the one proposed by Bertrand et al. (2004).”
MethodTesting Statistical Hypotheses
(2021) · Wiley Series in Probability and Statistics · doi:10.1002/9781119243830.ch12 · Cited by 5,106
Bootstrap-Based Improvements for Inference With Clustered Errors
Cameron, A. C., Gelbach, J. B., & Miller, D. L. (2008) · Review of Economics and Statistics · doi:10.1162/rest.90.3.414 · Cited by 4,417 · cited 2×
“CGM: This test is the one proposed by Cameron et al. (2008) based on the wild bootstrap.”
Inference with “Difference in Differences” with a Small Number of Policy Changes
Conley, T. G., & Taber, C. R. (2011) · Review of Economics and Statistics · doi:10.1162/rest_a_00049 · Cited by 633 · cited 8×
“We first consider an asymptotic framework similar to the one in Conley and Taber (2011), where |J 1 | = q is fixed, |J 0 | → ∞, and min{|T 0 |, |T 1”
BackgroundMethodInfluentialA NEW ASYMPTOTIC THEORY FOR HETEROSKEDASTICITY-AUTOCORRELATION ROBUST TESTS
Kiefer, N. M., & Vogelsang, T. J. (2005) · Econometric Theory · doi:10.1017/s0266466605050565 · Cited by 419 · cited 1×
“Bester et al. (2011) and Ibragimov and Müller (2010) also find that their tests outperform the test proposed by Kiefer and Vogelsang (2002, 2005), in which the 1 − α2 quantile of the N(0, 1) distribution is replaced with an alternative critical value that does not require q →∞.”
BackgroundThe Large-Sample Power of Tests Based on Permutations of Observations
Hoeffding, W. (1952) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729436 · Cited by 365 · cited 2×
“Earlier work on the asymptotic behavior of randomization tests includes Hoeffding (1952), Romano (1989) and Romano (1990).”
BackgroundBias reduction in standard errors for linear regression with multi-stage samples
McCaffrey, D. F., & Bell, R. M. (2003) · Quality Engineering · Cited by 318
t-Statistic Based Correlation and Heterogeneity Robust Inference
Ibragimov, R., & Müller, U. K. (2010) · Journal of Business & Economic Statistics · doi:10.1198/jbes.2009.08046 · Cited by 299 · cited 13×
“With the exception of the t-test approach developed in Ibragimov and Müller (2010), the competing approaches we discuss in Section 5.1 below do not share this feature.”
BackgroundMethodResultInfluentialThe Effects of High Stakes High School Achievement Awards: Evidence from a Randomized Trial
Angrist, J., & Lavy, V. (2009) · American Economic Review · doi:10.1257/aer.99.4.1384 · Cited by 270 · cited 10×
“The bias correction is highlighted by Angrist and Pischke (2008, page 320) and is used by Angrist and Lavy (2009) without the degrees of freedom adjustment.”
InfluentialInference with dependent data using cluster covariance estimators
Bester, C. A., Conley, T. G., & Hansen, C. B. (2011) · Journal of Econometrics · doi:10.1016/j.jeconom.2011.01.007 · Cited by 258 · cited 7×
“By replacing the time index t with a vector index, as in Bester et al. (2011), we can accommodate more complicated dependence structures, such as those found in spatially dependent data or in panel data.”
BackgroundMethodResultInfluential
Cited by
Items 101–126 of 126
Inference in Differences-in-Differences: How Much Should We Trust in Independent Clusters?
Ferman, B. (2019) · Munich Personal RePEc Archive (Ludwig Maximilian University of Munich)
Dependence-Robust Inference Using Randomized SubsamplingPreprint
Leung, M. P. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3036626
Some Finite Sample Properties of the Sign TestPreprint
Cai, Y. (2021) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.2103.01412
“Such as approach has been fruitful in for inference for cluster-dependent data with few clusters (Canay et al. 2017; 2019) and specification tests for RDD involving few units near the cut-off (Canay and Kamat 2018; Bugni and Canay 2021), among others.”
BackgroundMethodInfluentialPelican, A., & Graham, B. S. (2022) · doi:10.47004/wp.cem.2022.1322
Robust Inference on Income Inequality: \textit{t-}Statistic Based ApproachesPreprint
Ibragimov, R., Kattuman, P. A., & Андреевич, С. А. (2021) · arXiv (Cornell University)
Panel Data with Unknown ClustersPreprint
Cai, Y. (2021) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.2106.05503
“When q is small, the approximate randomization tests of Canay et al. (2017) controls size.”
MethodInfluentialLiu, W., Bretz, F., & Holger Dette, A. (2020) · ePrints Soton (University of Southampton)
Inference with Correlated ClustersPreprint
Powell, D. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2786125
Using Multiple Outcomes to Adjust Standard Errors for Spatial CorrelationPreprint
DellaVigna, S., Imbens, G. W., Kim, W., & Ritzwoller, D. M. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5233074 · Cited by 1
Robust IV inference with clustering dependence
Cao, J. (2025) · Econometrics Journal · doi:10.1093/ectj/utaf021
“Ibragimov and Müller (2010), Canay et al. (2017), Cao et al. (2019), and Hagemann (2019a,b) have documented the robustness and good power properties of this approach.”
BackgroundComparison of Cuscore and EWMA in Detecting Mean Shifts of Lévy Stable Processes
Hu, S., & Li, T. (2025) · IEEE Advanced Information Technology, Electronic and Automation Control Conference (IAEAC) · doi:10.1109/iaeac65194.2025.11165991
Randomization tests for conditional group symmetry
Chiu, K., Sharp, A., & Bloem-Reddy, B. (2025) · Electronic Journal of Statistics · doi:10.1214/25-ejs2473
Gradient Wild Bootstrap for Instrumental Variable Quantile Regressions with Weak and Few Clusters
Wang, W., & Zhang, Y. (2026) · Journal of Business and Economic Statistics · doi:10.1080/07350015.2026.2618311
Drachal, K., & Tabash, M. I. (2026) · International May Conference on Strategic Management (IMCSM) · doi:10.5937/imcsm26407d
Chen, J. C., Guo, Y., & Wu, Y. (2026) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2026.148879
Cohort-Based Regression Discontinuity Designs with Fixed-Quota CutoffsPreprint
Zhuang, Z. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7006558
The Year-end Toll: Evidence from the Option Funding BasisPreprint
Shin, U. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7182178
Climate Risk, Artificial Intelligence, and Regional Industrial Chain Resilience
Wang, S., Ren, J., Peng, W., & Chen, T. (2026) · Sustainability · doi:10.3390/su18178752
Default Assumptions as Hidden Parameters: A Theory of Implicit Priors in Materials AI
Mansour, A., & Saeed, O. (2022) · Journal of Artificial Intelligence for Materials Science · doi:10.68159/j862290346
The Year-End Toll: Frictions Embedded in Option-Implied RatesPreprint
Shin, U. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.20224
Lee, J., Hsu, T.-W., Yang, H. J., & Lee, J. (2026) · Buildings · doi:10.3390/buildings16193777
Shi, Y., Liu, Y., Du, G., Liu, H., & Feng, S. (2026) · IEEE Access · doi:10.1109/access.2026.3736562
“We enumerated all 2 18 sign assignments for a two-sided exact family-level randomization test [21].”
Distributional Difference-in-Differences: Aggregation Before or After Quantile Inversion?Preprint
Hounyo, U. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.27944
A general randomized test for Alpha
Massacci, D., Sarno, L., Trapani, L., & Vallarino, P. (2026) · Journal of the American Statistical Association · doi:10.1080/01621459.2026.2735067
Bandyopadhyay, A. P. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7530080
Guerrini, A. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7669848
Metadata
- Authors
- Canay, I. A., Romano, J. P., & Shaikh, A. M.
- Venue
- Econometrica
- Publisher
- The Econometric Society
- Published
- 2017
- Volume
- 85
- Issue
- 3
- Pages
- 1013-1030
- ISSN
- 0012-9682 (ISSN-L)
- OpenAlex
- W2188869357
- Cited by
- 125
- References
- 24
- Citations by year
- 2026: 17 · 2025: 12 · 2024: 8 · 2023: 12 · 2022: 12 · 2021: 15 · 2020: 13 · 2019: 12 · 2018: 7 · 2017: 11
- FWCI
- 8.49
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Econometrica (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Spatial and Panel Data Analysis
- Keywords
- permutation tests, approximate symmetries, finite-sample inference, cluster randomization, heterogeneous clusters, statistical power, exam performance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.