Randomization Tests Under an Approximate Symmetry Assumption
The DOI matches its Crossref record; 23 of the 24 references it cites were found in a registry or an index.
Abstract
This paper develops a theory of randomization tests under an approximate symmetry as-sumption. Randomization tests provide a general means of constructing tests that control size in finite samples whenever the distribution of the observed data exhibits symmetry under the null hypothesis. Here, by exhibits symmetry we mean that the distribution remains invariant under a group of transformations. In this paper, we provide conditions under which the same construction can be used to construct tests that asymptotically control the probability of a false rejection whenever the distribution of the observed data exhibits approximate symmetry in the sense that the limiting distribution of a function of the data exhibits symmetry under the null hypothesis. An important application of this idea is in settings where the data may be grouped into a fixed number of “clusters ” with a large number of observations within each cluster. In such settings, we show that the distribution of the observed data satisfies our ap-proximate symmetry requirement under weak assumptions. In particular, our results allow for the clusters to be heterogeneous and also have dependence not only within each cluster, but also across clusters. This approach enjoys several advantages over other approaches in these settings. Among other things, it leads to a test that is asymptotically similar, which, as shown in a simulation study, translates into improved power at many alternatives. Finally, we use our results to revisit the analysis of Angrist and Lavy (2009), who examine the impact of a cash award on exam performance for low-achievement students in Israel.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Canay, Romano and Shaikh (2017), Ibragimov and Müller (2016), Hagemann (2021), and Chernozhukov, Wüthrich and Zhu (2021) respectively propose permutation-based, t -test based, adjusted permutation-based, and conformal inference-based procedures that allow one to relax distributional assumptions about common shocks and accommodate richer forms of heterogeneity.”
From What's trending in difference-in-differences? A synthesis of the recent econometrics literature (2023) · cites this paper as Canay, Romano and Shaikh (2017)
In the citing paper
“The key restriction is that one is comfortable limiting the time-series dependence of the cluster-specific-shocks, and strengthening the parallel trends assumption to hold in many pre- and post-treatment time periods.”
From What's trending in difference-in-differences? A synthesis of the recent econometrics literature (2023) · cites this paper as Canay et al. (2017)
Citations
Cites
How Much Should We Trust Differences-in-Differences Estimates?
Bertrand, M., Duflo, E., & Mullainathan, S. (2004) · The Quarterly Journal of Economics · doi:10.1162/003355304772839588 · Cited by 10,641 · cited 1×
“CCE: This test is the one proposed by Bertrand et al. (2004).”
MethodTesting Statistical Hypotheses
(2021) · Wiley Series in Probability and Statistics · doi:10.1002/9781119243830.ch12 · Cited by 5,106
Bootstrap-Based Improvements for Inference With Clustered Errors
Cameron, A. C., Gelbach, J. B., & Miller, D. L. (2008) · Review of Economics and Statistics · doi:10.1162/rest.90.3.414 · Cited by 4,417 · cited 2×
“CGM: This test is the one proposed by Cameron et al. (2008) based on the wild bootstrap.”
Inference with “Difference in Differences” with a Small Number of Policy Changes
Conley, T. G., & Taber, C. R. (2011) · Review of Economics and Statistics · doi:10.1162/rest_a_00049 · Cited by 633 · cited 8×
“We first consider an asymptotic framework similar to the one in Conley and Taber (2011), where |J 1 | = q is fixed, |J 0 | → ∞, and min{|T 0 |, |T 1”
BackgroundMethodInfluentialA NEW ASYMPTOTIC THEORY FOR HETEROSKEDASTICITY-AUTOCORRELATION ROBUST TESTS
Kiefer, N. M., & Vogelsang, T. J. (2005) · Econometric Theory · doi:10.1017/s0266466605050565 · Cited by 419 · cited 1×
“Bester et al. (2011) and Ibragimov and Müller (2010) also find that their tests outperform the test proposed by Kiefer and Vogelsang (2002, 2005), in which the 1 − α2 quantile of the N(0, 1) distribution is replaced with an alternative critical value that does not require q →∞.”
BackgroundThe Large-Sample Power of Tests Based on Permutations of Observations
Hoeffding, W. (1952) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729436 · Cited by 365 · cited 2×
“Earlier work on the asymptotic behavior of randomization tests includes Hoeffding (1952), Romano (1989) and Romano (1990).”
BackgroundBias reduction in standard errors for linear regression with multi-stage samples
McCaffrey, D. F., & Bell, R. M. (2003) · Quality Engineering · Cited by 318
t-Statistic Based Correlation and Heterogeneity Robust Inference
Ibragimov, R., & Müller, U. K. (2010) · Journal of Business & Economic Statistics · doi:10.1198/jbes.2009.08046 · Cited by 299 · cited 13×
“With the exception of the t-test approach developed in Ibragimov and Müller (2010), the competing approaches we discuss in Section 5.1 below do not share this feature.”
BackgroundMethodResultInfluentialThe Effects of High Stakes High School Achievement Awards: Evidence from a Randomized Trial
Angrist, J., & Lavy, V. (2009) · American Economic Review · doi:10.1257/aer.99.4.1384 · Cited by 270 · cited 10×
“The bias correction is highlighted by Angrist and Pischke (2008, page 320) and is used by Angrist and Lavy (2009) without the degrees of freedom adjustment.”
InfluentialInference with dependent data using cluster covariance estimators
Bester, C. A., Conley, T. G., & Hansen, C. B. (2011) · Journal of Econometrics · doi:10.1016/j.jeconom.2011.01.007 · Cited by 258 · cited 7×
“By replacing the time index t with a vector index, as in Bester et al. (2011), we can accommodate more complicated dependence structures, such as those found in spatially dependent data or in panel data.”
BackgroundMethodResultInfluential
Cited by
What's trending in difference-in-differences? A synthesis of the recent econometrics literature
Roth, J., Sant’Anna, P. H. C., Bilinski, A., & Poe, J. (2023) · Journal of Econometrics · doi:10.1016/j.jeconom.2023.03.008 · Cited by 2,027
“When in fact the number of time periods is small, as frequently occurs in DiD applications, one can still use some of these methods, but the underlying assumptions are stronger — see, e.g., Remark 4.5 and Section 4.2 of Canay et al. (2017).”
InfluentialCluster-robust inference: A guide to empirical practice
MacKinnon, J. G., Nielsen, M. Ø., & Webb, M. D. (2023) · Journal of Econometrics · doi:10.1016/j.jeconom.2022.04.001 · Cited by 407
“The results of Canay et al. (2021) and MacKinnon and Webb (2020b) strongly suggest that homogeneity across clusters is more important for RI than for the”
BackgroundRandomization inference for difference-in-differences with few treated clusters
MacKinnon, J. G., & Webb, M. D. (2020) · Journal of Econometrics · doi:10.1016/j.jeconom.2020.04.024 · Cited by 167
Inference in Differences-in-Differences with Few Treated Groups and Heteroskedasticity
Ferman, B., & Pinto, C. (2019) · The Review of Economics and Statistics · doi:10.1162/rest_a_00759 · Cited by 178
Practical and Effective Approaches to Dealing With Clustered Data
Esarey, J., & Menger, A. (2019) · Political Science Research and Methods · doi:10.1017/psrm.2017.42 · Cited by 141
The Wild Bootstrap with a “Small” Number of “Large” Clusters
Canay, I. A., Santos, A., & Shaikh, A. M. (2021) · The Review of Economics and Statistics · doi:10.1162/rest_a_00887 · Cited by 104
“This distinction prevents the application of results in Canay et al. (2017). In fact, sn and ŝn do not even tend in distribution to the same limit.”
BackgroundSelf-citationMeasuring Success in Education: The Role of Effort on the Test Itself
Uri Gneezy, B., List, J. A., Livingston, J. A., Sadoff, S., Qin, X., & Xu, Y. (2017) · National Bureau of Economic Research · doi:10.3386/w24004 · Cited by 117
“and Shanghai, which we calculate using a randomization test (Canay et al., 2017).”
MethodInference with few heterogeneous clusters
Ibragimov, R., & Müller, U. K. (2016) · Review of Economics and Statistics · doi:10.1162/rest_a_00545 · Cited by 99
Synthetic Control and Inference
Hahn, J., & Shi, R. (2017) · Econometrics · doi:10.3390/econometrics5040052 · Cited by 90
“Although we will later use the estimator in (7) in our Monte Carlo simulations, we do not need to restrict ourselves to this particular estimator.”
MethodThe standard errors of persistence
Conley, T. G., & Kelly, M. (2024) · Journal of International Economics · doi:10.1016/j.jinteco.2024.104027 · Cited by 120
Inference with Dependent Data in Accounting and Finance Applications
Conley, T. G., Gonçalves, Ś., & Hansen, C. (2017) · Journal of Accounting Research · doi:10.1111/1475-679x.12219 · Cited by 100
Asymptotic Theory for Clustered Samples
Hansen, B. E., & Lee, S. (2019) · Journal of Econometrics · doi:10.1016/j.jeconom.2019.02.001 · Cited by 81
“A sufficient condition for the CLT (9) is that for all ε > 0”
Approximate Permutation Tests and Induced Order Statistics in the Regression Discontinuity Design
Canay, I. A., & Kamat, V. (2017) · The Review of Economic Studies · doi:10.1093/restud/rdx062 · Cited by 90
Self-citationThe Mechanics of the Industrial Revolution
Kelly, M., Mokyr, J., & Ó Gráda, C. (2023) · Journal of Political Economy · doi:10.1086/720890 · Cited by 70
Improved central limit theorem and bootstrap approximations in high dimensions
Chernozhuokov, V., Chetverikov, D., Kato, K., & Koike, Y. (2022) · The Annals of Statistics · doi:10.1214/22-aos2193 · Cited by 138
Measuring Success in Education: The Role of Effort on the Test Itself
Livingston, J. (2018) · AEA Randomized Controlled Trials · doi:10.1257/rct.3657 · Cited by 61
Placebo inference on treatment effects when the number of clusters is small
Hagemann, A. (2019) · Journal of Econometrics · doi:10.1016/j.jeconom.2019.04.011 · Cited by 70
“h. I consider (i) placebo inference as in Algorithm2.5, (ii) two-sample inference with theIbragimov and Muller(2016) ttest, (iii) randomized and non-randomized inference with the permutation ttest ofCanay et al. (2014), (iv) cluster-robust wild bootstrap inference as described inCameron et al.(2008) and Webb(2014), an…”
BackgroundMethodTesting Attrition Bias in Field Experiments
Ghanem, D., Hirshleifer, S., & Ortiz‐Becerra, K. (2023) · The Journal of Human Resources · doi:10.3368/jhr.0920-11190r2 · Cited by 70
A mixed placebo test for synthetic control method
Chen, Q., & Yan, G. (2023) · Economics Letters · doi:10.1016/j.econlet.2023.111004 · Cited by 52
Debiasing and t -Tests for Synthetic Control Inference on Average Causal Effects
Chernozhukov, V., Wüthrich, K., & Zhu, Y. (2026) · Journal of Political Economy · doi:10.1086/742424 · Cited by 67
Powell, D. (2022) · Journal of Business & Economic Statistics · doi:10.1080/07350015.2021.1927743 · Cited by 43
“Given asymptotically independent and symmetric21 functions with the same mean under the null hypothesis, the inference approach follows directly from Canay, Romano, and Shaikh (2017).”
MethodEvaluating Stochastic Seeding Strategies in Networks
Chin, A., Eckles, D., & Ugander, J. H. (2021) · Management Science · doi:10.1287/mnsc.2021.3963 · Cited by 50
Covariate-adjusted Fisher randomization tests for the average treatment effect
Zhao, A., & Ding, P. (2021) · Journal of Econometrics · doi:10.1016/j.jeconom.2021.04.007 · Cited by 40
“There is an increasing interest in economics and related fields to use frts to analyze various types of empirical data (Freedman and Lane 1983; Kennedy 1995; Cattaneo et al. 2015; Canay et al. 2017; Ganong and Jäger 2018; Athey et al. 2018; Bugni et al. 2018; Young 2019; Heckman and Karapakula 2019; MacKinnon and Webb…”
BackgroundTesting continuity of a density via g-order statistics in the regression discontinuity design
Bugni, F. A., & Canay, I. A. (2020) · Journal of Econometrics · doi:10.1016/j.jeconom.2020.02.004 · Cited by 39
Self-citationExact P-values for Network Interference
Athey, S. C., Eckles, D., & Imbens, G. W. (2015) · National Bureau of Economic Research · doi:10.3386/w21313 · Cited by 48
Estimation and Inference for Linear Models with Two-Way Fixed Effects and Sparsely Matched Data
Verdier, V. (2018) · The Review of Economics and Statistics · doi:10.1162/rest_a_00807 · Cited by 45
Inference With Dyadic Data: Asymptotic Behavior of the Dyadic-Robust t -Statistic
Tabord‐Meehan, M. (2017) · Journal of Business and Economic Statistics · doi:10.1080/07350015.2017.1409630 · Cited by 48
“…arises in the clustered-data setting, and recent papers have studied solutions for inference with few or even finitely many clusters (see Bester, Conley, and Hansen 2011; Cameron, Gelbach, andMiller 2008; Canay, Romano, and Shaikh 2017; Ibragimov andMüller 2010, 2016, for recent work in this area).”
BackgroundRobust Testing in Generalized Linear Models by Sign Flipping Score Contributions
Hemerik, J., Goeman, J. J., & Finos, L. (2020) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/rssb.12369 · Cited by 31
“sider α1,α2 ∈ {0/w,1/w,...,(w −1)/w}. Under H0, as n → ∞, P h Tn 1< T n (α1w+1) ∪ Tn 1 > T n ((1−α2)w) i → α +α2. Note that our test does not rely on an approximate symmetry assumption (as e.g. Canay et al., 2017). Indeed, even if the scores are very skewed, asymptotically the test of Theorem 1 is exact. However…”
MethodRandomization Tests for Peer Effects in Group Formation Experiments
Basse, G., Ding, P., Feller, A., & Toulis, P. (2024) · Econometrica · doi:10.3982/ecta20134 · Cited by 54
“…randomized group formation designs, building on a growing literature within economics and econometrics (see Lehmann and Romano, 2005; Imbens and Rubin, 2015; Canay et al., 2017; Young, 2019) that seeks to use the randomization itself as the source of uncertainty when analyzing randomized trials.”
BackgroundMethodThe local approach to causal inference under network interference
Auerbach, E., Guo, H., & Tabord-Meehan, M. (2026) · Quantitative Economics · doi:10.3982/qe2484 · Cited by 31
Approximate permutation tests and induced order statistics in the regression discontinuity design
Kamat, V., & Canay, I. A. (2015) · doi:10.1920/wp.cem.2015.2715 · Cited by 29
“This is possible in our context due to the recent asymptotic framework developed by Canay et al. (2017) for randomization tests, although we introduce novel modifications to make it work in the RDD setting — see Section 4.2.”
BackgroundMethodInfluentialSelf-citationNormal Approximation in Large Network Models
Leung, M. P., & Moon, H. R. (2026) · The Review of Economic Studies · doi:10.1093/restud/rdag022 · Cited by 33
The Labor Supply Consequences of the Opioid CrisisPreprint
Powell, D. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3899329 · Cited by 35
Not Too Late: Improving Academic Outcomes Among Adolescents
Guryan, J., Ludwig, J., Bhatt, M., Cook, P., Davis, J. M. V., Dodge, K., Farkas, G., Fryer, R., Mayer, S., Pollack, H., et al. (2021) · National Bureau of Economic Research · doi:10.3386/w28531 · Cited by 25
“We test the null hypothesis that there is no difference in the relationship between ITT effects and question difficulty between students with above and below median test scores using a permutation test (Canay, Romano, and Shaikh 2017).”
Inference with a Single Treated Cluster
Hagemann, A. (2025) · The Review of Economic Studies · doi:10.1093/restud/rdaf002 · Cited by 26
“More results in this direction can be found in Canay et al. (2017), who come to the same conclusion in their experiments.”
BackgroundWang, R., & Xu, W. (2022) · Biometrika · doi:10.1093/biomet/asac014 · Cited by 22
“, Romano (1990), Chung and Romano (2013) and Canay et al. (2017). In the proposed procedure, the conditional distribution L{TCQ(E; X̃1, X̃2) | X̃1, X̃2} is used to approximate the null distribution of TCQ(X1,X2). We have E{TCQ(E; X̃1, X̃2) | X̃1, X̃2} = 0. From Lemma S.10, under Assumptions 1-3, we have var{TCQ(E; X̃1…”
MethodInfluentialComparing forecasting performance with panel data
Qu, R., Timmermann, A., & Zhu, Y. (2023) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2023.08.001 · Cited by 28
Wild Bootstrap Randomization Inference for Few Treated Clusters
MacKinnon, J. G., & Webb, M. D. (2019) · The Econometrics of Complex Survey Data · doi:10.1108/s0731-905320190000039003 · Cited by 21
Network Cluster‐Robust Inference
Leung, M. P. (2023) · Econometrica · doi:10.3982/ecta19816 · Cited by 37
Inference in difference‐in‐differences: How much should we trust in independent clusters?
Ferman, B. (2023) · Journal of Applied Econometrics · doi:10.1002/jae.2955 · Cited by 24
Randomization Inference: Theory and Applications
Ritzwoller, D. M., Romano, J. P., & Shaikh, A. M. (2026) · Journal of Political Economy Microeconomics · doi:10.1086/743457 · Cited by 23
“Canay et al. (2017) employ such an approach in the context of differences-in-differences designs.”
InfluentialSelf-citationConsistency of invariance-based randomization tests
Dobriban, E. (2022) · The Annals of Statistics · doi:10.1214/22-aos2200 · Cited by 28
“, Canay et al. (2017); Hemerik and Goeman (2018a), but we will not pursue this. Noise invariance and robustness. The advantage of randomization tests compared to a rejection region of the form fm(Xm) > c̃m for a fixed c̃m is that it does not require the manual specification of the critical value c̃m. The critical valu…”
BackgroundInfluentialInference with a single treated cluster
Hagemann, A. (2020) · arXiv · doi:10.48550/arxiv.2010.04076 · Cited by 18
“More results in this direction can be found in Canay et al. (2017), who come to the same conclusion in their experiments.”
BackgroundSimple and trustworthy cluster-robust GMM inference
Hwang, J. (2020) · Journal of Econometrics · doi:10.1016/j.jeconom.2020.07.048 · Cited by 19
“J(θ̂ c 2) d ! J1 := G n B̄m − ΓΛ ( ΓΛS̄ ΓΛ )−1 ΓΛS̄ B̄m o0 S̄−1 (11) × n B̄m − ΓΛ ( ΓΛS̄ ΓΛ )−1 ΓΛS̄ B̄m o /q.”
On Inference When Using State Corporate Laws for IdentificationPreprint
Spamann, H. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3499101 · Cited by 19
Permutation Tests at Nonparametric Rates
Bertanha, M., & Chung, E. (2022) · Journal of the American Statistical Association · doi:10.1080/01621459.2022.2087660 · Cited by 14
Comparing Forecasting Performance with Panel DataPreprint
Timmermann, A., & Zhu, Y. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3380755 · Cited by 19
“However, as pointed out in Canay et al. (2017), the bound on the size distortion in Theorem 2 implies that, provided the number of clusters is not too small, the null rejection probability will be at least α− 2−K .”
BackgroundMethodWild bootstrap inference for instrumental variables regressions with weak and few clusters
Wang, W., & Zhang, Y. (2024) · Journal of Econometrics · doi:10.1016/j.jeconom.2024.105727 · Cited by 20
“We consider a simulation design similar to that in Section IV of Canay et al. (2021) and extend theirs to the IV model.”
MethodResultInfluentialMilitary training exercises, pollution, and their consequences for health
Bobonis, G. J., Stabile, M., & Tovar, L. (2020) · Journal of Health Economics · doi:10.1016/j.jhealeco.2020.102345 · Cited by 12
Cai, Y., Canay, I. A., Kim, D., & Shaikh, A. M. (2022) · Journal of Econometric Methods · doi:10.1515/jem-2021-0030 · Cited by 11
“This situation arises, for example, in the empirical application considered by Canay, Romano, and Shaikh (2017b) where j ∈ J indexes schools and the variable of interest is a treatment indicator at the school level.”
InfluentialSelf-citationPublic Works Programmes and Cooperation for the Common Good: Evidence from Malawi
Beierl, S., & Dodlova, M. (2022) · European Journal of Development Research · doi:10.1057/s41287-022-00525-1 · Cited by 13
Measuring the Graph Concordance of Locally Dependent Observations
Song, K. (2017) · The Review of Economics and Statistics · doi:10.1162/rest_a_00714 · Cited by 11
“More recently, Canay, Romano, and Shaikh (2017) investigated asymptotic validity of randomized tests under approximate symmetry conditions, where the symmetry conditions are imposed not on the finite sample distribution but on the asymptotic distribution of the statistic.”
BackgroundThe Wild Bootstrap with a Small Number of Large Clusters
Canay, I. A., Santos, A., & Shaikh, A. M. (2019) · doi:10.1920/wp.cem.2019.4019 · Cited by 11
Self-citationA simple way to assess inference methodsPreprint
Ferman, B. (2019) · arXiv (Cornell University) · Cited by 11
Matching estimators with few treated and many control observations
Ferman, B. (2021) · Journal of Econometrics · doi:10.1016/j.jeconom.2021.07.005 · Cited by 10
“Following Canay et al. (2017), we consider a test statistic given by”
BackgroundMethodInfluentialOn the performance of the Neyman Allocation with small pilots
Cai, Y., & Rafi, A. (2024) · Journal of Econometrics · doi:10.1016/j.jeconom.2024.105793 · Cited by 24
Robust inference on income inequality: t- statistic based approach
Ibragimov, R., Kattuman, P. A., & Андреевич, С. А. (2025) · Econometric Reviews · doi:10.1080/07474938.2024.2432362 · Cited by 9
“These tests outperform other asymptotic and bootstrap methods currently available (see also Canay, Romano, and Shaikh, 2017, for permutation tests of equality of two general parameters of interest under heterogeneity and clustered dependence), but do not provide a way of drawing inferences on any…”
InfluentialPermutation Inference with a Finite Number of Heterogeneous Clusters
Hagemann, A. (2023) · The Review of Economics and Statistics · doi:10.1162/rest_a_01300 · Cited by 10
“Several other papers show that inference with a fixed number of clusters is possible under a variety of conditions: Canay, Romano, and Shaikh (2017) obtain null distributions by permuting the signs of cluster-level statistics under symmetry assumptions.”
BackgroundInfluentialTesting continuity of a density via g -order statistics in the regression discontinuity design
Bugni, F. A., & Canay, I. A. (2018) · doi:10.1920/wp.cem.2018.2018 · Cited by 9
“The test φ(Sn) could be alternatively characterized as an “approximate” randomization test, see Canay et al. (2017) for a general description of such tests.”
BackgroundInfluentialSelf-citationDynamic synthetic control method for evaluating treatment effects in auto-regressive processes
Zheng, X., & Chen, S. X. (2023) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1093/jrsssb/qkad103 · Cited by 8
Clustering Methods for Statistical Inference
MacKinnon, J. G., & Webb, M. D. (2020) · Springer eBooks · doi:10.1007/978-3-319-57365-6_43-1 · Cited by 10
A Theory of Quasi-Experimental Evaluation of School Quality
Narita, Y. (2021) · Management Science · doi:10.1287/mnsc.2020.3742 · Cited by 10
Residual permutation test for regression coefficient testing
Wen, K., Wang, T., & Wang, Y. (2025) · The Annals of Statistics · doi:10.1214/24-aos2479 · Cited by 6
Dependence‐robust inference using resampled statistics
Leung, M. P. (2021) · Journal of Applied Econometrics · doi:10.1002/jae.2865 · Cited by 12
Asymptotic validity and finite-sample properties of approximate randomization tests
Toulis, P. (2025) · Biometrika · doi:10.1093/biomet/asaf085 · Cited by 8
“In a similar vein, we contribute new results to the theory of randomization-based tests employing studentized test statistics [Janssen, 1997, Chung and Romano, 2013, DiCiccio and Romano, 2017, Canay et al., 2017, Wu and Ding, 2021, Fogarty, 2023].”
MethodRANDOMIZATION TESTS OF COPULA SYMMETRY
Beare, B. K., & Seo, J. (2020) · Econometric Theory · doi:10.1017/s0266466619000410 · Cited by 6
“A recent paper by Canay, Romano, and Shaikh (2017) is related to ours in that it studies the behavior of randomization tests when symmetry is only approximately satisfied.”
BackgroundResultInfluentialA Modified Randomization Test for the Level of Clustering
Cai, Y. (2023) · Journal of Business and Economic Statistics · doi:10.1080/07350015.2023.2261567 · Cited by 6
“Imposing weak dependence within a (sub-)cluster is not an uncommon assumption (see for instance the discussion in Canay, Romano, and Shaikh 2017; Bester, Conley, and Hansen 2011).”
BackgroundMethodInfluentialScreening with Multitasking: Theory and Empirical Evidence from Teacher Tenure ReformPreprint
Dinerstein, M., & Opper, I. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4177557 · Cited by 13
Inference for Dependent Data with Learned Clusters
Cao, J., Hansen, C., Kozbur, D., & Villacorta, L. (2024) · The Review of Economics and Statistics · doi:10.1162/rest_a_01460 · Cited by 5
“Note, the CRS test was developed in Canay et al. (2017) for a single fixed partition and relies on a notion of approximate symmetry.”
BackgroundMethodInfluentialRandomization Tests for Equality in Dependence Structure
Seo, J. (2020) · Journal of Business and Economic Statistics · doi:10.1080/07350015.2020.1753527 · Cited by 5
“y, and this suggests that we may only rely on asymptotic group invariance conditions in our application of the permutation method. In this respect, our results in the next section complement those of Canay et al. (2017) or Beare and Seo (2017), who investigated the behavior of randomization tests under approximate sym…”
BackgroundResultInfluentialInference on quantile processes with a finite number of clusters
Hagemann, A. (2024) · Journal of Econometrics · doi:10.1016/j.jeconom.2024.105672 · Cited by 6
Dealing with imperfect randomization: Inference for the highscope perry preschool program
Heckman, J. J., Pinto, R. R., & Shaikh, A. M. (2024) · Journal of Econometrics · doi:10.1016/j.jeconom.2024.105683 · Cited by 6
Self-citationRandomization-based confidence sets for the local average treatment effect
Aronow, P. M., Chang, H., & Lopatto, P. (2026) · Biometrika · doi:10.1093/biomet/asag010 · Cited by 4
Inference with Modern Difference-in-Differences MethodsPreprint
Mizushima, Y., & Powell, D. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5221387 · Cited by 11
Robust Permutation Tests in Linear Instrumental Variables Regression
Tuvaandorj, P. (2024) · Journal of the American Statistical Association · doi:10.1080/01621459.2024.2412363 · Cited by 3
Norm-Referenced Scoring on Real Data: A Comparative Study of GRiSTEN and STEN
Günver, M. G. (2022) · SAGE Open · doi:10.1177/21582440221091253 · Cited by 3
“@ : r e p r e s e n t s t h e r e s p o n d e n t ’ s s c o r e . trimming the data, Box-Cox transformation (Sakia, 1992), and most frequently logarithmic transformation (Smith, 1993) in order to form a symmetry (Canay et al., 2017) and make both the arithmetic mean and standard deviation useful.”
BackgroundEducation and Internal Migration: Evidence From a Child Labor Reform in Spain
Chapela, J. G., Jiménez‐Martín, S., & Castelló, J. V. (2021) · SERIEs · doi:10.1007/s13209-023-00272-4 · Cited by 4
“However, the alternative inference procedures of Ibragimov and Müller (2016) and Canay et al. (2017) do not seem applicable in our setting as they rely on cluster-level estimators of the coefficients,”
BackgroundMethodAsymptotic Theory for Clustered SamplesPreprint
Hansen, B. E., & Lee, S. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3099187 · Cited by 3
Matching Estimators with Few Treated and Many Control ObservationsPreprint
Ferman, B. (2019) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.1909.05093 · Cited by 3
“Following Canay et al. (2017), we consider a test statistic given by”
BackgroundMethodInfluentialInference in Differences-in-Differences with Few Treated Units and Spatial CorrelationPreprint
Ferman, B. (2020) · arXiv (Cornell University) · Cited by 4
Counting Defiers: Examples from Health CarePreprint
Kowalski, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.06739 · Cited by 4
A novel method to identify differential pathways in uterine leiomyomata based on network strategy
Wang, H., Liu, J., & Qin, Z. (2017) · Oncology Letters · doi:10.3892/ol.2017.6928 · Cited by 2
“Randomization tests provide a general means of constructing tests that control size in finite samples whenever the distribution of the observed data exhibits symmetry under the null hypothesis (33).”
Network strategy to investigate differential pathways in sporadic amyotrophic lateral sclerosis
Han, L.-H., Fan, X., Guo, H. W., Wei, W., Chen, M.-M., & Shu-fang, Y. (2018) · Journal of Cancer Research and Therapeutics · doi:10.4103/0973-1482.199453 · Cited by 2
Directing Power Towards Conic Parameter SubspacesPreprint
Koning, N. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.05077 · Cited by 2
“The solution provided in the literature is to estimate the covariance matrix by imposing restrictions on its structure, through regularization or other means (see e.g. Srivastava and Du, 2008; Chen et al., 2011).”
BackgroundPelican, A., & Graham, B. S. (2020) · National Bureau of Economic Research · doi:10.3386/w27793 · Cited by 2
Using Multiple Outcomes to Adjust Standard Errors for Spatial Correlation
DellaVigna, S., Imbens, G. W., Kim, W., & Ritzwoller, D. M. (2025) · National Bureau of Economic Research · doi:10.3386/w33716 · Cited by 6
Testing Conditional Stochastic Dominance at Target PointsPreprint
Bugni, F., Canay, I. A., & Kim, D. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6215764 · Cited by 2
Self-citationSome impossibility results for inference with cluster dependence with large clusters
Kojevnikov, D., & Song, K. (2023) · Journal of Econometrics · doi:10.1016/j.jeconom.2023.105524 · Cited by 1
Randomized empirical likelihood test for ultra-high dimensional means under general covariances
Chen, Y., Zhu, L., & Xu, W. (2025) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1093/jrsssb/qkaf004 · Cited by 1
Powell, D. (2026) · Journal of Applied Econometrics · doi:10.1002/jae.70035 · Cited by 1
“ISCM produces multiple estimates of the same treatment effect, permitting the use of “finite inference” methods such as those introduced in Ibragimov and Müller (2010) and Canay et al. (2017).”
InfluentialInference with Correlated Clusters
Powell, D. (2017) · RAND Corporation eBooks · doi:10.7249/wr1137-1 · Cited by 1
Tests for group-specific heterogeneity in high-dimensional factor models
Djogbenou, A., & Sufana, R. (2023) · Journal of Multivariate Analysis · doi:10.1016/j.jmva.2023.105233 · Cited by 1
On the Term “Randomization Test”
Hemerik, J. (2024) · The American Statistician · doi:10.1080/00031305.2024.2319182 · Cited by 1
“Later works that adopt such a definition of “randomization test” are Lehmann and Romano (2005), Romano and Wolf (2005), Canay, Romano, and Shaikh (2017), and Dobriban (2022).”
Nonparametric methods controlling the median of the false discovery proportion
Hemerik, J. (2026) · Electronic Journal of Statistics · doi:10.1214/26-ejs2586 · Cited by 1
“Nonparametric and semiparametric tests can be applied in many situations (Winkler et al., 2014; Canay et al., 2017; Hemerik et al., 2020; Berrett et al., 2020; Hemerik et al., 2021; Hemerik and Goeman, 2021; Dobriban, 2022; Liu et al., 2022; Zhang and Zhao, 2023).”
Hasan, S. A., Syed, I. A., & Klimova, A. (2025) · Economic Record · doi:10.1111/1475-4932.12875 · Cited by 1
Policy choice in experiments with unknown interferencePreprint
Viviano, D. (2020) · RePEc: Research Papers in Economics · Cited by 1
(Non)Randomization: A Theory of Quasi-Experimental Evaluation of School QualityPreprint
Narita, Y. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3023376 · Cited by 2
Dealing with Imperfect Randomization: Inference for the HighScope Perry Preschool Program
Heckman, J. J., Pinto, R. R., & Shaikh, A. M. (2023) · National Bureau of Economic Research · doi:10.3386/w31982 · Cited by 1
Self-citationInference with Correlated ClustersPreprint
Powell, D. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2793555
“This test statistic has similarities to those proposed in Ibragimov and Müller [2010] and Canay et al. [2014]. For those test statistics, the papers propose using the mean and variance of the estimated parameters across clusters.”
MethodHwang, J. (2016) · eScholarship (California Digital Library)
Metadata
- Authors
- Canay, I. A., Romano, J. P., & Shaikh, A. M.
- Venue
- Econometrica
- Publisher
- The Econometric Society
- Published
- 2017
- Volume
- 85
- Issue
- 3
- Pages
- 1013-1030
- ISSN
- 0012-9682 (ISSN-L)
- OpenAlex
- W2188869357
- Cited by
- 125
- References
- 24
- Citations by year
- 2026: 17 · 2025: 12 · 2024: 8 · 2023: 12 · 2022: 12 · 2021: 15 · 2020: 13 · 2019: 12 · 2018: 7 · 2017: 11
- FWCI
- 8.49
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Econometrica (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Spatial and Panel Data Analysis
- Keywords
- permutation tests, approximate symmetries, finite-sample inference, cluster randomization, heterogeneous clusters, statistical power, exam performance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.