CoChat
Sign Up
Cited by 40Advanced Causal Inference TechniquesStatistical Methods in Clinical TrialsOptimal Experimental Design Methods

Covariate-adjusted Fisher randomization tests for the average treatment effect

Open access
Zhao, Anqi, Ding, Peng·2021·Journal of Econometrics
DOI 10.1016/j.jeconom.2021.04.007
Open this paper in CoChat — free account

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.jeconom.2021.04.007

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Randomization inference yields valid p-values for two null hypotheses: finite-sample exact inference on the sharp null of no treatment effect on all individuals, and asymptotically valid inference on the null hypothesis of no average treatment effect.

    “Note that randomization inference yields valid p-values for two null hypotheses.First, it provides finite-sample exact inference on the sharp null of no treatment effect on all individuals.Second, when the sharp null hypothesis is incorrect (e.g., some units are impacted positively while others negatively), randomization inference still provides asymptotically valid inference on the null hypothesi”

    From Can Restorative Justice Conferencing Reduce Recidivism? Evidence From the Make-it-Right Program (2021) · cites this paper as Zhao and Ding (2021)

    Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 66

  1. M. J. Anderson and P. Legendre. An empirical comparison of permutation methods for tests of partial regression coefficients in a linear model. Journal of Statistical Computation and Simulation , 62:271-303, 1999.
  2. M. J. Anderson and J. Robinson. Permutation tests for linear models. Australian and New Zealand Journal of Statistics , 43:75-88, 2001.
  3. J. D. Angrist and J.-S. Pischke. Mostly Harmless Econometrics . Princeton University Press, 2009.
  4. P. Aronow, D. Green, and D. Lee. Sharp bounds on the variance in randomized experiments. Annals of Statistics , 42:850-871, 2014.
  5. S. Athey, D. Eckles, and G. W. Imbens. Exact p-values for network interference. Journal of the American Statistical Association , 113:230-240, 2018.
  6. R. R. Bahadur and L. J. Savage. The nonexistence of certain statistical procedures in nonparametric problems. Annals of Mathematical Statistics , 27:1115-1122, 1956.
  7. Y. H. Bai, A. M. Shaikh, and J. P. Romano. Inference in experiments with matched pairs. Journal of the American Statistical Association , page to appear, 2021.
  8. A. V. Banerjee, S. Chassang, S. Montero, and E. Snowberg. A theory of experimenters: Robustness, randomization, and balance. American Economic Review , 110:1206-1230, 2020.
  9. G. Basse, P. Ding, A. Feller, and P. Toulis. Randomization tests for peer effects in group formation experiments. arXiv , page 1904.02308, 2019.
  10. R. Berk, E. Pitkin, L. Brown, A. Buja, E. George, and L. Zhao. Covariance adjustments for the analysis of randomized field experiments. Evaluation Review , 37:170-196, 2013.
  11. M. A. C. Bind and D. B. Rubin. When possible, report a Fisher-exact P value and display its underlying null randomization distribution. Proceedings of the National Academy of Sciences of the United States of America , 117:19151-19158, 2020.
  12. A. Bloniarz, H. Liu, C. Zhang, J. Sekhon, and B. Yu. Lasso adjustments of treatment effect estimates in randomized experiments. Proceedings of the National Academy of Sciences of the United States of America , 113:7383-7390, 2016.
  13. M. Bruhn and D. McKenzie. In pursuit of balance: Randomization in practice in development field experiments. American Economic Journal: Applied Economics , 1:200-232, 2009.
  14. F. A. Bugni, I. A. Canay, and A. M. Shaikh. Inference under covariate-adaptive randomization. Journal of the American Statistical Association , 113:1784-1796, 2018.
  15. M. D. Cattaneo, B. R. Frandsen, and R. Titiunik. Randomization inference in the regression discontinuity design: An application to party advantages in the US Senate. Journal of Causal Inference , 3:1-24, 2015.
  16. X. Chen, Y. Liu, S. Ma, and Z. Zhang. Efficient estimation of general treatment effects using neural networks with a diverging number of confounders. arXiv preprint arXiv:2009.07055 , 2020. · arXiv
  17. A. Chong, I. Cohen, E. Field, E. Nakasone, and M. Torero. Iron deficiency and schooling attainment in Peru. American Economic Journal: Applied Economics , 8:222-55, 2016.
  18. E. Chung and J. P. Romano. Exact and asymptotically robust permutation tests. Annals of Statistics , 41:484-507, 2013.
  19. E. Chung and J. P. Romano. Asymptotically valid and exact permutation tests based on two-sample U -statistics. Journal of Statistical Planning and Inference , 168:97-105, 2016.
  20. P. L. Cohen and C. B. Fogarty. Gaussian prepivoting for finite population causal inference. https://arxiv.org/abs/2002.06654 , 2020. · arXiv
  21. D. R. Cox. Randomization and concomitant variables in the design of experiments. In P. R. Krishnaiah G. Kallianpur and J. K. Ghosh, editors, Statistics and Probability: Essays in Honor of C. R. Rao , pages 197-202. North-Holland, Amsterdam, 1982.
  22. T. Dasgupta, N. Pillai, and D. B. Rubin. Causal inference from 2 K factorial designs by using potential outcomes. Journal of the Royal Statistical Society, Series B (Statistical Methodology) , 77:727-753, 2015.
  23. C. J. DiCiccio and J. P. Romano. Robust permutation tests for correlation and regression coefficients. Journal of the American Statistical Association , 112:1211-1220, 2017.
  24. P. Ding. The Frisch-Waugh-Lovell theorem for standard errors. Statistics and Probability Letters , 168:108945, 2020.
  25. F. Eicker. Limit theorems for regressions with unequal and dependent errors. In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability , volume 1, pages 59-82. Berkeley, CA: University of California Press, 1967.
  26. R. A. Fisher. The Design of Experiments . Edinburgh, London: Oliver and Boyd, 1st edition, 1935.
  27. C. B. Fogarty. On mitigating the analytical limitations of finely stratified experiments. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 80:1035-1056, 2018a.
  28. D. Freedman and D. Lane. A nonstochastic interpretation of reported significance levels. Journal of Business and Economic Statistics , 1:292-298, 1983.
  29. P. Ganong and S. J¨ ager. A permutation test for the regression kink design. Journal of the American Statistical Association , 113:494-504, 2018.
  30. K. Guo and G. Basse. The generalized Oaxaca-Blinder estimator. arXiv , page 2004.11615, 2020.
  31. J. H´ ajek. Some extensions of the Wald-Wolfowitz-Noether theorem. Annals of Mathematical Statistics , 32:506-523, 1961.
  32. J. J. Heckman and G. Karapakula. Using a satisficing model of experimenter decision-making to guide finite-sample inference for compromised experiments. Econometrics Journal , page in press, 2021.
  33. J. Hennessy, T. Dasgupta, L. Miratrix, C. Pattanayak, and P. Sarkar. A conditional randomization test to account for covariate imbalance in randomized experiments. Journal of Causal Inference , 4:61-80, 2016.
  34. W. Hoeffding. The large-sample power of tests based on permutations of observations. The Annals of Mathematical Statistics , 23:169-192, 1952.
  35. P. J. Huber. The behavior of maximum likelihood estimates under nonstandard conditions. In Lucien M. Le Cam and Jerzy Neyman, editors, Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability , volume 1, pages 221-233. Berkeley, California: University of California Press, 1967.
  36. A. Janssen. Studentized permutation tests for non-iid hypotheses and the generalized BehrensFisher problem. Statistics and Probability Letters , 36:9-21, 1997.
  37. F. Jiang, L. Tian, H. Fu, T. Hasegawa, and L. J. Wei. Robust alternatives to ANCOVA for estimating the treatment effect via a randomized comparative study. Journal of the American Statistical Association , 114:1854-1864, 2019.
  38. P. E. Kennedy. Randomization tests in econometrics. Journal of Business and Economic Statistics , 13:85-94, 1995.
  39. S. Lee and A. M. Shaikh. Multiple testing and heterogeneous treatment effects: Re-evaluating the effect of progresa on school enrollment. Journal of Applied Econometrics , 29:612-626, 2014.
  40. E. L. Lehmann and J. P. Romano. Testing Statistical Hypotheses . New York: Springer, 3rd edition, 2005.
  41. L. Lei and P. J. Bickel. An assumption-free exact test for fixed-design linear models with exchangeable errors. Biometrika , page in press, 2020.
  42. L. Lei and P. Ding. Regression adjustment in completely randomized experiments with a diverging number of covariates. Biometrika , page in press, 2020.
  43. X. Li and P. Ding. General forms of finite population central limit theorems with applications to causal inference. Journal of the American Statistical Association , 112:1759-1169, 2017.
  44. X. Li and P. Ding. Rerandomization and regression adjustment. Journal of the Royal Statistical Society, Series B (Methodological) , 82:241-268, 2020.
  45. X. Li, P. Ding, and D. B. Rubin. Asymptotic theory of rerandomization in treatment-control experiments. Proceedings of the National Academy of Sciences of the United States of America , 115:9157-9162, 2018.
  46. W. Lin. Agnostic notes on regression adjustments to experimental data: Reexamining Freedman's critique. Annals of Applied Statistics , 7:295-318, 2013.
  47. J. Lu. Covariate adjustment in randomization-based causal inference for 2 K factorial designs. Statistics and Probability Letters , 119:11-20, 2016.
  48. J. G. MacKinnon and M. D. Webb. Randomization inference for difference-in-differences with few treated clusters. Journal of Econometrics , 218:435-450, 2020.
  49. J. A. Middleton. A unified theory of regression adjustment for design-based inference. arXiv preprint arXiv:1803.06011 , 2018. · arXiv
  50. K. L. Morgan and D. B. Rubin. Rerandomization to improve covariate balance in experiments. Annals of Statistics , 40:1263-1282, 2012.
  51. R. Mukerjee, T. Dasgupta, and D. B. Rubin. Using standard tools from finite population sampling to improve causal inference for complex experiments. Journal of the American Statistical Association , 113:868-881, 2018.
  52. J. Neyman. Statistical problems in agricultural experimentation (with discussion). Supplement to the Journal of the Royal Statistical Society , 2:107-180, 1935.
  53. M. Pauly, E. Brunner, and F. Konietschke. Asymptotic permutation tests in general factorial designs. Journal of the Royal Statistical Society, Series B (Statistical Methodology) , 77:461-473, 2015.
  54. J. Raz. Testing for no effect when estimating a smooth function by nonparametric regression: a randomization approach. Journal of the American Statistical Association , 85:132-138, 1990.
  55. J. P. Romano. On the behavior of randomization tests without a group invariance assumption. Journal of the American Statistical Association , 85:686-692, 1990.
  56. A. J. Stephens, E. J. Tchetgen Tchetgen, and V. De Gruttola. Flexible covariate-adjusted exact tests of randomized treatment effects with application to a trial of HIV education. Annals of Applied Statistics , 7:2106-2137, 2013.
  57. F. Su and P. Ding. Model-assisted analyses of cluster-randomized experiments. arXiv , page 2104.04647, 2021.
  58. C. J. F. ter Braak. Permutation versus bootstrap significance tests in multiple regression and ANOVA. In K.H. J¨ ockel, G. Rothe, and W. Sendler, editors, Bootstrapping and Related Techniques , pages 79-85. Berlin: Springer-Verlag, 1992.
  59. A. W. van der vaart and J. Wellner. Weak Convergence and Empirical Processes: With Applications to Statistics . New York: Springer Verlag, 1996.
  60. S. Wager, W. Du, J. Taylor, and R. J. Tibshirani. High-dimensional regression adjustments in randomized experiments. Proceedings of the National Academy of Sciences of the United States of America , 113:12673-12678, 2016.
  61. E. Wu and J. A. Gagnon-Bartsch. The LOOP estimator: Adjusting for covariates in randomized experiments. Evaluation Review , 42:458-488, 2018.
  62. J. Wu and P. Ding. Randomization tests for weak null hypotheses in randomized experiments. Journal of American Statistical Association , 105:in press, 2020.
  63. T. Ye, J. Shao, and Q. Zhao. Principles for covariate adjustment in analyzing randomized clinical trials. arXiv preprint arXiv:2009.11828 , 2020a. · arXiv
  64. T. Ye, Y. Yi, and Q. Zhao. Inference on average treatment effect under minimization and other covariate-adaptive randomization methods. arXiv preprint arXiv:2007.09576 , 2020b. · arXiv
  65. A. Young. Channeling Fisher: Randomization tests and the statistical insignificance of seemingly significant experimental results. Quarterly Journal of Economics , 134:557-598, 2019.
  66. L. Zheng and M. Zelen. Multi-center clinical trials: Randomization and ancillary statistics. Annals of Applied Statistics , 2:582-600, 2008.

Cited by

Sorted by citations

Authors
Zhao, A., & Ding, P.
Affiliations
  • National University of Singapore (SG)
  • University of California, Berkeley (US)
Corresponding author
Ding, Peng
ORCID
Venue
Journal of Econometrics
Publisher
Elsevier BV
Published
2021-12
Volume
225
Issue
2
Pages
278-294
ISSN
0304-4076 (ISSN-L)
DOI
10.1016/j.jeconom.2021.04.007
OpenAlex
W3166452571
Cited by
40
References
3
Citations by year
2026: 2 · 2025: 7 · 2024: 11 · 2023: 5 · 2022: 7 · 2021: 8
FWCI
5.79
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Econometrics (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • arXiv (Cornell University)
Funders
National Science Foundation Division of Mathematical Sciences
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Optimal Experimental Design Methods
Keywords
covariate adjustment, average treatment effect, ordinary least squares, cluster randomization, stratified randomization, re-randomization, design-based inference, asymptotic validity, model misspecification

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review