CoChat

A Paradox from Randomization-Based Causal Inference

Open access
Ding, Peng · 2017 · Statistical Science
Cited by 87

The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.

Abstract

Under the potential outcomes framework, causal effects are defined as comparisons between potential outcomes under treatment and control. To infer causal effects from randomized experiments, Neyman proposed to test the null hypothesis of zero average causal effect (Neyman’s null), and Fisher proposed to test the null hypothesis of zero individual causal effect (Fisher’s null). Although the subtle difference between Neyman’s null and Fisher’s null has caused a lot of controversies and confusions for both theoretical and practical statisticians, a careful comparison between the two approaches has been lacking in the literature for more than eighty years. We fill this historical gap by making a theoretical comparison between them and highlighting an intriguing paradox that has not been recognized by previous researchers. Logically, Fisher’s null implies Neyman’s null. It is therefore surprising that, in actual completely randomized experiments, rejection of Neyman’s null does not imply rejection of Fisher’s null for many realistic situations, including the case with constant causal effect. Furthermore, we show that this paradox also exists in other commonly-used experiments, such as stratified experiments, matched-pair experiments and factorial experiments. Asymptotic analyses, numerical examples and real data examples all support this surprising phenomenon. Besides its historical and theoretical importance, this paradox also leads to useful practical implications for modern researchers.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1214/16-sts571

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 50

  1. Dasgupta, T., Pillai, N. S. and Rubin, D. B. (2015). Causal inference from $2^{K}$ factorial designs by using potential outcomes. J. R. Stat. Soc. Ser. B. Stat. Methodol. 77 727–753. · DOI
  2. Gail, M. H., Mark, S. D., Carroll, R. J., Green, S. B. and Pee, D. (1996). On design considerations and randomization-based inference for community intervention trials. Stat. Med. 15 1069–1092. · DOI · Paper page
  3. Lin, W. (2013). Agnostic notes on regression adjustments to experimental data: Reexamining Freedman’s critique. Ann. Appl. Stat. 7 295–318. · DOI · Paper page
  4. Ding, P. and Dasgupta, T. (2016). A potential tale of two-by-two tables from completely randomized experiments. J. Amer. Statist. Assoc. 111 157–168. · DOI
  5. Kempthorne, O. (1955). The randomization theory of experimental inference. J. Amer. Statist. Assoc. 50 946–967. · DOI · Paper page
  6. White, H. (1980). A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica 48 817–838. · DOI · Paper page
  7. Welch, B. L. (1937). On the $z$-test in randomized blocks and Latin squares. Biometrika 29 21–52. · DOI · Paper page
  8. Samii, C. and Aronow, P. M. (2012). On equivalencies between design-based and regression-based variance estimators for randomized experiments. Statist. Probab. Lett. 82 365–370. · DOI
  9. Schochet, P. Z. (2010). Is regression adjustment supported by the Neyman model for causal inference? J. Statist. Plann. Inference 140 246–259. · DOI
  10. Fisher, R. A. (1926). The arrangement of field experiments. Journal of the Ministry of Agriculture of Great Britain 33 503–513.
  11. Anscombe, F. J. (1948). The validity of comparative experiments. J. Roy. Statist. Soc. Ser. A 111 181–200; discussion, 200–211. · DOI
  12. Chung, E. and Romano, J. P. (2013). Exact and asymptotically robust permutation tests. Ann. Statist. 41 484–507. · DOI · Paper page
  13. Rubin, D. B. (1990). Comment on J. Neyman and causal inference in experiments and observational studies: “On the application of probability theory to agricultural experiments. Essay on principles. Section 9” [Ann. Agric. Sci. 10 (1923), 1–51]. Statist. Sci. 5 472–480. · DOI · Paper page
  14. Hoeffding, W. (1952). The large-sample power of tests based on permutations of observations. Ann. Math. Stat. 23 169–192. · DOI · Paper page
  15. Rosenbaum, P. R. (2002). Observational Studies, 2nd ed. Springer, New York. · DOI · Paper page
  16. Nelsen, R. B. (2006). An Introduction to Copulas, 2nd ed. Springer, New York.
  17. Wu, C. F. J. and Hamada, M. S. (2009). Experiments: Planning, Analysis, and Optimization, 2nd ed. Wiley, Hoboken, NJ.
  18. Kempthorne, O. (1952). The Design and Analysis of Experiments. Wiley, New York; Chapman & Hall, London. · DOI · Paper page
  19. Reid, C. (1982). Neyman—From Life. Springer, New York. · DOI
  20. Scheffé, H. (1959). The Analysis of Variance. Wiley, New York; Chapman & Hall, London.
  21. Huber, P. J. (1967). The behavior of maximum likelihood estimates under nonstandard conditions. In Proc. Fifth Berkeley Sympos. Math. Statist. and Probability (Berkeley, Calif., 1965/66), Vol. I: Statistics 221–233. Univ. California Press, Berkeley, CA.
  22. Cox, D. R. (2012). Statistical causality: Some historical remarks. In Causality: Statistical Perspectives and Applications (C. Berzuini, P. Dawid and L. Bernardinelli, eds.) 1–5. Wiley, New York. · doi:10.1002/9781119945710.ch1
  23. Angrist, J. D. and Pischke, J. S. (2008). Mostly Harmless Econometrics: An Empiricist’s Companion. Princeton Univ. Press, Princeton, NJ. · DOI · Paper page
  24. Cox, D. R. (1992). Planning of Experiments. Wiley, New York. Reprint of the 1958 original.
  25. Hinkelmann, K. and Kempthorne, O. (2008). Design and Analysis of Experiments, Vol. 1: Introduction to Experimental Design, 2nd ed. Wiley, New York.
  26. Cox, D. R. (1958). The interpretation of the effects of non-additivity in the Latin square. Biometrika 45 69–73. · DOI · Paper page
  27. Wilk, M. B. and Kempthorne, O. (1957). Non-additivities in a Latin square design. J. Amer. Statist. Assoc. 52 218–236. · DOI
  28. Greenland, S. (1991). On the logical justification of conditional tests for two-by-two contingency tables. Amer. Statist. 45 248–251. · DOI
  29. Pitman, E. J. G. (1938). Significance tests which can be applied to samples from any populations. III. The analysis of variance test. Biometrika 29 322–335. · DOI · Paper page
  30. Agresti, A. and Min, Y. (2004). Effects and non-effects of paired identical observations in comparing proportions with binary matched-pairs data. Stat. Med. 23 65–75. · DOI · Paper page
  31. Barnard, G. A. (1947). Significance tests for $2\times 2$ tables. Biometrika 34 123–138. · DOI · Paper page
  32. Box, G. E. P. (1992). Teaching engineers experimental design with a paper helicopter. Qual. Eng. 4 453–459. · doi:10.1080/08982119208918925
  33. Ding, P. (2017). Supplement to “A paradox from randomization-based causal inference.” DOI:10.1214/16-STS571SUPP. · doi:10.1214/16-sts571supp
  34. Ding, P., Feller, A. and Miratrix, L. W. (2016). Randomization inference for treatment effect variation. J. R. Stat. Soc. Ser. B. Stat. Methodol. 78 655–671. · DOI
  35. Eberhardt, K. R. and Fligner, M. A. (1977). Comparison of two tests for equality of two proportions. Amer. Statist. 31 151–155. · doi:10.1080/00031305.1977.10479225
  36. Eden, T. and Yates, F. (1933). On the validity of Fisher’s $z$-test when applied to an actual example of non-normal data. J. Agric. Sci. 23 6–17. · doi:10.1017/s0021859600052862
  37. Fienberg, S. E. and Tanur, J. M. (1996). Reconsidering the fundamental contributions of Fisher and Neyman on experimentation and sampling. Int. Stat. Rev. 64 237–253. · DOI
  38. Imai, K. (2008). Variance identification and efficiency analysis in randomized experiments under the matched-pair design. Stat. Med. 27 4857–4873. · DOI
  39. Janssen, A. (1997). Studentized permutation tests for non-i.i.d. hypotheses and the generalized Behrens–Fisher problem. Statist. Probab. Lett. 36 9–21. · DOI · Paper page
  40. Lang, J. B. (2015). A closer look at testing the “no-treatment-effect” hypothesis in a comparative experiment. Statist. Sci. 30 352–371. · DOI
  41. Li, X. and Ding, P. (2016). Exact confidence intervals for the average causal effect on a binary outcome. Stat. Med. 35 957–960. · doi:10.1002/sim.6764
  42. Lin, W., Halpern, S. D., Prasad Kerlin, M. and Small, D. S. (2017). A “placement of death” approach for studies of treatment effects on ICU length of stay. Stat. Methods Med. Res. 26 292–311. · DOI
  43. Neuhaus, G. (1993). Conditional rank tests for the two-sample problem under random censorship. Ann. Statist. 21 1760–1779. · DOI
  44. Neyman, J. (1990). On the application of probability theory to agricultural experiments. Essay on principles. Section 9. Statist. Sci. 5 465–472. Translated from the 1923 Polish original and edited by D. M. Dabrowska and T. P. Speed. · DOI · Paper page
  45. Pauly, M., Brunner, E. and Konietschke, F. (2015). Asymptotic permutation tests in general factorial designs. J. R. Stat. Soc. Ser. B. Stat. Methodol. 77 461–473. · DOI
  46. Pitman, E. J. G. (1937). Significance tests which may be applied to samples from any populations. Suppl. J. R. Stat. Soc. 4 119–130. · DOI · Paper page
  47. Rigdon, J. and Hudgens, M. G. (2015). Randomization inference for treatment effects on a binary outcome. Stat. Med. 34 924–935. · doi:10.1002/sim.6384
  48. Robins, J. M. (1988). Confidence intervals for causal parameters. Stat. Med. 7 773–785. · DOI · Paper page
  49. Rubin, D. B. (2004). Teaching statistical inference for causal effects in experiments and observational studies. J. Educ. Behav. Stat. 29 343–367. · DOI
  50. Sabbaghi, A. and Rubin, D. B. (2014). Comments on the Neyman–Fisher controversy and its consequences. Statist. Sci. 29 267–284. · DOI

Cited by

See all 87

Metadata

Authors
Ding, P.
Venue
Statistical Science
Published
2017-08-01
Volume
32
Issue
3
Pages
331-345
ISSN
0883-4237 (ISSN-L)
OpenAlex
W2963193871
Cited by
87
References
10
FWCI
6.78
Type
journal article
Language
en
Open access
bronze

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.