A Paradox from Randomization-Based Causal Inference
Open accessThe DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
Under the potential outcomes framework, causal effects are defined as comparisons between potential outcomes under treatment and control. To infer causal effects from randomized experiments, Neyman proposed to test the null hypothesis of zero average causal effect (Neyman’s null), and Fisher proposed to test the null hypothesis of zero individual causal effect (Fisher’s null). Although the subtle difference between Neyman’s null and Fisher’s null has caused a lot of controversies and confusions for both theoretical and practical statisticians, a careful comparison between the two approaches has been lacking in the literature for more than eighty years. We fill this historical gap by making a theoretical comparison between them and highlighting an intriguing paradox that has not been recognized by previous researchers. Logically, Fisher’s null implies Neyman’s null. It is therefore surprising that, in actual completely randomized experiments, rejection of Neyman’s null does not imply rejection of Fisher’s null for many realistic situations, including the case with constant causal effect. Furthermore, we show that this paradox also exists in other commonly-used experiments, such as stratified experiments, matched-pair experiments and factorial experiments. Asymptotic analyses, numerical examples and real data examples all support this surprising phenomenon. Besides its historical and theoretical importance, this paradox also leads to useful practical implications for modern researchers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Estimating causal effects of treatments in randomized and nonrandomized studies
Rubin, D. B. (1974) · Journal of Educational Psychology · doi:10.1037/h0037350 · Cited by 9,436 · cited 1×
“Ever since Neyman’s seminal work, the potential outcomes framework (Neyman, 1923; Rubin, 1974) has been widely used for causal inference in randomized experiments (e.g., Neyman, 1935; Hinkelmann and Kempthorne, 2007; Imbens and Rubin, 2015).”
MethodCausal Inference for Statistics, Social, and Biomedical Sciences: An Introduction
Imbens, G. W., & Rubin, D. B. (2015) · doi:10.1017/cbo9781139025751 · Cited by 4,210
Edgington, E., Edgington, E., & Onghena, P. (2007) · doi:10.1201/9781420011814 · Cited by 2,030
Randomization Analysis of Experimental Data: The Fisher Randomization Test Comment
Rubin, D. B. (1980) · Journal of the American Statistical Association · doi:10.2307/2287653 · Cited by 1,368
Sampling Techniques. 3rd Edition
Cochran, W. G. (1977) · Zenodo · doi:10.5281/zenodo.15338807 · Cited by 1,136
Outline of a Theory of Statistical Estimation Based on the Classical Theory of Probability
Neyman, J. (1937) · Philosophical Transactions of the Royal Society of London. Series A, Mathematical and Physical Sciences · doi:10.1098/rsta.1937.0005 · Cited by 1,087
Elements of Large-Sample Theory
(1999) · Springer Texts in Statistics · doi:10.1007/b98855 · Cited by 677
On regression adjustments to experimental data
Freedman, D. A. (2008) · Advances in Applied Mathematics · doi:10.1016/j.aam.2006.12.003 · Cited by 417 · cited 2×
“Although linear regression has been criticized for analyzing experimental data (Freedman, 2008), the least square estimator β̂OLS = τ̂ is unbiased for the average causal effect τ .”
MethodGeneral Forms of Finite Population Central Limit Theorems with Applications to Causal Inference
Li, X., & Ding, P. (2017) · Journal of the American Statistical Association · doi:10.1080/01621459.2017.1295865 · Cited by 167
Self-citationSharp bounds on the variance in randomized experiments
Aronow, P. M., Green, D. P., & Lee, D. K. K. (2014) · The Annals of Statistics · doi:10.1214/13-aos1200 · Cited by 49 · cited 1×
“Recently, for general outcomes Aronow et al. (2014) derived the sharp bound of S2τ based on the marginal distributions of the treatment and control potential outcomes using the Frechét–Hoeffding bounds (Nelsen, 2007); for binary outcomes Robins (1988) and Ding and Dasgupta (2016) gave simple forms.”
Method
Research works printed but not linked: 50
- Dasgupta, T., Pillai, N. S. and Rubin, D. B. (2015). Causal inference from $2^{K}$ factorial designs by using potential outcomes. J. R. Stat. Soc. Ser. B. Stat. Methodol. 77 727–753. · DOI
- Gail, M. H., Mark, S. D., Carroll, R. J., Green, S. B. and Pee, D. (1996). On design considerations and randomization-based inference for community intervention trials. Stat. Med. 15 1069–1092. · DOI · Paper page
- Lin, W. (2013). Agnostic notes on regression adjustments to experimental data: Reexamining Freedman’s critique. Ann. Appl. Stat. 7 295–318. · DOI · Paper page
- Ding, P. and Dasgupta, T. (2016). A potential tale of two-by-two tables from completely randomized experiments. J. Amer. Statist. Assoc. 111 157–168. · DOI
- Kempthorne, O. (1955). The randomization theory of experimental inference. J. Amer. Statist. Assoc. 50 946–967. · DOI · Paper page
- White, H. (1980). A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica 48 817–838. · DOI · Paper page
- Welch, B. L. (1937). On the $z$-test in randomized blocks and Latin squares. Biometrika 29 21–52. · DOI · Paper page
- Samii, C. and Aronow, P. M. (2012). On equivalencies between design-based and regression-based variance estimators for randomized experiments. Statist. Probab. Lett. 82 365–370. · DOI
- Schochet, P. Z. (2010). Is regression adjustment supported by the Neyman model for causal inference? J. Statist. Plann. Inference 140 246–259. · DOI
- Fisher, R. A. (1926). The arrangement of field experiments. Journal of the Ministry of Agriculture of Great Britain 33 503–513.
- Anscombe, F. J. (1948). The validity of comparative experiments. J. Roy. Statist. Soc. Ser. A 111 181–200; discussion, 200–211. · DOI
- Chung, E. and Romano, J. P. (2013). Exact and asymptotically robust permutation tests. Ann. Statist. 41 484–507. · DOI · Paper page
- Rubin, D. B. (1990). Comment on J. Neyman and causal inference in experiments and observational studies: “On the application of probability theory to agricultural experiments. Essay on principles. Section 9” [Ann. Agric. Sci. 10 (1923), 1–51]. Statist. Sci. 5 472–480. · DOI · Paper page
- Hoeffding, W. (1952). The large-sample power of tests based on permutations of observations. Ann. Math. Stat. 23 169–192. · DOI · Paper page
- Rosenbaum, P. R. (2002). Observational Studies, 2nd ed. Springer, New York. · DOI · Paper page
- Nelsen, R. B. (2006). An Introduction to Copulas, 2nd ed. Springer, New York.
- Wu, C. F. J. and Hamada, M. S. (2009). Experiments: Planning, Analysis, and Optimization, 2nd ed. Wiley, Hoboken, NJ.
- Kempthorne, O. (1952). The Design and Analysis of Experiments. Wiley, New York; Chapman & Hall, London. · DOI · Paper page
- Reid, C. (1982). Neyman—From Life. Springer, New York. · DOI
- Scheffé, H. (1959). The Analysis of Variance. Wiley, New York; Chapman & Hall, London.
- Huber, P. J. (1967). The behavior of maximum likelihood estimates under nonstandard conditions. In Proc. Fifth Berkeley Sympos. Math. Statist. and Probability (Berkeley, Calif., 1965/66), Vol. I: Statistics 221–233. Univ. California Press, Berkeley, CA.
- Cox, D. R. (2012). Statistical causality: Some historical remarks. In Causality: Statistical Perspectives and Applications (C. Berzuini, P. Dawid and L. Bernardinelli, eds.) 1–5. Wiley, New York. · doi:10.1002/9781119945710.ch1
- Angrist, J. D. and Pischke, J. S. (2008). Mostly Harmless Econometrics: An Empiricist’s Companion. Princeton Univ. Press, Princeton, NJ. · DOI · Paper page
- Cox, D. R. (1992). Planning of Experiments. Wiley, New York. Reprint of the 1958 original.
- Hinkelmann, K. and Kempthorne, O. (2008). Design and Analysis of Experiments, Vol. 1: Introduction to Experimental Design, 2nd ed. Wiley, New York.
- Cox, D. R. (1958). The interpretation of the effects of non-additivity in the Latin square. Biometrika 45 69–73. · DOI
- Wilk, M. B. and Kempthorne, O. (1957). Non-additivities in a Latin square design. J. Amer. Statist. Assoc. 52 218–236. · DOI
- Greenland, S. (1991). On the logical justification of conditional tests for two-by-two contingency tables. Amer. Statist. 45 248–251. · DOI
- Pitman, E. J. G. (1938). Significance tests which can be applied to samples from any populations. III. The analysis of variance test. Biometrika 29 322–335. · DOI · Paper page
- Agresti, A. and Min, Y. (2004). Effects and non-effects of paired identical observations in comparing proportions with binary matched-pairs data. Stat. Med. 23 65–75. · DOI · Paper page
- Barnard, G. A. (1947). Significance tests for $2\times 2$ tables. Biometrika 34 123–138. · DOI · Paper page
- Box, G. E. P. (1992). Teaching engineers experimental design with a paper helicopter. Qual. Eng. 4 453–459. · doi:10.1080/08982119208918925
- Ding, P. (2017). Supplement to “A paradox from randomization-based causal inference.” DOI:10.1214/16-STS571SUPP. · doi:10.1214/16-sts571supp
- Ding, P., Feller, A. and Miratrix, L. W. (2016). Randomization inference for treatment effect variation. J. R. Stat. Soc. Ser. B. Stat. Methodol. 78 655–671. · DOI
- Eberhardt, K. R. and Fligner, M. A. (1977). Comparison of two tests for equality of two proportions. Amer. Statist. 31 151–155. · doi:10.1080/00031305.1977.10479225
- Eden, T. and Yates, F. (1933). On the validity of Fisher’s $z$-test when applied to an actual example of non-normal data. J. Agric. Sci. 23 6–17. · doi:10.1017/s0021859600052862
- Fienberg, S. E. and Tanur, J. M. (1996). Reconsidering the fundamental contributions of Fisher and Neyman on experimentation and sampling. Int. Stat. Rev. 64 237–253. · DOI
- Imai, K. (2008). Variance identification and efficiency analysis in randomized experiments under the matched-pair design. Stat. Med. 27 4857–4873. · DOI
- Janssen, A. (1997). Studentized permutation tests for non-i.i.d. hypotheses and the generalized Behrens–Fisher problem. Statist. Probab. Lett. 36 9–21. · DOI · Paper page
- Lang, J. B. (2015). A closer look at testing the “no-treatment-effect” hypothesis in a comparative experiment. Statist. Sci. 30 352–371. · DOI
- Li, X. and Ding, P. (2016). Exact confidence intervals for the average causal effect on a binary outcome. Stat. Med. 35 957–960. · doi:10.1002/sim.6764
- Lin, W., Halpern, S. D., Prasad Kerlin, M. and Small, D. S. (2017). A “placement of death” approach for studies of treatment effects on ICU length of stay. Stat. Methods Med. Res. 26 292–311. · DOI
- Neuhaus, G. (1993). Conditional rank tests for the two-sample problem under random censorship. Ann. Statist. 21 1760–1779. · DOI
- Neyman, J. (1990). On the application of probability theory to agricultural experiments. Essay on principles. Section 9. Statist. Sci. 5 465–472. Translated from the 1923 Polish original and edited by D. M. Dabrowska and T. P. Speed. · DOI · Paper page
- Pauly, M., Brunner, E. and Konietschke, F. (2015). Asymptotic permutation tests in general factorial designs. J. R. Stat. Soc. Ser. B. Stat. Methodol. 77 461–473. · DOI
- Pitman, E. J. G. (1937). Significance tests which may be applied to samples from any populations. Suppl. J. R. Stat. Soc. 4 119–130. · DOI · Paper page
- Rigdon, J. and Hudgens, M. G. (2015). Randomization inference for treatment effects on a binary outcome. Stat. Med. 34 924–935. · doi:10.1002/sim.6384
- Robins, J. M. (1988). Confidence intervals for causal parameters. Stat. Med. 7 773–785. · DOI · Paper page
- Rubin, D. B. (2004). Teaching statistical inference for causal effects in experiments and observational studies. J. Educ. Behav. Stat. 29 343–367. · DOI
- Sabbaghi, A. and Rubin, D. B. (2014). Comments on the Neyman–Fisher controversy and its consequences. Statist. Sci. 29 267–284. · DOI
Cited by
Econometric Methods for Program Evaluation
Abadie, A., & Cattaneo, M. D. (2018) · Annual Review of Economics · doi:10.1146/annurev-economics-080217-053402 · Cited by 431
General Forms of Finite Population Central Limit Theorems with Applications to Causal Inference
Li, X., & Ding, P. (2017) · Journal of the American Statistical Association · doi:10.1080/01621459.2017.1295865 · Cited by 167
“…because the treatment and control groups are simple random samples from the finite experimental units, the CLTs for sampling surveys are sometimes adequate for establishing asymptotic distributions of the causal effect estimators (e.g., Liu and Hudgens 2014; Ding 2017; Ding and Dasgupta 2016).”
BackgroundMethodInfluentialSelf-citationPapay, J. P., Taylor, E. S., Tyler, J. H., & Laski, M. E. (2020) · American Economic Journal: Economic Policy · doi:10.1257/pol.20170709 · Cited by 148
Papay, J., Taylor, E., Tyler, J., & Laski, M. (2016) · doi:10.3386/w21986 · Cited by 148
Systemic Discrimination Among Large U.S. Employers
Kline, P., Rose, E. K., & Walters, C. (2021) · doi:10.2139/ssrn.3898669 · Cited by 3
Metadata
- Authors
- Ding, P.
- Venue
- Statistical Science
- Publisher
- Institute of Mathematical Statistics
- Published
- 2017-08-01
- Volume
- 32
- Issue
- 3
- Pages
- 331-345
- ISSN
- 0883-4237 (ISSN-L)
- OpenAlex
- W2963193871
- Cited by
- 87
- References
- 10
- FWCI
- 6.78
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A Paradox from Randomization-Based Causal Inference. https://app.cochat.ai/papers/10.1214/16-sts571
- The paper
- Ding, P. (2017). A Paradox from Randomization-Based Causal Inference. Statistical Science. https://doi.org/10.1214/16-sts571
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.