Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 641

The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.

Agnostic notes on regression adjustments to experimental data: Reexamining Freedman’s critique

Open access
Lin, Winston·2013·The Annals of Applied Statistics
DOI 10.1214/12-aoas583
Open this paper in CoChat — free account

Abstract

Freedman [Adv. in Appl. Math. 40 (2008) 180–193; Ann. Appl. Stat. 2 (2008) 176–196] critiqued ordinary least squares regression adjustment of estimated treatment effects in randomized experiments, using Neyman’s model for randomization inference. Contrary to conventional wisdom, he argued that adjustment can lead to worsened asymptotic precision, invalid measures of precision, and small-sample bias. This paper shows that in sufficiently large samples, those problems are either minor or easily fixed. OLS adjustment cannot hurt asymptotic precision when a full set of treatment–covariate interactions is included. Asymptotically valid confidence intervals can be constructed with the Huber–White sandwich standard error estimator. Checks on the asymptotic approximations are illustrated with data from Angrist, Lang, and Oreopoulos’s [Am. Econ. J.: Appl. Econ. 1:1 (2009) 136–163] evaluation of strategies to improve college students’ achievement. The strongest reasons to support Freedman’s preference for unadjusted estimates are transparency and the dangers of specification search.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1214/12-aoas583

Cites

12 of 12 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 12 references

Research works printed but not linked: 59

  1. Holt, D. and Smith, T. M. F. (1979). Post stratification. J. Roy. Statist. Soc. Ser. A 142 33–46. · DOI · Paper page
  2. Gail, M. H., Mark, S. D., Carroll, R. J., Green, S. B. and Pee, D. (1996). On design considerations and randomization-based inference for community intervention trials. Stat. Med. 15 1069–1092. · DOI · Paper page
  3. Hansen, B. B. and Bowers, J. (2009). Attributing effects to a cluster-randomized get-out-the-vote campaign. J. Amer. Statist. Assoc. 104 873–885. · DOI
  4. Imbens, G. W. and Wooldridge, J. M. (2009). Recent developments in the econometrics of program evaluation. J. Econ. Lit. 47 5–86. · DOI · Paper page
  5. MacKinnon, J. G. and White, H. (1985). Some heteroskedasticity-consistent covariance matrix estimators with improved finite sample properties. J. Econometrics 29 305–325. · DOI · Paper page
  6. Angrist, J. D. and Imbens, G. W. (2002). Comment on “Covariance adjustment in randomized experiments and observational studies” by P. R. Rosenbaum. Statist. Sci. 17 304–307. · DOI · Paper page
  7. Angrist, J. D., Lang, D. and Oreopoulos, P. (2009). Incentives and services for college achievement: Evidence from a randomized trial. Am. Econ. J.: Appl. Econ. 1 136–163. · DOI · Paper page
  8. Ashenfelter, O. and Plant, M. W. (1990). Nonparametric estimates of the labor-supply effects of negative income tax programs. J. Labor Econ. 8 S396–S415. · DOI
  9. Berk, R., Barnes, G., Ahlman, L. and Kurtz, E. (2010). When second best is good enough: A comparison between a true experiment and a regression discontinuity quasi-experiment. J. Exp. Criminol. 6 191–208. · DOI
  10. Chamberlain, G. (1982). Multivariate regression models for panel data. J. Econometrics 18 5–46. · DOI · Paper page
  11. Cochran, W. G. (1942). Sampling theory when the sampling-units are of unequal sizes. J. Amer. Statist. Assoc. 37 199–212. · DOI
  12. Cochran, W. G. (1957). Analysis of covariance: Its nature and uses. Biometrics 13 261–281. · DOI · Paper page
  13. Cochran, W. G. (1969). The use of covariance in observational studies. J. R. Stat. Soc. Ser. C. Appl. Stat. 18 270–275. · doi:10.2307/2346587
  14. Cox, D. R. and McCullagh, P. (1982). Some aspects of analysis of covariance. Biometrics 38 541–561. · DOI
  15. Deaton, A. (2010). Instruments, randomization, and learning about development. J. Econ. Lit. 48 424–455. · DOI · Paper page
  16. Fienberg, S. E. and Tanur, J. M. (1987). Experimental and sampling structures: Parallels diverging and meeting. Internat. Statist. Rev. 55 75–96. · DOI
  17. Freedman, D. A. (1991). Statistical models and shoe leather (with discussion). Socio. Meth. 21 291–358. · DOI · Paper page
  18. Freedman, D. A. (2008b). On regression adjustments in experiments with several treatments. Ann. Appl. Stat. 2 176–196. · DOI · Paper page
  19. Freedman, D. A. (2008c). Editorial: Oasis or mirage? Chance 21(1) 59–61. Annotated references at http://www.stat.berkeley.edu/~census/chance.pdf. · doi:10.1007/s00144-008-0047-x
  20. Hinkley, D. V. (1977). Jacknifing in unbalanced situations. Technometrics 19 285–292. · DOI · Paper page
  21. Hinkley, D. V. and Wang, S. (1991). Efficiency of robust standard errors for regression coefficients. Comm. Statist. Theory Methods 20 1–11. · doi:10.1080/03610929108830479
  22. Imbens, G. W. (2010). Better LATE than nothing: Some comments on Deaton (2009) and Heckman and Urzua (2009). J. Econ. Lit. 48 399–423. · DOI · Paper page
  23. Klar, N. and Darlington, G. (2004). Methods for modelling change in cluster randomization trials. Stat. Med. 23 2341–2357. · DOI
  24. Kline, P. (2011). Oaxaca–Blinder as a reweighting estimator. Am. Econ. Rev. 101(3) 532–537. · DOI · Paper page
  25. Kline, P. and Santos, A. (2012). Higher order properties of the wild bootstrap under misspecification. J. Econometrics 171 54–70. · doi:10.1016/j.jeconom.2012.06.001
  26. Lin, W. (2013). Supplement to “Agnostic notes on regression adjustments to experimental data: Reexamining Freedman’s critique.” DOI:10.1214/12-AOAS583SUPP. · doi:10.1214/12-aoas583supp
  27. Miratrix, L. W., Sekhon, J. S. and Yu, B. (2012). Adjusting treatment effect estimates by post-stratification in randomized experiments. J. R. Stat. Soc. Ser. B. Stat. Methodol. 75 369–396. · DOI
  28. Raudenbush, S. W. (1997). Statistical analysis and optimal design for cluster randomized trials. Psychol. Meth. 2 173–185. · DOI · Paper page
  29. Reichardt, C. S. and Gollob, H. F. (1999). Justifying the use and increasing the power of a $t$ test for a randomized experiment with a convenience sample. Psychol. Meth. 4 117–128. · DOI
  30. Rosenbaum, P. R. (2002). Covariance adjustment in randomized experiments and observational studies. Statist. Sci. 17 286–327. · DOI · Paper page
  31. Royall, R. M. and Cumberland, W. G. (1978). Variance estimation in finite population sampling. J. Amer. Statist. Assoc. 73 351–358. · DOI
  32. Samii, C. and Aronow, P. M. (2012). On equivalencies between design-based and regression-based variance estimators for randomized experiments. Statist. Probab. Lett. 82 365–370. · DOI
  33. Schochet, P. Z. (2010). Is regression adjustment supported by the Neyman model for causal inference? J. Statist. Plann. Inference 140 246–259. · DOI
  34. Senn, S. J. (1989). Covariate imbalance and random allocation in clinical trials. Stat. Med. 8 467–475. · DOI · Paper page
  35. Stock, J. H. (2010). The other transformation in econometric practice: Robust tools for inference. J. Econ. Perspect. 24(2) 83–94. · DOI
  36. Stonehouse, J. M. and Forrester, G. J. (1998). Robustness of the $t$ and $U$ tests under combined assumption violations. J. Appl. Stat. 25 63–74. · DOI
  37. Tibshirani, R. (1986). Discussion of “Jackknife, bootstrap and other resampling methods in regression analysis” by C. F. J. Wu. Ann. Statist. 14 1335–1339. [Correction: (1988) 16 479.] · doi:10.1214/aos/1176350716
  38. Tsiatis, A. A., Davidian, M., Zhang, M. and Lu, X. (2008). Covariate adjustment for two-sample treatment comparisons in randomized clinical trials: A principled yet flexible approach. Stat. Med. 27 4658–4677. · DOI · Paper page
  39. Tukey, J. W. (1991). Use of many covariates in clinical trials. Internat. Statist. Rev. 59 123–137. · DOI
  40. Tukey, J. W. (1993). Tightening the clinical trial. Contr. Clin. Trials 14 266–285. · DOI
  41. Watson, D. J. (1937). The estimation of leaf area in field crops. J. Agr. Sci. 27 474–483. · doi:10.1017/s002185960005173x
  42. White, H. (1980a). Using least squares to approximate unknown regression functions. Internat. Econom. Rev. 21 149–170. · DOI · Paper page
  43. White, H. (1980b). A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica 48 817–838. · DOI · Paper page
  44. Wu, C. F. J. (1986). Jackknife, bootstrap and other resampling methods in regression analysis. Ann. Statist. 14 1261–1350. · DOI · Paper page
  45. Yang, L. and Tsiatis, A. A. (2001). Efficiency study of estimators for a treatment effect in a pretest-posttest trial. Amer. Statist. 55 314–321. · DOI · Paper page
  46. Cochran, W. G. (1977). Sampling Techniques, 3rd ed. Wiley, New York.
  47. Fuller, W. A. (2009). Sampling Statistics. Wiley, Hoboken, NJ. · DOI · Paper page
  48. Cox, D. R. and Reid, N. (2000). The Theory of the Design of Experiments. CRC Press, Boca Raton, FL. · DOI · Paper page
  49. Davidson, R. and MacKinnon, J. G. (1993). Estimation and Inference in Econometrics. Oxford Univ. Press, New York.
  50. Eicker, F. (1967). Limit theorems for regressions with unequal and dependent errors. In Proc. Fifth Berkeley Sympos. Math. Statist. and Probability (Berkeley, Calif., 1965/66), Vol. I 59–82. Univ. California Press, Berkeley, CA.
  51. Freedman, D. A. (2010). Survival analysis: An epidemiological hazard? In Statistical Models and Causal Inference: A Dialogue with the Social Sciences (D. Collier, J. S. Sekhon and P. B. Stark, eds.) 169–192. Cambridge Univ. Press, Cambridge. · DOI
  52. Green, D. P. and Aronow, P. M. (2011). Analyzing experimental data using regression: When is bias a practical concern? Working paper, Yale Univ. · DOI
  53. Greenberg, D. and Shroder, M. (2004). The Digest of Social Experiments, 3rd ed. Urban Institute Press, Washington, DC.
  54. Huber, P. J. (1967). The behavior of maximum likelihood estimates under nonstandard conditions. In Proc. Fifth Berkeley Sympos. Math. Statist. and Probability (Berkeley, Calif., 1965/66), Vol. I: Statistics 221–233. Univ. California Press, Berkeley, CA.
  55. MacKinnon, J. G. (2013). Thirty years of heteroskedasticity-robust inference. In Recent Advances and Future Directions in Causality, Prediction, and Specification Analysis: Essays in Honor of Halbert L. White Jr. (X. Chen and N. R. Swanson, eds.) 437–461. Springer, New York. · DOI
  56. Middleton, J. A. and Aronow, P. M. (2012). Unbiased estimation of the average treatment effect in cluster-randomized experiments. Working Paper, Yale Univ. · DOI
  57. Miller, R. G. Jr. (1986). Beyond ANOVA, Basics of Applied Statistics. Wiley, New York.
  58. Rubin, D. B. (1984). William G. Cochran’s contributions to the design, analysis, and evaluation of observational studies. In W. G. Cochran’s Impact on Statistics (P. S. R. S. Rao andJ. Sedransk, eds.) 37–69. Wiley, New York.
  59. Rubin, D. B. and van der Laan, M. J. (2011). Targeted ANCOVA estimator in RCTs. In Targeted Learning: Causal Inference for Observational and Experimental Data (M. J. van der Laan and S. Rose, eds.) 201–215. Springer, New York. · doi:10.1007/978-1-4419-9782-1_12

Cited by

See all 641

Metadata

Authors
Lin, W.
Venue
The Annals of Applied Statistics
Publisher
Institute of Mathematical Statistics
Published
2013-03-01
Volume
7
Issue
1
Pages
295-318
ISSN
1932-6157 (ISSN-L)
DOI
10.1214/12-aoas583
OpenAlex
W2063910719
Cited by
641
References
12
FWCI
19.50
Type
journal article
Language
en
Open access
bronze

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Agnostic notes on regression adjustments to experimental data: Reexamining Freedman’s critique. https://app.cochat.ai/papers/10.1214/12-aoas583
The paper
Lin, W. (2013). Agnostic notes on regression adjustments to experimental data: Reexamining Freedman’s critique. The Annals of Applied Statistics. https://doi.org/10.1214/12-aoas583

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.