The DOI matches its Crossref record; 76 of the 76 references it cites were found in a registry or an index.
Agnostic notes on regression adjustments to experimental data: Reexamining Freedman’s critique
Open accessAbstract
Freedman [Adv. in Appl. Math. 40 (2008) 180–193; Ann. Appl. Stat. 2 (2008) 176–196] critiqued ordinary least squares regression adjustment of estimated treatment effects in randomized experiments, using Neyman’s model for randomization inference. Contrary to conventional wisdom, he argued that adjustment can lead to worsened asymptotic precision, invalid measures of precision, and small-sample bias. This paper shows that in sufficiently large samples, those problems are either minor or easily fixed. OLS adjustment cannot hurt asymptotic precision when a full set of treatment–covariate interactions is included. Asymptotically valid confidence intervals can be constructed with the Huber–White sandwich standard error estimator. Checks on the asymptotic approximations are illustrated with data from Angrist, Lang, and Oreopoulos’s [Am. Econ. J.: Appl. Econ. 1:1 (2009) 136–163] evaluation of strategies to improve college students’ achievement. The strongest reasons to support Freedman’s preference for unadjusted estimates are transparency and the dangers of specification search.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity
White, H. (1980) · Econometrica · doi:10.2307/1912934 · Cited by 25,861
Mostly Harmless Econometrics: An Empiricist's Companion
Angrist, J. D., & Pischke, J.-S. (2009) · doi:10.1515/9781400829828 · Cited by 9,467
Estimating causal effects of treatments in randomized and nonrandomized studies
Rubin, D. B. (1974) · Journal of Educational Psychology · doi:10.1037/h0037350 · Cited by 9,436 · cited 4×
“(iv) The tyranny estimator can also be seen as a one-step variant of Rubin and van der Laan’s (2011) two-step “targeted ANCOVA.” Their estimator is equivalent to the difference in means of the residuals from a weighted least squares regression of Yi on zi , with the same weights as in remark (iii). (v) When is the usu…”
BackgroundMethodInfluentialMoher, D., Hopewell, S., Schulz, K. F., Montori, V., Gotzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c869 · Cited by 8,883 · cited 2×
“By Winston Lin”
MethodBootstrap Methods and their Application
Davison, A. C., & Hinkley, D. V. (1997) · doi:10.1017/cbo9780511802843 · Cited by 7,350 · cited 2×
“12Finite-population bootstrap methods [Davison and Hinkley (1997), pp. 92–100, 125] may also be useful for estimating the bias of ÂTEinteract, but similar caveats would apply.”
MethodBootstrap Methods and Their Application
Loughin, T. M., Davison, A. C., & Hinkley, D. V. (1999) · Journal of the American Statistical Association · doi:10.2307/2669707 · Cited by 5,418
Estimation and Inference in Econometrics.
Chambers, M. J., Davidson, R., & MacKinnon, J. G. (1994) · The Economic Journal · doi:10.2307/2234656 · Cited by 5,413
Statistics and Causal Inference
Holland, P. W. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478354 · Cited by 5,293 · cited 2×
“Assume that each subject has two potential outcomes [Neyman (1923), Rubin (1974, 2005), Holland (1986)], ai and bi, which would be observed under treatments A and B, respectively.3 Thus, the observed outcome is Yi = aiTi + bi(1− Ti), where Ti is a dummy variable for treatment A. Random assignment…”
BackgroundMostly Harmless Econometrics: An Empiricist's Companion
Angrist, J. D., & Pischke, J.-S. (2008) · doi:10.2307/j.ctvcm4j72 · Cited by 5,051
Recent Developments in the Econometrics of Program Evaluation
Imbens, G. W., & Wooldridge, J. M. (2009) · Journal of Economic Literature · doi:10.1257/jel.47.1.5 · Cited by 4,926
(1967) · Project Euclid (Cornell University) · Cited by 4,720
Mostly Harmless Econometrics: An Empiricist's Companion
Kmenta, J. (2010) · Business Economics · doi:10.1057/be.2009.37 · Cited by 2,495
Estimation and Inference in Econometrics.
Levin, A., Davidson, R., & MacKinnon, J. G. (1994) · Journal of the American Statistical Association · doi:10.2307/2290953 · Cited by 2,396
Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gøtzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2012) · International Journal of Surgery · doi:10.1016/j.ijsu.2011.10.001 · Cited by 2,345
Causal Inference Using Potential Outcomes: Design, Modeling, Decisions
Rubin, D. B. (2005) · Journal of the American Statistical Association · doi:10.1198/016214504000001880 · Cited by 2,173
MacKinnon, J. G., & White, H. (1985) · Journal of Econometrics · doi:10.1016/0304-4076(85)90158-7 · Cited by 1,879
Jackknife, Bootstrap and Other Resampling Methods in Regression Analysis
Wu, C. F. J. (1986) · The Annals of Statistics · doi:10.1214/aos/1176350142 · Cited by 1,724
Multivariate regression models for panel data
Chamberlain, G. (1982) · Journal of Econometrics · doi:10.1016/0304-4076(82)90094-x · Cited by 1,292
Instruments, Randomization, and Learning about Development
Deaton, A. (2010) · Journal of Economic Literature · doi:10.1257/jel.48.2.424 · Cited by 1,286
Design of Observational Studies
Rosenbaum, P. R. (2010) · Springer Series in Statistics · doi:10.1007/978-1-4419-1213-8 · Cited by 1,150
Statistical Methods for Research Workers.
Hotelling, H., & Fisher, R. A. (1933) · Journal of the American Statistical Association · doi:10.2307/2278451 · Cited by 839
Complex Surveys: A Guide to Analysis Using R
Lumley, T. (2010) · doi:10.1002/9780470580066 · Cited by 647
Statistical analysis and optimal design for cluster randomized trials.
Raudenbush, S. W. (1997) · Psychological Methods · doi:10.1037/1082-989x.2.2.173 · Cited by 636
(1986) · CERN Document Server (European Organization for Nuclear Research) · Cited by 583
Complex Surveys: A Guide to Analysis Using R
Lumley, T. (2010) · Cited by 571
Using Least Squares to Approximate Unknown Regression Functions
White, H. (1980) · International Economic Review · doi:10.2307/2526245 · Cited by 571
The Theory of the Design of Experiments
Cox, D. R., & Reid, N. (2000) · Chapman & Hall/CRC eBooks · doi:10.1201/9781420035834 · Cited by 559
Recent Developments in the Econometrics of Program Evaluation
Imbens, G., & Wooldridge, J. (2008) · National Bureau of Economic Research · doi:10.3386/w14251 · Cited by 533
Analysis of Covariance: Its Nature and Uses
Cochran, W. G. (1957) · Biometrics · doi:10.2307/2527916 · Cited by 504
On The So-Called “Huber Sandwich Estimator” and “Robust Standard Errors”
Freedman, D. A. (2006) · The American Statistician · doi:10.1198/000313006x152207 · Cited by 498
Covariance Adjustment in Randomized Experiments and Observational Studies
Rosenbaum, P. R. (2002) · Statistical Science · doi:10.1214/ss/1042727942 · Cited by 477
Hays, W. L. (1988) · Cited by 462
Lessons from the U.S. Unemployment Insurance Experiments
Meyer, B. (2007) · RePEc: Research Papers in Economics · Cited by 426
On regression adjustments to experimental data
Freedman, D. A. (2008) · Advances in Applied Mathematics · doi:10.1016/j.aam.2006.12.003 · Cited by 417
Incentives and Services for College Achievement: Evidence from a Randomized Trial
Angrist, J., Lang, D., & Oreopoulos, P. (2009) · American Economic Journal: Applied Economics · doi:10.1257/app.1.1.136 · Cited by 405
Tsiatis, A. A., Davidian, M., Zhang, M., & Lu, X. (2008) · Statistics in Medicine · doi:10.1002/sim.3113 · Cited by 349
Statistical models and shoe leather
Freedman, D. A. (1991) · Sociological Methodology · doi:10.2307/270939 · Cited by 333
Beyond ANOVA, Basics of Applied Statistics
Amster, S. J. (1987) · Technometrics · doi:10.1080/00401706.1987.10488261 · Cited by 289
Covariate imbalance and random allocation in clinical trials
Senn, S. J. (1989) · Statistics in Medicine · doi:10.1002/sim.4780080410 · Cited by 279
Better LATE Than Nothing: Some Comments on Deaton (2009) and Heckman and Urzua (2009)
Imbens, G. W. (2010) · Journal of Economic Literature · doi:10.1257/jel.48.2.399 · Cited by 278
Limit Theorems for Regressions with Unequal and Dependent Errors
Eicker, F. (1967) · Project Euclid (Cornell University) · Cited by 261
Holt, D., & Smith, T. M. F. (1979) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2344652 · Cited by 257
ON DESIGN CONSIDERATIONS AND RANDOMIZATION-BASED INFERENCE FOR COMMUNITY INTERVENTION TRIALS
GAIL, M. H., MARK, S. D., CARROLL, R. J., GREEN, S. B., & PEE, D. (1996) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19960615)15:11<1069::aid-sim220>3.0.co;2-q · Cited by 248
Jackknifing in Unbalanced Situations
Hinkley, D. V. (1977) · Technometrics · doi:10.1080/00401706.1977.10489550 · Cited by 243
Fuller, W. A. (2009) · Wiley eBooks · doi:10.1002/9780470523551 · Cited by 237
Exact and asymptotically robust permutation tests
Chung, E., & Romano, J. P. (2013) · The Annals of Statistics · doi:10.1214/13-aos1090 · Cited by 236
Oaxaca-Blinder as a Reweighting Estimator
Kline, P. (2011) · American Economic Review · doi:10.1257/aer.101.3.532 · Cited by 186
Efficiency Study of Estimators for a Treatment Effect in a Pretest–Posttest Trial
Yang, L., & Tsiatis, A. A. (2001) · The American Statistician · doi:10.1198/000313001753272466 · Cited by 184
On regression adjustments in experiments with several treatments
Freedman, D. A. (2008) · The Annals of Applied Statistics · doi:10.1214/07-aoas143 · Cited by 177
A Biometrics Invited Paper with Discussion. Some Aspects of Analysis of Covariance
Cox, D. R., & McCullagh, P. (1982) · Biometrics · doi:10.2307/2530040 · Cited by 164
Thirty Years of Heteroskedasticity-Robust Inference
MacKinnon, J. G. (2012) · Springer eBooks · doi:10.1007/978-1-4614-1653-1_17 · Cited by 157
Adjusting Treatment Effect Estimates by Post-Stratification in Randomized Experiments
Miratrix, L. W., Sekhon, J. S., & Yu, B. (2012) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.1467-9868.2012.01048.x · Cited by 156
Some aspects of analysis of covariance.
Cox, D. R., & McCullagh, P. (1982) · PubMed · Cited by 144
The digest of social experiments
(2005) · Choice Reviews Online · doi:10.5860/choice.42-3801 · Cited by 140
Variance Estimation in Finite Population Sampling
Royall, R. M., & Cumberland, W. G. (1978) · Journal of the American Statistical Association · doi:10.1080/01621459.1978.10481581 · Cited by 137
Sampling Theory When the Sampling-Units are of Unequal Sizes
Cochran, W. G. (1942) · Journal of the American Statistical Association · doi:10.1080/01621459.1942.10500626 · Cited by 136
Tukey, J. W. (1993) · Controlled Clinical Trials · doi:10.1016/0197-2456(93)90225-3 · Cited by 135
Better LATE Than Nothing: Some Comments on Deaton (2009) and Heckman and Urzua (2009)
Imbens, G. W. (2009) · National Bureau of Economic Research · doi:10.3386/w14896 · Cited by 118
Design of Observational Studies
Bedogni, G. (2011) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2010.00676_8.x · Cited by 108
Unbiased Estimation of the Average Treatment Effect in Cluster-Randomized Experiments
Middleton, J. A., & Aronow, P. M. (2015) · Statistics Politics and Policy · doi:10.1515/spp-2013-0002 · Cited by 98
Robustness of the t and U tests under combined assumption violations
Stonehouse, J. M., & Forrester, G. J. (1998) · Journal of Applied Statistics · doi:10.1080/02664769823304 · Cited by 96
Berk, R. A., Barnes, G. C., Ahlman, L., & Kurtz, E. (2010) · Journal of Experimental Criminology · doi:10.1007/s11292-010-9095-3 · Cited by 88
The Other Transformation in Econometric Practice: Robust Tools for Inference
Stock, J. H. (2010) · The Journal of Economic Perspectives · doi:10.1257/jep.24.2.83 · Cited by 83
Attributing Effects to a Cluster-Randomized Get-Out-the-Vote Campaign
Hansen, B. B., & Bowers, J. (2009) · Journal of the American Statistical Association · doi:10.1198/jasa.2009.ap06589 · Cited by 82
Nonparametric Estimates of the Labor-Supply Effects of Negative Income Tax Programs
Ashenfelter, O., & Plant, M. W. (1990) · Journal of Labor Economics · doi:10.1086/298255 · Cited by 81
Samii, C., & Aronow, P. M. (2011) · Statistics & Probability Letters · doi:10.1016/j.spl.2011.10.024 · Cited by 74
Is regression adjustment supported by the Neyman model for causal inference?
Schochet, P. Z. (2009) · Journal of Statistical Planning and Inference · doi:10.1016/j.jspi.2009.07.008 · Cited by 65
Methods for modelling change in cluster randomization trials
Klar, N. S., & Darlington, G. (2004) · Statistics in Medicine · doi:10.1002/sim.1858 · Cited by 65
Asymptotically valid and exact permutation tests based on two-sample U -statistics
Chung, E., & Romano, J. P. (2015) · Journal of Statistical Planning and Inference · doi:10.1016/j.jspi.2015.07.004 · Cited by 61
Experimental and Sampling Structures: Parallels Diverging and Meeting
Fienberg, S. E., & Tanur, J. M. (1987) · International Statistical Review · doi:10.2307/1403272 · Cited by 54
Regression Estimation for Survey Samples
Fuller, W. A. (2002) · Cited by 51
Reichardt, C. S., & Gollob, H. F. (1999) · Psychological Methods · doi:10.1037/1082-989x.4.1.117 · Cited by 38
Use of Many Covariates in Clinical Trials
Tukey, J. W. (1991) · International Statistical Review · doi:10.2307/1403439 · Cited by 27
William G. Cochran's Contributions to the Design, Analysis, and Evaluation of Observational Studies
Rubin, D. B. (2006) · Cambridge University Press eBooks · doi:10.1017/cbo9780511810725.004 · Cited by 18
Analyzing Experimental Data Using Regression: When is Bias a Practical Concern?Preprint
GREEN, D. P., & Aronow, P. M. (2009) · SSRN Electronic Journal · doi:10.2139/ssrn.1466886 · Cited by 14
Survival Analysis: An Epidemiological Hazard?
Collier, D. (2009) · Cambridge University Press eBooks · doi:10.1017/cbo9780511815874.013 · Cited by 6
Research works printed but not linked: 10
- Cochran, W. G. (1969). The use of covariance in observational studies. J. R. Stat. Soc. Ser. C. Appl. Stat. 18 270–275. · doi:10.2307/2346587
- Freedman, D. A. (2008c). Editorial: Oasis or mirage? Chance 21(1) 59–61. Annotated references at http://www.stat.berkeley.edu/~census/chance.pdf. · doi:10.1007/s00144-008-0047-x
- Hinkley, D. V. and Wang, S. (1991). Efficiency of robust standard errors for regression coefficients. Comm. Statist. Theory Methods 20 1–11. · doi:10.1080/03610929108830479
- Kline, P. and Santos, A. (2012). Higher order properties of the wild bootstrap under misspecification. J. Econometrics 171 54–70. · doi:10.1016/j.jeconom.2012.06.001
- Lin, W. (2013). Supplement to “Agnostic notes on regression adjustments to experimental data: Reexamining Freedman’s critique.” DOI:10.1214/12-AOAS583SUPP. · doi:10.1214/12-aoas583supp
- Rosenbaum, P. R. (2002). Covariance adjustment in randomized experiments and observational studies. Statist. Sci. 17 286–327. · DOI · Paper page
- Tibshirani, R. (1986). Discussion of “Jackknife, bootstrap and other resampling methods in regression analysis” by C. F. J. Wu. Ann. Statist. 14 1335–1339. [Correction: (1988) 16 479.] · doi:10.1214/aos/1176350716
- Watson, D. J. (1937). The estimation of leaf area in field crops. J. Agr. Sci. 27 474–483. · doi:10.1017/s002185960005173x
- Cochran, W. G. (1977). Sampling Techniques, 3rd ed. Wiley, New York.
- Rubin, D. B. and van der Laan, M. J. (2011). Targeted ANCOVA estimator in RCTs. In Targeted Learning: Causal Inference for Observational and Experimental Data (M. J. van der Laan and S. Rose, eds.) 201–215. Springer, New York. · doi:10.1007/978-1-4419-9782-1_12
Cited by
What's trending in difference-in-differences? A synthesis of the recent econometrics literature
Roth, J., Sant’Anna, P. H. C., Bilinski, A., & Poe, J. (2023) · Journal of Econometrics · doi:10.1016/j.jeconom.2023.03.008 · Cited by 2,027
Hainmueller, J., Mummolo, J., & Xu, Y. (2019) · Political Analysis · doi:10.1017/pan.2018.46 · Cited by 959
How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It
Montgomery, J. M., Nyhan, B., & Torres, M. (2018) · American Journal of Political Science · doi:10.1111/ajps.12357 · Cited by 800
Regression Discontinuity Designs Using Covariates
Calonico, S., Cattaneo, M. D., Farrell, M. H., & Titiunik, R. (2019) · The Review of Economics and Statistics · doi:10.1162/rest_a_00760 · Cited by 678
Sources of Authoritarian Responsiveness: A Field Experiment in China
Chen, J., Pan, J., & Xu, Y. (2016) · American Journal of Political Science · doi:10.1111/ajps.12207 · Cited by 636
Metadata
- Authors
- Lin, W.
- Venue
- The Annals of Applied Statistics
- Publisher
- Institute of Mathematical Statistics
- Published
- 2013-03-01
- Volume
- 7
- Issue
- 1
- Pages
- 295-318
- ISSN
- 1932-6157 (ISSN-L)
- OpenAlex
- W2063910719
- Cited by
- 641
- References
- 76
- FWCI
- 19.50
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.