Site Selection Bias in Program Evaluation *
The DOI matches its Crossref record; 44 of the 47 references it cites were found in a registry or an index.
Abstract
“Site selection bias” can occur when the probability that a program is adopted or evaluated is correlated with its impacts. I test for site selection bias in the context of the Opower energy conservation programs, using 111 randomized control trials involving 8.6 million households across the United States. Predictions based on rich microdata from the first 10 replications substantially overstate efficacy in the next 101 sites. Several mechanisms caused this positive selection. For example, utilities in more environmentalist areas are more likely to adopt the program, and their customers are more responsive to the treatment. Also, because utilities initially target treatment at higher-usage consumer subpopulations, efficacy drops as the program is later expanded. The results illustrate how program evaluations can still give systematically biased out-of-sample predictions, even after many replications.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“most interventions that are effective in initial efficacy trials go on to show weaker or no effects when they are scaled up in effectiveness trials that deliver treatments under everyday conditions to more heterogeneous samples21–23.”
From A national experiment reveals where a growth mindset improves achievement (2019) · cites this paper as [22]
In the citing paper
“Site selection in experimental evaluations of training programs (Joseph Hotz, Imbens, and Mortimer 2005) and energy conservation programs (Allcott 2015) matches this pattern.”
From Difference-in-differences with variation in treatment timing (2021) · cites this paper as Allcott 2015
Citations
Cites
Multiple Imputation for Nonresponse in Surveys
Rubin, D. B. (1987) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316696 · Cited by 20,701
Estimating causal effects of treatments in randomized and nonrandomized studies
Rubin, D. B. (1974) · Journal of Educational Psychology · doi:10.1037/h0037350 · Cited by 9,436 · cited 1×
“Consider the standard Rubin (1974) Causal Model.”
Donabedian, A. (1988) · JAMA · doi:10.1001/jama.260.12.1743 · Cited by 5,895 · cited 1×
“Remaining characteristics are grouped according to the standard Donabedian (1988) triad of clinical quality measures: structure, process, and outcomes.”
Identification and estimation of local average treatment effects
Imbens, G. W., & Angrist, J. D. (1994) · Econometrica · doi:10.2307/2951620 · Cited by 4,101
Selection on observed and unobserved variables: Assessing the effectiveness of Catholic schools
Altonji, J. G., Elder, T. E., & Taber, C. R. (2005) · Journal of Political Economy · doi:10.1086/426036 · Cited by 3,706 · cited 1×
“This echoes the result from Table VI that observables are not very informative about unobservables in this context.14 Thus, the logic of inferring the direction and magnitude of bias from unobservables (Altonji, Elder, and Taber 2005) would not work well here.”
The Constructive, Destructive, and Reconstructive Power of Social Norms
Schultz, P. W., Nolan, J. M., Cialdini, R. B., Goldstein, N. J., & Griskevicius, V. (2007) · Psychological Science · doi:10.1111/j.1467-9280.2007.01917.x · Cited by 3,643
Does matching overcome lalonde’s critique of nonexperimental estimators?
A. Smith, J., & E. Todd, P. (2005) · Journal of Econometrics · doi:10.1016/j.jeconom.2004.04.011 · Cited by 3,176
Identification and Estimation of Local Average Treatment Effects
Angrist, J., & Imbens, G. (1995) · National Bureau of Economic Research · doi:10.3386/t0118 · Cited by 2,808 · cited 1×
“Notice that site selection bias is quite distinct from the treatment effect heterogeneity relevant to defining local average treatment effects (Angrist and Imbens 1994): within-sample heterogeneity in τi is neither necessary nor sufficient for site selection bias.5 Notice also that site selection…”
Social norms and energy conservation
Allcott, H. (2011) · Journal of Public Economics · doi:10.1016/j.jpubeco.2011.03.003 · Cited by 2,802 · cited 1×
“Also closely related are the academic studies of early Opower programs, including Allcott (2011), Allcott and Rogers (2014), Ayres, Raseman, and Shih (2013), and Costa and Kahn (2013).”
Self-citationExperimental Analysis of Neighborhood Effects
Kling, J. R., Liebman, J. B., & Katz, L. F. (2007) · Econometrica · doi:10.1111/j.1468-0262.2007.00733.x · Cited by 2,719 · cited 1×
“For example, the Moving to Opportunity and RAND Health Insurance experiments were implemented in multiple cities chosen for diversity in size and geographic region (Kling, Liebman, and Katz 2007, Manning et al. 1988).”
Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs
Dehejia, R. H., & Wahba, S. (1999) · Journal of the American Statistical Association · doi:10.1080/01621459.1999.10473858 · Cited by 2,395
Characterizing Selection Bias Using Experimental Data
Heckman, J., Ichimura, H., Smith, J., & Todd, P. (1998) · Econometrica · doi:10.2307/2999630 · Cited by 2,286
Harrison, G. W., & List, J. A. (2004) · Journal of Economic Literature · doi:10.1257/0022051043004577 · Cited by 2,099
Normative social influence is underdetected
Nolan, J. M., Schultz, P. W., Cialdini, R. B., Goldstein, N. J., & Griskevicius, V. (2008) · Personality and Social Psychology Bulletin · doi:10.1177/0146167208316691 · Cited by 1,772
Angrist, J. D., & Pischke, J.-S. (2010) · Journal of Economic Perspectives · doi:10.1257/jep.24.2.3 · Cited by 1,566
Allcott, H., & Rogers, T. (2014) · American Economic Review · doi:10.1257/aer.104.10.3003 · Cited by 1,340 · cited 2×
“Complementarity is possible because one way that consumers respond to the Opower treatment is by participating in other utility programs, such as energy efficient insulation and lighting replacement (Allcott and Rogers 2014).”
Self-citationStructural Equations, Treatment Effects, and Econometric Policy Evaluation1
Heckman, J. J., & Vytlacil, E. (2005) · Econometrica · doi:10.1111/j.1468-0262.2005.00594.x · Cited by 1,220
Dealing with limited overlap in estimation of average treatment effects
Crump, R. K., Hotz, V. J., Imbens, G. W., & Mitnik, O. A. (2009) · Biometrika · doi:10.1093/biomet/asn055 · Cited by 1,007 · cited 14×
“Site: (1) (2) (3) (4) (5) (6) (7) (8) (9) (10)”
InfluentialSix Randomized Evaluations of Microcredit: Introduction and Further Steps
Banerjee, A., Karlan, D., & Zinman, J. (2015) · American Economic Journal: Applied Economics · doi:10.1257/app.20140287 · Cited by 988
Understanding Instrumental Variables in Models with Essential Heterogeneity
Heckman, J. J., Urzua, S., & Vytlacil, E. (2006) · Review of Economics and Statistics · doi:10.1162/rest.88.3.389 · Cited by 874 · cited 1×
“Thus, there is no need to model essential heterogeneity or household-level selection into the treatment (Heckman, Urzua, and Vytlacil 2006), and the treatment effect is a Policy-Relevant Treatment Effect in the sense of Heckman and Vytlacil (2001).”
Evaluating the Econometric Evaluations of Training Programs with Experimental DataPreprint
Lalonde, R. (1984) · RePEc: Research Papers in Economics · Cited by 839
Costa, D. L., & Kahn, M. E. (2013) · Journal of the European Economic Association · doi:10.1111/jeea.12011 · Cited by 656 · cited 1×
“Also closely related are the academic studies of early Opower programs, including Allcott (2011), Allcott and Rogers (2014), Ayres, Raseman, and Shih (2013), and Costa and Kahn (2013).”
Crépon, B., Duflo, E., Gurgand, M., Rathelot, R., & Zamora, P. (2013) · The Quarterly Journal of Economics · doi:10.1093/qje/qjt001 · Cited by 558
Ayres, I., Raseman, S., & Shih, A. (2013) · Journal of Law, Economics, and Organization · doi:10.1093/jleo/ews020 · Cited by 542
Wennberg, D. E., Lucas, F. L., Birkmeyer, J. D., Bredenberg, C. E., & Fisher, E. S. (1998) · JAMA · doi:10.1001/jama.279.16.1278 · Cited by 511 · cited 2×
“Wennberg et al. (1998) use a broader sample of administrative data to show that mortality rates were significantly higher at non-trial hospitals, and for some classes of patients and hospitals, treatment with drugs instead of the surgical procedure might have been preferred.”
Field, E., Pande, R., Papp, J., & Rigol, N. (2013) · American Economic Review · doi:10.1257/aer.103.6.2196 · Cited by 375
Productivity Spillovers in Health Care: Evidence from the Treatment of Heart Attacks
Chandra, A., & Staiger, D. O. (2007) · Journal of Political Economy · doi:10.1086/512249 · Cited by 366 · cited 1×
“Chandra and Staiger (2007) show that due to productivity spillovers, surgical procedures are more effective in areas that perform more surgeries, and they point out that this may compromise the external validity of randomized control trials.”
Indicator and Stratification Methods for Missing Explanatory Variables in Multiple Linear Regression
Jones, M. P. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476680 · Cited by 361 · cited 1×
“In earlier drafts, I instead used the missing indicator method, which is only unbiased under stronger assumptions (Jones 1996) but gives very similar results.”
Policy-Relevant Treatment Effects
Heckman, J. J., & Vytlacil, E. (2001) · American Economic Review · doi:10.1257/aer.91.2.107 · Cited by 285 · cited 1×
“Thus, there is no need to model essential heterogeneity or household-level selection into the treatment (Heckman, Urzua, and Vytlacil 2006), and the treatment effect is a Policy-Relevant Treatment Effect in the sense of Heckman and Vytlacil (2001).”
Giné, X., & Karlan, D. S. (2014) · Journal of Development Economics · doi:10.1016/j.jdeveco.2013.11.003 · Cited by 268 · cited 1×
“For example, the RCTs summarized by Banerjee, Karlan, and Zinman (2015) study various effects on borrowers, while Gine and Karlan (2014) study how group vs. individual liability impacts default rates, Field and Pande (2008) study how repayment frequency affects default rates, and Field, Pande,…”
Evaluating Welfare and Training Programs.
Crane, J., Manski, C. F., & Garfinkel, I. (1992) · Contemporary Sociology · doi:10.2307/2075695 · Cited by 253
Repayment Frequency and Default in Microfinance: Evidence From India
Field, E., & Pande, R. (2008) · Journal of the European Economic Association · doi:10.1162/jeea.2008.6.2-3.501 · Cited by 239 · cited 1×
“…and Zinman (2015) study various effects on borrowers, while Gine and Karlan (2014) study how group vs. individual liability impacts default rates, Field and Pande (2008) study how repayment frequency affects default rates, and Field, Pande, Papp, and Rigol (2013) study how delayed repayment…”
Nonparametric Tests for Treatment Effect Heterogeneity
Crump, R. K., Hotz, V. J., Imbens, G. W., & Mitnik, O. A. (2008) · Review of Economics and Statistics · doi:10.1162/rest.90.3.389 · Cited by 235
Predicting the efficacy of future training programs using past experiences at other locations
Joseph Hotz, V., Imbens, G. W., & Mortimer, J. H. (2005) · Journal of Econometrics · doi:10.1016/j.jeconom.2004.04.009 · Cited by 190
Scaling Up What Works: Experimental Evidence on External Validity in Kenyan Education
Bold, T., Kimenyi, M., Mwabu, G., Alice Ng'ang'a, A., & Sandefur, J. (2013) · SSRN Electronic Journal · doi:10.2139/ssrn.2241240 · Cited by 186 · cited 1×
“For example, “job training” often takes different forms at different sites (Dehejia 2003, Hotz, Imbens, and Klerman 2006), and the effects of “contract teachers” could depend markedly on the teacher’s ability and even who employs them (Bold et al. 2013).”
It pays to be ignorant: A simple political economy of rigorous program evaluation
Pritchett, L. (2002) · The Journal of Policy Reform · doi:10.1080/1384128032000096832 · Cited by 178 · cited 2×
“Pritchett (2002) shows how such incentives could lead to an equivalent of site 6All of Opower’s first ten sites had fee-for-service contracts without performance incentives.”
Imposing Moment Restrictions from Auxiliary Data by Weighting
Hellerstein, J. K., & Imbens, G. W. (1999) · Review of Economics and Statistics · doi:10.1162/003465399557860 · Cited by 163
Hotz, V. J., Imbens, G. W., & Klerman, J. A. (2006) · Journal of Labor Economics · doi:10.1086/505050 · Cited by 158
(1991) · SSRN Electronic Journal · Cited by 128
Health Insurance and the Demand for Medical Care
Manning, W. G., Newhouse, J. P., Duan, N., Keeler, E. B., Benjamin, B., Leibowitz, A. A., Marquis, M. S., & Zwanziger, J. (1987) · Cited by 120
Was There a Riverside Miracle? A Hierarchical Framework for Evaluating Programs With Grouped Data
Dehejia, R. (2003) · Journal of Business and Economic Statistics · doi:10.1198/073500102288618702 · Cited by 45
The Sensitivity of Experimental Impact Estimates: Evidence from the National JTPA Study
Heckman, J. J., & Smith, J. (1997) · National Bureau of Economic Research · doi:10.3386/w6105 · Cited by 26
Cost-Effectiveness of Electricity Energy Efficiency ProgramsPreprint
Arimura, T. H., Li, S., Newell, R. G., & Palmer, K. (2011) · RePEc: Research Papers in Economics · Cited by 5
Designing Experimental Evaluations of Social Programs: The Case of the U.S. National JTPA StudyPreprint
Hotz, V. J. (1992) · RePEc: Research Papers in Economics · Cited by 3
Research works printed but not linked: 2
- Imbens, 2014, Causal Inference in Statistics and the Social Sciences
- Keim, “The Hot New Frontier of Energy Research Is Human Behavior,”, Wired
Cited by
Difference-in-differences with variation in treatment timing
Goodman-Bacon, A. (2021) · Journal of Econometrics · doi:10.1016/j.jeconom.2021.03.014 · Cited by 8,046
A national experiment reveals where a growth mindset improves achievement
Yeager, D. S., Hanselman, P., Walton, G. M., Murray, J. S., Crosnoe, R., Muller, C., Tipton, E., Schneider, B., Hulleman, C. S., Hinojosa, C. P., et al. (2019) · Nature · doi:10.1038/s41586-019-1466-y · Cited by 1,513
The State of Applied Econometrics: Causality and Policy Evaluation
Athey, S., & Imbens, G. W. (2017) · Journal of Economic Perspectives · doi:10.1257/jep.31.2.3 · Cited by 1,397
Allcott, H., & Rogers, T. (2014) · American Economic Review · doi:10.1257/aer.104.10.3003 · Cited by 1,340
Self-citationInterpreting effect sizes of education interventions
Kraft, M. A. (2020) · Educational Researcher · doi:10.3102/0013189x20912798 · Cited by 1,281
“Students, teachers, schools, and districts are more likely to participate in a study when they think they will benefit, causing selection bias (Allcott, 2015).”
Metadata
- Authors
- Allcott, H.
- Affiliations
- New York University (US)
- Corresponding author
- Allcott, Hunt
- Venue
- The Quarterly Journal of Economics
- Publisher
- Oxford University Press (OUP)
- Published
- 2015-03-18
- Volume
- 130
- Issue
- 3
- Pages
- 1117-1165
- ISSN
- 0033-5533 (ISSN-L)
- 1531-4650
- OpenAlex
- W2143326716
- Cited by
- 426
- References
- 47
- Citations by year
- 2026: 33 · 2025: 34 · 2024: 34 · 2023: 36 · 2022: 39 · 2021: 35 · 2020: 59 · 2019: 40 · 2018: 31 · 2017: 33 · 2016: 27 · 2015: 12
- FWCI
- 31.14
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- The Quarterly Journal of Economics (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Physical Sciences › Environmental Science › Management, Monitoring, Policy and Law › Pro-Environmental Behavior and Attitudes
- Keywords
- program evaluation, energy conservation programs, randomized controlled trials, out-of-sample forecasting, program adoption, program replication
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.