The DOI matches its Crossref record; 44 of the 47 references it cites were found in a registry or an index.
Site Selection Bias in Program Evaluation *
Abstract
“Site selection bias” can occur when the probability that a program is adopted or evaluated is correlated with its impacts. I test for site selection bias in the context of the Opower energy conservation programs, using 111 randomized control trials involving 8.6 million households across the United States. Predictions based on rich microdata from the first 10 replications substantially overstate efficacy in the next 101 sites. Several mechanisms caused this positive selection. For example, utilities in more environmentalist areas are more likely to adopt the program, and their customers are more responsive to the treatment. Also, because utilities initially target treatment at higher-usage consumer subpopulations, efficacy drops as the program is later expanded. The results illustrate how program evaluations can still give systematically biased out-of-sample predictions, even after many replications.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“most interventions that are effective in initial efficacy trials go on to show weaker or no effects when they are scaled up in effectiveness trials that deliver treatments under everyday conditions to more heterogeneous samples21–23.”
From A national experiment reveals where a growth mindset improves achievement (2019) · cites this paper as [22]
In the citing paper
“Site selection in experimental evaluations of training programs (Joseph Hotz, Imbens, and Mortimer 2005) and energy conservation programs (Allcott 2015) matches this pattern.”
From Difference-in-differences with variation in treatment timing (2021) · cites this paper as Allcott 2015
Citations
Cites
Multiple Imputation for Nonresponse in Surveys
Rubin, D. B. (1987) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316696 · Cited by 20,701
Estimating causal effects of treatments in randomized and nonrandomized studies
Rubin, D. B. (1974) · Journal of Educational Psychology · doi:10.1037/h0037350 · Cited by 9,436 · cited 1×
“Consider the standard Rubin (1974) Causal Model.”
Donabedian, A. (1988) · JAMA · doi:10.1001/jama.260.12.1743 · Cited by 5,895 · cited 1×
“Remaining characteristics are grouped according to the standard Donabedian (1988) triad of clinical quality measures: structure, process, and outcomes.”
Identification and estimation of local average treatment effects
Imbens, G. W., & Angrist, J. D. (1994) · Econometrica · doi:10.2307/2951620 · Cited by 4,101
Selection on observed and unobserved variables: Assessing the effectiveness of Catholic schools
Altonji, J. G., Elder, T. E., & Taber, C. R. (2005) · Journal of Political Economy · doi:10.1086/426036 · Cited by 3,706 · cited 1×
“This echoes the result from Table VI that observables are not very informative about unobservables in this context.14 Thus, the logic of inferring the direction and magnitude of bias from unobservables (Altonji, Elder, and Taber 2005) would not work well here.”
The Constructive, Destructive, and Reconstructive Power of Social Norms
Schultz, P. W., Nolan, J. M., Cialdini, R. B., Goldstein, N. J., & Griskevicius, V. (2007) · Psychological Science · doi:10.1111/j.1467-9280.2007.01917.x · Cited by 3,643
Does matching overcome lalonde’s critique of nonexperimental estimators?
A. Smith, J., & E. Todd, P. (2005) · Journal of Econometrics · doi:10.1016/j.jeconom.2004.04.011 · Cited by 3,176
Identification and Estimation of Local Average Treatment Effects
Angrist, J., & Imbens, G. (1995) · National Bureau of Economic Research · doi:10.3386/t0118 · Cited by 2,808 · cited 1×
“Notice that site selection bias is quite distinct from the treatment effect heterogeneity relevant to defining local average treatment effects (Angrist and Imbens 1994): within-sample heterogeneity in τi is neither necessary nor sufficient for site selection bias.5 Notice also that site selection…”
Social norms and energy conservation
Allcott, H. (2011) · Journal of Public Economics · doi:10.1016/j.jpubeco.2011.03.003 · Cited by 2,802 · cited 1×
“Also closely related are the academic studies of early Opower programs, including Allcott (2011), Allcott and Rogers (2014), Ayres, Raseman, and Shih (2013), and Costa and Kahn (2013).”
Self-citationExperimental Analysis of Neighborhood Effects
Kling, J. R., Liebman, J. B., & Katz, L. F. (2007) · Econometrica · doi:10.1111/j.1468-0262.2007.00733.x · Cited by 2,719 · cited 1×
“For example, the Moving to Opportunity and RAND Health Insurance experiments were implemented in multiple cities chosen for diversity in size and geographic region (Kling, Liebman, and Katz 2007, Manning et al. 1988).”
Research works printed but not linked: 2
- Imbens, 2014, Causal Inference in Statistics and the Social Sciences
- Keim, “The Hot New Frontier of Energy Research Is Human Behavior,”, Wired
Cited by
Difference-in-differences with variation in treatment timing
Goodman-Bacon, A. (2021) · Journal of Econometrics · doi:10.1016/j.jeconom.2021.03.014 · Cited by 8,046
A national experiment reveals where a growth mindset improves achievement
Yeager, D. S., Hanselman, P., Walton, G. M., Murray, J. S., Crosnoe, R., Muller, C., Tipton, E., Schneider, B., Hulleman, C. S., Hinojosa, C. P., et al. (2019) · Nature · doi:10.1038/s41586-019-1466-y · Cited by 1,513
The State of Applied Econometrics: Causality and Policy Evaluation
Athey, S., & Imbens, G. W. (2017) · Journal of Economic Perspectives · doi:10.1257/jep.31.2.3 · Cited by 1,397
Allcott, H., & Rogers, T. (2014) · American Economic Review · doi:10.1257/aer.104.10.3003 · Cited by 1,340
Self-citationInterpreting effect sizes of education interventions
Kraft, M. A. (2020) · Educational Researcher · doi:10.3102/0013189x20912798 · Cited by 1,281
“Students, teachers, schools, and districts are more likely to participate in a study when they think they will benefit, causing selection bias (Allcott, 2015).”
Metadata
- Authors
- Allcott, H.
- Affiliations
- New York University (US)
- Corresponding author
- Allcott, Hunt
- Venue
- The Quarterly Journal of Economics
- Publisher
- Oxford University Press (OUP)
- Published
- 2015-03-18
- Volume
- 130
- Issue
- 3
- Pages
- 1117-1165
- ISSN
- 0033-5533 (ISSN-L)
- 1531-4650
- OpenAlex
- W2143326716
- Cited by
- 426
- References
- 47
- Citations by year
- 2026: 33 · 2025: 34 · 2024: 34 · 2023: 36 · 2022: 39 · 2021: 35 · 2020: 59 · 2019: 40 · 2018: 31 · 2017: 33 · 2016: 27 · 2015: 12
- FWCI
- 31.14
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- The Quarterly Journal of Economics (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Physical Sciences › Environmental Science › Management, Monitoring, Policy and Law › Pro-Environmental Behavior and Attitudes
- Keywords
- program evaluation, energy conservation programs, randomized controlled trials, out-of-sample forecasting, program adoption, program replication
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Site Selection Bias in Program Evaluation *. https://app.cochat.ai/papers/10.1093/qje/qjv015
- The paper
- Allcott, H. (2015). Site Selection Bias in Program Evaluation *. The Quarterly Journal of Economics. https://doi.org/10.1093/qje/qjv015
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.