Evaluating energy behavior change programs using randomized controlled trials: Best practice guidelines for policymakers
Open accessThe DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Abstract
Governments and policymakers around the globe are becoming increasingly interested in how to effectively change the behavior of energy consumers. In the residential sector, numerous programs are attempting to shift the behavior of individuals and households in the public interest—for example toward more energy efficient practices, greater uptake of demand-side management technology, increased use of renewable energy, and better responsiveness to new tariffs (e.g., dynamic pricing), to name but a few. However, the effectiveness of such behavior change interventions is often limited, or even unknown, due to weaknesses in program design and evaluation of program impact on behavior. To help policymakers avoid such pitfalls, this paper outlines some practical guidelines for designing, conducting and, most importantly, evaluating the impact of energy-related behavior change programs and initiatives. We explain why randomized controlled trials are generally the optimal approach for obtaining scientifically valid estimates of a behavioral program’s efficacy and effectiveness. In parallel, we offer specific guidelines for strengthening the validity, reliability and generalizability of empirical findings about program impact on behavior. Adopting these guidelines will help to improve program design and delivery, thereby allowing more accurate evaluation of the true cost-effectiveness, utility and mass-scalability of future energy-related behavioral interventions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistics and Causal Inference
Holland, P. W. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478354 · Cited by 5,293
Encouraging pro-environmental behaviour: An integrative review and research agenda
Steg, L., & Vlek, C. (2009) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2008.10.004 · Cited by 5,089
A Brief Tutorial on the Development of Measures for Use in Survey Questionnaires
Hinkin, T. R. (1998) · Organizational Research Methods · doi:10.1177/109442819800100106 · Cited by 4,095
Orne, M. T. (1962) · American Psychologist · doi:10.1037/h0043424 · Cited by 4,041
CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials
Schulz, K. F., Altman, D. G., Moher, D., & for the CONSORT Group* (2010) · Annals of Internal Medicine · doi:10.7326/0003-4819-152-11-201006010-00232 · Cited by 3,824
Webb, T. L., & Sheeran, P. (2006) · Psychological Bulletin · doi:10.1037/0033-2909.132.2.249 · Cited by 3,732
Intention—Behavior Relations: A Conceptual and Empirical Review
Sheeran, P. (2002) · European Review of Social Psychology · doi:10.1080/14792772143000003 · Cited by 3,688
The Constructive, Destructive, and Reconstructive Power of Social Norms
Schultz, P. W., Nolan, J. M., Cialdini, R. B., Goldstein, N. J., & Griskevicius, V. (2007) · Psychological Science · doi:10.1111/j.1467-9280.2007.01917.x · Cited by 3,643
Impression management: A literature review and two-component model.
Leary, M. R., & Kowalski, R. M. (1990) · Psychological Bulletin · doi:10.1037/0033-2909.107.1.34 · Cited by 3,334
A Review of Scale Development Practices in the Study of Organizations
Hinkin, T. R. (1995) · Journal of Management · doi:10.1177/014920639502100509 · Cited by 3,193
Self-reports: How the questions shape the answers.
Schwarz, N. (1999) · American Psychologist · doi:10.1037/0003-066x.54.2.93 · Cited by 2,846
A structural approach to selection bias
Hernán, M. A., Hernández-Díaz, S., & Robins, J. M. (2004) · Epidemiology · doi:10.1097/01.ede.0000135174.63482.43 · Cited by 2,844
Social norms and energy conservation
Allcott, H. (2011) · Journal of Public Economics · doi:10.1016/j.jpubeco.2011.03.003 · Cited by 2,802
Statistical methods in psychology journals: Guidelines and explanations.
Wilkinson, L., & Task Force on Statistical Inference (1999) · American Psychologist · doi:10.1037/0003-066x.54.8.594 · Cited by 2,791
A review of intervention studies aimed at household energy conservation
Abrahamse, W., Steg, L., Vlek, C., & Rothengatter, T. (2005) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2005.08.002 · Cited by 2,668
Bellg, A. J., Treatment Fidelity Workgroup of the NIH Behavior Change Consortium, Borrelli, B., Resnick, B., Hecht, J., Minicucci, D. S., Ory, M., Ogedegbe, G., Orwig, D., Ernst, D., et al. (2004) · Health Psychology · doi:10.1037/0278-6133.23.5.443 · Cited by 2,620
External validity of randomised controlled trials: “To whom do the results of this trial apply?”
Rothwell, P. M. (2005) · The Lancet · doi:10.1016/s0140-6736(04)17670-8 · Cited by 2,544
Causal Inference Using Potential Outcomes: Design, Modeling, Decisions
Rubin, D. B. (2005) · Journal of the American Statistical Association · doi:10.1198/016214504000001880 · Cited by 2,173
What is meant by intention to treat analysis? Survey of published randomised controlled trials
Hollis, S., & Campbell, F. (1999) · BMJ · doi:10.1136/bmj.319.7211.670 · Cited by 2,160
Energy efficiency and consumption—The rebound effect—A survey
A. Greening, L., Greene, D. L., & Difiglio, C. (2000) · Energy Policy · doi:10.1016/s0301-4215(00)00021-5 · Cited by 2,077
Normative social influence is underdetected
Nolan, J. M., Schultz, P. W., Cialdini, R. B., Goldstein, N. J., & Griskevicius, V. (2008) · Personality and Social Psychology Bulletin · doi:10.1177/0146167208316691 · Cited by 1,772
Allcott, H., & Rogers, T. (2014) · American Economic Review · doi:10.1257/aer.104.10.3003 · Cited by 1,340
Standards of Evidence: Criteria for Efficacy, Effectiveness and Dissemination
Flay, B. R., Biglan, A., Boruch, R. F., Castro, F. G., Gottfredson, D., Kellam, S., Mościcki, E. K., Schinke, S., Valentine, J. C., & Ji, P. (2005) · Prevention Science · doi:10.1007/s11121-005-5553-y · Cited by 1,338
Blake, J. (1999) · Local Environment · doi:10.1080/13549839908725599 · Cited by 1,289
The Hawthorne effect: A reconsideration of the methodological artifact.
Adair, J. G. (1984) · Journal of Applied Psychology · doi:10.1037/0021-9010.69.2.334 · Cited by 1,095
Understanding Self-Report Bias in Organizational Behavior Research
Donaldson, S. I., & Grant-Vallone, E. J. (2002) · Journal of Business and Psychology · doi:10.1023/a:1019637632584 · Cited by 1,068
For objective causal inference, design trumps analysis
Rubin, D. B. (2008) · The Annals of Applied Statistics · doi:10.1214/08-aoas187 · Cited by 831
Models for Sample Selection Bias
Winship, C., & Mare, R. D. (1992) · Annual Review of Sociology · doi:10.1146/annurev.so.18.080192.001551 · Cited by 812
Understanding controlled trials: Why are randomised controlled trials important?
Sibbald, B., & Roland, M. (1998) · BMJ · doi:10.1136/bmj.316.7126.201 · Cited by 807
Borrelli, B., Sepinwall, D., Ernst, D., Bellg, A. J., Czajkowski, S., Breger, R., DeFrancesco, C., Levesque, C., Sharp, D. L., Ogedegbe, G., et al. (2005) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.73.5.852 · Cited by 803
Social influence approaches to encourage resource conservation: a meta-analysis
Abrahamse, W., & Steg, L. (2013) · Global Environmental Change · doi:10.1016/j.gloenvcha.2013.07.029 · Cited by 802
Treatment fidelity in outcome studies
Moncher, F. J., & Prinz, R. J. (1991) · Clinical Psychology Review · doi:10.1016/0272-7358(91)90103-2 · Cited by 797
Osbaldiston, R., & Schott, J. P. (2012) · Environment and Behavior · doi:10.1177/0013916511402673 · Cited by 754
Delmas, M. A., Fischlein, M., & Asensio, O. I. (2013) · Energy Policy · doi:10.1016/j.enpol.2013.05.109 · Cited by 672
Sample size slippages in randomised trials: exclusions and the lost and wayward.
Schulz, K. F., & Grimes, D. A. (2002) · The Lancet · doi:10.1016/s0140-6736(02)07882-0 · Cited by 642
Standardized or simple effect size: what should be reported?
Baguley, T. (2009) · British Journal of Psychology · doi:10.1348/000712608x377117 · Cited by 619
The good-subject effect: investigating participant demand characteristics
Nichols, A. L., & Maner, J. K. (2008) · The Journal of General Psychology · doi:10.3200/genp.135.2.151-166 · Cited by 615
Do consumers really care about corporate responsibility? Highlighting the attitude—behaviour gap
Boulstridge, E., & Carrigan, M. (2000) · Journal of Communication Management · doi:10.1108/eb023532 · Cited by 559
Sample size calculations in randomised trials: mandatory and mystical
Schulz, K. F., & Grimes, D. A. (2005) · The Lancet · doi:10.1016/s0140-6736(05)61034-3 · Cited by 549
Strategies to improve recruitment to randomised controlled trials
Treweek, S., Mitchell, E., Pitkethly, M., Cook, J., Kjeldstrøm, M., Johansen, M., Taskila, T. K., Sullivan, F., Wilson, S., Jackson, C., et al. (2010) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000013.pub5 · Cited by 378
Statistics notes. Treatment allocation in controlled trials: why randomise?
Altman, D. G., & Bland, J. M. (1999) · BMJ · doi:10.1136/bmj.318.7192.1209 · Cited by 362
The Intervention Selection Bias: An Underrecognized Confound in Intervention Research.
Larzelere, R. E., Kuhn, B. R., & Johnson, B. (2004) · Psychological Bulletin · doi:10.1037/0033-2909.130.2.289 · Cited by 144
Consumer End-Use Energy Efficiency and Rebound Effects
Azevedo, I. M. L. (2014) · Annual Review of Environment and Resources · doi:10.1146/annurev-environ-021913-153558 · Cited by 144
Cited by
Sovacool, B. K., Axsen, J., & Sorrell, S. (2018) · Energy Research & Social Science · doi:10.1016/j.erss.2018.07.007 · Cited by 1,092
Nudging pro‐environmental behavior: evidence and opportunities
Byerly, H., Balmford, A., Ferraro, P. J., Hammond Wagner, C., Palchak, E., Polasky, S., Ricketts, T. H., Schwartz, A. J., & Fisher, B. (2018) · Frontiers in Ecology and the Environment · doi:10.1002/fee.1777 · Cited by 361
Punzo, G., Panarello, D., Pagliuca, M. M., Castellano, R., & Aprile, M. C. (2019) · Environmental Science & Policy · doi:10.1016/j.envsci.2019.09.006 · Cited by 132
Metadata
- Authors
- Frederiks, E. R., Stenner, K., Hobman, E. V., & Fischle, M.
- Venue
- Energy Research & Social Science
- Publisher
- Elsevier BV
- Published
- 2016-12
- Volume
- 22
- Pages
- 147-164
- ISSN
- 2214-6296 (ISSN-L)
- OpenAlex
- W2525326387
- Cited by
- 81
- References
- 43
- FWCI
- 17.05
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.