A CONCEPTUAL FRAMEWORK FOR STUDYING THE SOURCES OF VARIATION IN PROGRAM EFFECTS
Open accessThe DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
Evaluations of public programs in many fields reveal that different types of programs—or different versions of the same program—vary in their effectiveness. Moreover, a program that is effective for one group of people might not be effective for other groups, and a program that is effective in one set of circumstances may not be effective in other circumstances. This paper presents a conceptual framework for research on such variation in program effects and the sources of this variation. The framework is intended to help researchers—both those who focus mainly on studying program implementation and those who focus mainly on estimating program effects—see how their respective pieces fit together in a way that helps to identify factors that explain variation in program effects, and thereby support more systematic data collection. The ultimate goal of the framework is to enable researchers to offer better guidance to policymakers and program operators on the conditions and practices that are associated with larger and more positive effects.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009) · Implementation Science · doi:10.1186/1748-5908-4-50 · Cited by 14,496
Estimating causal effects of treatments in randomized and nonrandomized studies
Rubin, D. B. (1974) · Journal of Educational Psychology · doi:10.1037/h0037350 · Cited by 9,436
Grit: perseverance and passion for long-term goals
Duckworth, A. L., Peterson, C., Matthews, M. D., & Kelly, D. R. (2007) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.92.6.1087 · Cited by 6,911
Durlak, J. A., & DuPre, E. P. (2008) · American Journal of Community Psychology · doi:10.1007/s10464-008-9165-0 · Cited by 5,304
Statistics and Causal Inference
Holland, P. W. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478354 · Cited by 5,293
Fisher, R. A. (1936) · BMJ · doi:10.1136/bmj.1.3923.554-a · Cited by 4,980
Identification of Causal Effects Using Instrumental Variables
Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476902 · Cited by 4,230
A general approach to causal mediation analysis.
Imai, K., Keele, L., & Tingley, D. (2010) · Psychological Methods · doi:10.1037/a0020761 · Cited by 4,143
A conceptual framework for implementation fidelity
Carroll, C., Patterson, M., Wood, S., Booth, A., Rick, J., & Balain, S. (2007) · Implementation Science · doi:10.1186/1748-5908-2-40 · Cited by 2,559
Dane, A. V., & Schneider, B. H. (1998) · Clinical Psychology Review · doi:10.1016/s0272-7358(97)00043-3 · Cited by 1,557
Durlak, J. A., Weissberg, R. P., & Pachan, M. (2010) · American Journal of Community Psychology · doi:10.1007/s10464-010-9300-6 · Cited by 1,216
Yes, but what’s the mechanism? (don’t expect an easy answer).
Bullock, J. G., Green, D. P., & Ha, S. E. (2010) · Journal of Personality and Social Psychology · doi:10.1037/a0018933 · Cited by 1,063
Bettinger, E. P., Long, B. T., Oreopoulos, P., & Sanbonmatsu, L. (2012) · The Quarterly Journal of Economics · doi:10.1093/qje/qjs017 · Cited by 953
Subgroup analysis in randomised controlled trials: importance, indications, and interpretation
Rothwell, P. M. (2005) · The Lancet · doi:10.1016/s0140-6736(05)17709-5 · Cited by 942
ROLLNICK, S., HEATHER, N., GOLD, R., & HALL, W. (1992) · British Journal of Addiction · doi:10.1111/j.1360-0443.1992.tb02720.x · Cited by 818
Treatment fidelity in outcome studies
Moncher, F. J., & Prinz, R. J. (1991) · Clinical Psychology Review · doi:10.1016/0272-7358(91)90103-2 · Cited by 797
Glisson, C., Landsverk, J., Schoenwald, S., Kelleher, K., Hoagwood, K. E., Mayberg, S., Green, P., & The Research Network on Youth Mental Health (2008) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-007-0148-5 · Cited by 608
Cook, T. D., Shadish, W. R., & Wong, V. C. (2008) · Journal of Policy Analysis and Management · doi:10.1002/pam.20375 · Cited by 604
Statistical power and optimal design for multisite randomized trials.
Raudenbush, S. W., & Liu, X. (2000) · Psychological Methods · doi:10.1037/1082-989x.5.2.199 · Cited by 593
Measurement of Fidelity in Psychiatric Rehabilitation
Bond, G. R., Evans, L., Salyers, M. P., Williams, J., & Kim, H.-W. (2000) · Mental Health Services Research · doi:10.1023/a:1010153020697 · Cited by 302
The fidelity‐adaptation debate: Implications for the implementation of public sector social programs
Blakely, C. H., Mayer, J. P., Gottschalk, R. G., Schmitt, N., Davidson, W. S., Roitman, D. B., & Emshoff, J. G. (1987) · American Journal of Community Psychology · doi:10.1007/bf00922697 · Cited by 297
Moving From the Lab to the Field: The Role of Fidelity and Achieved Relative Intervention Strength
Hulleman, C. S., & Cordray, D. S. (2009) · Journal of Research on Educational Effectiveness · doi:10.1080/19345740802539325 · Cited by 291
What is the best way to achieve broader reach of improved practices in higher education?
Kezar, A. (2011) · Innovative Higher Education · doi:10.1007/s10755-011-9174-z · Cited by 124
Rewarding Persistence: Effects of a Performance-Based Scholarship Program for Low-Income Parents
Richburg-Hayes, L., Brock, T., LeBlanc, A., Paxson, C. H., Rouse, C. E., & Barrow, L. (2009) · doi:10.2139/ssrn.1353360 · Cited by 104
Self-citationBloom, H. S., & Michalopoulos, C. (2013) · Prevention Science · doi:10.1007/s11121-010-0198-x · Cited by 83
Self-citationCastleman, B. L., Arnold, K., & Wartman, K. L. (2012) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2011.618214 · Cited by 73
Cited by
Behavioural science is unlikely to change the world without a heterogeneity revolution
Bryan, C. J., Tipton, E., & Yeager, D. S. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01143-3 · Cited by 537
Why Interventions to Influence Adolescent Behavior Often Fail but Could Succeed
Yeager, D. S., Dahl, R. E., & Dweck, C. S. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691617722620 · Cited by 512
Yeager, D. S., Fong, C. J., Lee, H. Y., & Espelage, D. L. (2015) · Journal of Applied Developmental Psychology · doi:10.1016/j.appdev.2014.11.005 · Cited by 472
on behalf of the CONSORT-SPI Group, Montgomery, P., Grant, S., Mayo-Wilson, E., Macdonald, G., Michie, S., Hopewell, S., & Moher, D. (2018) · Trials · doi:10.1186/s13063-018-2733-1 · Cited by 327
Walton, G. M., & Yeager, D. S. (2020) · Current Directions in Psychological Science · doi:10.1177/0963721420904453 · Cited by 298
Metadata
- Authors
- Weiss, M. J., Bloom, H. S., & Brock, T.
- Venue
- Journal of Policy Analysis and Management
- Publisher
- Wiley
- Published
- 2014-03-28
- Volume
- 33
- Issue
- 3
- Pages
- 778-808
- ISSN
- 0276-8739 (ISSN-L)
- 1520-6688
- OpenAlex
- W2154869759
- Cited by
- 174
- References
- 26
- FWCI
- 20.40
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.