Integrating a Comparison Group Design into a Theory of Change Evaluation: The Case of the Urban Health Initiative
Abstract
This paper describes how we strengthened the theory of change approach to evaluating a complex social initiative by integrating it with a quasi-experimental, comparison group design. We also demonstrate the plausibility of selecting a credible comparison group through the use of cluster analysis, and describe our work in validating that analysis with additional measures. The integrated evaluation design relies on two points of comparison: (1) program theory to program experience; and (2) program cities to comparison cities. We describe how we are using this integrated design to evaluate the Robert Wood Johnson Foundation’s Urban Health Initiative, an effort that aims to improve health and safety outcomes for children and youth in five distressed urban areas through a process of citywide, multi-sector planning and changed public and private systems. We also discuss how the use of two research frameworks and multiple methods can enrich our ability to test underlying assumptions and evaluate overall program effects. Using this integrated approach has provided evidence that the earliest phases of this initiative are unfolding as the theory would predict, and that the comparison cities are not undergoing a similar experience to those in UHI. Despite many remaining limitations, this integrated evaluation can provide greater confidence in assessing whether future changes in health and safety outcomes may have resulted from the Urban Health Initiative (UHI).
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Qualitative data collection methods included interviews, programme observation, programme documentation and visual evidence.
“Qualitative data collection methods included interviews [13, 27, 35, 47, 71, 73, 75, 79], programme observation [13, 27, 35], programme documentation [13, 35, 71, 75, 79] and visual evidence [32].”
From Using theory of change to design and evaluate public health interventions: a systematic review (2015) · cites this paper as [13]
Data collected for the evaluation included routinely collected data, custom-designed surveys and qualitative data.
“Data collected for the evaluation included routinely collected data [33, 44], custom-designed surveys [13, 16, 32, 72, 76, 79] and qualitative data.”
From Using theory of change to design and evaluate public health interventions: a systematic review (2015) · cites this paper as [13]
Cites
Lipsey, M. W. (1991) · Evaluation and Program Planning · doi:10.1016/0149-7189(91)90027-e · Cited by 592 · see citing works
How Can Theory-Based Evaluation Make Greater Headway?
Weiss, C. H. (1997) · Evaluation Review · doi:10.1177/0193841x9702100405 · Cited by 484 · see citing works
Urban Schools: The Challenge Of Location And Poverty
Laura, L. (2004) · Cited by 277 · see citing works
Program theory evaluation: Practice, promise, and problems
Rogers, P. J., Petrosino, A., Huebner, T. A., & Hacsi, T. A. (2000) · New Directions for Evaluation · doi:10.1002/ev.1177 · Cited by 274 · see citing works
Indicators of children's well-being
(1998) · Choice Reviews Online · doi:10.5860/choice.35-5947 · Cited by 152 · see citing works
Evaluation Models: New Directions for Evaluation
Stufflebeam, D. L. (2001) · Cited by 148 · see citing works
New approaches to evaluating community initiatives: Theory, measurement, and analysis
Connell, K. F.-A. A. A. C. K. A. J. P. (1998) · Cited by 136 · see citing works
What We Argue about When We Argue about Regionalism
Swanstrom, T. (2001) · Journal of Urban Affairs · doi:10.1111/0735-2166.00102 · Cited by 122 · see citing works
Program theory in evaluation : challenges and opportunities
Rogers, P. (2000) · Jossey-Bass eBooks · Cited by 93 · see citing works
Regional Politics: America in a Post-City Age
Savitch, H. V., & Vogel, R. K. (1996) · Sage eBooks · doi:10.4135/9781483327617 · Cited by 71 · see citing works
The false choice between theory‐based evaluation and experimentation
Cook, T. D. (2000) · New Directions for Evaluation · doi:10.1002/ev.1179 · Cited by 61 · see citing works
Program analysis for State and local governments
Hatry, H. P. (1976) · Medical Entomology and Zoology · Cited by 21 · see citing works
Setting an Agenda for Local Action
Silver, D. R., Weitzman, B. C., & Brecher, C. (2002) · Policy Studies Journal · doi:10.1111/j.1541-0072.2002.tb02152.x · Cited by 19 · see citing works
Self-citationMeasuring Community Indicators
Gruenewald, P. J., Treno, A. J., Taff, G., et al. (1997) · Sage eBooks · doi:10.4135/9781412984959 · Cited by 19 · see citing works
Using program theory to replicate successful programs
Hacsi, T. A. (2000) · New Directions for Evaluation · doi:10.1002/ev.1183 · Cited by 14 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
De Silva, M. J., Breuer, E., Lee, L., Asher, L., Chowdhary, N., Lund, C., & Patel, V. (2014) · Trials · doi:10.1186/1745-6215-15-267 · Cited by 738 · see citing works
A Systematic Review of Theory-Driven Evaluation Practice From 1990 to 2009
Coryn, C. L. S., Noakes, L. A., Westine, C. D., & Schröter, D. C. (2011) · American Journal of Evaluation · doi:10.1177/1098214010389321 · Cited by 531 · see citing works
Using theory of change to design and evaluate public health interventions: a systematic review
Breuer, E., Lee, L., De Silva, M., & Lund, C. (2015) · Implementation Science · doi:10.1186/s13012-016-0422-6 · Cited by 425 · see citing works
Theories of Change and Realistic Evaluation: Peas in a Pod or Apples and Oranges?
Blamey, A., & Mackenzie, M. (2007) · Evaluation · doi:10.1177/1356389007082129 · Cited by 383 · see citing works
Breuer, E., De Silva, M. J., Fekadu, A., Luitel, N. P., Murhar, V., Nakku, J., Petersen, I., & Lund, C. (2014) · International Journal of Mental Health Systems · doi:10.1186/1752-4458-8-15 · Cited by 93 · see citing works
- Authors
- Weitzman, B. C., Silver, D., & Dillman, K.-N.
- Affiliations
- New York University (US)
- Corresponding author
- Weitzman, Beth C.
- Venue
- American Journal of Evaluation
- Publisher
- SAGE Publications
- Published
- 2002-12
- Volume
- 23
- Issue
- 4
- Pages
- 371-385
- ISSN
- 1098-2140 (ISSN-L)
- 1557-0878
- OpenAlex
- W2100229763
- Cited by
- 52
- References
- 15
- Citations by year
- 2024: 3 · 2022: 2 · 2021: 1 · 2019: 1 · 2018: 4 · 2017: 2 · 2015: 1 · 2014: 3 · 2013: 3 · 2012: 1
- FWCI
- 7.13
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- American Journal of Evaluation (publishedVersion)
- Funders
- Robert Wood Johnson Foundation
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Evaluation and Performance Assessment
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Keywords
- quasi-experimental evaluation, Robert Wood Johnson Foundation, clustering, program evaluation design, urban health
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
