Unpacking black boxes: mechanisms and theory building in evaluation

Astbury, Brad; Leeuw, Frans L.

doi:10.1177/1098214010371972

Cited by 850Evaluation and Performance AssessmentQualitative Comparative Analysis ResearchAdvanced Causal Inference Techniques

Unpacking black boxes: mechanisms and theory building in evaluation

Astbury, Brad, Leeuw, Frans L.·2010·American Journal of Evaluation

DOI 10.1177/1098214010371972

Open this paper in CoChat — free account

Abstract

There is growing interest in the concept of ‘‘mechanism’’ across many areas of the social sciences. In the field of program and policy evaluation, a number of scholars have also emphasized the importance of causal mechanisms for explaining how and why programs work. However, there appears to be some ambiguity about the meaning and uses of mechanism-based thinking in both the social science and evaluation literature. In this article we attempt to clarify what is meant by mechanisms in the context of program evaluation by identifying three main characteristics of mechanisms and outlining a possible typology of mechanisms. A number of theoretical and practical implications for evaluators are also discussed, along with some precautions to consider when investigating mechanisms that might plausibly account for program outcomes.

Integrity signals

Registry

Crossref · 10.1177/1098214010371972

Cites

105 of 105 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Both a published literature review and critical discussions in the applied literature cite several methodological challenges associated with a RE approach, including deciding what constitutes a mechanism, and differentiating between what is a mechanism and what is context.
“Both a published literature review 112 and critical discussions in the applied literature cite several methodological challenges associated with a RE approach, including deciding what constitutes a mechanism, 127 and indeed differentiating between what is a mechanism and what is context.”
From ReseArch with Patient and Public invOlvement: a RealisT evaluation – the RAPPORT study (2015) · cites this paper as [127]

The Strength of Weak Ties
Granovetter, M. S. (1973) · American Journal of Sociology · doi:10.1086/225469 · Cited by 40,491 · see citing works
Diffusion of Innovations 1
Rogers, E. M., Singhal, A., & Quinlan, M. M. (2019) · Routledge eBooks · doi:10.4324/9780203710753-35 · Cited by 37,557 · see citing works
A Theory of Cognitive Dissonance
Festinger, L. (1957) · doi:10.1515/9781503620766 · Cited by 24,241 · see citing works
A theory of social comparison processes
Festinger, L. (1954) · Human Relations · doi:10.1177/001872675400700202 · Cited by 20,926 · see citing works
Diffusion of innovations
Valente, T. W. (2003) · Genetics in Medicine · doi:10.1097/01.gim.0000061743.67794.c4 · Cited by 20,555 · see citing works
A Theory of Cognitive Dissonance
Morvan, C., & O’Connor, A. (2017) · Macat Library eBooks · doi:10.4324/9781912282432 · Cited by 18,284 · see citing works
Social Theory and Social Structure
Ross, E. J., & Merton, R. Κ. (1958) · The American Catholic Sociological Review · doi:10.2307/3709394 · Cited by 10,318 · see citing works
The Agenda-Setting Function of Mass Media
McCombs, M. E., & Shaw, D. L. (1972) · Public Opinion Quarterly · doi:10.1086/267990 · Cited by 9,537 · see citing works
Untitled work
(2005) · Cited by 9,479 · see citing works
Reproduction in Education, Society and Culture
Erben, M., Bourdıeu, P., & Passeron, J.-C. (1979) · British Journal of Sociology · doi:10.2307/589547 · Cited by 8,157 · see citing works
Foundations of Social Theory
Gamst, F. C. (1991) · Anthropology of Work Review · doi:10.1525/awr.1991.12.3.19 · Cited by 8,036 · see citing works
Social Theory and Social Structure.
Bock, K. E., & Merton, R. Κ. (1969) · American Sociological Review · doi:10.2307/2091967 · Cited by 6,721 · see citing works
Reproduction in Education, Society and Culture
Broadfoot, T. (1978) · Comparative Education · doi:10.1080/0305006780140109 · Cited by 6,685 · see citing works
A Realist Theory of Science
Bhaskar, R. (2013) · Routledge eBooks · doi:10.4324/9780203090732 · Cited by 4,035 · see citing works
Pygmalion in the classroom
Rosenthal, R., & Jacobson, L. (1968) · The Urban Review · doi:10.1007/bf02322211 · Cited by 3,878 · see citing works
Framing as a Theory of Media Effects
Scheufele, D. A. (1999) · Journal of Communication · doi:10.1111/j.1460-2466.1999.tb02784.x · Cited by 3,843 · see citing works
Thinking about Mechanisms
Machamer, P., Darden, L., & Craver, C. F. (2000) · Philosophy of Science · doi:10.1086/392759 · Cited by 3,179 · see citing works
Realism and Social Science
Sayer, A. (2000) · doi:10.4135/9781446218730 · Cited by 2,259 · see citing works
The Logic of Collective Action: Public Goods and the Theory of Groups.
Perrow, C. B., & Olson, M. (1973) · Social Forces · doi:10.2307/2576430 · Cited by 2,137 · see citing works
Nuts and Bolts for the Social Sciences
Elster, J. (1989) · Cambridge University Press eBooks · doi:10.1017/cbo9780511812255 · Cited by 2,026 · see citing works
Economics and Reality
Lawson, T. (1997) · Routledge eBooks · doi:10.4324/9780203195390 · Cited by 1,910 · see citing works
Methodology of Scientific Research Programmes
(1998) · Edward Elgar Publishing eBooks · doi:10.4337/9781781954249.00075 · Cited by 1,817 · see citing works
Evidence-based Policy
Pawson, R. (2006) · doi:10.4135/9781849209120 · Cited by 1,791 · see citing works
Social Mechanisms
(1998) · Cambridge University Press eBooks · doi:10.1017/cbo9780511663901 · Cited by 1,774 · see citing works
Realism and social science
Sayer, A. (2000) · Cited by 1,753 · see citing works
Informal social communication.
Festinger, L. (1950) · Psychological Review · doi:10.1037/h0056932 · Cited by 1,732 · see citing works
Social Theory, Social Research, and a Theory of Action
Coleman, J. S. (1986) · American Journal of Sociology · doi:10.1086/228423 · Cited by 1,576 · see citing works
Informal social communication.
Festinger, L. (1950) · Research Center for Dynamics Institute for Social Research eBooks · doi:10.1037/10646-001 · Cited by 1,317 · see citing works
Theory-driven evaluations
Chen, H.-T. (1990) · Medical Entomology and Zoology · Cited by 1,227 · see citing works
<i>Conjectures and Refutations</i>
Popper, K. R., & Hudson, G. E. (1963) · Physics Today · doi:10.1063/1.3050617 · Cited by 1,071 · see citing works
On a distinction between hypothetical constructs and intervening variables.
Maccorquodale, K., & Meehl, P. (1948) · Psychology Review · doi:10.1037/h0056029 · Cited by 1,035 · see citing works
On Education, opportunity, and social inequality
Ramsøy, N. R. (1975) · Social Science Information · doi:10.1177/053901847501400606 · Cited by 937 · see citing works
The Logic of Collective Action: Public Goods and the Theory of Groups
Midgley, L., & Olson, M. (1969) · The Western Political Quarterly · doi:10.2307/446187 · Cited by 920 · see citing works
Assessing the impact of planned social change
Campbell, D. T. (1979) · Evaluation and Program Planning · doi:10.1016/0149-7189(79)90048-x · Cited by 867 · cited 3× · see citing works
“Campbell’s Law, which relates to the mechanism of ‘‘corruptibility of social indicators’’15 (Campbell, 1979), also shows that in some situations, a too strong belief in the relevance of indicators and performance measures and their contribution to ‘‘enlightenment’’ can produce undesired effects…”
BackgroundInfluential
Explaining Social Behavior
Elster, J. (2007) · Cambridge University Press eBooks · doi:10.1017/cbo9780511806421 · Cited by 853 · see citing works
Dissecting the Social: On the Principles of Analytical Sociology
Hedstrom, P. (2005) · doi:10.1017/cbo9780511488801 · Cited by 816 · see citing works
Untitled work
(2005) · Cited by 772 · see citing works
Logic models: a tool for telling your programs performance story
McLaughlin, J. A., & Jordan, G. (1999) · Evaluation and Program Planning · doi:10.1016/s0149-7189(98)00042-1 · Cited by 740 · see citing works
Mechanisms in Political Processes
Tilly, C. (2001) · Annual Review of Political Science · doi:10.1146/annurev.polisci.4.1.21 · Cited by 674 · see citing works
Mechanism and Explanation
Bunge, M. (1997) · Philosophy of the Social Sciences · doi:10.1177/004839319702700402 · Cited by 620 · see citing works
Mechanisms in the Analysis of Social Macro-Phenomena
Mayntz, R. (2004) · Philosophy of the Social Sciences · doi:10.1177/0048393103262552 · Cited by 614 · see citing works
Theory-driven evaluations
Lipsey, M. W. (1991) · Evaluation and Program Planning · doi:10.1016/0149-7189(91)90027-e · Cited by 598 · see citing works
How Does It Work?
Bunge, M. (2004) · Philosophy of the Social Sciences · doi:10.1177/0048393103262550 · Cited by 501 · see citing works
How Can Theory-Based Evaluation Make Greater Headway?
Weiss, C. H. (1997) · Evaluation Review · doi:10.1177/0193841x9702100405 · Cited by 484 · see citing works
Outcome pattern matching and program theory
Trochim, W. M. K. (1989) · Evaluation and Program Planning · doi:10.1016/0149-7189(89)90052-9 · Cited by 481 · see citing works
The Foundations of Social Theory
Delanty, G. (2008) · doi:10.1002/9781444304992.ch1 · Cited by 477 · see citing works
The Conditions of Fruitfulness of Theorizing About Mechanisms in Social Science
Stinchcombe, A. L. (1991) · Philosophy of the Social Sciences · doi:10.1177/004839319102100305 · Cited by 461 · see citing works
Evaluation : An Integrated Framework for Understanding, Guiding, and Improving Policies and Programs
Mark, M. M., Henry, G. T., & Julnes, G. (2000) · Cited by 395 · see citing works
Practical Program Evaluation
Chen, H.-T. (2005) · doi:10.4135/9781412985444 · Cited by 365 · see citing works
The Mechanismic Worldview: Thinking Inside the Box
Gerring, J. (2007) · British Journal of Political Science · doi:10.1017/s0007123408000082 · Cited by 361 · see citing works
The functions of program theory
Bickman, L. (1987) · New Directions for Program Evaluation · doi:10.1002/ev.1443 · Cited by 347 · cited 1× · see citing works
“Regardless of the way program theory is developed and depicted, it should constitute a ‘‘plausible and sensible model of how a program is supposed to work’’ (Bickman, 1987, p. 5).”
Method
Theory as method: small theories of treatments
Lipsey, M. W. (1993) · New Directions for Program Evaluation · doi:10.1002/ev.1637 · Cited by 322 · see citing works
Varieties of Social Explanation: An Introduction to the Philosophy of Social Science.
Ruben, D., & Little, D. (1993) · The Philosophical Review · doi:10.2307/2185667 · Cited by 316 · see citing works
The Mechanisms and Outcomes of Evaluation Influence
Mark, M. M., & Henry, G. T. (2004) · Evaluation · doi:10.1177/1356389004042326 · Cited by 311 · see citing works
Evaluation--promise and performance
Wholey, J. S. (1979) · Cited by 303 · see citing works
Measuring Program Outcomes: A Practical Approach
Roberts, N. S. (1998) · Journal of Park and Recreation Administration · Cited by 262 · see citing works
Evaluation and Effective Public Management
Goldoff, A. C., & Wholey, J. S. (1983) · Public Productivity Review · doi:10.2307/3379834 · Cited by 260 · see citing works
Reconstructing Program Theories: Methods Available and Problems to be Solved
Leeuw, F. L. (2003) · American Journal of Evaluation · doi:10.1016/s1098-2140(02)00271-0 · Cited by 254 · see citing works
Self-citation
Evaluating the ?logical framework approach? towards learning-oriented development evaluation
Gasper, D. (2000) · Public Administration and Development · doi:10.1002/1099-162x(200002)20:1 · Cited by 246 · see citing works
Evaluative Research: Principles and Practice in Public Service and Social Action Progr
Suchman, E. A. (1968) · Project Muse (Johns Hopkins University) · Cited by 223 · see citing works
Theory-Based Evaluation in Practice
Birckmayer, J. D., & Weiss, C. H. (2000) · Evaluation Review · doi:10.1177/0193841x0002400404 · Cited by 217 · see citing works
Program Theory-Driven Evaluation Science: Strategies and Applications
Donaldson, S. I. (2007) · Cited by 216 · see citing works
WHAT WORKS IN EVALUATION RESEARCH?
Pawson, R., & Tilley, N. (1994) · The British Journal of Criminology · doi:10.1093/oxfordjournals.bjc.a048424 · Cited by 213 · see citing works
Varieties of social explanation: an introduction to the philosophy of social science
(1991) · Choice Reviews Online · doi:10.5860/choice.29-2048 · Cited by 209 · see citing works
The theory-driven approach to validity
Chen, H.-T., & Rossi, P. H. (1987) · Evaluation and Program Planning · doi:10.1016/0149-7189(87)90025-5 · Cited by 204 · see citing works
Program Theory-Driven Evaluation Science
Donaldson, S. I. (2007) · Routledge eBooks · doi:10.4324/9780203809730 · Cited by 201 · see citing works
The Philosophies of Science: An Introductory Survey
Harré, R. (1972) · Medical Entomology and Zoology · Cited by 194 · see citing works
The methodology of scientific research programmes
Feyerabend, P. K. (1981) · Cambridge University Press eBooks · doi:10.1017/cbo9781139171588.011 · Cited by 176 · see citing works
Evaluability assessment: Developing program theory
Wholey, J. S. (1987) · New Directions for Program Evaluation · doi:10.1002/ev.1447 · Cited by 176 · see citing works
Realism in Research
House, E. R. (1991) · Educational Researcher · doi:10.3102/0013189x020006002 · Cited by 175 · see citing works
Reconstructing Program Theories: Methods Available and Problems to be Solved
Leeuw, F. L. (2003) · American Journal of Evaluation · doi:10.1177/109821400302400102 · Cited by 170 · see citing works
Self-citation
A Realist Theory of Science.
Sharpe, R. A., & Bhaskar, R. (1976) · The Philosophical Quarterly · doi:10.2307/2219031 · Cited by 158 · see citing works
Social mechanisms and social dynamics
Schelling, T. C. (1998) · Cambridge University Press eBooks · doi:10.1017/cbo9780511663901.002 · Cited by 151 · see citing works
Theory‐based evaluation: Reflections ten years on: Theory‐based evaluation: Past, present, and future
Rogers, P., & Weiss, C. H. (2007) · New Directions for Evaluation · doi:10.1002/ev.225 · Cited by 150 · see citing works
Social Mechanisms and Causal Inference
Steel, D. P. (2004) · Philosophy of the Social Sciences · doi:10.1177/0048393103260775 · Cited by 148 · see citing works
Middle-range realism
Pawson, R. (2000) · European Journal of Sociology · doi:10.1017/s0003975600007050 · Cited by 145 · see citing works
Understanding Mechanisms in Organizational Research
Anderson, P., Blatt, R., Christianson, M. K., Grant, A. M., Marquis, C., Neuman, E. J., Sonenshein, S. B., & Sutcliffe, K. M. (2006) · Journal of Management Inquiry · doi:10.1177/1056492605280231 · Cited by 135 · see citing works
Roles for Theory in Contemporary Evaluation Practice: Developing Practical Knowledge
Donaldson, S. I., & Lipsey, M. W. (2006) · SAGE Publications Ltd eBooks · doi:10.4135/9781848608078.n2 · Cited by 126 · see citing works
A Measure for Measures
Pawson, R. (2004) · Routledge eBooks · doi:10.4324/9780203393338 · Cited by 121 · see citing works
Abstraction: a realist interpretation
Sayer, A. (1981) · Radical philosophy · Cited by 101 · see citing works
A Measure for Measures: A Manifesto for Empirical Sociology.
Smith, H. L., & Pawson, R. (1992) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2074815 · Cited by 98 · see citing works
Program theory and implementation theory: Implications for evaluators
Scheirer, M. A. (1987) · New Directions for Program Evaluation · doi:10.1002/ev.1446 · Cited by 97 · see citing works
Minimalist theory: The least theory that practice requires
Scriven, M. (1998) · American Journal of Evaluation · doi:10.1016/s1098-2140(99)80180-5 · Cited by 82 · see citing works
Ascertaining causality in theory‐based evaluation
Davidson, E. J. (2000) · New Directions for Evaluation · doi:10.1002/ev.1178 · Cited by 74 · see citing works
The Fine Line between Evaluation and Explanation
Scriven, M. (1994) · Evaluation Practice · doi:10.1177/109821409401500108 · Cited by 66 · see citing works
Minimalist Theory: The Least Theory That Practice Requires
Scriven, M. (1998) · American Journal of Evaluation · doi:10.1177/109821409801900105 · Cited by 64 · see citing works
Explaining Human Change
Morén, S., & Blom, B. (2003) · Journal of Critical Realism · doi:10.1558/jocr.v2i1.37 · Cited by 58 · see citing works
The conceptual framework of the theory-driven perspective
Chen, H.-T. (1989) · Evaluation and Program Planning · doi:10.1016/0149-7189(89)90057-8 · Cited by 58 · see citing works
Policy theories, knowledge utilization, and evaluation
Leeuw, F. L. (1991) · Knowledge, technology & policy/Knowledge in society/Knowledge, technology, & policy · doi:10.1007/bf02693089 · Cited by 49 · see citing works
Self-citation
GUILT, SHAME AND SITUATIONAL CRIME PREVENTION
Wortley, R. (1996) · Cited by 47 · see citing works
Applying theory-driven evaluation to the British Crime Reduction Programme
Tilley, N. (2004) · Criminal Justice · doi:10.1177/1466802504048465 · Cited by 42 · see citing works
The Legal Analyst: A Toolkit for Thinking about the Law
Farnsworth, W. (2007) · Medical Entomology and Zoology · Cited by 37 · see citing works
Middle Range Theory and Program Theory Evaluation: From Provenance to Practice 1
Pawson, R. (2017) · Routledge eBooks · doi:10.4324/9781315124537-11 · Cited by 37 · see citing works
The fine line between evaluation and explanation
Scriven, M. (1994) · Evaluation Practice · doi:10.1016/0886-1633(94)90062-0 · Cited by 37 · see citing works
Designed blindness: an action science perspective on program theory evaluation
Friedman, V. J. (2001) · American Journal of Evaluation · doi:10.1016/s1098-2140(01)00121-7 · Cited by 33 · see citing works
The Legal Analyst
Farnsworth, W. (2007) · University of Chicago Press eBooks · doi:10.7208/chicago/9780226238364.001.0001 · Cited by 30 · see citing works
From Nuts and Bolts to Toolkits
Weber, K. (2006) · Journal of Management Inquiry · doi:10.1177/1056492605280237 · Cited by 27 · see citing works
Using path analysis to develop and evaluate program theory and impact
Smith, N. L. (1990) · New Directions for Program Evaluation · doi:10.1002/ev.1554 · Cited by 27 · see citing works
Designed Blindness: An Action Science Perspective on Program Theory Evaluation
Friedman, V. J. (2001) · American Journal of Evaluation · doi:10.1177/109821400102200203 · Cited by 27 · see citing works
Evaluators as applied theorists
McClintock, C. C. (1990) · Evaluation Practice · doi:10.1016/0886-1633(90)90003-v · Cited by 21 · see citing works
Invisible Mechanisms
Pawson, R. (2008) · Evaluation Journal of Australasia · doi:10.1177/1035719x0800800202 · Cited by 18 · see citing works
Introduction
Clarke, M. (1992) · doi:10.1007/978-1-349-22122-6_1 · Cited by 15 · see citing works
The "Baggaging" of Theory-Based Evaluation
Davidson, E. J. (2006) · Journal of MultiDisciplinary Evaluation · doi:10.56645/jmde.v3i4.73 · Cited by 10 · see citing works
Seeing off the danger
Tilley, N. (1995) · European Journal on Criminal Policy and Research · doi:10.1007/bf02242926 · Cited by 2 · see citing works
Nothing as Practical as a Good Theory
Kurvink, C. (2019) · Princeton University Press eBooks · doi:10.2307/j.ctvb938mm.5 · Cited by 2 · see citing works

Flagged first, then by citations

Sorted by citations

RAMESES publication standards: realist syntheses
Wong, G., Greenhalgh, T., Westhorp, G., Buckingham, J., & Pawson, R. (2013) · BMC Medicine · doi:10.1186/1741-7015-11-21 · Cited by 1,374 · see citing works
Research co-design in health: a rapid overview of reviews
Slattery, P., Saeri, A. K., & Bragge, P. (2020) · Health Research Policy and Systems · doi:10.1186/s12961-020-0528-9 · Cited by 1,313 · see citing works
Is realist evaluation keeping its promise? A review of published empirical studies in the field of health systems research
Marchal, B., van Belle, S., van Olmen, J., Hoerée, T., & Kegels, G. (2012) · Evaluation · doi:10.1177/1356389012442444 · Cited by 476 · see citing works
RAMESES publication standards: realist syntheses
Wong, G., Greenhalgh, T., Westhorp, G., Buckingham, J., & Pawson, R. (2013) · Journal of Advanced Nursing · doi:10.1111/jan.12095 · Cited by 385 · see citing works
Realist Synthesis for Public Health: Building an Ontologically Deep Understanding of How Programs Work, For Whom, and In Which Contexts
Jagosh, J. (2019) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-031816-044451 · Cited by 346 · see citing works
ReseArch with Patient and Public invOlvement: a RealisT evaluation – the RAPPORT study
Wilson, P., Mathie, E., Keenan, J., McNeilly, E., Goodman, C., Howe, A., Poland, F., Staniszewska, S., Kendall, S., Munday, D., et al. (2015) · Health Services and Delivery Research · doi:10.3310/hsdr03380 · Cited by 297 · see citing works
Development of methodological guidance, publication standards and training materials for realist and meta-narrative reviews: the RAMESES (Realist And Meta-narrative Evidence Syntheses – Evolving Standards) project
Wong, G., Greenhalgh, T., Westhorp, G., & Pawson, R. (2014) · Health Services and Delivery Research · doi:10.3310/hsdr02300 · Cited by 276 · see citing works
Sustaining organizational culture change in health systems
Willis, C. D., Saul, J., Bevan, H., Scheirer, M. A., Best, A., Greenhalgh, T., Mannion, R., Cornelissen, E., Howland, D., Jenkins, E., et al. (2016) · Journal of Health Organization and Management · doi:10.1108/jhom-07-2014-0117 · Cited by 178 · see citing works
Critical reflections on realist review: insights from customizing the methodology to the needs of participatory research assessment: Realist Review Methodology Reflection
Jagosh, J., Pluye, P., Wong, G., Cargo, M., Salsberg, J., Bush, P. L., Herbert, C. P., Green, L. W., Greenhalgh, T., & Macaulay, A. C. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1099 · Cited by 165 · see citing works
Why National eHealth Programs Need Dead Philosophers: Wittgensteinian Reflections on Policymakers’ Reluctance to Learn from History
GREENHALGH, T., RUSSELL, J., ASHCROFT, R. E., & PARSONS, W. (2011) · Milbank Quarterly · doi:10.1111/j.1468-0009.2011.00642.x · Cited by 119 · see citing works
Does therapeutic writing help people with long-term conditions? Systematic review, realist synthesis and economic considerations
Nyssen, O. P., Taylor, S. J., Wong, G., Steed, E., Bourke, L., Lord, J., Ross, C. A., Hayman, S., Field, V., Higgins, A., et al. (2016) · Health Technology Assessment · doi:10.3310/hta20270 · Cited by 115 · see citing works
Making contribution analysis work: A practical framework for handling influencing factors and alternative explanations
Lemire, S. T., Nielsen, S. B., & Dybdal, L. (2012) · Evaluation · doi:10.1177/1356389012450654 · Cited by 66 · see citing works
Evaluating the effectiveness of a multifaceted, multilevel continuous quality improvement program in primary health care: developing a realist theory of change
Schierhout, G., Hains, J., Si, D., Kennedy, C., Cox, R., Kwedza, R., O’Donoghue, L., Fittock, M., Brands, J., Lonergan, K., et al. (2013) · Implementation Science · doi:10.1186/1748-5908-8-119 · Cited by 67 · see citing works
Improving the Impact and Implementation of Disaster Education: Programs for Children Through Theory‐Based Evaluation
Johnson, V. A., Ronan, K. R., Johnston, D. M., & Peace, R. (2016) · Risk Analysis · doi:10.1111/risa.12545 · Cited by 49 · see citing works
“Therefore, when theorybased evaluation is applied, central assumptions need to be unpacked retrospectively.(32) There are a number of frameworks for developing or reconstructing the underlying theories of how a program is intended to work and what it is intended to achieve.”
Background
A systematic review of interventions to support adults with ADHD at work—Implications from the paucity of context-specific research for theory and practice
Lauder, K., McDowall, A., & Tenenbaum, H. R. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.893469 · Cited by 36 · see citing works

Authors: Astbury, B., & Leeuw, F. L.
Affiliations: The University of Melbourne (AU)
Maastricht University (NL)
Corresponding author: Astbury, Brad
ORCID: Astbury, Brad
Leeuw, Frans L.
Venue: American Journal of Evaluation
Publisher: SAGE Publications
Published: 2010-09
Volume: 31
Issue: 3
Pages: 363-381
ISSN: 1098-2140 (ISSN-L)
1557-0878
DOI: 10.1177/1098214010371972
OpenAlex: W2028200801
Semantic Scholar: 2097836b832fe93938565ae987d89e17fd7f64ef
Cited by: 850
References: 105
Citations by year: 2026: 44 · 2025: 48 · 2024: 55 · 2023: 68 · 2022: 70 · 2021: 87 · 2020: 87 · 2019: 81 · 2018: 69 · 2017: 50 · 2016: 50 · 2015: 44 · 2014: 35 · 2013: 23 · 2012: 24
FWCI: 26.16
Type: journal article
Language: en
Open access: closed
Licence: journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations: American Journal of Evaluation (publishedVersion)
Topics: Social Sciences › Decision Sciences › Management Science and Operations Research › Evaluation and Performance Assessment
Social Sciences › Sociology and Political Science › Qualitative Comparative Analysis Research
Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
Keywords: causal mechanisms, program evaluation, policy evaluation, theory development, program outcomes, evaluator practice

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Unpacking black boxes: mechanisms and theory building in evaluation

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Related

Record

About these checks