The contribution of process tracing to theory-based evaluations of complex aid instruments
Open accessThe DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Abstract
This article focuses on methodological challenges in evaluating complex program aid interventions like budget support. We show that recent innovations in process-tracing methodology can help solve the identified challenges and increase the strength of causal inference made when using case studies in demanding settings. For the specific task of evaluating the governance effectiveness of budget support interventions, we developed a more fine-grained causal mechanism for a sub-set of the comprehensive program theory of budget support. Moreover, based on the informal use of Bayesian logic, we have elaborated on how to increase the conclusiveness of empirical evidence for one part of the theorized causal mechanism. We argue that by establishing an explicit theorized mechanism prior to empirical research and by critically judging our evidence according to an informal Bayesian logic we can remedy some of the problems at hand in much case-study research and increase the inferential leverage in complex within-case evaluation studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Machamer, P., Darden, L., & Craver, C. F. (2000) · Philosophy of Science · doi:10.1086/392759 · Cited by 3,179 · cited 2×
“Entities are what engage in activities, where the activities are the producers of change or what transmits causal forces through a mechanism (Machamer et al., 2000).”
BackgroundProcess-Tracing Methods: Foundations and Guidelines
Beach, D., & Pedersen, R. (2013) · doi:10.3998/mpub.2556282 · Cited by 1,140
Self-citationProcess-Tracing Methods: Foundations and Guidelines
(2013) · Choice Reviews Online · doi:10.5860/choice.51-1711 · Cited by 921 · cited 4×
“However, when engaging in PT we typically focus on the link(s) between a cause (X) and a particular outcome instead of multiple different outcomes (Beach and Pedersen, 2013: 14–16).”
BackgroundMethodInfluentialContext and Causal Mechanisms in Political Analysis
Falleti, T. G., & Lynch, J. F. (2009) · Comparative Political Studies · doi:10.1177/0010414009331724 · Cited by 909
Mechanisms and the nature of causation
Glennan, S. (1996) · Erkenntnis · doi:10.1007/bf00172853 · Cited by 894
Unpacking black boxes: mechanisms and theory building in evaluation
Astbury, B., & Leeuw, F. L. (2010) · American Journal of Evaluation · doi:10.1177/1098214010371972 · Cited by 850 · cited 4×
“…are treated as either a) a series of events or narratives that link the occurrence of X and the occurrence of Y in a particular case (e.g. Astbury and Leeuw, 2010), or b) are theorized in a fashion where the critical causal links in the process are ‘grey-boxed’ by being treated as…”
BackgroundMethodInfluentialBunge, M. (1997) · Philosophy of the Social Sciences · doi:10.1177/004839319702700402 · Cited by 620 · cited 2×
“parts of a mechanism to produce an outcome (Bunge, 1997; Glennan, 1996; Mayntz, 2004).”
BackgroundMechanisms in the Analysis of Social Macro-Phenomena
Mayntz, R. (2004) · Philosophy of the Social Sciences · doi:10.1177/0048393103262552 · Cited by 614 · cited 1×
“So ur ce : O EC D /D A C ( 20 12 : 9 ). parts of a mechanism to produce an outcome (Bunge, 1997; Glennan, 1996; Mayntz, 2004).”
Process Tracing: From Metaphor to Analytic Tool
(2014) · Cambridge University Press · doi:10.1017/cbo9781139858472 · Cited by 495
Contribution analysis: Coming of age?
Mayne, J. (2012) · Evaluation · doi:10.1177/1356389012451663 · Cited by 387 · cited 2×
“In CA, stakeholders are interviewed to find out whether they believe the program worked (Blamey and McKenzie, 2007: 449; Mayne, 2012).”
BackgroundTheories of Change and Realistic Evaluation: Peas in a Pod or Apples and Oranges?
Blamey, A., & Mackenzie, M. (2007) · Evaluation · doi:10.1177/1356389007082129 · Cited by 383
Broadening the range of designs and methods for impact evaluations
Stern, E., Stame, N., Mayne, J., Forss, K., Davies, R., & Befani, B. (2012) · doi:10.22163/fteval.2012.100 · Cited by 369 · cited 4×
“Advocates of the counterfactual approach to causality have had a strong influence on the debate on evaluation designs and ‘succeeded in challenging all evaluators to address questions of causal claims and explaining the effectiveness of development interventions’ (Stern et al., 2012: 8).”
BackgroundInfluentialGerring, J. (2010) · Comparative Political Studies · doi:10.1177/0010414010376911 · Cited by 279
Program theory evaluation: Practice, promise, and problems
Rogers, P. J., Petrosino, A., Huebner, T. A., & Hacsi, T. A. (2000) · New Directions for Evaluation · doi:10.1002/ev.1177 · Cited by 274 · cited 1×
“Following the definition by Funnel and Rogers (2011: 31), it cannot be termed a program theory because it does not make explicit the underlying causal mechanisms to explain how the program causes the intended outcomes (see also Rogers et al., 2000: 5).”
Activities and Causation: The Metaphysics and Epistemology of Mechanisms
Machamer, P. (2004) · International Studies in the Philosophy of Science · doi:10.1080/02698590412331289242 · Cited by 237 · cited 1×
“Machamer et al. (2000) state that when theorizing a mechanism, we should describe how parts of the mechanism are bound together by ‘productive continuity’, i.e. there are no logical holes in our explanation of the causal process (Machamer et al., 2000; also Machamer, 2004).”
BackgroundComparative Hypothesis Testing Via Process Tracing
Rohlfing, I. (2014) · Sociological Methods & Research · doi:10.1177/0049124113503142 · Cited by 85 · cited 3×
“Taken together, the levels of theoretical certainty and uniqueness of predicted evidence in relation to a case results in four different types of empirical tests (Beach and Pedersen, 2013; Rohlfing, 2014; Van Evera, 1997).”
BackgroundMethodLemire, S. T., Nielsen, S. B., & Dybdal, L. (2012) · Evaluation · doi:10.1177/1356389012450654 · Cited by 66 · cited 6×
“Most importantly, developing how each part of a mechanism is logically linked to each other in terms of entities engaging in activities results in the explicit theorization of causal links instead of relegating them to ‘assumptions’ (e.g. Lemire et al., 2012).”
BackgroundMethodInfluentialMechanistic explanation at the limit
Waskan, J. (2011) · Synthese · doi:10.1007/s11229-010-9869-1 · Cited by 32
Cited by
Cairney, P., & Oliver, K. (2017) · Health Research Policy and Systems · doi:10.1186/s12961-017-0192-x · Cited by 526
Multi-Method Research in the Social Sciences: A Review of Recent Frameworks and a Way Forward
Beach, D. (2020) · Government and Opposition · doi:10.1017/gov.2018.53 · Cited by 43
Self-citationPallas, S. W., & Ruger, J. P. (2017) · Social Science & Medicine · doi:10.1016/j.socscimed.2017.01.004 · Cited by 16
Metadata
- Authors
- Schmitt, J., & Beach, D.
- Venue
- Evaluation
- Publisher
- SAGE Publications
- Published
- 2015-10
- Volume
- 21
- Issue
- 4
- Pages
- 429-447
- ISSN
- 1356-3890 (ISSN-L)
- 1461-7153
- OpenAlex
- W2337550509
- Cited by
- 53
- References
- 18
- FWCI
- 13.29
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.