Broadening the range of designs and methods for impact evaluations

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Method combination has become easier as the boundary between quantitative and qualitative methods has blurred, and computer tools have facilitated non-statistical quantitative methods and new within-case analyses.

“Combining methods has also become easier because the clear distinctions between quantitative and qualitative methods have become blurred, with quantitative methods that are non-statistical and new forms of within-case analysis made easier by computer aided tools.”

May combine two claims.

Reviews of existing evaluations suggest that sometimes the methods chosen cannot answer the evaluation questions posed, or that programme characteristics are not considered when choosing designs or evaluation questions.

“Reviews of existing evaluations suggests that sometimes methods chosen are unable to answer the evaluation questions posed; or the characteristics of development programmes are not taken into account when choosing designs or deciding on evaluation questions.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Counterfactual logics, which underpin experimental impact evaluation, are better understood than other causal inference approaches.

“There is greater understanding of counterfactual logics, the approach to causal inference that underpins experimental approaches to IE.”

Classified as probably not a substantial claim. May combine two claims.

Focused impact evaluation definitions from Poverty Action Lab, the World Bank, 3ie, and White are methods-led, seek a given effect and often a particular cause, treat attribution as a direct treatment-effect link, rely on counterfactual logic, often assume experimental or quasi-experimental methods are default, and do not specifically address long-term effects.

“In these definitions: • tend to be methods-led rather than content-ledfoot_1 ; • the search is for a given effect (the indicator of interest) and often a particular cause; • the emphasis on attribution suggests a direct link between a cause (often described in medical terms as a 'treatment') and an effect; • there is a reliance on counterfactual logic -asking the question what would have happened”

• cites White 2010Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Several of the most promising causal inference approaches are unfamiliar or unused in international development evaluation; multiple causality, configurations, and theory-based evaluation of causal mechanisms are especially underdeveloped.

“Some of the most potentially useful approaches to causal inference are not generally known or applied in the evaluation of international development and aid.Multiple causality and configurations; and theory-based evaluation that can analyse causal mechanisms are particularly weak.”

Probably not a substantial claim: may not pass S4 (based).

Approaches to causal inference widely understood in social sciences are not well established in IE.

“• Approaches to 'causal inference' widely understood in the social sciences are not well established in IE.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The wish of EBP evaluators to establish a clear link between causes and subsequent effects has not been sufficiently emphasised by evaluators hitherto.

“In particular their wish as far as possible to establish a clear link between causes (interventions) and subsequent effects has not been sufficiently emphasised by evaluators hitherto.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The review conducted for this study distinguished four approaches to causal inference, each drawing on particular theories of causation.

“There are various well-rehearsed classifications of these different approaches, for example in a review conducted for this study (see Appendix) we distinguished between four approaches each of which draws on particular theories of causation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Many development agencies, NGOs, consulting firms and individual consultants approached for examples of IEs beyond experimental and quantitative methods said they did not undertake IEs.

“It is worth noting that many of those who were approached and who responded said that they did not undertake IEs.”

Probably not a substantial claim: may not pass S4 (based).

Up to now IE has been less successful dealing with multiple and overlapping interventions that require system-wide evaluation strategies and designs.

“Up to now IE has been less successful dealing with these multiple and overlapping interventions which require system-wide evaluation strategies and designs.”

May combine two claims.

The analysis of qualitative data in the reviewed evaluations was especially weak; it was generally seen as supporting quantitative data rather than a data source in its own terms.

“• The analysis of qualitative data was especially weak.It was generally seen as 'supporting' quantitative data rather than a data source in its own terms.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

IE guidance from a cross-section of development agencies was scanned as a complement to the case work; it was useful in explaining approaches taken in the reviewed cases, which mainly followed currently available guidance.

“As a complement to this work on cases IE guidance put out by a cross-section of development agencies was also scanned7.This was useful in explaining the approaches taken in the cases reviewed: these cases mainly followed the guidance that is currently available.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In reviewed guides, qualitative methods were generally regarded as complementary to quantitative methods.

“• Qualitative methods were generally regarded as complementary to quantitative methods.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Evaluations take place either ex-post or ongoing; in the latter case because programmes last several years.

“• Evaluations take place either ex-post or ongoing; in the latter case, it is because programmes last several years.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The dominant model of QA even for qualitative methods in reviewed evaluations was derived from quantitative rather than qualitative logics.

“However the dominant model of QA even for qualitative methods was derived from quantitative rather than qualitative logics.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In some reviewed cases there was a sense that evaluators felt obliged to use certain terms they barely understood as often as possible.

“In some cases there was a sense that the evaluators felt obliged to use certain terms they barely understood as often as possible.”

May combine two claims.

Many impacts chosen in reviewed evaluations would normally be classified as outcomes or intermediate effects, with few attempts to link shorter-term with longer-term.

“• Many of the 'impacts' chosen would normally be classified as outcomes or intermediate effects and there were few attempts to links the shorter-term with the longer term.”

May combine two claims.

Theory-based evaluations were seen in reviewed guides mainly as a way of interpreting quantitative results or deepening understanding of causal mechanisms derived from experiments (3ie, NONIE Guidance); only in two cases (GEF and NONIE Subgroup 2) was theory-based evaluation seen as a core approach.

“• Theory based evaluations were also seen as mainly a way of interpreting quantitative results or deepening understanding of causal mechanisms derived from experiments (3ie, NONIE Guidance).In only two cases (GEF and NONIE Subgroup 2) was theory based evaluation seen as a core approach.”

• cites 3ie, NONIE GuidanceNot checked yet

  Not checked in the free run.

• cites GEF and NONIE Subgroup 2Not checked yet

  Not checked in the free run.

May combine two claims.

The reviewed evaluations mostly focused on short-term or intermediate outcomes and impacts; in some cases longer-term effects could have been linked with intermediates using a time-related Theory of Change.

“• These evaluations mostly focussed on short-term or intermediate outcomes and impacts.In some cases it would have been possible to have linked longer term effects with intermediates, by using a time-related Theory of Change.”

May combine two claims.

Even where potentially useful qualitative data appeared to have been collected (e.g., via case studies or focus groups), it was not always reported, and where analysed did not use state-of-the-art designs and methods.

“Even where potentially useful data appears to have been collected (e.g. via case studies or focus groups) this data was not always reported and where it was analysis did not use stateof-the-art designs and methods.”

May combine two claims.

Standards, when addressed in reviewed evaluations, are mostly understood as statistical robustness and bias control; in some cases validity, appropriateness for causal questions, relevance to goals, good practice for mixed methods and quality of Theory of Change are also recommended.

“• Standards (when they are addressed) are mostly understood as statistical robustness and bias control.In some cases, validity, appropriateness for causal questions and relevance to goals are also recommended, as well as good practice for mixed methods research and ensuring the quality of the Theory of Change.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The importance of addressing inequality and poverty seemed especially strong in DFID commissioned evaluations.

“The importance of addressing inequality and poverty seemed especially strong in DFID commissioned evaluations.”

May combine two claims.

Several reviewed IEs had relatively sophisticated ToRs or Approach Papers that were not followed through; weaknesses evident in the evaluation report were anticipated.

“• Several IEs had relatively sophisticated ToRs or 'Approach Papers' that were not followed through.Weaknesses that were evident in the evaluation report were anticipated.”

May combine two claims.

Participatory methods received relatively little attention in reviewed guides; one guide suggested they were not appropriate, and another that while providing insight they were not rigorous or a sound design.

“• Participatory methods received relatively little attention.One guide suggested it was not appropriate; and in another that whilst providing insight it was not a rigorous or a sound design.”

May combine two claims.

In reviewed guides, it was almost always assumed that only experimental and statistical methods could be rigorous; the term was generally associated with counterfactual methods rather than the underlying logic of an approach or how IE was conducted.

“• It was almost always assumed that only experimental and statistical methods could be rigorous.In general this term was associated with counterfactual methods rather than to the underlying logic of an approach or the way IE was conducted.”

May combine two claims.

The review encountered examples of all causal patterns in the evaluation cases, but these patterns did not appear to be reflected in the design of those evaluations.

“We encountered examples of all of these 'causal patterns' in the evaluation cases reviewed however they did not appear to be reflected in the design of these evaluations.”

May combine two claims.

Evaluation questions in the reviewed cases were poorly expressed-sometimes too narrow, sometimes too broad, sometimes absent; even when clearly formulated they were only loosely connected with the methodologies adopted.

“• Evaluation questions were poorly expressed -sometimes too narrow, at other times too broad and on occasions completely absent.Even when evaluation questions were clearly formulated they were only loosely connected with the methodologies adopted which may have been unable to answer the questions put forward.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Common terms used in IE were used inconsistently in the reviewed cases; 'rigour', 'Theory of Change' and even 'impact' were variously defined.

“• Common terms that are used in IE were used in inconsistent ways.For example 'rigour', 'Theory of Change' and even 'impact' were variously defined.”

Probably not a substantial claim: may not pass S3 (load-bearing).

NONIE Guidance associated participatory methods with testing for validity and identifying alternative explanations.

“NONIE Guidance -NONIE Guidance Subgroup 2 associated participatory methods with testing for validity and identifying alternative explanations.”

• cites NONIE Guidance -NONIE Guidance Subgroup 2Not checked yet

  Not checked in the free run.

May combine two claims.

Stakeholders are seldom involved in the IE process; participatory evaluation is usually a module within an overall design rather than an overarching principle.

“• Stakeholders are seldom involved in the IE process; participatory evaluation is usually a module within an overall design rather than an overarching principle.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The review mainly underlined weaknesses in current IE practice outside experimental and statistical practice, but also pointed to important gaps addressed in later stages of the study.

“Although this review mainly underlined weaknesses in current IE practice outside of experimental and statistical practice, it also pointed to important gaps that have been addressed in later stages of the study.”

May combine two claims.

Measurement worked best when impacts were specific and easily measured (access to services, transport, income levels) and less when dealing with abstract concepts such as governance, conflict and cultural change.

“Measurement worked best when 'impacts' were specific and could be easily measured (access to services, transport, income levels etc.) and less when dealing with more abstract concepts such as governance, conflict and cultural change.”

May combine two claims.

There is little assessment of contexts in the reviewed evaluations; context was used inconsistently and without typologies or theories, often spatially defined but rarely linked to causal analysis, limiting potential for causal inference.

“• There is little assessment of contexts.The term is certainly used but inconsistently and without any typologies or theories.Context was often spatially defined (the country or district) but rarely linked to a causal analysis.This limited the potential for causal inference.”

May combine two claims.

Capacity development was widely recognised as key to longer-term sustainability of intervention effects but was not then evaluated; there was limited understanding about how to evaluate capacity development in the reviewed cases.

“There were references to 'capacity development' which seems to be widely recognised as key to the longer term sustainability of the effects of interventions but capacity development was not then evaluated.There seemed to be limited understanding about how to evaluate capacity development from the cases we reviewed.”

Classified as probably not a substantial claim. May combine two claims.

Some programmes were complex because of their own characteristics, while others were embedded in complex and sometimes dangerous country environments.

“Some were 'complex' because of their own characteristics whilst others were embedded in complex and sometimes dangerous country environments.”

Classified as probably not a substantial claim.

The programmes given by DFID to consider were variously described as complex, challenging or difficult to evaluate, including empowerment, accountability, governance, state-building, climate change and humanitarian relief.

“The kinds of programmes that we were given to consider by DFID were variously described as complex, challenging or difficult to evaluate.They included programmes concerned with empowerment, accountability, governance, state-building, climate change and humanitarian relief.”

May combine two claims.

The analysis of specific cases suggests that breaking down complex systems into smaller subsystems for evaluation purposes is sometimes justified, but often not.

“The analysis of specific cases (see below), suggests that whilst breaking down complex systems into smaller subsystems for evaluation purposes is sometimes justified, it is often not.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Some large-scale complex initiatives reviewed led to a conclusion similar to Marra's from a bottom-up perspective.

“some of the large scale complex initiatives that we reviewed led to a similar conclusion from a 'bottom-up' perspective”

May combine two claims.

The likely impacts of the reviewed programmes were long term, but their planned life-cycle was often relatively short, so the most that could be expected were intermediate outcomes.

“The likely impacts of programmes were long term.However even though end goals for intended beneficiaries were long term the planned life-cycle of the programme itself was often relatively short.The most that could be expected were intermediate outcomes.”

May combine two claims.

The six programmes reviewed in depth were not atypical, and the attributes identified are likely to be of wider relevance.

“On the basis of the above we are confident that the six programmes reviewed in depth were not a-typical and the attributes identified would be likely to be of wider relevance.”

May combine two claims.

The reviewed programmes commonly worked in areas of limited understanding or experience, where they could be unable to articulate their Theory of Change or assumptions, or a ToC was put forward but not backed by evidence.

“Working in areas of limited understanding/experience.Such programmes could be innovative to the point of being unable to clearly articulate their Theory of Change or their assumptions.Alternatively a ToC was put forward but was not backed by evidence.”

May combine two claims.

The reviewed programmes commonly had stated impacts that were difficult to measure, possibly intangible, and often composite different goals into one heading.

“Stated impacts are difficult to measure, possibly intangible and often composite different goals into one heading.”

May combine two claims.

Several programmes shared key characteristics, such as setting up a Fund that selects and supports a portfolio of projects in line with programme aims, which appears to be a common modality post Paris Declaration.

“Several programmes shared key characteristics, even if they were not in the same broad policy heading.An example of this was setting up a 'Fund' that would then select and support a portfolio of projects in line with programme aims.This appears to be a common modality post Paris Declaration.”

Classified as probably not a substantial claim.

Evaluation still lacks some simulation and modelling approaches for complex systems that have been applied in other domains.

“evaluation still lacks some of the simulation and modelling approaches for complex systems that have been applied in other domains.”

May combine two claims.

The reviewed programmes commonly worked in areas of high risk or uncertainty, including dangerous security situations, post-disaster settings, or ambitious social programmes where progress was unclear and setbacks could be expected.

“Working in areas of high risk or uncertainty.This could include dangerous security situations or post disaster.It could also include ambitious social programmes where how progress would be achieved was unclear and set-backs could be expected.”

May combine two claims.

The reviewed programmes commonly worked indirectly through agents and often at different levels and stages.

“Programmes working 'indirectly' through 'agents' and often at different levels and stages.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Similar attributes can be identified in specific programmes reviewed as part of this study.

“similar attributes can be identified in specific programmes that have been reviewed as part of this study.”

Probably not a substantial claim: may not pass S3 (load-bearing).

From reviews of actual programmes, it is possible to structure design decisions related to interdependence by identifying potentially related programmes, whether within a single large-scale programme, in nearby or overlapping programmes, or in separate programmes targeting the same population.

“From reviews of actual programmes it seems to be possible to structure design decisions related to interdependence by identifying potentially related programmes whether in a single large scale programmes or in other 'nearby' or overlapping programmes or in separate programmes that target the same population.”

May combine two claims.

Development agencies either noted that they did not undertake IE or identified IE entirely with experimental methods rather than a broader range of methods.

“development agencies which either noted that they did not undertake IE or identified IE entirely with experimental methods rather than a broader range of methods.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Many of those with relevant knowledge and skills are not currently engaged in IE evaluations in international development.

“Many of those with relevant knowledge and skills are not currently engaged in IE evaluations in international development.”

Probably not a substantial claim: may not pass S4 (based).

One of the main barriers to extending the range of IE designs is policy makers' concerns that evaluation findings are reliable and defensible.

“One of the main barriers to extending the range of IE designs is policy makers' concerns that evaluation findings are reliable and defensible.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

There are few established rules for combining methods, though one can distinguish parallel from sequential combinations.

“There are few ground-rules for how methods should be combined although distinctions can be made between 'parallel' and 'sequential' combinations of methods.”

Classified as probably not a substantial claim.

Quality assurance content is typically expressed as a series of questions for evaluators, managers, or policy makers to consider.

“Third, the content of QA whether described as a 'guideline' or a 'standard' is mainly operationalised as a set of questions that evaluators, managers or policy makers should ask themselves ('How has this evaluation involved stakeholders?',or 'Has the design that is used been justified and explained?').”

May combine two claims.

Designs and methods identified in the report are immature, still being used mainly by their originators and a small group of enthusiasts, and have not been routinized.

“designs are immature, still being used mainly by their originators and a small group of enthusiasts and have not been routinized in any way.”

Probably not a substantial claim: may not pass S4 (based).

Much of the relevant ground for impact evaluation is already covered by existing QA systems.

“Existing QA systems cover much of the territory relevant to IE.”

Probably not a substantial claim: may not pass S4 (based).

Many current programmes are not narrowly defined; they are ambitious, wide-ranging, and composed of multiple subprogrammes.

“The reality is that many contemporary programmes are not narrowly specified: they are ambitious, broad in scope and made up of many subprogrammes.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

IE methods using theory based evaluation approaches are particularly evident in public health; realist evaluation approaches seem strongest in health and criminology.

“IE methods using theory based evaluation approaches are particularly evident in public health; whilst realist evaluation approaches seem strongest in health and criminology.”

Classified as probably not a substantial claim. May combine two claims.

While useful resources are available to support the use of experimental methods in IE at a country level, fewer resources are available to encourage a wider range of designs and methods.

“whilst there are useful resources available to support the use of experimental methods in IE at a country level, fewer resources are available to encourage a wider range of designs and methods.”

Classified as probably not a substantial claim. May combine two claims.

Across the social sciences, agreement on causal inference is limited, though each design has recognized strengths and weaknesses.

“There is only limited consensus across the social sciences although each design24 has its acknowledged strengths and weaknesses.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Mixed methods are now widely used.

“Mixed methods are now commonplace.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Necessary skills are often in the research rather than the evaluation or consulting community.

“Necessary skills are often in the research rather than the evaluation or consulting community.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Emerging methods such as 'Agent based modelling' appear strongest in areas like climate change and other environmental fields; systems dynamics and other forms of modelling are scattered across engineering, urban spatial analysis and corporate and product planning.

“Emerging methods such as 'Agent based modelling' appear strongest in areas like climate change and other environmental fields; and systems dynamics and other forms of modelling are scattered across engineering, urban spatial analysis and corporate and product planning.”

Classified as probably not a substantial claim. May combine two claims.

RCTs can answer whether something works but not how to improve it; experiments can answer whether it worked here but not whether it will work elsewhere.

“For example, Jonas et al (2009) observes that RCTs can answer 'does it work?' questions but not 'how could it be improved?'.Cartwright and Munro (2010: 265) note that experiments can answer the question 'did it work here?' but not 'will it work for us (elsewhere)?'.”

• cites Jonas et al (2009)Not checked yet

  Not checked in the free run.

• cites Cartwright and Munro (2010: 265)Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

Evaluators are most capable when intervention-policy links are direct and short term, interventions are self-contained and standardised; long-term programmes in changing contexts with extended causal chains are especially challenging.

“Evaluators are at their best when links between 'interventions' and policy goal are relatively direct and short term; and when policy interventions are relatively self contained -parallel and overlapping interventions make it difficult to disentangle causes from effects.We find it easier to deal with standardised interventions that go about things in the same way whatever the circumstances.And we”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Splitting programmes into sub-parts risks missing synergies among parts and basing success or failure claims on incomplete analysis.

“The temptation to break programmes down into sub-parts is therefore strong, however this risks failing to evaluate synergies between programme parts and basing claims of success or failure on incomplete analysis.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Difficult-to-evaluate development programmes, including governance, democracy strengthening, and climate change mitigation, are commonly called complex; for humanitarian relief or state-building, complexity may also describe the setting.

“Development programmes that are difficult to evaluate such as those concerned with governance, democracy strengthening or climate change mitigation, are often described as 'complex'.In the case of humanitarian relief or state-building the term complexity can also be applied to the setting in which the programme is located.”

May combine two claims.

The study's identified designs and methods are often complex and costly, meaning elaborate impact evaluations will likely be uncommon and require justification.

“However many of the designs and methods identified in this study are complicated and expensive.Whilst agency managers and policy makers may want the kinds of answers that only these kinds of designs can deliver, they may need persuading that the extra effort and resource is justified.One approach that we favour is to acknowledge that elaborate IEs (of whatever design) are likely to be rare and”

May combine two claims.

Counterfactual assumptions often fail, and even when they hold, counterfactuals link a single cause to an effect without explaining how the effect is produced.

“But the assumptions of counterfactual thinking do not always hold (e.g.finding an identical match for the factual world, e.g. the world where the cause and the effect have been observed, may be difficult or impossible); and even when they do, counterfactuals associate a single cause with a given effect without providing information on what happens in-between: how the effect is produced.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The OECD-DAC impact definition is broad and reflects the consensus of most international development actors.

“The OECD-DAC definition is broad in scope and represents the consensus view of most international development actors.”

• cites OECD-DAC Glossary (2002)Not checked yet

  Not checked in the free run.

May combine two claims.

Counterfactuals as a causal inference strategy rest on Mill's Method of Difference, which does not require experimental methods.

“However, as a strategy for causal inference, counterfactuals are based on Mill's Method of Difference, which does not necessarily imply adoption of experimental methods.”

May combine two claims.

Even flawless counterfactual inference can mislead; counterfactuals answer setting-specific causal questions and cannot generalise to other settings or times without finer knowledge of operating causal mechanisms.

“Even when counterfactual inference is possible and can be applied flawlessly, it can still be misleading.Counterfactuals answer contingent, setting-specific causal questions 'did it work there and then' and cannot be used for generalization to other settings and timeframes, unless they are accompanied by more fine-grained knowledge on the causal mechanisms actually operating within the process”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Showing that interventions produce development effects requires theories and rules of causal inference capable of backing causal claims.

“Demonstrating that interventions cause development effects depends on theories and rules of causal inference that can support causal claims.”

Classified as probably not a substantial claim.

Advocates of counterfactual and experimental approaches have pushed evaluators to address causal claims and explain intervention effectiveness reliably so lessons can be learned.

“The proponents of counterfactual and experimental ideas have succeeded in challenging all evaluators to address questions of causal claims and explaining the effectiveness of development interventions in ways that because they are reliable can allow lessons to be learnt.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Impact and impact evaluation definitions are embedded in policy debates: the OECD-DAC definition reflects changing aid values and donor-partner relations, while methods-led definitions emerged from evidence-based policy debates.

“Different definitions of 'impact' and 'impact evaluation' are themselves embedded in wider policy-related debates.Thus the OECD-DAC definitions of impact reflects important shifts in understanding and values surrounding aid and development including shifts in the relationship between donors and partner countries -the beneficiaries of aid.On the other hand the methods-led definitions of impact”

May combine two claims.

Impact evaluation has mostly invested in a narrow range of experimental and statistical methods and designs, which DFID has found applicable to only a small proportion of its current programme portfolio.

“Up to now most investment in IE has gone into a narrow range of mainly experimental and statistical methods and designs that according to the study's Terms of Reference, DFID has found are only applicable to a small proportion of their current programme portfolio.”

Classified as probably not a substantial claim.

A significant share of development spending is not amenable to randomised controlled trials.

“'Some have suggested that only 5% of development programmes of donors such as DFID are suitable for randomised controlled trials.Even if this is a conservative estimate, it is clear that this will be the case for a significant proportion of the profile of development spending.'”

Classified as probably not a substantial claim. May combine two claims.

Some impact evaluation proponents, including Poverty Action Lab and the Bill and Melinda Gates Foundation, advocate only experimental and quasi-experimental methods; others say RCTs are useful only in some circumstances.

“Whilst as noted some proponents of IE (e.g.Poverty Action Lab and the Bill and Melinda Gates Foundation) only advocate experimental and quasi-experimental methods others argue that Randomised Control Trials (RCTs) are useful in some but not all circumstances.”

• cites Poverty Action LabNot checked yet

  Not checked in the free run.

• cites Bill and Melinda Gates FoundationNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Asking whether an intervention made a difference is often more informative than asking whether it worked, because the former permits combinations of causes while the latter assumes the intervention acts alone.

“It is often more informative to ask: 'Did the intervention make a difference?' which allows space for combinations of causes rather than 'Did the intervention work?' which expects an intervention to be cause acting on its own.”

Probably not a substantial claim: may not pass S2 (checkable).

When impact evaluation is unsuitable, other evaluation strategies-results monitoring, real-time, operational, action-research oriented, and formative evaluations-can still enhance results and impacts.

“It is important to recognise that even when IE is inappropriate, enhancing results and impacts can still be addressed through evaluation strategies other than IE.Results monitoring can make major contributions to accountability driven evaluations.There will also be occasions, when real-time, operational, action-research oriented and formative evaluations can all make serious contributions to”

May combine two claims.

RCTs are strong on internal validity-what works here and now-but much weaker on external validity, or whether an intervention would work in another setting.

“Ravallion (2009) makes a similar argument that RCTs are strong on internal validity (what works here and now) but are far weaker on external validity -whether an intervention would work in a similar or contrasting setting.”

• cites Ravallion (2009)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Causation seeks to connect cause with effect, and different approaches to causal inference do this in different ways.

“Causation seeks to connect cause with effect and different approaches to 'causal inference' do this in different ways.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Other ways remain available to answer accountability questions, e.g., monitoring results and estimating distance travelled; measuring sub-parts of programmes using hard-edged methods may be appropriate even if a full large-scale programme cannot be evaluated the same way.

“For example, monitoring of results and estimations of 'distance travelled' remains possible; and the measurement of sub-parts of programmes using 'hard-edged' methods (electoral interventions which are just one part of a wider 'Governance' or 'Democracy' programme) may also be appropriate even if a full large-scale programme cannot be evaluated in the same way.”

May combine two claims.

The distinction between true experimental designs based on control/manipulation and observational studies accounts for important differences in how far causal inference is possible and underpins distinctive claims of RCTs.

“This distinction accounts for important differences in how far it is possible to draw causal inference and underpins the distinctive claims of RCTs.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Theory-based, case-based studies and participatory approaches can offer IE real potential to link causes and effects.

“• 'Theory-based', 'Case-based studies' and 'Participatory' approaches can offer IE real potential to link causes and effects.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The focus on poverty reduction, MDGs and IADGs such as gender equality highlights the equity question 'development for whom?' and places limits on how far aid can be managed by donors in traditional ways.

“• The focus on poverty reduction, MDGs and other 'Internationally Agreed Development Goals' (IADGs) such as gender equality, highlights the equity question: 'development for whom?'.It also places limits on how far aid can be managed by donors in traditional ways.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Both experiments and regularity/statistical association approaches work best when causal factors are independent.

“• Both experiments and regularity/statistical association approaches work best when causal factors are independent.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The frequency of humanitarian emergencies, post-conflict stabilisation and containment of terrorism have focused attention on governance, peace-building and empowerment of civil society actors.

“• The frequency of humanitarian emergencies, post conflict stabilisation and the containment of terrorism have all focussed attention on governance, peace-building and the empowerment of civil society actors.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Many development interventions are now longer-term and high risk, with few pre-tested policy models or Theories of Change.

“Many development interventions are now longer-term and high risk for which there are few pre-tested policy models or Theories of Change.”

Probably not a substantial claim: may not pass S2 (checkable).

Following a definition of IE that emphasises causal links between interventions and effects, evaluation is inevitably pushed towards norms and principles of scientific research.

“However we would argue following a definition of IE that specifically emphasises the causal links between interventions and 'effects', evaluation is inevitably pushed towards the norms and principles of scientific research.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Development programmes now rarely involve direct delivery of services by donors; indirect modes and reliance on country systems (e.g., Sector Wide and General Budget Support) are more common and create specific challenges for accountability and assessing VFM.

“Development programmes now rarely involve the direct delivery of services by donors.Indirect modes of delivery and reliance on 'country systems' as in Sector Wide and General Budget Support are more common and create specific challenges for accountability and when assessing VFM.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Neither experiments nor statistical models are good at dealing with contextualisation.

“• Neither experiments or statistical models are good at dealing with contextualisation.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

With growing importance of qualitative methods, the distinction between quantitative and qualitative approaches is both clarified and challenged.

“As the ToR for this study emphasises, with the growing importance of 'qualitative' methods the distinction between quantitative and qualitative approaches is both clarified and challenged.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Recent developments in methodology have blurred distinctions between quantitative and qualitative methods, making combinations more feasible.

“• Recent developments in methodology have blurred the distinctions between quantitative and qualitative methods making combinations of methods more feasible.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Evidence Based Policy originated in medical drug-trial quality assurance protocols, which helps explain the experimental causality approach favoured by EBP and first-generation impact evaluation advocates.

“However the origins of EBP were medical or more specifically based on protocols for quality assuring drug-trials.This goes some way to explaining the particular approach to judging causality through experiments that EBP and advocates of first generation IE favoured.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Causation is at the heart of traditional science and social science.

“Causation is at the heart of traditional science -and social science.”

Classified as probably not a substantial claim.

Short-term accountability evaluations that do not address long-term development effects are on the margins of the paper's definition of IE.

“However short-term accountability evaluations that do not address long-term development effects (even if only by anticipating longer-term causal chains) are on the margins of how we have defined IE.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Methods that strongly support causal inference work best for narrowly specified interventions, while methods that work well for broadly specified interventions are less strong on causal inference even though they can support plausible judgements of effectiveness.

“• Methods that strongly support causal inference work best for narrowly specified interventions while methods that work well for broadly specified interventions are less strong on causal inference even though they can support plausible judgements of effectiveness.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Many difficulties with mixed methods are at the level of mixed designs (underlying assumptions) rather than at a more technical level.

“For example, it maintains attention on causality which in our understanding is at the heart of IE; and it opens up the possibility that many of the difficulties with mixed methods are at the level of mixed designs, i.e. underlying assumptions rather than at a more technical level.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

In circumstances where manipulation is possible, causal claims are strong.

“In those circumstances where 'manipulation' is possible causal claims are indeed strong.”

Classified as probably not a substantial claim. May combine two claims.

The Evidence Based Policy movement began with two assumptions: that only certain methods can rigorously produce evidence of policy intervention efficacy, with biomedical gold-standard methods favouring RCTs for attribution and generalisation; and that policymakers will use high-quality impact evidence and learn from systematic research or evaluation.

“The EBP movement started from two main assumptions.First, it is assumed that there are only some specific methods that can 'rigorously' produce evidence of the efficacy of policy interventions.These superior methods (described by proponents as a 'gold standard') and coming from the bio-medical tradition favour RCTs both as a way of answering questions of attribution but also as a foundation for”

Classified as probably not a substantial claim. May combine two claims.

The OECD-DAC definition of impact is broad enough to accommodate complex and long-term interventions, acknowledges unintended consequences and uncertainty, and does not restrict range of designs/methods.

“It is broad enough to accommodate the complex and long-term nature of the interventions we have been asked to work with; it acknowledges unintended consequences and uncertainty; and does not place restrictions on the range of designs and methods that could be used.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Methods and designs are fit for different purposes; when well-executed all have strengths and weaknesses.

“Rather it is generally understood that methods and designs are fit for different purposes and when well-executed all have their strengths and weaknesses -one of the justifications of so called 'mixed-methods'.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Explanation depends on using theory alongside appropriate methods, because without theory there is no basis for interpretation.

“Explanation depends on the use of theory alongside appropriate methods, because without theory there is no basis for interpretation.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

These programmes are emergent and their impacts are often difficult to evaluate or measure using established tools.

“These programmes are 'emergent' and their impacts are often difficult to evaluate or measure using established tools.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Most real-world evaluations will involve hybrid designs.

“It is important to emphasise that consistent with a commitment to mixed methods we are not suggesting that these designs are alternatives: most real-world evaluations will involve hybrid designs.”

Probably not a substantial claim: may not pass S2 (checkable).

Focusing only on manipulation risks blurring the distinction between contributions of many qualitative and quantitative designs which, in right circumstances and properly implemented, can support causal and explanatory analysis.

“However this risks blurring the distinction between the different contributions of many different types of qualitative and quantitative design which in the right circumstances and when properly implemented are able to support causal and explanatory analysis.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Studies of cases that combine within-case analysis and comparisons across cases are especially suited to IE in complex settings.

“• Studies of 'cases' that combine within-case analysis and comparisons across cases are especially suited to IE in complex settings.”

Probably not a substantial claim: may not pass S2 (checkable).

Choice of methods and overall designs should follow from the kinds of questions asked and the settings where applied.

“Furthermore the choice of methods (and overall designs) needs to follow from the kinds of questions that are being asked; whilst also taking into account the settings in which they are to be applied.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Even sound methods have strengths and weaknesses and need to be combined in mixed or hybrid forms to ensure rigour.

“• that even sound methods have their strengths and weaknesses and need to be combined in 'mixed' or 'hybrid' forms to ensure rigour; and”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Theory alongside methods is needed to permit explanation, which is necessary for evidence to be useful outside the single case.

“• there is a need for theory alongside methods to permit explanation, which is necessary for evidence to be useful outside of the single case.”

Probably not a substantial claim: may not pass S2 (checkable). The quote only partly states this claim.

These circumstances require multi-dimensional interventions at policy, programme and project level.

“These circumstances as in developed countries require multi-dimensional interventions at the policy, programme and project level.”

Probably not a substantial claim: may not pass S2 (checkable).

Because policy makers often need to generalise, the paper has emphasised learning and explanation in its definition of IE.

“Because policy makers often need to generalise, we have emphasised learning and hence explanation in the way IE is defined.”

• AssumptionRests on an assumption: Policy makers often need to generalise.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

In all policy domains, some priorities (e.g., international development, humanitarian relief, failing states) are overriding and cannot wait to be proved before implementation.

“But more routinely in all policy domains there are some priorities -in international development, humanitarian relief and concerns about 'failing' States -that are overriding and cannot wait to be proved before being implemented.”

Classified as probably not a substantial claim. May combine two claims.

Different designs may share similar methods and techniques; what holds them together is the fundamental logic of a design, not their methods.

“Different designs may share similar methods and techniques: both experiments and case studies may use interview data, draw on focus groups and analyse statistical data.What holds them together is the fundamental logic of a design not their methods.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Middle income countries and those coming out of lowest quartiles of extreme poverty face particular problems of inequality and deprivation.

“• Middle income countries and those coming out of the lowest quartiles of extreme poverty face particular problems of inequality and deprivation.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Real-world designs are rarely pure types; hybrid designs combining statistical models with in-depth case-studies or participatory designs with statistical surveys have become common in development and other evaluation areas.

“Real-world designs are also rarely pure types: as we discuss below 'hybrid' designs combining, for example, statistical models with in-depth case-studies or participatory designs with statistical surveys, have become common in development as in other areas of evaluation.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Aid is no longer seen as the sole or even main determinant of development; since mid-1990s other drivers such as world trade, governments' own resources and political economy are seen as working in combination with aid rather than alone.

“• Aid is no longer seen as the sole or even main determinant of development.Since the mid-1990s (OECD-DAC 1996, World Bank 1998) other development drivers such as world trade, governments' own resources and the political economy of developing countries, are seen as working in combination with aid rather than alone.”

• cites OECD-DAC 1996Not checked yet

  Not checked in the free run.

• cites World Bank 1998Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

This includes how impacts of aid can be evaluated in partner countries that take responsibility for their own development, placing new emphasis on capacity development, decentralised evaluation and participatory/empowerment approaches.

“This includes how the 'impacts' of aid can be evaluated in partner countries that take responsibility for their own development, thus placing a new emphasis on capacity development, decentralised evaluation and participatory and empowerment approaches to aid delivery and evaluation.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The Paris Declaration has led to an increase in devolved and locally customised interventions which increasingly rely on devolved evaluations often jointly conducted by donors and beneficiary countries.

“The Paris Declaration has led to an increase in devolved and locally customised interventions which increasingly rely on devolved evaluations often jointly conducted by donors and beneficiary countries.”

Probably not a substantial claim: may not pass S2 (checkable).

IE designs need to address a full range of evaluation questions policy makers are interested in, including explanation and lesson-learning as well as accountability.

“Finally we have introduced the argument that IE designs need to be able to address a full range of evaluation questions that policy makers are interested in.These include explanation and lesson-learning questions as well as the accountability question 'has this intervention worked?'.”

May combine two claims.

In real-world policy settings, what counts as evidence is often ambiguous, and evidence from practitioners, beneficiaries, partners and policy-makers competes.

“What constitutes evidence is often not clear-cut.Different forms of 'evidence' from practitioners, beneficiaries, partners and policy-makers themselves, vie with each other in real-world settings.”

• cites Pawson 2006Not checked yet

  Not checked in the free run.

• cites Cartwright andHardie 2012Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Experiments are good at answering whether a particular intervention made a difference here but weak on generalisation (external validity).

“• Experiments are good at answering the question; 'has this particular intervention made a difference here?' but weak on generalisation (external validity).”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

A contribution-based logic is consistent with the complex and multi-dimensional nature of many contemporary development interventions.

“A contribution-based logic is also consistent with the complex and multi-dimensional nature of many contemporary development interventions.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Generative causation is strong on explanation but weak on estimating quantities or extent of impact.

“• Generative causation is strong on explanation but weak on estimating quantities or extent of impact.”

Classified as probably not a substantial claim.

The evolving nature of aid relationship and shifts in aid priorities/modalities has consequences for IE designs.

“The evolving nature of the aid relationship and shifts in aid priorities and modalities has many consequences for IE designs.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Questions for most evaluations are set by policy makers or commissioners, not scientists as in research.

“Questions for most evaluations are set by policy makers or commissioners not scientists as they are for research.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Multiple causalities are good at dealing with limited complexity and interdependence but not at unpicking highly complex combinations.

“• Multiple causalities are good at dealing with limited complexity and interdependence but not at unpicking highly complex combinations.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Each of the four causal approaches has requirements, potential strengths and potential weaknesses.

“Each of these causal approaches has: • Requirements, i.e. conditions under which they do and do not apply. • Potential strengths. • Potential weaknesses.”

Classified as probably not a substantial claim.

The definition of IE emphasises the logic of contribution as much as attribution.

“To that extent our definition of IE emphasises the logic of contribution as much as attribution.”

• AssumptionThe authors state this as an assumption: Causes are more likely to contribute to impacts than to be exclusively connected as the term attribution implies.

Probably not a substantial claim: may not pass S4 (based).

Long-term interventions and more complex programmes around governance, accountability and democracy may not be amenable to measurement with interval scale data and estimates of net-effects.

“However as discussed further in Chapters 3 and 4 long-term interventions and more complex programmes around governance, accountability and democracy may not be amenable to measurement with interval scale data and estimates of net-effects.”

May combine two claims.

EBP was steered by researchers who chose questions amenable to research answers, and researcher questions can be distant from complex policy questions.

“EBP was steered by researchers who understandably chose questions amenable to answers that research could provide.This highlights a further weakness of EBPquestions posed by researchers can be distant from the more complex and even messy questions that policy makers need answers to.”

Classified as probably not a substantial claim. May combine two claims.

Long implementation chains, partnerships with governments and civil society, and bottom-up or devolved programmes are characteristic of today's complex interventions.

“Long implementation chains; partnerships with governments and civil society; and bottom-up or devolved programmes are characteristic of today's 'complex' interventions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

There continue to be different schools with respectable foundations that take fundamentally different positions about causation.

“These debates are very much alive today: there continue to be different schools each with respectable foundations that take fundamentally different position about causations.”

Classified as probably not a substantial claim.

The paper's definition of IE looks to establishing cause and effect relationships between development interventions and development results, although not exclusively through counterfactual-based methods.

“The definition of IE looks to establishing cause and effect relationships between development interventions and development results, although not exclusively through counterfactual-based methods.”

• AssumptionThe authors state this as an assumption: Causes are more likely to contribute to impacts than to be exclusively connected as the term attribution implies.

Probably not a substantial claim: may not pass S4 (based).

Most social scientists now recognise that causation without explanation is insufficient for policy learning because policy-makers need to understand why as well as how to use findings for future policy-making.

“Finally, most social scientists now recognise that causation without explanation is insufficient for policy learning because policy-makers need to understand why as well as how if they are to use findings from research or evaluation for future policy-making.”

Classified as probably not a substantial claim.

The distinctive character of a contribution perspective focuses less on a single cause and more on how different causal and supporting or contextual factors combine.

“However as the discussion in Chapters 3 and 4 argues as we understand it, the distinctive character of a contribution perspective focuses less on a single cause and more on the way different causal and supporting or contextual factors combine together.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Evidence and possibilities of causal inference can be found through many methods and designs.

“• that evidence and the possibilities of causal inference can be found through many methods and designs;”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In evaluation there are other design choices about purpose, utilisation, stakeholder involvement, country ownership, and ethical dilemmas.

“In evaluation there are also other 'design' choices -about the purpose of an evaluation (which can include lesson-learning as well as demonstrating impacts); about how to ensure utilisation; how to involve stakeholders and, in development settings in particular, to acknowledge country ownership; and how to address ethical dilemmas when working with very poor and marginalised people.”

Classified as probably not a substantial claim. May combine two claims.

The evolution of the EBP debate implies for IE that different kinds of evidence often clash and must be reconciled or judged between conflicting policy signposts.

“• there are different kinds of evidence which often clash and have to be either reconciled or judgements have to be made between conflicting policy signposts;”

Probably not a substantial claim: may not pass S3 (load-bearing).

Notions of partnership at heart of aid relationship are spelt out in Paris Declaration and Accra Agenda for Action with commitment to country ownership/leadership and donor harmonisation.

“• Notions of partnership at the heart of the aid-relationship are most clearly spelt out in the Paris Declaration and the Accra Agenda for Action with their commitment to country ownership and leadership and to donor harmonisation.”

• cites Paris DeclarationNot checked yet

  Not checked in the free run.

• cites Accra Agenda for ActionNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

Without designs and methods, theories and quality standards, there is no firm foundation for identifying causes and offering explanation.

“Without designs and methods, theories and quality standards there is no firm foundation for identifying causes and offering explanation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Aid works together with other factors and rarely leads to development results on its own.

“Aid works together with other factors and rarely leads to development results on its own.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

Regularity requires high numbers of diverse cases; without this it is not possible to capture sufficient diversity.

“• 'Regularity' requires high numbers of diverse cases; without this it is not possible to capture sufficient diversity (or difference).”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

The distinction between learning and accountability purposes of evaluation is not always clear-cut; accountability can and should reinforce policy learning.

“This distinction is not always clear-cut, for example potentially accountability can and should reinforce policy learning.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The emphasis on contributory causes is a major theme in this study, consistent with broad consensus that development aid interventions work best in combination with other non-aid factors.

“This emphasis on 'contributory' causes is a major theme in this study consistent with the broad consensus that development aid interventions work best in combination with other non-aid factors.”

• AssumptionRests on an assumption: Development aid interventions work best in combination with other non-aid factors.

Probably not a substantial claim: may not pass S3 (load-bearing).

Stronger versions of programme theory take into account other understandings.

“Stronger versions do take into account these other understandings.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Causal mechanisms operate in specific contexts, making it important to analyse contexts.

“Causal mechanisms operate in specific contexts making it important to analyse contexts.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Mechanism-based designs look for the connection between cause and effect through in-depth theoretical analysis, rather than by demonstrating regularity or inevitability.

“Designs that are mechanism-based look for the connection between cause and effect through in-depth theoretical analysis, rather than by demonstrating regularity or inevitability.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In QCA, causal inference depends more on conditions necessary and sufficient for an outcome, based on comparisons of configurations of cases and their attributes.

“For example in QCA causal inference depends more on the conditions that are necessary and sufficient for an outcome basing this on comparisons of 'configurations' of cases and their attributes.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Theory is needed to bridge the gap between data and interpretation, and in impact evaluation between causes and effects.

“In order to explain we need theory to bridge the gap between data and interpretation of that data; and in the case of IE to bridge the gap between 'causes' and 'effects'.”

May combine two claims.

These designs are beginning to be applied in some areas of policy research, but only being spoken of in international development, and are entirely absent from IE in this sector.

“These designs are beginning to be applied in some areas of policy research, but only being spoken of in international development (Porter 2011), and are entirely absent from IE in this sector.”

• cites Porter 2011Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The single case features in case-based approaches especially through within-case analysis.

“The single case also features in case based approaches especially through 'within-case' analysis -examining the attributes of a case so as to establish configurations.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Causal inferences are only reasonable if measurement is valid, which depends on careful attention to conceptualisation; issues of conceptualisation and measurement are more fundamental than generalising from a sample to a population.

“'Causal inferences are only reasonable if measurement is valid … this depends on careful attention to conceptualisation…… Issues of conceptualisation and measurement are more fundamental than the conventional problem of generalising from a sample to a population' (Collier, Brady and Seawright (2010;197)).”

• cites Collier, Brady and Seawright (2010;197)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

This suggests that real-world constraints such as budget, skill capacities and timescale may have had some role in how the evaluation was conducted.

“This suggests that 'real-world' constraints (Bamberger et al 2006) such as budget, skill capacities and timescale may have had some role in how the evaluation was conducted.”

• cites Bamberger et al 2006Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

There has been a tradition in evaluation of naturalistic, constructivist, and interpretative case studies that focus on unique characteristics of a single case.

“There has been a tradition in evaluation of 'naturalistic', 'constructivist' and 'interpretative' case studies that generally focus on the unique characteristics of a single case.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Process-oriented theory-based approaches regard a programme as a sequence of causal links from initiation to intended outcomes.

“It regards the programme as a 'conjunction' of causes that follows a sequence.It follows the pathway of a programme from its initiation through various causal links in a chain of implementation, until intended outcomes are reached.”

May combine two claims.

Case-based causal inference may test a theory and also contribute to theory building.

“Case based causal inference may 'test' a theory (as in George and Bennett's notion of 'congruence') and also contribute to 'theory building'.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Evaluation, and impact evaluation in particular, provides an opportunity to test a programme's theory through the links in the causal chain.

“Evaluation and IE in particular, is an opportunity to test a programme's theory through the links in the causal chain.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Typologies of context are a useful intermediate step towards generalisation in mechanisms-based evaluations.

“This is often to be done only in an ad hoc way but typologies of context are a useful intermediate step towards generalisation in mechanisms based evaluations.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

They are interested in generalising beyond the single case, though not in universalising.

“They are also interested in generalising that goes beyond the single case -though not in 'universalising' (Byrne 2009).”

• cites Byrne 2009Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

In some programmes there is one main intervention and in others several are implemented as part of an overall programme; some aim for a single outcome/impact while others aim for several; interventions may be multiple or single, interdependent rather than independent.

“For example in some programmes there is one main intervention and in others several are implemented as part of an overall programme.Some programmes aim for a single outcome or impact whilst others aim for several.Interventions may be multiple or single, interdependent rather than independent.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

There are raised questions about how IEs are structured, e.g., ensuring enough time at the beginning to construct a sensible analytic framework and Theory of Change, or proceeding iteratively and revising an initial evaluation plan as the evaluation unfolds.

“There are also raised questions about how IEs are structured.For example ensuring that enough time is allocated at the beginning of an evaluation to construct a sensible analytic framework and Theory of Change; or to proceed iteratively and revise an initial evaluation plan as the evaluation unfolds and new issues become clear.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Mechanism-focused understandings build on the assumption that plausible causal claims require identification of a mechanism.

“Mechanism focused understandings, build on the widespread assumption in the philosophy of science that in order to be able to make plausible causal claims there needs to be an identification of a mechanism (or what Cartwright calls 'capacities') that make things happen.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Theory-based approaches can be divided into two main tendencies: process-oriented and mechanism-focused.

“However we can identify two main tendencies in theory-based approaches likely to have traction for IE purposes.”

• cites Funnell and Rogers 2011Not checked yet

  Not checked in the free run.

May combine two claims.

These approaches tend to generalise 'under certain conditions' and identify clusters or subsets of cases for similar causal inferences.

“These approaches tend to generalise 'under certain conditions' and identify clusters or subsets of cases about which it is possible to make similar causal inferences.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Newer forms of simulation or modelling, such as Systems Dynamics and Agent based modelling, are underdeveloped designs that could be useful for virtual experiments with single-case complex interventions.

“One noticeably underdeveloped design that has been included in the table above is newer forms of simulation or modelling.'Systems Dynamics' and 'Agent based modelling' in particular could be a useful design to conduct 'virtual' experiments with single-case complex interventions.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

New methodologies for systematic causal analysis of cases emerged around the turn of the century.

“New methodologies for the systematic causal analysis of 'cases' emerged around the turn of the century.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Weak forms of programme theory can be little more than a logic model that ignores other stakeholders and assumptions about conditions.

“In weak forms it can be little more than a logic model that expresses the intentions of policy makers ignoring the actions and intentions of other stakeholders; and not making clear the assumptions that are made about the conditions under which the programme success is assumed.”

May combine two claims.

These case studies eschew causal analysis even though they contribute to such analysis in several ways.

“These case studies eschew causal analysis even though they contribute to such analysis in several ways.”

Probably not a substantial claim: may not pass S4 (based).

Different designs and methods are best able to deal with different kinds of causal relations.

“Different designs and methods are best able to deal with different kinds of causal relations.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Generative causality explains how causal factors interact; the association as a whole is explained, not just one element's influence on another.

“Instead of comparing cases generative causality explains how causal factors interact.The causal explanation is not a matter of one element (X), or a combination of elements (X1.X2) asserting influence on another (Y); rather it is the association as a whole that is explained (Pawson 2007).”

• cites Pawson 2007Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Programme mechanisms take the step from asking whether a programme works to understanding what it is about the programme that makes it work.

“Programme mechanisms 'take the step from asking whether a programme works to understanding what it is about the programme that makes it work' (Pawson and Tilley 1997:66).”

• cites Pawson and Tilley 1997:66Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

Locating variables in the context of the case and conducting within-case analysis alongside comparisons across cases has opened up major new opportunities for causal analysis that are still largely ignored in evaluation practice.

“Locating variables in the context of the 'case' and conducting within-case analysis alongside comparisons across cases has opened up major new opportunities for causal analysis that are still largely ignored in evaluation practice.”

May combine two claims.

Developing typologies of context can support 'under these conditions' types generalizations, though not universal generalization.

“Developing typologies of context whilst not supporting universal generalization can support 'under these conditions' types generalizations.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

It should be possible to conduct single case studies within this framework and locate the case in relation to a previously established cluster of similar programmes or institutions.

“Notionally it should be possible to conduct single 'case' studies within this framework and locate the case in relation to a previously established cluster of other similar programmes or similar institutions.”

May combine two claims.

RCTs work best under stringent conditions: one primary cause and effect, an identifiable control group with controlled contamination, interest in a specific setting rather than wider generalisation, enough cases for statistical analysis, and a greater need for causal than explanatory analysis.

“Where there is likely to be one primary cause and one primary effect of interest although there are many possible causes -hence difficulties with complex interventions.”

• Also stated in: § 4.10 · plain

  “Where a 'control' group can be identified and contamination between treatment and comparison groups can be controlled.”

• AssumptionThe authors state this as an assumption: Expected outcomes and the intervention itself are clearly understood and specifiable.
• AssumptionThe authors state this as an assumption: There is a likelihood of a primary cause and a primary effect.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

New forms of participation through joint donor/beneficiary IEs may be one solution.

“New forms of 'participation' through joint donor/beneficiary IEs may be one solution.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Combining methods is a way to overcome limitations and enhance strengths.

“Combining methods is a way to overcome limitations and enhance strengths.”

Classified as probably not a substantial claim.

Three main designs that show promise to reinforce existing IE practice when dealing with complex programmes are theory-based, case-based, and participatory.

“Three main designs that show promise to reinforce existing IE practice when dealing with complex programmes -theory-based; case-based and participatory -were identified and introduced.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

It should be possible to interpret impacts in terms of an intervention's participatory content, e.g., the extent to which involvement, ownership, and commitment improve development outcomes.

“It should be possible to interpret impacts in terms of an interventions participatory content: for example the extent to which involvement, ownership and commitment improve development outcomes.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

There are challenges for transferable learning through IE in today's development architecture built around indirect delivery of aid through partnerships.

“There are challenges for transferable learning through IE in today's development architecture built around indirect delivery of aid through partnerships.”

May combine two claims.

Designs and methods applied to narrowly specified interventions may lead to strong causal claims, but this will not be so for broadly specified interventions.

“The implications of the above is that whilst designs and methods applied to narrowly specified interventions may lead to strong causal claims this will not be so for broadly specified interventions.”

Probably not a substantial claim: may not pass S4 (based).

New understandings of case-based rather than variable-based methods, and challenges to a simple quantitative/qualitative distinction, have opened up new methodological possibilities for IE.

“New understandings of case-based rather than variable-based methods; and challenges to a simple distinction between quantitative and qualitative methods have opened up new methodological possibilities for IE.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The case-based view regards the case as a complex entity in which multiple causes interact not always predictably.

“The case-based view on the other hand, regards the case as a complex entity in which multiple 'causes' interact not always predictably.”

Probably not a substantial claim: may not pass S3 (load-bearing).

True experiments (RCTs) are a powerful method for answering intervention-specific questions about effectiveness.

“True experimental designs (i.e.RCTs) are a powerful way to answer intervention specific questions about effectiveness.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

It is how these causes interact as a set that allows an understanding of causes.

“It is how these causes interact as a set that allows an understanding of causes (for example, the way that women owned businesses contributes to economic or income growth can only be understood in the context of sources of finance, stage in the economic cycle and family structure).”

Probably not a substantial claim: may not pass S4 (based).

Statistical evaluation approaches that compensate for incomplete data or deploy quasi-experimental methods when control is not possible have less statistical power than straightforward experiments.

“At the same time various statistical evaluation approaches that seek to compensate for incomplete data or deploy quasi-experimental methods when control is not possible, also have less statistical power than straightforward experiments.”

Probably not a substantial claim: may not pass S4 (based).

Some methods are stronger on internal than external validity.

“As foreshadowed in Chapter 3, some methods are stronger on internal than external validity.”

Probably not a substantial claim: may not pass S2 (checkable).

Beneficiaries have agency and can help cause successful outcomes by their own actions and decisions.

“Within this understanding, beneficiaries have 'agency' and can help 'cause' successful outcomes by their own actions and decisions.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Contribution makes a causal claim about whether and how an intervention has contributed to an observed impact.

“This contrasts with 'contribution', which makes a causal claim about whether and how an intervention has contributed to an observed impact.”

May combine two claims.

When the conditions for RCTs do not hold, comparative case-based designs such as QCA and participatory methods combined with comparative case designs can help answer counterfactual questions.

“If all these conditions do not hold, some of the comparative, case-based designs described in Chapter 3 could also help answer these questions.”

• cites Rihoux and RaginNot checked yet

  Not checked in the free run.

• AssumptionThe authors state this as an assumption: There are several relevant causes that need to be disentangled.
• AssumptionThe authors state this as an assumption: Interventions are just one part of a causal package.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

This perspective is consistent with David Ellerman (2005) that development is only possible through self-directed actions, and supporters should enhance autonomy.

“This perspective is also consistent with David Ellerman (2005), that it is only through self-directed actions that development is possible; and that it is the role of those supporting development to enhance that autonomy.”

• cites David Ellerman (2005)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Classical quantitative research draws conclusions from de-contextualised variables that may or may not be true to the real world.

“Classical quantitative research draws conclusions from variables that are de-contextualised and represent phenomena that may or may not be true to the real world (is the state rated as 'legitimate'; how many small-businesses are owned by women etc.?).”

Probably not a substantial claim: may not pass S4 (based).

There are now new possibilities and frameworks to combine different methods and designs in ways that allow causal inference to be systematically assessed.

“In particular there are now new possibilities and frameworks to combine different methods and different designs in ways that allow causal inference to be systematically assessed.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Experimental attribution typically compares similar situations with and without the intervention to show that the intervention was sufficient for a net change and that without it there was no impact beyond existing trends, so the change could only come from the intervention.

“Typically, this is approached by comparing like situations with and without the intervention, showing both that with the intervention there was an impact i.e. that the intervention was sufficient for a net change in the impact; and that without the intervention, there was no impact (necessity), i.e. that net changebeyond pre-existing trends or business as usual -in those conditions could only be a”

• cites Mill's Method of DifferenceNot checked yet

  Not checked in the free run.

• AssumptionThe authors state this as an assumption: For experimental attribution, without the intervention there would be no impact or a different impact, with the focus on additional change.

Classified as probably not a substantial claim.

Case-centred methods are not limited to small-N research but can offer alternatives for large datasets where limitations of mainstream quantitative methods restrict conclusions.

“In fact it is argued that: 'Case centred methods are by no means limited to small-N research, but can also offer alternative for large datasets in situations where the limitations and assumptions of mainstream […] quantitative methods restrict conclusions that can be drawn from the analysis of quantitative data' (Ray Kent 2009).”

• cites Ray Kent 2009Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

A commonly expressed concern is about the strength of causal claims that can be made with other than experimental designs.

“A commonly expressed concern is about the strength of the causal claims that can be made with other than experimental designs.”

Probably not a substantial claim: may not pass S4 (based).

In development policy, the characteristics or general forms of causes and effects vary with the characteristics of interventions.

“In development policy, the characteristics or general forms of causes and effects vary with the characteristics of interventions.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Methods can be combined in different ways: triangulation, complementarity, new start.

“Methods can be combined in different ways (Greene, Caracelli and Graham 1997foot_11) :”

• cites Greene, Caracelli and Graham 1997Not checked yet

  Not checked in the free run.