Essential components of public health evidence reviews: capturing intervention complexity, implementation, economics and equity

Abstract

Carefully developed recommendations for conducting studies of programme effectiveness have provided an extremely useful framework for researchers and articulated the different components required to answer not only the question of effectiveness (does it work?), but also the equally important questions of how, why and for whom was the programme effective.1 However, the continued emphasis on the use of the term ‘complex’ in describing the intervention itself or the system within which it is contextualized can detract from our ability to focus on the strategies required to better understand the strengths or limitations of the evidence base for decision-making. Social and public health interventions operate in a context that demands explicit recognition of politics, service systems, funding flows and shortages, staff competencies and multi-strategic approaches. For those immersed in the complicated political and scientific acrobatics of coordinating studies of programme implementation and evaluation, any real or perceived misalignment in the connection between research effectiveness trials versus policy relevant implementation knowledge must be overcome. Devising relatively simple approaches to understand complexity can assist in making complexity more manageable so that meaningful answers to important policy and practice questions can emerge. In this paper we argue that if reviews of intervention evidence are to be useful to decision-makers at all, contextual and implementation information is an essential, non-negotiable component of the review process. We highlight steps evidence review authors can take to capture and interpret this information. With relatively small changes or additions to the evidence review process, practical, meaningful and rigorous public health evidence can be generated. Within the context of rigorous and transparent review methodology, there is a strong need for the scope of public health evidence reviews to be shaped by end users: stakeholders who are uniquely placed to have a deep understanding of the nuances inherent in the research and policy environment context. Having this perspective early on in the development of the protocol for the evidence review will go a long way to ensuring relevance of the evidence captured and reviewed in the process. For example, in an upcoming Cochrane public health review funded by 3ie,2 end users have been involved in a review advisory group (RAG) to assist in defining slum upgrading interventions: a range of strategies from installing basic infrastructure, home improvements, securing land tenure, improving access to health and basic services through to integrating physical, social, economic, organizational and environmental improvements in slums. Given that the range of strategies is too broad to be assessed in one review, identifying the core component of all programmes to be included was essential in narrowing the scope. In this case, physical environment interventions (with or without other approaches) usually represent the most basic level of slum upgrading, thus enabling the review findings to be relevant to a broad range of end users. The RAG served an important role in guiding this decision. Logic models can be used for many purposes including designing, implementing and evaluating interventions. They can also be used in a systematic review to detail the process of how an intervention might be expected to achieve particular outcomes and what aspects of the process are under review. Coming to a shared understanding of how the intervention is likely to improve public health can be difficult when stakeholders are involved from different disciplines and sectors, however achieving this ensures that the outcomes will be useful and have wide relevance. Mapping pathways and presenting logic models also assists the reader of the review by ‘setting the scene’ and explicitly articulating the theoretical assumptions of the review authors. This logical model can also guide the presentation of results and synthesis of findings by the review authors, potentially making the review more readable. The inclusion of logic models in systematic reviews of complex interventions has been described in detail elsewhere.3 We echo these recommendations and suggest that logic models are a useful and important way of demystifying a complex process and making clear the ‘logic’ of the review parameters, in ways that are immediately (and visually) explicit to the reader of the review. Implementation and effectiveness are inextricably linked. For example, an intervention may not have achieved its desired outcomes, and therefore deemed ineffective, however on closer examination it may become apparent that the programme was not implemented as planned. By collecting and examining information about implementation, researchers and subsequently review authors can gain insights into both why an intervention may not have shown effect (for instance, the intervention may have not reac