Evidence-based Policy: In Search of a Method

“Simply producing a short summary or profile of each study does not guarantee helping those readers who might have to make important decisions based on the study findings (Pawson, 2002).”

“For example, ‘naming and shaming’ works well in many contexts, but when used in the non-payment of council tax in the UK, it was found for certain communities to create ‘heroes’ rather than ‘villains’ (Pawson 2002a).”

“(2011: 59) Passing quickly over the problem of measuring baselines and outcomes at such high levels of aggregation (Pawson, 2002) we come to a key aspect of Kazi et al.’s article.”

“These often use broad inclusion criteria for evidence (qualitative and quantitative) to facilitate comparison of impact evaluation methods, develop a transferable theory, and attempt to provide policy-makers with knowledge in response to time sensitive and emerging issues (Victora et al., 2011; Saul et al., 2013; Paws…”

“In other words, how does a change in context generate a particular mechanism that in turn produces specific outcomes? In developing these program theories, realist reviews assume a broad inclusion of evidence, both quantitative and qualitative, which is used to develop an understanding of ‘what works, for whom, in wha…”

“It requires in-depth knowledge of the studies included in the review but is also capable of generating findings that can be generalized to other situations by means of developing and examining theory (Pawson, 2002a).”

“Acknowledging the problems of observational (non‐manipulated) studies, critics stress the limited scope of RCT methods for studying important social and organizational issues (Pawson, 2002a , 2002b ), noting the progress made in statistics and social science in working with observational data (eg, Nagin, 2005 ; Rubin,…”

“While at the sociopolitical level Pawson remains recognisably in the social engineering rather than critical mode, his synthesis offers considerably more sophistication than that which informs most reviews ( Pawson, 2002a ), which base their quality criteria on the method of RCTs (Torgerson & Torgerson, 2001) as such,…”

“It is possible – in fact common – for a promising social project to be ‘let down’ by failings in another part of the support system that the target group need if they are to have a reasonable prospect of improving their situation (Pawson, 2002).”

“Synthesis is particularly useful in areas of contested findings, to present evidence, consolidate knowledge and therefore progress an area of inquiry (Pawson, 2002).”

“We believe that more is to be learned by analyzing the same means applied to different ends (Pawson, 2002).”

“Therefore the synthesis is by definition narrative and descriptive, rather than statistical (Pawson, 2002; Shadish, Cook, & Campbell, 2002).”

“Focusing on the use of meta-analysis Pawson (2002a) puts forward a number of objections.”

“In a couple of recent articles (Pawson, 2002a, b) I have joined the imbroglio and argued that an approach termed ‘realist synthesis’ has many advantages over the more standard approaches of ‘meta-analysis’ and ‘narrative review’.”

“…certainly does link (and should take us back) to some rather hoary debates in public policy such as decision-making (Russel and Jordan 2009), policy co-ordination (Schout and Jordan 2007), and the development and (non-)utilization of evidence in politics (Dowie 1996; Pawson 2002; Sanderson 2006).”

“In turning to a discussion of this question, we draw from prior analyses of arguments for and against EBP (Gambrill, 1999, 2001, 2003; Gibbs & Gambrill, 2002; Pawson, 2002; Sackett et al., 2000; Straus & McAlister, 2000; Webb, 2001).”

“While police research has its roots in the social and behavioral sciences, the recent adoption of the “medical evidence” model can be seen to sidestep the differences between medical and social phenomena, as well as presenting a narrow focus of “evidence-based” medicine, that is focusing on increased application of ch…”

“The narrative reviews approach (Pawson, 2002) was used to extract data on study characteristics, including author(s), year of publication, study location, study populations, study objectives, research design, ACEs measurement, child outcomes, mechanisms tested, and main findings (See Supplemental…”

“142 The information was extracted from the selected articles using the descriptive143 analytical method.(25) The collected data included author(s), year of publication, 144 research design, level of evidence, number of participants, participant characteristics, 145 age (range), intervention, parameters for interventio…”

“This component, which is part of the descriptive analytical method within the narrative review tradition, ensured data were collected in a standard way, enhancing their usefulness [23].”

“The ‘narrative review’ approach by Pawson [31], adopted in this review, aims to include a broader context of the results to provide readers with a better understanding.”

“Seven sources address conceptualization of synthesizing opportunities (The Joanna Briggs Institute 2014; Thomas et al. 2004; Popay et al. 2006; Jones 2005; Pope et al. 2007; Pawson 2002; Lucas et al. 2007).”

“2000, Sanderson, 2002a and 2002b, Campbell, 2002, Packwood, 2002) and within the evaluation (Pawson, 2002), statistics (Smith, 1996) and public finance (Nutley and Davies, 1999) communities.”

“Pawson (2002a, 2002b, 2002c, 2002d, 2003) has recently been pushing the issue of generalization.”

“or synthesis [20–23], which may be more helpful to knowledge users [21, 22] in public health for synthesizing evidence on effective implementation [23–26].”

“Realist synthesis adopts this logic for systematic review (Pawson, 2002b).”

“The commissioning of evaluations of major policy initiatives has become routine under New Labour with an increased interest in ‘what works’ (Pawson, 2002 ).”

“The aim is to minimize the risk of bias and inaccuracy in policy makers’ reasoning by providing an antidote to “crowd pleasing, political pandering, window dressing, and god-acting” (Pawson 2002:160).”

“It involves approaches such as systematic review, realist synthesis (PAWSON, 2002a, b), action learning (LINCOLN, 2001) and meta-evaluation.”

“It proposes that programmes work (i.e. have successful outcomes) only where they introduce appropriate ideas and opportunities (mechanisms) into appropriate contexts (Pawson and Tilley, 1997, Pawson, 2002).”

“The findings are mapped into these themes as they relate to the drug GHB and presented as a broadbased “narrative review” design (Pawson 2002: 171) of existing literature on the drug GHB as situated within certain user groups.”

“In line with other researchers' views, and as a realist synthesis approach for the data synthesis was adopted, every contribution was additionally judged based on its t for purpose [24], whether it added anything important to our understanding of the phenomenon under review [25], and on its quality as it was establish…”

“The charting approach was akin to a ‘narrative review’ [12] to include detailed information of population characteristics according to identified indicators of health disparities.”

“…focus on the importance of wider contextual factors, such as policy domain, bureaucratic structures, or participant behaviour, capable of influencing whether or not a particular mechanism is are successful in delivering a desired outcome (Pawson, 2002b; 2006; Pawson, Greenhalgh et al., 2005).”

“From the guidance of Arksey and O'Malley,(17) a narrative synthesis was chosen to summarize the data because it allowed the data to be analyzed and contextualized, improving the likelihood it would be understood by the reader.(23,24) In summarizing the findings on fear and its impact on activity, the template set out…”

“Aggregative review methods, typified by meta-analysis, are not currently useful for analysing evidence about suicide in later life because of the lack of trials, and would be problematic even when more trial results are published because meta-analysis produces decontextualized lessons (Pawson, 2002).”

“In areas where the judgements involve a long causal chain from the intervention to the outcome as in public health or other social care or educational interventions (Kelly et al, 2010; Kelly and Moore, 2010), the problem of external validity is still more vexing (See also Pawson, 2002, 2006).”

“literature by drawing on Pawson’s (2002) descriptive-”

“[43–45] The process of navigating and redefining the findings was iterative, and with each stage reflexive, by fine tuning and repeating steps so as to ensure comprehensive coverage of available literature.”

“In reality, there are demands from quick returns for research within the criminal justice system (Pawson, 2002).”

“Equally, quantitative meta-evaluation has the potential to be far more than the simplistic parody suggested by some critics (Pawson, 2002a, 2002b).”

“Pawson R. Evidence-based policy: in search of a method.”

“The first, somewhat contentiously, is as a social movement representing the efforts of persons inside and outside formal decision-making processes to improve policy outcomes by applying rational means and techniques (Jackson, 2007; Mintrom, 2007; Packwood, 2002; Pawson, 2002).”

“…and research syntheses are becoming increasingly popular in the social sciences (Arksey & O’Malley, 2005; Davies, Nutley, & Smith, 1999; Grant & Booth, 2009; Hansen & Rieper, 2009; Learmonth & Harding, 2006; Pawson, 2002; Petticrew & Roberts, 2006; Sandelowski & Barroso, 2006; Sanderson, 2002).”

“Well-designed randomized controlled trials frequently show no difference or a small and possibly unimportant differences between two treatments [8, 19].”

“Charting the data To collect data systematically, standard information was extracted from each article using the descriptive-analytical method.(18) The collected data included author(s), year of publication, study location, area of health or QOL, participant information, study design, and specific identified issues.”

“Adapting descriptive–analytical methods traditionally used in narrative reviews and in published guidelines for scoping reviews similar to this project, information from selected papers was systematically extracted using a standard data abstraction form [23].”

“Many authors have pointed out the importance of basing an evaluation on a theory, on a precise understanding of the mechanisms operating in the programs being studied (Chen, 1990; Chen and Rossi, 1983; Jordan et al., 2008; Pawson, 2002; Pawson and Tilley, 1997; Shadish et al., 1991).”

“The approach adopted in this study in ontological terms is related to the Realist Evaluation approach of Pawson and his colleagues (Pawson, 2001; Pawson, 2006; Pawson, 2013; Pawson & Tilley, 2004).”

“However, ‘[o]utcome measurement is a tourniquet of compression’, squeezing out variation and diversity within and between studies through repeated averaging (Pawson 2002, p. 166).”

“But what kinds of knowledge or evidence are used, how and by whom in this process is unclear a priori (Feldman, 1989; Howlett, Tan, Wellstead, Migone, & Evans, 2014; Nutley, Walter, & Davies, 2007; Pawson, 2002; Sanderson, 2009).”

“This approach follows what Pawson (2002) describes as a ‘descriptiveanalytical’ method, ‘where studies are examined in relation to a common analytical framework, so the same template of features is applied to each study scrutinized’ (p. 172).”

“Again, we now know that this is not as straightforward as we might think in either syntheses of narrative evaluations or the integration of statistical studies (Pawson, 2002) The presence of evaluation in social policy levels is identified at an earlier stage in this paper.”