Translational research in South Africa: evaluating implementation quality using a factorial design

Probably not a substantial claim: may not pass S3 (load-bearing).

The HW program has developed since 2000 from pilot testing to an efficacy trial with more than 7,000 youth in the Cape Town area.

“Since 2000, this program has evolved from pilot testing through an efficacy trial involving over 7,000 youth in the Cape Town area.”

Classified as probably not a substantial claim.

A factorial experimental design was selected so that the separate effect of each of the three factors could be estimated.

“An innovative factorial experimental design was chosen to enable estimation of the individual effect of each of the three factors.”

Probably not a substantial claim: may not pass S3 (load-bearing).

One estimate is that an evidence-based intervention takes 17 years to enter practice.

“Some have estimated that it takes 17 years for an EBI to be put into practice (Bales and Soren 2000)”

• cites Bales and Soren 2000Not checked yet

  Not checked in the free run.

May combine two claims.

HW has shown encouraging effects on sexual risk behavior and substance use and is listed by AIDSTAR-One as a promising practice with good evidence and high promise.

“HW showed promising positive results on sexual risk behaviors and substance use (described subsequently in more detail) and is listed on the AIDSTAR-One website as a promising practice, category 3 with good evidence and high promise”

• cites AIDSTAR-OneNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Garet and colleagues also found that coherence-helping teachers see how the curriculum fits the school's mission-matters.

“Garet and colleagues also found that helping teachers understand how the curriculum fits into the school’s mission (i.e., coherence) is important.”

• cites Garet et al. 2001 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

An interview study found that teachers considered training and detailed instruction manuals essential for implementation fidelity and quality.

“One implementation fidelity-focused study based on interviews with teachers (as well as other key individual) identified that teachers viewed training and detailed instruction manuals were critical to implementation fidelity and quality (Dusenbury et al. 2004).”

• cites Dusenbury et al. 2004Not checked yet

  Not checked in the free run.

May combine two claims.

Both sexes in HWStudy1 reported less substance use at sexual encounters and less sex with someone recently met.

“both males and females reported less use of substances at sexual encounters and were less likely to report having sex with someone they had recently met (Tibbits et al. 2009).”

• cites Tibbits et al. 2009 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Time with friends predicted higher lifetime alcohol use, while hobbies and music/singing predicted lower lifetime alcohol use.

“Spending time with friends positively predicted lifetime alcohol use while participating in hobbies and music and singing activities negatively predicted lifetime alcohol use (Tibbits et al. 2009).”

• cites Tibbits et al. 2009 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

HWStudy1 results indicated that HW reduced both substance use and sexual risk.

“Results from HWStudy1 suggest that HW was effective in reducing both substance use and sexual risk (Smith et al. 2008).”

• cites Smith et al. 2008 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Research on teacher training and fidelity shows a positive but weaker-than-expected association.

“Although studies examining the relationship between teacher training and fidelity have shown a positive association, the relationship is weaker than might be expected (e.g., Gottfredson and Gottfredson 2002).”

• cites Gottfredson and Gottfredson 2002 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The design is not an eight-arm trial; each effect is estimated from all 56 schools by combining conditions.

“It is important to note that this design should not be considered an 8-arm trial in which each condition is compared in turn to a control condition. Such a design would have extremely low power with n = 7 per group. Instead, in this factorial experiment each effect is estimated based on the entire N = 56 schools.”

Classified as probably not a substantial claim.

Monitoring how well the treatment factors were implemented at each school is necessary to rule out poor factor implementation if results are unexpected.

“For the same reasons it is necessary to document teacher implementation fidelity and quality, it is necessary to monitor how well we implemented the treatment factors at each school. This will allow us to answer the question, “If our results are not as anticipated, can we rule out the possibility that the treatment factors were not well implemented?””

Probably not a substantial claim: may not pass S3 (load-bearing).

A US national study of mathematics teachers found that sustained professional development focused on content and active learning is important.

“Another US nationally representative study on mathematics teachers provided evidence that sustained professional development that focuses on content and provides opportunities for active learning are important (Garet et al. 2001).”

• cites Garet et al. 2001 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The culture or climate of an intervention setting is frequently ignored during dissemination.

“although it is often overlooked in the dissemination process (Adelman and Taylor 2003; Burns and Hoagwood 2005; Dusenbury et al. 2003; Greenberg et al. 2005; Rohrbach et al. 2006; Wandersman et al. 2008).”

• cites Adelman and Taylor 2003 sourceNot checked yet

  Not checked in the free run.

• cites Burns and Hoagwood 2005Not checked yet

  Not checked in the free run.

• cites Dusenbury et al. 2003 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The MSED is starting to adopt the Health Promoting Schools Framework.

“the MSED (as in all provinces of South Africa) is beginning to adopt the Health Promoting Schools Framework (HPSF), which is based on empowerment and a more comprehensive community approach to education (Department of Health 2000; National Conference on Health Promoting Schools 2006).”

• cites Department of Health 2000Not checked yet

  Not checked in the free run.

• cites National Conference on Health Promoting Schools 2006Not checked yet

  Not checked in the free run.

May combine two claims.

HW changed boredom and motivation, with the strongest effects at the school showing the highest implementation fidelity.

“HW was effective in changing levels of boredom and motivation, but the strongest effects were found for the school that had the highest degree of implementation fidelity (Caldwell et al. 2010).”

• cites Caldwell et al. 2010 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Randomized controlled trials can estimate the combined causal impact of enhancement strategies, yet they are not efficient for testing each strategy separately.

“This approach enables assessment of the causal effects of the enhancement strategies as a package. However, it is not an efficient means of evaluating the effectiveness of individual enhancement strategies (Collins et al. 2005, 2007, 2009a, 2011).”

• cites Collins et al. 2005 sourceNot checked yet

  Not checked in the free run.

• cites Collins et al. 2007 sourceNot checked yet

  Not checked in the free run.

• cites Collins et al. 2009aNot checked yet

  Not checked in the free run.

May combine two claims.

HWStudy1 also increased perceived condom availability, condom skills, and partner conversations about condom use.

“There was also increased perception of condom availability, condom skills, and conversations with partners surrounding condom use (Coffman et al. 2011).”

• cites Coffman et al. 2011 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Few studies have examined which training elements are critical for successful teacher training on prevention program fidelity and quality, and little professional development research links teacher development to student outcomes.

“There have not been many studies that have specifically studied what elements of training are critical to successful teacher training regarding prevention program fidelity and quality, although there is a body of literature on professional development in general, there is little that focus on professional development of teachers as it relates to student outcomes (Garet et al. 2001).”

• cites Garet et al. 2001 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Although controlled studies can achieve high fidelity, strict fidelity is usually lost when programs are scaled up in real-world settings.

“Although high levels of fidelity have occurred in well controlled studies (e.g., Mihalic et al. 2004), when programs go to scale in the “real world,” strict fidelity to implementation typically is lost (Dusenbury et al. 2003; Bumbarger and Perkins 2008).”

• cites Dusenbury et al. 2003 sourceNot checked yet

  Not checked in the free run.

• cites Bumbarger and Perkins 2008 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The teachers the authors collaborate with appreciate guidance and support but require flexibility.

“It is motivated by the observation that the teachers with whom we work welcome support and guidance, but need flexibility (Wegner et al. 2008).”

• cites Wegner et al. 2008 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The study uses a randomized factorial design with three factors-teacher training enhancement, SSS, and school environment enhancement-each set at present or absent.

“HWStudy2 employs an RCT using a factorial design manipulating the three independent variables described previously: enhanced teacher training, SSS, and enhanced school environment. Each independent variable has two levels: present and absent.”

Classified as probably not a substantial claim.

It is possible that under some conditions or in some settings, training is not the best way to convey knowledge about an intervention.

“Thus, it is possible that in certain environments or under certain conditions, training is not the most effective method of giving knowledge about the intervention.”

May combine two claims.

Observational studies let variables vary uncontrolled, making individual causal effects hard to identify and permitting only tentative causal conclusions.

“In an observational study, variables hypothesized to impact the success of dissemination are allowed to vary in an uncontrolled, naturalistic manner.”

• Also stated in: § Rationale for Choice of Innovative Factorial Design · hedged

  “However, because this approach is not an RCT, it is difficult to discern the individual effects of putative causal variables on mediators or outcome variables (Shadish et al. 2002).”

• Also stated in: § Rationale for Choice of Innovative Factorial Design · plain

  “Thus, the observational approach has the disadvantage of enabling only tentative causal inference.”

• cites Shadish et al. 2002Not checked yet

  Not checked in the free run.

May combine two claims.

Factorial designs may save resources, whereas separate experiments and single-factor designs tend to demand more resources and be less efficient.

“Third, a factorial experiment can be economical. Collins et al. (2009b) showed that although many behavioral scientists turn to individual experiments and single-factor (e.g., comparative treatment) approaches in order to conserve resources, in reality these designs are often much more resource-intensive and much less efficient than factorial designs.”

• cites Collins et al. 2009bNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

In three Australian primary teachers, feedback on performance led them to apply more of the methods taught in a workshop.

“Leach and Conto (1999) found that among a sample of three primary school teachers in Australia, performance feedback increased teachers’ use of techniques learned in a training workshop.”

• cites Leach and Conto (1999) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Programs that are well developed and theory-based ought to contain core lessons and techniques essential to effectiveness; these should inform quality measurement and assessment of adaptation effects.

“Well developed, theoretically based programs should possess a core set of lessons and techniques (mechanisms of change) that are central to the effectiveness of the intervention; these should be used to measure quality as well as the impact of program adaptations (Domitrovich et al. 2008).”

• cites Domitrovich et al. 2008 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

A coding system and manual were created for the four target lessons, with codes for dosage and adherence.

“We have developed a coding system and manual similar to the work of others (Forgatch et al. 2005; Giles et al. 2008; Knutson et al. 2003). Specific codes have been developed for each of the four lessons with regard to dosage and adherence.”

• cites Forgatch et al. 2005 sourceNot checked yet

  Not checked in the free run.

• cites Giles et al. 2008 sourceNot checked yet

  Not checked in the free run.

• cites Knutson et al. 2003Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Organizational changes and challenges should be expected.

“Another important lesson that we have learned is to expect organizational changes and challenges; this was also noted by August et al.”

• cites August et al. 2004 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Cultural sensitivity matters greatly in developing, delivering, and evaluating programs, especially in African Black settings.

“In particular the need to be culturally sensitive in program development, implementation and evaluation is critical, particularly as we move into African Black contexts (as opposed to mixed race or Coloured contexts where we have been).”

• cites August et al. 2004 sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Knowing how to measure intervention aspects depends centrally on explicit theories of program content and process.

“Snyder et al. (2006) contended that explicit program content and process theories are central to determining how to measure aspects of an intervention.”

• cites Snyder et al. 2006 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Teacher implementation fidelity is measured by three sources: logs completed daily by teachers, a yearly teacher survey, and observations; principal and student data also contribute.

“Three sources of data are used to assess teacher implementation fidelity: daily self-report teacher logs, an annual teacher survey, and observations. Data from principals and students also help assess implementation fidelity.”

Probably not a substantial claim: may not pass S2 (checkable).

Economic considerations need close attention.

“August et al. caution for the need to pay close attention to economic considerations.”

• cites August et al. 2004 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Multiple data sources are used to triangulate observations.

“In addition to the observational protocol, other measures provide a secondary source of data with which to triangulate the observations.”

May combine two claims.

A random sample of lessons is recorded besides the four target lessons; this acclimates teachers to the assistant and equipment, so they do not prepare specially for target lessons.

“Therefore, we are video recording a randomly selected set of lessons in addition to the four target lessons. A research assistant is assigned to specific schools and is responsible for video and audio recording the teachers teaching the lessons. This process also sensitizes the teachers to the research assistant and video and audio equipment, thus preventing special preparation for the target”