Sustained Implementation of Evidence‐based Programs in Disadvantaged Communities: A Conceptual Framework of Supporting Factors

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Assessment processes included interviews (14; 50%), self-report measures (13; 46%), observation or program adherence assessment (8; 29%), and record review (6; 21%).

“Assessment processes included interviews (14; 50%), self-report measures (13; 46%), observation or program adherence assessment (8; 29%), and record review (6; 21%).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Twenty-one countries were represented, including high-income countries USA, Australia, and Israel, and low- and middle-income countries in Africa, Latin America, Asia, and Papua New Guinea.

“Twenty-one countries were represented including highincome countries: USA, Australia, Israel; and LMICs: Africa (12), Latin America (3), Asia (2), and Papua New Guinea.”

May combine two claims.

For measuring sustainment outcomes, studies used qualitative (12; 43%), quantitative (10; 21% longitudinal, 14% cross-sectional), and mixed methods (6; 21%).

“To measure sustainment outcomes, the studies employed qualitative (12; 43%), quantitative (10; 21% longitudinal, 14% cross-sectional), and mixed methods (6; 21%).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Among articles defining sustainment, the most common definition was continued or discontinued practice, project, or activity (12; 92%), with some also including continued participant benefits, continued capacity, or program anchored in the organization system.

“Of these, the most common definition was “continued or discontinued practice/project/activity” (12; 92%), with some of these also including continued benefits for participants, continued capacity or program anchored in the organization system.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The final sample comprised 28 empirical studies spanning various locations, program foci, and designs.

“The final sample of articles included 28 empirical studies representing a range of locations, program foci, and designs (see supplementary materials for details).”

• Also stated in: § Review Method · plain

  “Following the full text screening, 28 articles were retained for data extraction and analysis (see supplementary material for article details).”

May combine two claims.

Most studies were naturalistic (20; 71%), while some involved experimental manipulation (2; 7%) or quasi-experimental evaluation (6; 21%).

“Most were naturalistic (20; 71%); however, some involved experimental manipulation (2; 7%) or quasi-experimental evaluation (6; 21%) of implementation approaches to analyze sustainment.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The most common term for continued practice, project, or activity was sustainability (7; 54%); other terms included sustainment, institutionalization or institutionalized sustainability, de-adoption, and ex-novation.

“The most common term used to describe continued practice, project, or activity was “sustainability” (7; 54%).Other terms were “sustainment”, “institutionalization” or “institutionalized sustainability”, “de-adoption” and “ex-novation”.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The reviewed studies covered family and child behavioral health (9; 32%), medical or healthcare (8; 29%), general primary health care (4; 14%), health promotion (3; 11%), and mental health (4; 14%).

“Nine articles (32%) examined family and child behavioral health programs such as prevention programs for family violence, child conduct problems, and emotional problems.Eight (29%) covered medical or healthcare programs for disease prevention or intervention, such as river blindness, diabetes and iodine-deficiency, and four (14%) addressed general primary health care.Three studies (11%) reported”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The current review's methodology and sustainment factor categories align with previous recommendations.

“The research methodology and sustainment factor categories found in the current literature review align with previous recommendations (e.g., Shediac-Rizkallah & Bone, 1998;Stirman et al., 2012), however, this review focuses on program sustainment studies specific to disadvantaged communities.”

• cites Shediac-Rizkallah & Bone, 1998Not checked yet

  Not checked in the free run.

• cites Stirman et al., 2012Not checked yet

  Not checked in the free run.

May combine two claims.

No factor evaluations were experimental, but a few studies used observational or triangulated approaches to strengthen analytic validity.

“None of the factor evaluations were experimental, but a few studies used an observational or triangulated approach to strengthen the validity of the analysis.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

For evaluating factors influencing sustainment, designs were qualitative (15; 54%), quantitative (7; 25%), and mixed methods (6; 21%).

“With regard to evaluating the factors that influence sustainment, the spread of designs was similar: qualitative (15; 54%), quantitative (7; 25%), and mixed-methods (6; 21%).”

May combine two claims.

The reviewed studies show that providers are more likely to sustain a new program if they have open attitudes, see it as a better way of working, observe benefits, find it compatible with their values and needs, find it adaptable to the local context, find it not too complex, and receive adequate training and practice to maintain knowledge and skills.

“In summary, in relation to program characteristics, the studies reviewed show that providers are more likely to sustain a new program if they have open attitudes toward the program, it offers them a better way of working, has observable benefits, is compatible with their values and needs, is adaptable for the local context, is not too complex, and offers adequate training and practice to maintain”

May combine two claims.

Maintenance of knowledge and skills gained at training has not been extensively evaluated, but recent work shows perceived competency in program skills can affect provider retention and sustained implementation.

“Maintenance of the knowledge and skills gained by providers at training has not been evaluated extensively, but recent work shows that perceived competency in program skills can impact provider retention (Boucar et al., 2011) and sustained implementation (Boucar et al., 2011;Cooper et al., 2015).”

• cites Boucar et al., 2011Not checked yet

  Not checked in the free run.

• cites Cooper et al., 2015Not checked yet

  Not checked in the free run.

May combine two claims.

For individual-level outcomes, five studies (16%) used one large community respondent group, two (7%) reported on small numbers of providers, and one (4%) aggregated scores at the team level.

“Reporting on outcomes at the individual level, five studies (16%) used one large group of respondents within the community, two (7%) reported on small numbers of providers, and one (4%) aggregated scores at the team level.”

May combine two claims.

Programs are more likely to be sustained if they fill a critical gap in the health system, become part of everyday practice, and are mandated service delivery.

“Studies in this review indicate that programs are more likely to be sustained if they fill a critical gap in the health system (Edwards & Roelofs, 2006;MacLean, Berends, & Mugavin, 2012), become a part of everyday practice (MacLean et al., 2012) and mandated service delivery (Shapiro et al., 2014).”

• cites Edwards & Roelofs, 2006Not checked yet

  Not checked in the free run.

• cites MacLean, Berends, & Mugavin, 2012Not checked yet

  Not checked in the free run.

• cites Shapiro et al., 2014Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Fourteen articles addressed program characteristics.

“Program Characteristics (n = 14)”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In analyzing sustainment factors, 12 articles (43%) drew on the same sustainment data, 16 (57%) used alternative measures or respondents, and two (7%) described factors using commentary on study data or reviewed studies.

“In terms of analyzing sustainment factors, of all 28 articles, 12 (43%) drew on the same sustainment data, 16 (57%) had alternative measures or respondents (e.g., health workers were questioned instead of villagers, or health leaders instead of patients), and two (7%) described factors that influence sustainment using commentary on the study data or reviewed studies.”

May combine two claims.

Of 20 articles reporting changes in implementation levels after initial implementation or funding ceased, 18 (90%) reported lower implementation levels and two (10%) reported program spread to other locations without details on extent.

“Of the 20 articles that reported on changes in implementation levels after initial implementation efforts or funding ceased, 18 (90%) reported lower levels of implementation and two (10%) reported program spread to other locations without providing details on extent of spread.”

May combine two claims.

Providers' decisions to implement often involve a cost-benefit analysis, and programs that are appealing, easy to implement, and visibly effective are sustained more often.

“A provider's decision to implement is often made through a cost-benefit analysis, with programs that are appealing (Spoth, Guyll, Redmond, Greenberg & Feinberg, 2011), easy to implement (Ahluwalia, Robinson, Vallely, Gieseker, & Kabakama, 2010;Amazigo et al., 2007), and visibly effective (Amazigo et al., 2007;Shapiro, Prinz, & Sanders, 2014) being sustained more often (Ahluwalia et al., 2010;”

• cites Spoth, Guyll, Redmond, Greenberg & Feinberg, 2011Not checked yet

  Not checked in the free run.

• cites Ahluwalia, Robinson, Vallely, Gieseker, & Kabakama, 2010Not checked yet

  Not checked in the free run.

• cites Amazigo et al., 2007Not checked yet

  Not checked in the free run.

May combine two claims.

A review of mental health evidence-based programs in the U.S. found that 88% of sustaining sites had adapted the program to meet local needs, primarily for agencies with limited resources related to lack of financial and staff support.

“A review of mental health EBPs in the U.S. found 88% of sustaining sites had adapted the EBP to meet local needs, primarily adaptations for agencies with limited resources related to lack of financial and staff support (e.g., discontinuation of fidelity assessment, reduced training, limiting provider time commitment, decreased supervision hours; Swain, Whitley, McHugo, & Drake, 2010).”

• cites Swain, Whitley, McHugo, & Drake, 2010Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Three studies (15%) divided the sample between two groups to compare sustainment with a larger number of respondents at two sites.

“Three studies (15%) divided the sample between two groups to compare sustainment for a larger number of respondents at two sites (e.g., providers in year 1 and year 2; north commune and central commune).”

May combine two claims.

Professional training systems that use active skills training methods are well accepted and are related to increased practitioner self-efficacy and skills in conducting parent support programs.

“Professional training systems that incorporate active skills training methods are well accepted and are related to increased practitioner self-efficacy and skills in conducting parent support programs (Turner, Nicholson & Sanders, 2011).”

• cites Turner, Nicholson & Sanders, 2011Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Three studies (15%) gathered data from two different sources using a large number of respondents per site.

“Three studies (15%) gathered data from two different sources (e.g., providers and community members) using a large number of respondents per site (e.g., 8-10 per village in 29 villages).”

May combine two claims.

Time and implementation burden from balancing a new program with daily work is a barrier to continuation and can lead to discontinuation.

“Time and implementation burden experienced by providers when balancing the commitment to deliver a new program and day-to-day work is a barrier to program continuation (Shapiro et al., 2014) and a factor that can lead to program discontinuation (Gaven & Schorer, 2013;Massatti, Sweeney, Panzano, & Roth, 2008).”

• cites Shapiro et al., 2014Not checked yet

  Not checked in the free run.

• cites Gaven & Schorer, 2013Not checked yet

  Not checked in the free run.

• cites Massatti, Sweeney, Panzano, & Roth, 2008Not checked yet

  Not checked in the free run.

May combine two claims.

Prior participation in a program at a smaller-scale demonstration site and greater knowledge of the program's logical model are both associated with increased sustainment.

“Having already participated in program at a smaller scale demonstration site (Boucar et al., 2011), and greater knowledge of the program's logical model (Cooper, Bumbarger, & Moore, 2015), have both been associated with increased sustainment.”

• cites Boucar et al., 2011Not checked yet

  Not checked in the free run.

• cites Cooper, Bumbarger, & Moore, 2015Not checked yet

  Not checked in the free run.

May combine two claims.

Ten studies (36%) used measures that had been pilot tested, and only four of those measures were validated before use.

“Ten studies (36%) used measures that were pilot tested and only four of these measures were validated before use in the study.”

May combine two claims.

For the remaining studies, the validity and reliability of assessment measures were unclear, and most failed to mention psychometric properties.

“For the remainder, the validity and reliability of assessment measures was unclear, with the large majority failing to mention psychometric properties at all.”

May combine two claims.

Program sustainability is threatened when a program is not viewed as a regular part of service delivery or when it is difficult to balance program and work commitments.

“However, program sustainability is threatened when a program is not viewed as regular part of service delivery (Massatti et al., 2008) or if it is difficult to balance program and work commitments (Gaven & Schorer, 2013).”

• cites Massatti et al., 2008Not checked yet

  Not checked in the free run.

• cites Gaven & Schorer, 2013Not checked yet

  Not checked in the free run.

May combine two claims.

Many different sustainment outcomes were reported, and most studies reported more than one outcome.

“Many different sustainment outcomes were reported and most studies reported more than one outcome.”

May combine two claims.

For evidence-based programs, providing program supervision and support improves staff retention, reduces burnout, and sustains adherence to program protocol.

“For EBPs, staff retention improves, burn out reduces and adherence to program protocol is sustained when program supervision and support is provided (Novins et al., 2013).”

• cites Novins et al., 2013Not checked yet

  Not checked in the free run.

May combine two claims.

Involving all staff and community members or community systems in new programs is important for integration and is an appropriate indicator for sustainment.

“Involving all staff and community members or community systems (i.e., through community assembly meetings; reports) in new programs can be important for integration and are also appropriate indicators for sustainment (Ahluwalia et al., 2010;Bustamante et al., 2012;Savaya & Spiro, 2012).”

• cites Ahluwalia et al., 2010Not checked yet

  Not checked in the free run.

• cites Bustamante et al., 2012Not checked yet

  Not checked in the free run.

• cites Savaya & Spiro, 2012Not checked yet

  Not checked in the free run.

May combine two claims.

Lack of supervision can lead to significant increases in staff attrition and decreases in program activity and effectiveness.

“Lack of supervision can lead to significant increases in staff attrition (Emukah et al., 2008) and decrease in program activity and effectiveness (G€ urtler et al., 2007).”

• cites Emukah et al., 2008Not checked yet

  Not checked in the free run.

• cites G€ urtler et al., 2007Not checked yet

  Not checked in the free run.

May combine two claims.

Pragmatic support such as guidance through collaboration, provision of space, time for training, financial support, and a vocal mandate for continuation is related to successful sustainment and is often missing where sustainment fails.

“Pragmatic support (e.g., guidance through the collaboration process, provision of space for the practice, time for training, financial support, and a vocal mandate for practice continuation) is related to successful sustainment (Kay et al., 2010;Loman, Rodriguez, & Horner, 2010), and these supports are often missing in sites where sustainment fails (Swain et al., 2010).”

• cites Kay et al., 2010Not checked yet

  Not checked in the free run.

• cites Loman, Rodriguez, & Horner, 2010Not checked yet

  Not checked in the free run.

• cites Swain et al., 2010Not checked yet

  Not checked in the free run.

May combine two claims.

Support through supervision is important for maintaining program benefits, activity, and fidelity, with refresher training and proper supervision needed to maintain program activity.

“Many articles highlighted the importance of support in program delivery through supervision to maintain program benefits, activity, and fidelity, with a need for refresher training and proper supervision to maintain program activity (Ahluwalia et al., 2010;Boucar et al., 2011;Emukah et al., 2008;G€ urtler, Kitron, Cecere, Segura, & Cohen, 2007)”

• cites Ahluwalia et al., 2010Not checked yet

  Not checked in the free run.

• cites Boucar et al., 2011Not checked yet

  Not checked in the free run.

• cites Emukah et al., 2008Not checked yet

  Not checked in the free run.

May combine two claims.

Implementing a new program with practitioner feedback and easy integration with existing programs creates a smooth implementation process that can lead to program sustainability.

“If a new program is implemented with practitioner feedback (August, Blooquist, Lee, Realmuto, & Hektner, 2006;Boucar et al., 2011) and easily integrated with existing programs in the organization or community (Ahluwalia et al., 2010), a smooth implementation process is created that can lead to program sustainability (Boucar et al., 2011)”

• cites August et al., 2006Not checked yet

  Not checked in the free run.

• cites Boucar et al., 2011Not checked yet

  Not checked in the free run.

• cites Ahluwalia et al., 2010Not checked yet

  Not checked in the free run.

May combine two claims.

Sustainment research supports this theory: programs in Indigenous communities and low- and middle-income countries operating in supportive workplaces are more likely to be sustained, while programs with weak support functions are less likely to maintain activity.

“Sustainment research has supported this theory with studies showing programs in Indigenous communities (Aitaoto et al., 2009) and LMICs (Ahluwalia et al., 2009) operating in a supportive workplace are more likely to be sustained, and programs with weak support functions (e.g., information and communication, capacity building, rewarding quality program delivery) at any level (from management to”

• cites Aitaoto et al., 2009Not checked yet

  Not checked in the free run.

• cites Ahluwalia et al., 2009Not checked yet

  Not checked in the free run.

• cites Swain et al., 2010Not checked yet

  Not checked in the free run.

May combine two claims.

Staff awareness of workplace values, mission, and goals, and program congruence and integration with those values, are important factors for sustainment.

“staff awareness of workplace values, mission, and goals and program congruence and integration with those values appear to be important factors for sustainment (Ahluwalia et al., 2010;Tibbits et al., 2010).”

• cites Ahluwalia et al., 2010Not checked yet

  Not checked in the free run.

• cites Tibbits et al., 2010Not checked yet

  Not checked in the free run.

May combine two claims.

Programs can create barriers to sustainment and fail when they envision large-scale implementation simply as a training cascade, do not offer ongoing training and support when moving from training to program delivery, and do not allow for appropriate learning and pilot phases.

“Programs can create barriers to sustainment and fail when they envision large-scale implementation simply as a training cascade (i.e., do not offer ongoing training and support when they move from the training room to program delivery; Edwards & Roelofs, 2006;Gaven & Schorer, 2013;and do not allow for appropriate learning and pilot phases).”

• cites Edwards & Roelofs, 2006Not checked yet

  Not checked in the free run.

• cites Gaven & Schorer, 2013Not checked yet

  Not checked in the free run.

May combine two claims.

Supervision is regarded as integral to sustaining a practice, but supervision time often decreases after the initial implementation period due to time constraints.

“Supervision is regarded as integral to sustaining a practice; however, supervision time often decreases after the initial implementation period due to time constraints.”

May combine two claims.

Ongoing supervision that allows time to practice program delivery skills, review clinical cases, and receive feedback can prevent burnout and improve program delivery confidence and fidelity.

“Ongoing supervision that allows time to practice program delivery skills, review clinical cases, and receive feedback can prevent burnout and improve program delivery confidence and fidelity (Beidas & Kendall, 2010;Pallas et al., 2013;Sanders & Turner, 2005).”

• cites Beidas & Kendall, 2010Not checked yet

  Not checked in the free run.

• cites Pallas et al., 2013Not checked yet

  Not checked in the free run.

• cites Sanders & Turner, 2005Not checked yet

  Not checked in the free run.

May combine two claims.

Lack of engagement effort and rapid geographic spread of program activity with little time for engagement can create major hurdles.

“Good team member engagement can support continuous implementation, however, lack of engagement effort and rapid geographic spread of program activity with little time for engagement, can create major hurdles (Hodgins et al., 2013).”

• cites Hodgins et al., 2013Not checked yet

  Not checked in the free run.

May combine two claims.

Missing opportunities to engage individuals and lack of community education can have a large negative effect on sustained implementation.

“Missing opportunities to engage individuals (August et al., 2006;Boucar et al., 2011;Cooper et al., 2015;Gaven & Schorer, 2013) and lack of community education (Sebotsa, Dannhauser, Jooste, & Joubert, 2007) can have a large negative effect on sustained implementation.”

• cites August et al., 2006Not checked yet

  Not checked in the free run.

• cites Boucar et al., 2011Not checked yet

  Not checked in the free run.

• cites Cooper et al., 2015Not checked yet

  Not checked in the free run.

May combine two claims.

Providing technical assistance is intended to build provider confidence in program delivery, enhance capacity, promote local problem solving efforts, and motivate providers with regard to program delivery.

“Providing technical assistance is intended to build provider confidence in program delivery, enhance capacity (Bustamante et al., 2012;Loman et al., 2010;Savaya & Spiro, 2012;Wright, 2009), promote local problem solving efforts (McDermott et al., 2004), and motivate providers with regard to program delivery (Boucar et al., 2011).”

• cites Bustamante et al., 2012Not checked yet

  Not checked in the free run.

• cites Loman et al., 2010Not checked yet

  Not checked in the free run.

• cites Savaya & Spiro, 2012Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

The climate is gauged by staff awareness of agency mission and goals, workplace trust, unity and stability, effective leadership, integration and support for adopted programs, and provision of supervision and peer support.

“The climate is gaged by staff awareness of agency mission and goals, workplace trust, unity and stability, effective leadership, integration and support for adopted programs, and the provision of supervision and peer support (Brown, Feinberg & Greenberg, 2011;Simpson & Flynn, 2007).”

• cites Brown, Feinberg & Greenberg, 2011Not checked yet

  Not checked in the free run.

• cites Simpson & Flynn, 2007Not checked yet

  Not checked in the free run.

May combine two claims.

Peer-assisted supervision is helpful for managing workload, sharing costs, engaging attendees, administering and scoring outcome assessment instruments, sharing learnings, and increasing confidence.

“Peer-assisted supervision is helpful for managing workload, sharing costs, engaging attendees, administering and scoring outcome assessment instruments, sharing learnings and increasing confidence (Gaven & Schorer, 2013).”

• cites Gaven & Schorer, 2013Not checked yet

  Not checked in the free run.

May combine two claims.

Lack of resources such as transportation and essential materials and uneven support for program training can create issues for sustainment.

“Lack of resources such as transportation and essential materials (Amazigo et al., 2007; e.g., medical treatment supplies) and uneven support for the program training can create issues for sustainment (Amazigo et al., 2007;August et al., 2006;Edwards and Roelofs, 2006;Emukah et al., 2008).”

• cites Amazigo et al., 2007Not checked yet

  Not checked in the free run.

• cites August et al., 2006Not checked yet

  Not checked in the free run.

• cites Edwards and Roelofs, 2006Not checked yet

  Not checked in the free run.

May combine two claims.

Planning from the outset is similarly important for community ownership; creating a handover plan from the beginning can emphasize a real plan for community ownership.

“This planning from the outset is similarly important for community ownership; creating a handover plan from the beginning can emphasize a real plan for community ownership (Edwards & Roelofs, 2006).”

• cites Edwards & Roelofs, 2006Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The fundamental objective of sustainment planning is to design a course of action early to promote effective implementation by collectively building local expertise for using the program.

“The fundamental objective of sustainment planning is to design a course of action early to promote effective implementation by collectively building local expertise for using the program (Gruen et al., 2008).”

• cites Gruen et al., 2008Not checked yet

  Not checked in the free run.

May combine two claims.

Partnership models are beneficial for implementation and sustainment because they build upon a community or organization's pre-existing capacity and provide a support system to help them perform at their optimal level.

“Partnership models are beneficial for implementation and sustainment because they build upon a community or organization's pre-existing capacity (Kay et al., 2010) and provide a support system to help them perform at their optimal level (Spoth et al., 2004).”

• cites Kay et al., 2010Not checked yet

  Not checked in the free run.

• cites Spoth et al., 2004Not checked yet

  Not checked in the free run.

May combine two claims.

Continued monitoring and evaluation can allow for continuous performance improvements and was noted as a reinforcer by program sustainers who perceived this approach as a positive opportunity for consultation.

“Continued monitoring and evaluation can allow for continuous performance improvements and was noted as a reinforcer by program sustainers who perceived this approach as a positive opportunity for consultation (Swain et al., 2010).”

• cites Swain et al., 2010Not checked yet

  Not checked in the free run.

May combine two claims.

When a capacity building framework is used, it can enhance the chance of program sustainment success, especially if partnership members are all trained in the program.

“It has been demonstrated that when this type of capacity building framework is used, it can enhance the chance of program sustainment success, especially if the partnership members are all trained in the program (Brown, Feinberg, Shapiro & Greenberg, 2015).”

• cites Brown, Feinberg, Shapiro & Greenberg, 2015Not checked yet

  Not checked in the free run.

May combine two claims.

Developing strong, transparent partnerships across organization, community, district, state, and national levels; involvement of universities and commitment of the Ministry of Health; and greater coalition functioning are all supports for sustainment.

“Developing strong, transparent partnerships across the board, that is at the organization, community (Gaven & Schorer, 2013), district (Loman et al., 2010), state (Swain et al., 2010), and national level (Eliason, 1999;Sebotsa et al., 2007); involvement of universities and commitment of the Ministry of Health (Quality Assurance Project and UNICEF/Nicaragua, 2006); and greater coalition functioning”

• cites Gaven & Schorer, 2013Not checked yet

  Not checked in the free run.

• cites Loman et al., 2010Not checked yet

  Not checked in the free run.

• cites Swain et al., 2010Not checked yet

  Not checked in the free run.

May combine two claims.

In our review, abandonment of team meetings and indicators that external partner agents did not support the agency's effort to implement the program were barriers to sustainment.

“In our review of the literature, abandonment of team meetings (Boucar et al., 2011) and indicators that external partner agents did not support the agency's effort to implement the program were barriers to sustainment (Massatti et al., 2008).”

• cites Boucar et al., 2011Not checked yet

  Not checked in the free run.

• cites Massatti et al., 2008Not checked yet

  Not checked in the free run.

May combine two claims.

Both program and community partnership sustainment involves financial and operational plans that will establish independently sustainable programs.

“Both program and community partnership sustainment involves financial and operational plans that will establish independently sustainable programs (Feinberg, Bontempo & Greenberg, 2008).”

• cites Feinberg, Bontempo & Greenberg, 2008Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

It is suggested that large-scale community health programs should be planned with at least a 10-year timeframe.

“It is suggested that large-scale community health programs should be planned with at least a 10-year timeframe (Hodgins et al., 2013).”

• cites Hodgins et al., 2013Not checked yet

  Not checked in the free run.

May combine two claims.

It is important that partnerships are high functioning and involve stakeholders who contribute to and gain from the partnership to reduce the deterioration of the capacity built.

“However, it is important that partnerships are high functioning (Cooper et al., 2015), and involve stakeholders who contribute to and gain from the partnership to reduce the deterioration of the capacity built (Boucar et al., 2011).”

• cites Cooper et al., 2015Not checked yet

  Not checked in the free run.

• cites Boucar et al., 2011Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

These factor categories align with recommendations from Shediac-Rizkallah and Bone and Stirman et al. for program sustainability and evaluation.

“These factor categories align with recommendations from Shediac-Rizkallah and Bone (1998) and Stirman et al. (2012) for program sustainability and evaluation”

• cites Shediac-Rizkallah and Bone, 1998Not checked yet

  Not checked in the free run.

• cites Stirman et al., 2012Not checked yet

  Not checked in the free run.

May combine two claims.

Financial resources and support for program sustainment are important.

“financial resources (Boucar et al., 2011;Bustamante et al., 2012), and support for program sustainment (Massatti et al., 2008;Swain et al., 2010).”

• cites Boucar et al., 2011Not checked yet

  Not checked in the free run.

• cites Bustamante et al., 2012Not checked yet

  Not checked in the free run.

• cites Massatti et al., 2008Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

One measure developed in parallel with this review is the Sustained Implementation Support Scale.

“One such measure, developed in parallel with this review, is the Sustained Implementation Support Scale (Hodge, Turner, Sanders & Filus, 2016).”

• cites Hodge, Turner, Sanders & Filus, 2016Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

Before this review, such a framework had not been established to guide work in disadvantaged communities.

“Before this review, such a framework had not been established to guide work in disadvantaged communities.”

May combine two claims.

Our review found a greater focus on program burden, program familiarity, and perceived competence in program skills, workplace support for the program, staff mobility and turnover, supervision and peer support, and ongoing technical assistance.

“Our review found a greater focus on program burden, program familiarity, and perceived competence in program skills, workplace support for the program, staff mobility and turnover, supervision and peer support, and ongoing technical assistance.”

May combine two claims.

Only a few studies included measures that were pilot tested and validated, with most failing to mention psychometric properties completely.

“given that only a few studies included measures that were pilot tested and validated, with most failing to mention psychometric properties completely.”

May combine two claims.

Ensuring long-term financial support and political commitment is essential to sustain programs, but many factors impact this with regard to financial arrangements and stability of the socioeconomic and political environment.

“Ensuring long-term financial support and political commitment is essential to sustain programs, but many factors impact this with regard to financial arrangements and stability of the socioeconomic and political environment (Gruen et al., 2008;Hodgins et al., 2013;Novins et al., 2013).”

• cites Gruen et al., 2008Not checked yet

  Not checked in the free run.

• cites Hodgins et al., 2013Not checked yet

  Not checked in the free run.

• cites Novins et al., 2013Not checked yet

  Not checked in the free run.

May combine two claims.

A few studies looked at multiple levels of implementation, but did not have a large enough sample size within each level or enough respondents per site to evaluate the data appropriately.

“In an attempt to thoroughly capture the complex system that programs operate within in disadvantaged settings, a few studies looked at multiple levels of implementation, but did not have a large enough sample size within each level or enough respondents per site to evaluate the data appropriately.”

May combine two claims.

Diversity of funding sources, use of political fundraising strategies and number of strategies employed, and funding body involvement all distinguish between continued and non-continued projects.

“Diversity of funding sources, use of political fundraising strategies (and number of strategies employed), and funding body involvement all distinguish between continued and non-continued projects (Savaya & Spiro, 2012).”

• cites Savaya & Spiro, 2012Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

A few studies used a multimethod approach to triangulate qualitative data with quantitative data.

“To strengthen the analysis, a few studies used a multimethod approach to triangulate qualitative data with quantitative data (Quality Assurance Project and UNICEF /Nicaragua, 2006;Savaya & Spiro, 2012;Wright, 2009).”

• cites Quality Assurance Project and UNICEF /Nicaragua, 2006Not checked yet

  Not checked in the free run.

• cites Savaya & Spiro, 2012Not checked yet

  Not checked in the free run.

• cites Wright, 2009Not checked yet

  Not checked in the free run.

May combine two claims.

National laws that support a program can reinforce sustainment as long as there is a political commitment to carry out the new legislation or regulations.

“however, national laws that support a program can reinforce sustainment (Quality Assurance Project and UNICEF/Nicaragua, 2006) as long as there is a political commitment to carry out the new legislation or regulations (Sebotsa et al., 2007).”

• cites Quality Assurance Project and UNICEF/Nicaragua, 2006Not checked yet

  Not checked in the free run.

• cites Sebotsa et al., 2007Not checked yet

  Not checked in the free run.

May combine two claims.

The empirical studies examined in this review suggest that, of the program factors that influence sustainability in disadvantaged settings, provider perceptions of the program and program fit were most frequently cited and have the strongest empirical evidence.

“The empirical studies examined in this review suggest that, of the program factors that influence sustainability in disadvantaged settings, provider perceptions of the program (benefits, burden) (n = 8) and program fit (n = 5) were most frequently cited and have the strongest empirical evidence.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Staff mobility and turnover was highlighted as a workplace factor that has not been previously mentioned in other commonly referenced literature reviews.

“Staff mobility and turnover (n = 6) was highlighted as a workplace factor that has not been previously mentioned in other commonly referenced literature reviews.”

May combine two claims.

Engagement and technical assistance with ongoing support from program disseminators were the most frequently cited factors, each in 17 studies.

“The most frequently cited factors were engagement (n = 17) and technical assistance and ongoing support from program disseminators (n = 17).”

May combine two claims.

Program adaptation and program knowledge or familiarity are also reported influences.

“Program adaptation and program knowledge or familiarity are also reported influences.”

May combine two claims.

External support such as funding and policy imperatives, and collaborative partnerships, were commonly identified factors, each in 13 studies.

“Other commonly identified factors include external support, such as funding and policy imperatives (n = 13) and collaborative partnerships (n = 13).”

May combine two claims.

Of the workplace factors examined, the most frequently cited and empirically supported factors were organizational and leadership support, integration of the program into the workplace, and supervision and peer support.

“Of the workplace factors examined, the most frequently cited and empirically supported factors were organizational and leadership support (n = 13), integration of the program into the workplace (n = 8), and supervision and peer support (n = 8).”

Probably not a substantial claim: may not pass S4 (based).

Evidence-based programs can often take a linear approach to program implementation by providing one-off training for providers to develop the skills to put a program into practice.

“EBPs can often take a linear approach to program implementation by providing one-off training for providers to develop the skills to put a program into practice.”

Probably not a substantial claim: may not pass S4 (based).

Ongoing technical assistance had not been identified in previous literature reviews as a factor impacting sustainment.

“Ongoing technical assistance was not previously identified in other literature reviews as a factor that impacts sustainment”

Classified as probably not a substantial claim.

Proctor and colleagues call for efforts to advance implementation science and merge learnings into a comprehensive model with defined constructs, a measurement model, and an analytic model linking measured constructs.

“Proctor and colleagues illustrate the need for concerted efforts to advance implementation science and merge learnings to form a comprehensive model with clearly defined constructs, a measurement model for the key constructs and, an analytic model hypothesizing links between measured constructs (Proctor et al., 2009).”

• cites Proctor et al., 2009Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Different research disciplines have historically held different views of implementation and sustainment, leading to different approaches to studying program sustainability.

“Different research disciplines have historically had different perspectives of implementation and sustainment, leading to different approaches for investigating program sustainability (Greenhalgh, Robert, Macfarlane, Bate & Kyriakidou, 2004).”

• cites Greenhalgh, Robert, Macfarlane, Bate & Kyriakidou, 2004Not checked yet

  Not checked in the free run.

May combine two claims.

Low-resourced areas often face additional issues including lack of trained professionals, lack of practitioner supervision, practitioner resistance to innovation, lack of access, trust and engagement in services, and high attrition.

“This is particularly relevant in lowresourced areas as there are often additional issues such as: lack of availability of trained professionals; lack of practitioner supervision; practitioner resistance to innovation; lack of access, trust and engagement in services; and high attrition rates (Turner & Sanders, 2007).”

• cites Turner & Sanders, 2007Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

Sustained implementation is measured by how long a project continues or how well a program becomes institutionalized within the organization system.

“This is measured by how long an individual project continues or how well a program becomes institutionalized within the organization system (Chaudoir, Dugan & Barr, 2013).”

• cites Chaudoir, Dugan & Barr, 2013Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Organization change, community coalition, and innovation research define sustained implementation as continued use of programs in usual practice.

“Organization change, community coalition, and innovation research define sustained implementation as the continued use of programs in usual practice (Chambers, Glasgow & Stange, 2013;Gomez, Greenberg & Feinberg, 2005).”

• cites Chambers, Glasgow & Stange, 2013Not checked yet

  Not checked in the free run.

• cites Gomez, Greenberg & Feinberg, 2005Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

These frameworks generally propose that providers are more likely to adopt, implement, and sustain a new program when essential provider, program, and setting elements exist.

“In general, these frameworks propose that providers are more likely to adopt, implement, and sustain a new program if a number of essential provider, program and setting elements (internal organization functioning and external variables) exist.”

May combine two claims.

Practical aspects relevant to program delivery in disadvantaged communities include community and provider needs and capacity, intergenerational trauma, provider complexity in moving from training to delivery, engagement barriers, workforce availability and nature, and community partnerships.

“When implementing a program with disadvantaged communities, the practical aspects of program delivery must be taken into consideration, including the needs and capacity of the community and provider, intergenerational trauma in the community, level of complexity for the provider when moving from training to delivery in their workplace, the barriers to provider and community engagement, the”

• cites Duong et al., 2015Not checked yet

  Not checked in the free run.

• cites Zhou et al., 2015Not checked yet

  Not checked in the free run.

May combine two claims.

Recent reviews of program sustainment factors have contributed to implementation science by reporting methodology and categorizing factors, but they are not specific to vulnerable communities.

“Recent reviews of factors that influence program sustainment (Novins, Green, Legha & Aarons, 2013;Shediac-Rizkallah & Bone, 1998;Stirman et al., 2012) contribute immensely to implementation science by reporting on research methodology and categorizing factors that impact sustainment, however, these studies are not specific to vulnerable communities.”

• cites Novins, Green, Legha & Aarons, 2013Not checked yet

  Not checked in the free run.

• cites Shediac-Rizkallah & Bone, 1998Not checked yet

  Not checked in the free run.

• cites Stirman et al., 2012Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

A child's chance of a healthy life can be determined before birth by their social status.

“Before a child is born, their chance at a healthy life can be determined based on their social status (National Institute of Child Health and Human Development Early Child Care Research Network, 2005).”

• cites National Institute of Child Health and Human Development Early Child Care Research Network, 2005Not checked yet

  Not checked in the free run.

May combine two claims.

Existing frameworks have limitations: they emphasize different areas, some were not developed for health program implementation, and they lack empirical support for sustainability.

“However, there are limitations as each emphasizes different areas relevant to implementation; some were not developed specifically for health program implementation, or although they highlight sustainment as a critical part of the implementation process, they lack empirical support in the area of sustainability.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Many conceptual models, frameworks, and theories have been developed in implementation science to guide successful evidence-based program implementation.

“A number of conceptual models, frameworks, and theories have been developed in the implementation science literature to guide successful EBP implementation (Damschroder et al., 2009;Meyers, Joseph & Wandersman, 2012).”

• cites Damschroder et al., 2009Not checked yet

  Not checked in the free run.

• cites Meyers, Joseph & Wandersman, 2012Not checked yet

  Not checked in the free run.

May combine two claims.

Very few studies reported non-respondents, and most had small sample sizes and retrospective data, raising questions about informants' memory accuracy and its effect on findings.

“Overall, very few studies reported on non-respondents and most studies had small sample sizes and collected data retrospectively raising questions about the accuracy of the informants' memories and how this may have affected the findings.”

May combine two claims.

There is a tension between allowing trained providers without research training to adapt programs, while rarely supervising EBP skill rehearsal or incentivizing fidelity, and maintaining fidelity to core EBP content to ensure effectiveness.

“However, the struggle is striking the balance between providing a program that can be adapted by trained providers that have no research training, rarely receive supervision involving EBP skill rehearsal nor are incentivized to deliver programs with fidelity (Onken, Carroll, Shoham, Cuthbert & Riddle, 2014), while maintaining fidelity to core EBP content to ensure program effectiveness (Mazzucche”

• cites Onken, Carroll, Shoham, Cuthbert & Riddle, 2014Not checked yet

  Not checked in the free run.

• cites Mazzucchelli & Sanders, 2010Not checked yet

  Not checked in the free run.

May combine two claims.

The workplace is important because it provides the base for program functioning; if that base is not solid, the program begins with an inadequate foundation.

“The workplace is important because it provides the base for program functioning.If the base is not solid, then the program will start out with an inadequate foundation (Steckler & Goodman, 1989).”

• cites Steckler & Goodman, 1989Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

High-quality and productive interpersonal relationships are proposed to promote trust and commitment in the workplace.

“It has been proposed that high-quality and productive interpersonal relationships promote trust and commitment in the workplace (Aarons, Ehrhart & Farahnak, 2014;Edmondson, 2003).”

• cites Aarons, Ehrhart & Farahnak, 2014Not checked yet

  Not checked in the free run.

• cites Edmondson, 2003Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

This suggests that implementation strategies should focus at both the workplace level and the provider level.

“This suggests that implementation strategies should be focused at both the workplace level and provider level (Beidas et al., 2015).”

• cites Beidas et al., 2015Not checked yet

  Not checked in the free run.

May combine two claims.

Practitioners' perception of their workplace's commitment to staff affects their work-related attitudes and behaviors, and organizations can influence employee attitudes and behaviors related to new programs by providing or withholding support.

“Practitioners' perception of their workplace's commitment to staff impacts their work-related attitudes and behaviors, and organizations can directly influence employee work attitudes and behaviors related to new programs or innovations by providing (or withholding) support (Eisenberger, Huntington, Hutchison & Sowa, 1986).”

• cites Eisenberger, Huntington, Hutchison & Sowa, 1986Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Three research assistants conducted data extraction with significant overlap to ensure consistency.

“Three research assistants conducted data extraction, with significant overlap to ensure consistency in information extracted.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The extracted data were checked for comprehensiveness by the first author, and additional detail on sustainment constructs was extracted as appropriate.

“The data extracted were checked for comprehensiveness by the first author and additional detail on sustainment constructs extracted as appropriate.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Hand searching and snowballing were used to check for articles missed by the database search.

“To check for articles missed by the database search, we used hand searching and snowballing strategies, which involved searching the reference lists of reviews and relevant articles from our initial search (Marquez, Silvia, Edward & Simon, 2014;Pallas, Minhas, P erez-Escamilla, Lauren & Curry, 2013;Stirman et al., 2012).”

• cites Marquez, Silvia, Edward & Simon, 2014Not checked yet

  Not checked in the free run.

• cites Pallas, Minhas, P erez-Escamilla, Lauren & Curry, 2013Not checked yet

  Not checked in the free run.

• cites Stirman et al., 2012Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

An agency may fail to sustain program funding because they are dissatisfied with the program or have inadequate workplace functioning.

“but it should also be taken into consideration that an agency may fail to sustain program funding because they are dissatisfied with the program or have inadequate workplace functioning.”

May combine two claims.

These factors primarily relate to support for program implementation, ease of implementation and assistance to build capacity, and speak to the importance of providing strong, tailored support when implementing programs in settings with complex systems and limited resources.

“These factors primarily relate to support for program implementation, ease of implementation and assistance to build capacity, and speak to the importance of providing strong, tailored support when implementing programs in settings with complex systems and limited resources.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Programs need some time to establish themselves in the community or workplace before being left to support and take care of themselves.

“echoing the message that programs need some time to establish themselves in the community, or workplace before being left to support and take care of themselves (Savaya & Spiro, 2012).”

• cites Savaya & Spiro, 2012Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

The reality of working and researching with disadvantaged communities is that rigorous controlled trials will not always be appropriate, as small sample size is very common, and carrying out rigorous data analytic procedures is not always possible.

“However, the reality of working and researching with disadvantaged communities is that rigorous controlled trials will not always be appropriate, as small sample size is very common, and carrying out rigorous data analytic procedures is not always possible.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Recent implementation literature highlights the importance of considering the multiple levels of general capacity to promote evidence-based practice.

“Recent implementation literature highlights the importance of considering the multiple levels of general capacity to promote evidence-based practice (Halgunseth et al., 2012).”

• cites Halgunseth et al., 2012Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Integrating a program with all parts of the organization and/or community systems can improve sustainability.

“Similarly, integrating a program with all parts of the organization (e.g., administrative staff to providers) and/or community systems (e.g., community assembly meetings) can improve sustainability.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

All levels of workplace support for a program may directly influence employee implementation behavior and improve staff retention.

“as all levels of workplace support for a program may directly influence employee implementation behavior and improve staff retention.”

Classified as probably not a substantial claim.

Such partnerships can help build capacity for providers, workplaces, and communities by supplying information before adoption, training before implementation, and technical assistance during use.

“This type of partnership can facilitate capacity building for providers, workplaces, and communities by providing information about an innovation before the organization decides to adopt, training in how to carry out an innovation before it implements, and technical assistance once the innovation is in use.”

Probably not a substantial claim: may not pass S2 (checkable).

Using an EBP sustained implementation support framework in disadvantaged communities, with a partnership approach and suitable workplace climate and program elements, can help organizations plan complex implementation and build professional capacity to deliver and sustain programs long term.

“Planning intervention implementation using an EBP sustained implementation support framework in disadvantaged communities, that involves a partnership approach with appropriate workplace climate and program elements, can allow organizations to carry out complex planning for implementation and build the capacity for professionals to implement interventions and sustain program delivery long term.”

• AssumptionRests on an assumption: Disadvantaged communities likely have distinctive features that must be considered, including community needs and capacity, available providers, intergenerational trauma, learning styles, language barriers, provider complexity in moving from training to practice, trust-related barriers to engagement, system structure and integrity, and community partnerships.

Probably not a substantial claim: may not pass S2 (checkable).