CoChat
Sign Up
Cited by 251Health Policy Implementation ScienceHealth Sciences Research and EducationPsychiatric care and mental health services

The Roles of System and Organizational Leadership in System-Wide Evidence-Based Intervention Sustainment: A Mixed-Method Study

Open access
Aarons, Gregory A., Green, Amy E., Trott, Elise, Willging, Cathleen E., Torres, Elisa M., Ehrhart, Mark G., Roesch, Scott C.·2016·Administration and Policy in Mental Health and Mental Health Services Research
DOI 10.1007/s10488-016-0751-4PMC PMC5494253
Open this paper in CoChat — free account

Abstract

If evidence-based interventions (EBIs) are not sustained, investments are wasted and public health impact is limited. Leadership has been suggested as a key determinant of implementation and sustainment; however, little empirical work has examined this factor. This mixed-methods study framed using the Exploration, Preparation, Implementation, Sustainment (EPIS) conceptual framework examines leadership in both the outer service system context and inner organizational context in eleven system-wide implementations of the same EBI across two U.S. states and 87 counties. Quantitative data at the outer context (i.e., system) and inner context (i.e., team) levels demonstrated that leadership predicted future sustainment and differentiated between sites with full, partial, or no sustainment. In the outer context positive sustainment leadership was characterized as establishing a project’s mission and vision, early and continued planning for sustainment, realistic project plans, and having alternative strategies for project survival. Inner context frontline transformational leadership predicted sustainment while passive-avoidant leadership predicted non-sustainment. Qualitative results found that sustainment was associated with outer context leadership characterized by engagement in ongoing supportive EBI championing, marketing to stakeholders; persevering in these activities; taking action to institutionalize the EBI with funding, contracting, and system improvement plans; and fostering ongoing collaboration between stakeholders at state and county, and community stakeholder levels. For frontline leadership the most important activities included championing the EBI and providing practical support for service providers. There was both convergence and expansion that identified unique contributions of the quantitative and qualitative methods. Greater attention to leadership in both the outer system and inner organizational contexts is warranted to enhance EBI implementation and sustainment.

AI snapshot

Written from the paper’s full text.

TL;DR

This mixed-methods study found that leadership in both the outer service system and inner organizational context predicted sustainment of SafeCare across 11 public-sector service systems in two U.S. states and 87 counties. Quantitative and qualitative results converged on the importance of system-level leadership competence and frontline transformational leadership, while passive-avoidant frontline leadership predicted non-sustainment.

Key findings

  • Outer-context leadership competence predicted future sustainment: β=.697, p<.001; model R²=.667; a one-unit increase was associated with 17.167 times greater odds of being in a higher sustainment category.
  • Inner-context transformational leadership predicted sustainment: β=.165, p<.05; R²=.461; odds ratio 1.563. Passive-avoidant leadership predicted non-sustainment: β=−.395, p<.001; R²=.608; odds ratio .609. Transactional leadership was marginally supported: β=−.143, p=.05; R²=.447.
  • Sites were classified as full sustainment (n=7), partial sustainment (n=1), and no sustainment (n=3). Qualitative themes for outer-context leadership included ongoing championing, institutionalizing SafeCare through funding/contracts/system improvement plans, and collaboration; inner-context themes included championing SafeCare and providing practical support to home visitors.
  • Mixed-methods integration showed convergence and expansion: both methods supported leadership’s importance, while qualitative data added detail on collaboration, day-to-day support, and active management by exception.

Methods

  • Mixed-methods study using the EPIS framework, examining 11 service systems implementing SafeCare across two states and 87 counties; systems included one statewide child-welfare system and 10 county-wide systems (public health n=1, child welfare n=8, mental health n=1).
  • Quantitative data: outer-context administrators (n=44) completed the Program Sustainability Index Leadership Competence scale; inner-context home visitors (n=162) in 39 teams rated supervisors using the Multifactor Leadership Questionnaire. Ordinal regressions nested by system or team predicted sustainment level (0=non, 1=partial, 2=full), controlling for time since implementation, system size, provider age, sex, and job tenure.
  • Qualitative data: semi-structured interviews and focus groups conducted 2012–2014 with administrators and providers, analyzed iteratively in NVivo using a priori and focused coding; mixed-methods integration assessed convergence and expansion.

Limitations

  • The paper states that leadership measures differed for outer and inner contexts, quantitative sample sizes were small for some groups, and interviews/focus groups covered broad issues rather than being tightly tied to quantitative measures.
  • Evident from the text: only 11 systems and 3 non-sustaining sites limit statistical power and generalizability; leadership ratings were self-reported by administrators or providers, and the design cannot fully establish causal direction.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s10488-016-0751-4

Cites

Statistics

Nothing to recompute.

Claims the paper makes

0 of 59 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    For frontline leadership, the most important activities were championing the EBI and giving practical support to service providers.

    “For frontline leadership the most important activities included championing the EBI and providing practical support for service providers.”
    Quote in the paperSubstantial claim
  • Findingown data

    The ongoing championing of SC at the system level was especially significant in contexts of turnover and system change.

    “The ongoing championing of SC at the system level was especially significant in contexts of turnover and system change.”
    Quote in the paperSubstantial claim
  • Findingown data

    In sustaining sites, home visitors described supervisors as gung ho and wonderful cheerleaders.

    “In sustaining sites, home visitors described their supervisors as “gung ho,” and “wonderful cheerleaders. I mean they’re so into it [SC].””
    Quote in the paperNot a substantial claim
  • Findingreasoning

    A one-unit increase in transactional leadership was associated with a .143 decrease in log odds of moving to the next sustainment level.

    “Therefore, it is expected that for every one unit increase in transactional leadership there is a .143 decrease in the log odds of moving from a given level of sustainment.”
    Quote in the paperSubstantial claim
  • Findingown data

    System-level leaders also worked to institutionalize SC by embedding the intervention within contracts and official plans.

    “System-level leaders also worked to institutionalize SC by embedding the intervention within contracts and official plans.”
    Quote in the paperSubstantial claim
  • May combine two claims. 49
  • Classified as probably not a substantial claim. 44

  • May combine two claims.

    Findingown data

    At the inner context, frontline transformational leadership forecast sustainment, while passive-avoidant leadership forecast non-sustainment.

    “Inner context frontline transformational leadership predicted sustainment while passive-avoidant leadership predicted non-sustainment.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In the quantitative analysis, leadership at both system and team levels forecast later sustainment and separated sites into full, partial, and no sustainment categories.

    “Quantitative data at the outer context (i.e., system) and inner context (i.e., team) levels demonstrated that leadership predicted future sustainment and differentiated between sites with full, partial, or no sustainment.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Positive outer-context leadership for sustainment involved setting a project's mission and vision, planning early and continuously for sustainment, making realistic project plans, and having alternative strategies for project survival.

    “In the outer context positive sustainment leadership was characterized as establishing a project’s mission and vision, early and continued planning for sustainment, realistic project plans, and having alternative strategies for project survival.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    The quantitative and qualitative methods showed both convergence and expansion, revealing unique contributions from each approach.

    “There was both convergence and expansion that identified unique contributions of the quantitative and qualitative methods.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Qualitative data identified importance of valuing EBIs, providing supportive and perseverant leadership, and institutionalizing EBI in system in multiple ways.

    “Qualitative data identified the importance of valuing EBIs, providing supportive and perseverant leadership, while also institutionalizing the EBI in the system in multiple ways.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Quantitative data suggested decision makers could provide funding, policies, and support for EBP; qualitative results expanded and described how leaders can not only create policies that provide funding, but also establish collaborations that support sustainment.

    “In regard to leadership findings across methods, quantitative data suggested that decision makers could provide funding, policies, and support for EBP, qualitative results expanded on this and described how leaders can not only create policies that provide funding, but also establish collaborations that support sustainment.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    System-level leaders in sustaining sites worked to strategically and proactively institutionalize the intervention locally, typically ensuring adequate funding.

    “In addition to functioning as SC champions into the sustainment phase, system-level leaders in the sustaining sites worked to strategically and proactively institutionalize the intervention locally. This typically took the form of making certain that adequate funding was available for SC.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For inner context, high transformational leadership was associated with sustainment and passive-avoidant leadership was associated with non-sustainment.

    “For the inner context, high transformational leadership was associated with sustainment and passive-avoidant leadership was associated with non-sustainment.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    These collaborative relationships reportedly engendered continuing commitment to SC and shared accountability, ensuring neither state/county administrators nor CBOs put a premature end to SC.

    “These relationships reportedly engendered continuing commitment to SC and a spirit of shared accountability, ensuring that neither state/county administrators nor CBOs put a premature end to SC.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    Full sustainment sites had certified SafeCare providers conducting sessions, ongoing coaching and fidelity monitoring, and SafeCare team meetings in accordance with model developer standards.

    “The “full sustainment” sites have certified SC providers who conduct SC sessions, there is ongoing coaching and fidelity monitoring, and SC team meetings in accordance with model developer standards.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Two primary themes were identified in frontline-level/inner context qualitative data: ongoing championing of EBIs and SC and practical support for home visitors.

    “Two primary themes were identified in the frontline-level/inner context qualitative data: ongoing championing of EBIs and SC and practical support for home visitors.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Many home visitors felt that social workers and other government staff lacked knowledge of SC.

    “Many home visitors felt that social workers and other government staff simply lacked knowledge of SC.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In contrast to quantitative finding that transactional leadership was not associated with sustainment, qualitative data identified that it was important for frontline leaders to attend to how EBI was being used, and intervene as needed when quality standards weren't met.

    “However, in contrast to the quantitative finding that transactional leadership was not associated with sustainment, qualitative data identified that it was important for frontline leaders to attend to how the EBI was being used, and intervene as needed when quality standards weren’t being met.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Such strategies included establishing formal funding, system improvement plans, and proactive planning.

    “Such strategies included establishing formal funding, system improvement plans, and proactive planning.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Transformational leadership significantly predicted sustainment; greater transformational leadership predicted sustainment.

    “The ordinal regression analyses showed that transformational leadership (β=.165, p <.05) significantly predicted sustainment, such that greater transformational leadership predicted sustainment.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Qualitative data expanded on these constructs finding that being knowledgeable about EBI, proactive in problem solving, and preserving through ups-and-downs of sustainment was important.

    “Qualitative data expanded on these constructs finding that being knowledgeable about the EBI, proactive in problem solving, and preserving through the ups-and-downs of sustainment was important.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Transformational leadership was associated with sustainment, while passive-avoidant leadership was associated with non-sustainment.

    “While transformational leadership was associated with sustainment, passive-avoidant leadership was associated with non-sustainment.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The partial sustainment site had providers trained and certified in the EBI who were actively providing services, but it no longer provided recommended ongoing coaching and fidelity monitoring.

    “The site classified as “partial sustainment” had providers trained and certified in the EBI who were actively providing services, but no longer provided recommended ongoing coaching and fidelity monitoring.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    Sites with no sustainment no longer provided SafeCare services.

    “Those sites with “no sustainment” no longer provided SC services.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    In one focus group in a sustaining site, home visitors shared conviction that they couldn't get guidance from higher ups, who were clueless and hadn't been in field.

    “For example, in one focus group in a sustaining site, home visitors shared their conviction that, “I couldn’t get [guidance] from the higher ups. I think they were clueless. They sit at a desk all day and they maybe haven’t been in the field in 15, 20 years. Don’t talk to me if you haven’t been in there in five years.””
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Supervisors were characterized as running interference or building relationships with government entities whose work sometimes conflicted with SC provision.

    “Supervisors were characterized as “running interference” or building relationships with government entities whose work sometimes conflicted with SC provision.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Qualitative data showed that system- and frontline-level leaders were proactive, persistent, informed, supportive during implementation, and attentive to funding issues affecting EBI delivery.

    “Consistent with implementation leadership theory, qualitative data also highlighted specific characteristics of leaders at the system and frontline levels, including being proactive, perseverant, knowledgeable, and supportive during implementation, while remaining mindful of funding considerations that could impact provision of the EBI.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    While expected transformational leadership would be associated with sustainment, it is notable that passive-avoidant leadership was strongly associated with non-sustainment.

    “While we expected that transformational leadership would be associated with sustainment, it is notable that passive-avoidant leadership was strongly associated with non-sustainment.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    At inner context level both transformational and passive-avoidant leadership were associated with sustainment or non-sustainment, respectively.

    “At the inner context level both transformational leadership and passive-avoidant leadership were associated with sustainment or non-sustainment, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Qualitative data showed that leaders should be informed about the EBI, proactive in solving problems, and persistent through sustainment challenges.

    “Consistent with emerging work on implementation leadership, qualitative data identified the importance of leaders being knowledgeable about the EBI, proactive in problem solving, and preserving through the ups-and-downs of sustainment (Aarons, Ehrhart, & Farahnak, 2014).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Aarons, Ehrhart, & Farahnak, 2014

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    Leadership mattered in both the outer system and inner team or organizational settings.

    “This was apparent in that leadership was important in both the outer system context and the inner team/organizational context.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    Qualitative analysis deepened understanding that leaders should address daily issues and remain consistently available and engaged in helping providers use EBIs.

    “Qualitative analyses enhanced a more nuanced understanding of the importance of leaders working on day-to-day issues that arose, but also being consistently available and engaged in supporting providers in their EBI use.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Decision makers could supply funding, policies, and EBP support; qualitative findings added that leaders can both create funding policies and build collaborations that aid sustainment.

    “As noted above, decision makers could provide funding, policies, and support for EBP, and qualitative results expanded describing how leaders can not only create policies that provide funding, but also establish collaborations that support sustainment.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Qualitative methods confirmed quantitative results and provided a richer, broader view of the leadership types in outer and inner contexts that can sustain EBI use.

    “However, the qualitative methods not only corroborated quantitative findings, but added a more nuanced and expanded perspective on the types of leadership in the outer and inner context, that can support sustained use of EBIs.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisenone given

    The role of leadership in EBI implementation is often discussed and almost universally considered critical, but it is rarely empirically analyzed.

    “While there is a vast literature on leadership, the role of leadership in EBI implementation is often discussed, and almost universally acknowledged as critical, but is rarely empirically analyzed.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Implementation scholars assert that leadership matters for obtaining funding, dispersing resources, and enforcing policies that support implementation.

    “Implementation scholars assert the importance of leadership in terms of obtaining funding, dispersing resources, and enforcing policies in support of implementation (Aarons, Horowitz, Dlugosz, & Ehrhart, 2012).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Aarons, Horowitz, Dlugosz, & Ehrhart, 2012

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Leader roles can include interpreting research evidence, applying it to organizational contexts, and making research-informed implementation decisions.

    “Other research suggests that leader roles can include interpreting research evidence, applying it to organizational contexts, and making research-informed implementation decisions (Kyratsis, Ahmad, & Holmes, 2012).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Kyratsis, Ahmad, & Holmes, 2012 source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Weiner's organizational theory of innovation implementation suggests leaders are critical for creating readiness for change, ensuring innovation-values fit, and developing plans, practices, structures, and strategies to support implementation.

    “Weiner’s organizational theory of innovation implementation suggests that leaders are critical in creating readiness for change, ensuring innovation-values fit, and developing plans, practices, structures, and strategies to support implementation (Weiner, 2009).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Weiner, 2009 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Research from the UK's Collaboration for Leadership in Applied Health Research and Care addresses the importance of leaders managing implementation projects, obtaining senior management support, serving as clinical opinion leaders, and fostering organizational learning climates.

    “For example, research from the United Kingdom’s Collaboration for Leadership in Applied Health Research and Care addresses the importance of leaders managing implementation projects, obtaining senior management support, serving as clinical opinion leaders, and fostering organizational learning climates (Harvey et al., 2011).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Harvey et al., 2011

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The EPIS framework emphasizes outer (system) and inner (organization) context factors and their interplay across each implementation phase.

    “The EPIS framework emphasizes the role of outer (i.e., system) and inner (i.e., organization) context factors (Aarons, Hurlburt, et al., 2011) and their interplay through each implementation phase.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Aarons, Hurlburt, et al., 2011

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Premisenone given

    There is consensus that leadership is important in implementation and change.

    “While studies vary in setting (e.g., human services, business etc.), sample size, number of organizational units, and rigor, there is consensus that leadership is important in implementation and change.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Transformational leadership may predict employees' reported use of an innovative practice being implemented in their organization.

    “For example, transformational leadership (i.e., the degree to which a leader can inspire and motivate others; (Bass & Avolio, 1995) may predict employees’ reported use of an innovative practice being implemented in their organization (Michaelis, Stegmaier, & Sonntag, 2009; Michaelis et al., 2010).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Michaelis, Stegmaier, & Sonntag, 2009 source

      Not checked in the free run.

    • cites Michaelis et al., 2010 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The Full-Range Leadership model includes empirically supported dimensions of transformational, transactional, and passive-avoidant leadership.

    “Regarding inner context leadership, the Full-Range Leadership (FRL) model includes empirically supported dimensions of transformational leadership, transactional leadership, and passive-avoidant leadership (Avolio, Bass, & Jung, 1999).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Avolio, Bass, & Jung, 1999 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Different leadership approaches may be more effective depending on whether an organization is in a stable-operating state or undergoing change.

    “Different leadership approaches may be more effective depending upon whether an organization is in a stable-operating state or undergoing change (Eggleston & Bhagat, 1993).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Eggleston & Bhagat, 1993 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    System and organization administrator respondents had to have involvement or knowledge of SafeCare implementation in their service system.

    “In particular, system and organization administrator level respondents must have had involvement or knowledge of the implementation of SafeCare in their service system.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The study was approved by an Institutional Review Board, and informed consent was obtained from all participants before data collection.

    “This study was approved by the [name withheld for blinding] Institutional Review board and informed consent was obtained from all participants prior to data collection.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    A purposive sampling approach was used to identify and recruit individuals with knowledge of EBI implementation and sustainment in each service system, consistent with mixed-methods sampling recommendations.

    “Consistent with recommendations for sampling in mixed-methods, a purposive sampling approach was used to identify and recruit the most relevant individuals with knowledge of the EBI implementation and sustainment in each service system (Aarons, Fettes, Sommerfeld, & Palinkas, 2012; Collins, Onwuegbuzie, & Jiao, 2007; Teddlie & Yu, 2007).”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Aarons, Fettes, Sommerfeld, & Palinkas, 2012

      Not checked in the free run.

    • cites Collins, Onwuegbuzie, & Jiao, 2007 source

      Not checked in the free run.

    • cites Teddlie & Yu, 2007 source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductcitation

    The PSI is associated with ongoing planning processes, confidence in a project's long-term survival, and meeting clients' needs.

    “In regard to validity, the PSI is associated with ongoing planning process, confidence in project long-term survival, and meeting the needs of clients (Mancini & Marek, 2004).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Mancini & Marek, 2004 source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Implementation factors that span outer and inner contexts include engaging strong leadership across system and organizational levels, using specific management strategies, attending to organizational and individual factors, and anchoring new programs across system levels.

    “A number of implementation factors span outer and inner contexts. These include engaging strong leadership across system and organizational levels, use of specific management strategies, attending to both organizational and individual factors, and anchoring new programs across system levels (Larsen & Samdal, 2007).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Larsen & Samdal, 2007 source

      Not checked in the free run.

  • May combine two claims.

    Study design and conductcitation

    The Program Sustainability Index comprises five subscales with good psychometric properties, with Cronbach's alphas ranging from .76 to .88.

    “It comprises 5 subscales that demonstrate good psychometric properties with Cronbach’s alphas ranging from. 76–.88.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Mancini & Marek, 2004 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation

    There is a need to identify unique sustainment factors that can lead to improvements in processes and efficiencies not evident during initial implementation.

    “There is a need to identify unique sustainment factors that can lead to improvements in processes and efficiencies not evident during initial implementation (Grimes, Kurns, & Tilly, 2006).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Grimes, Kurns, & Tilly, 2006 source

      Not checked in the free run.

  • May combine two claims.

    Study design and conductown data

    In this study, Cronbach's alphas indicated excellent reliability for transformational leadership (α=0.96) and passive-avoidant leadership (α=0.86), and good reliability for transactional leadership (α=0.76).

    “The Cronbach’s alphas from the present study indicated excellent reliability for transformational leadership (α=0.96) and passive-avoidant leadership (α=0.86), and good reliability for transactional leadership (α=0.76).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    These leadership aspects are well-researched and are associated with organizational functioning and attitudes toward adopting EBIs.

    “These aspects of leadership are well-researched and are associated with organizational functioning and attitudes toward adopting EBI (Aarons, 2006).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Aarons, 2006

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    SafeCare enhances problem solving and communication to improve caregiver skills and behaviors related to home safety, child health, and parent-child or parent-infant interactions.

    “SafeCare enhances problem solving and communications to improve parent or caregiver skills and behaviors to address home safety, child health, and parent-child or parent-infant interactions.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    A recent review highlights that leadership at outer and inner contexts may be critical for effective implementation and sustainment.

    “A recent review highlights how leadership at the outer and inner contexts may be critical for effective implementation and sustainment (Aarons, Ehrhart, Farahnak, & Sklar, 2014).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Aarons, Ehrhart, Farahnak, & Sklar, 2014

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    SafeCare is a manualized curriculum-based EBI that aims to reduce child maltreatment through home-based skills training and education for caregivers of children aged zero to five who are at risk for or reported for child neglect.

    “SafeCare is a manualized curriculum-based EBI to reduce child maltreatment through home-based skills training and education for caregivers of children, ages zero to five, who are at-risk for, or reported for, child neglect (Chaffin et al., 2012; Lutzker & Edwards, 2009).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Chaffin et al., 2012 source

      Not checked in the free run.

    • cites Lutzker & Edwards, 2009

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    In this study, the PSI-LC subscale had excellent internal consistency reliability (α=0.84).

    “The PSI-LC subscale has excellent internal consistency reliability in this study (α=0.84).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Research has shown that leaders influence employee attitudes toward EBIs, commitment to organizational change, and improvement in leader EBI support behaviors.

    “Research has also shed light on the role of leaders in influencing employee attitudes toward EBI (Aarons, 2006), commitment to organizational change (Hill, Seo, Kang, & Taylor, 2012), and improving leader EBI support behaviors (Aarons, Ehrhart, Farahnak, & Hurlburt, 2015).”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Aarons, 2006

      Not checked in the free run.

    • cites Hill, Seo, Kang, & Taylor, 2012 source

      Not checked in the free run.

    • cites Aarons, Ehrhart, Farahnak, & Hurlburt, 2015 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Site classification into sustainment categories was based on independent review and consensus of the first and second authors, with consultation from the EBI developers.

    “Classification of sites into each category was based on independent review and then consensus of the first and second author, supported by consultation from the EBI developers.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Small samples and few organizational units are frequent difficulties in leadership research, especially at higher organizational or system levels.

    “However, sample size and number of organizational units (e.g., service teams) are common challenges in studies of leadership, particularly as higher organizational or system levels are studied (Klein, Dansereau, & Hall, 1994; Klein & Kozlowski, 2000).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Klein, Dansereau, & Hall, 1994 source

      Not checked in the free run.

    • cites Klein & Kozlowski, 2000

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    Consistent with emerging work regarding implementation leadership, leaders should be knowledgeable about EBI, proactive and perseverant in sustainment process, and supportive of staff efforts to use EBIs.

    “Consistent with emerging work regarding implementation leadership, leaders should be knowledgeable about the EBI or EBIs being implemented, proactive and perseverant in the sustainment process, and should be supportive of their staff members’ efforts to use EBIs.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Provider age, job tenure, and sex were controlled for due to potential influence on leadership perceptions and SafeCare sustainment.

    “Provider age, job tenure, and sex were controlled for due to their potential influence on both leadership perceptions and SafeCare sustainment.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Separate outer and inner context ordinal regressions adjusted standard errors for relevant nesting unit.

    “Because there were different respondents for outer and inner context analyses, respondents, and leadership variables at the system and team levels, separate outer context and inner context ordinal regressions adjusting standard errors for the relevant nesting unit of analysis were utilized.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The present study adds to literature on sustainment leadership and provides support for predictive validity of PSI in predicting sustainment.

    “The present study also adds to the literature on sustainment leadership and provides support for the predictive validity of the PSI in predicting sustainment.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The dimensions of FRL are sensitive to context (Antonakis et al., 2003) so not surprising PSI-LC has conceptual similarities to aspects of FRL.

    “The dimensions of the FRL are sensitive to context (Antonakis, Avolio, & Sivasubramaniam, 2003) so it is not surprising that the PSI-LC has some conceptual similarities to aspects of the FRL model.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Antonakis, Avolio, & Sivasubramaniam, 2003 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Ordinal regressions accounted for nested data structure for system and organizational levels.

    “All analyses accounted for the nested data structure for system and organizational (i.e., team) levels, respectively.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Qualitative interviewers were ethnographers not local academic collaborators or intervention developers.

    “The qualitative interviewers were conducted by ethnographers that were not local academic collaborators or intervention developers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Interviews and focus groups were digitally recorded, professionally transcribed, and reviewed for accuracy by at least one author.

    “Interviews and focus groups were digitally-recorded, professionally transcribed, and reviewed for accuracy by at least one author.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Each team member independently coded sets of transcripts, created detailed memos, and shared work for review.

    “Each team member independently coded sets of transcripts, created detailed memos that both described and linked codes to each theme and shared their work with one another for review.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    For providers, delivering a suitable EBI with supportive fidelity coaching may reduce emotional exhaustion and improve staff retention.

    “For the provider workforce, the implementation of an appropriate EBI along with supportive fidelity coaching can lead to positive outcomes including lower emotional exhaustion and higher staff retention (Aarons, Fettes, Flores, & Sommerfeld, 2009; Aarons, Sommerfeld, Hecht, Silovsky, & Chaffin, 2009).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Aarons, Fettes, Flores, & Sommerfeld, 2009

      Not checked in the free run.

    • cites Aarons, Sommerfeld, Hecht, Silovsky, & Chaffin, 2009

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Mixed methods can reduce sample-size concerns when triangulation confirms findings and quantitative and qualitative data are combined to address key implementation questions.

    “Mixed-methods (quantitative-qualitative) can help to mitigate these concerns when triangulation can be used to corroborate findings and quantitative and qualitative data are integrated to answer key implementation questions (Aarons, Fettes, et al., 2012).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Aarons, Fettes, et al., 2012

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The study used different leadership measures for outer and inner contexts.

    “First, our measures of leadership were different for outer and inner contexts.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    This is consistent with other work in different settings that find stronger effects for transformational leadership (Spinelli, 2006).

    “However this is consistent with other work in different settings that find stronger effects for transformational leadership (Spinelli, 2006).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Spinelli, 2006 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    Present study is unique in highlighting multilevel leadership in relation to sustainment reported for both outer system context and inner context.

    “However, the present study is unique in highlighting multilevel leadership in relation to sustainment reported for both the outer system context and inner context.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Leadership congruence works by communicating EBI importance clearly and fostering a positive implementation climate among stakeholders.

    “We propose that such leadership congruence is effective because it sends a clear message about the importance of EBI and facilitates a positive implementation climate among stakeholders.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Although interviewers were aware of which sites had no providers, they were informed of sustainment status of sites for which there were current providers.

    “Although they were aware of which sites had no providers with whom to conduct focus groups, they were informed of the sustainment status of the sites for which there were current providers.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Using different leadership measures for outer and inner contexts enabled assessment of the leadership most relevant to each level.

    “However, this allowed us to assess leadership most pertinent to those levels.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Leadership types share commonalities across outer and inner contexts, though they may appear differently depending on level-specific roles and responsibilities.

    “Thus, there are some commonalities in the types of leadership across outer and inner context, but these may manifest differently in keeping with roles and responsibilities at different levels.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    FRL model has been shown applicable in large healthcare settings (Spinelli, 2006).

    “Indeed the FRL model has been shown to be applicable in large healthcare settings (Spinelli, 2006).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Spinelli, 2006 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Leadership training and similar approaches may improve EBI implementation, sustainment, and care quality and outcomes.

    “Such approaches hold promise to improve EBI implementation and sustainment and quality and outcomes of care.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    A more focused assessment linked to quantitative measures might have yielded greater convergence of findings.

    “More targeted assessment tied to quantitative measures may have provided greater convergence of findings.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The interviews and focus groups addressed a wide range of issues concerning factors that affect EBI sustainment.

    “Third, interviews and focus groups covered a broad range of issues related to factors impacting EBI sustainment.”
    Quote in the paperNot a substantial claim
77 of 77

  • The paper relies on this without support.

    If evidence-based interventions are not sustained, the investments made in developing and implementing them are wasted and their public health impact is limited.

    “If EBIs are not sustained, investments in their development and implementation are wasted and public health impact will be limited.”
1 of 1

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The second most cited article is "The Roles of System and Organizational Leadership in System-Wide Evidence-Based Intervention Sustainment: A Mixed-Method Study", published in Administration and Policy in Mental Health and Mental Health Services Research in 2016 by an international team from Asia, the USA, and Europe.

    “Second is the article “The Roles of System and Organizational Leadership in System-Wide Evidence-Based Intervention Sustainment: A Mixed-Method Study” [85], published in the Administration and Policy in Mental Health and Mental Health Services Research in 2016 by an international team from Asia, the USA, and Europe.”

    From Towards Green and Sustainable Healthcare: A Literature Review and Research Agenda for Green Leadership in the Healthcare Sector (2023) · cites this paper as [85]

    The paper’s title heading identifies it as the article cited. The passages from our copy do not establish that its authors formed a team from Asia, the USA, and Europe. The ‘second’ ranking is the citing document’s own ranking, not a finding attributed to this source. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Empirical research shows the presence of internal and external champions, supporters, and advocates influences program sustainment.

    “Empirical research also shows that the presence of internal and external champions, supporters, and advocates influences program sustainment [26, 51–53]”

    From Using qualitative comparative analysis to uncover multiple pathways to program sustainment: implications for community-based youth substance misuse prevention (2022) · cites this paper as [26]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Outer-context odds comparison reverses the reported effect

    Major

    The first statement says greater leadership competence increases the odds of moving toward sustainment. The second says it increases the odds of non-sustainment or partial sustainment relative to full sustainment-the opposite direction.

  • Transformational-leadership odds comparison reverses the reported effect

    Major

    The first statement says greater transformational leadership increases the odds of moving to a higher sustainment level. The second instead says it increases the odds of the lower sustainment levels relative to full sustainment.

  • Reported team size does not match the provider count

    Minor

    The 162 surveyed home visitors across 39 teams average about 4.15 per team, not 5.25. The stated mean would imply about 205 providers; the text does not clarify whether that mean includes providers outside the surveyed group.

    Reported 162 · recomputed 204.75

  • Passive-avoidant-leadership odds comparison reverses the reported effect

    Minor

    The first statement says greater passive-avoidant leadership reduces the odds of moving to a higher sustainment level. The second says it reduces the odds of the lower sustainment levels relative to full sustainment, which points toward greater sustainment instead.

4 of 4

The citation lists haven’t been fetched yet

References 108 · Cited by 251

Research works printed but not linked: 6

  1. Greene, J. C. (2006). Toward a methodology of mixed methods social inquiry. Research in the Schools, 13(1), 93–98.
  2. Lutzker, J. R., & Edwards, A. (2009). SafeCare®: Towards wide-scale implementation of a child maltreatment prevention program. International Journal of Child Health and Human Development, 2, 7–15.
  3. Muthén, L. K., & Muthén, B. O., (1998-2015).  Mplus User’s Guide. (7th ed.). Los Angeles, CA: Muthén & Muthén
  4. Patton, M. Q. (2002). Qualitative analysis and interpretation. In Qualitative research and evaluation methods (3rd ed., pp. 431–539). Thousand Oaks, CA: Sage.
  5. Skolarus, T. A., & Sales, A. E. (2014). Who uses what model? Assessing models for dissemination and implementation research using citation analysis. Paper presented at the 7th Annual NIH conference on the science of dissemination and implementation, Bethesda, MD.
  6. Wong, C., & Giallonardo, L. (2015). Leadership and its influence on patient outcomes. In Leadership and Nursing: Contemporary Perspectives (2nd ed., pp. 153–170). Sydney: Churchill Livingstone.

Authors
Aarons, G. A., Green, A. E., Trott, E., Willging, C. E., Torres, E. M., Ehrhart, M. G., & Roesch, S. C.
Affiliations
  • University of California San Diego (US)
  • University of New Mexico (US)
  • San Diego State University (US)
Corresponding author
Aarons, Gregory A.
ORCID
Venue
Administration and Policy in Mental Health and Mental Health Services Research
Publisher
Springer Science and Business Media LLC
Published
2016-07-20
Volume
43
Issue
6
Pages
991-1008
ISSN
  • 0894-587X (ISSN-L)
  • 1573-3289
DOI
10.1007/s10488-016-0751-4
PMC
PMC5494253
OpenAlex
W2483396302
Semantic Scholar
3841d8113ff8a46f236478bad36c78f56dc60c36
Cited by
251
References
108
Citations by year
2026: 13 · 2025: 21 · 2024: 30 · 2023: 30 · 2022: 23 · 2021: 41 · 2020: 34 · 2019: 24 · 2018: 20 · 2017: 13 · 2016: 2
FWCI
36.58
Type
journal article
Language
en
Open access
Licence
Locations
  • Administration and Policy in Mental Health and Mental Health Services Research (publishedVersion)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
  • eScholarship (California Digital Library) (submittedVersion)
Funders
  • National Institute of Mental Health (R01MH072961, R01MH092950)
  • National Center for Injury Prevention and Control (R01CE001556)
  • National Institute on Drug Abuse (R01DA038466)
Topics
  • Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
  • Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
  • Social Sciences › Psychology › Clinical Psychology › Psychiatric care and mental health services
Keywords
evidence-based interventions, intervention sustainability, leadership, systems leadership, organizational leadership, frontline leadership, transformational leadership, passive-avoidant leadership, EPIS framework, mental health system, stakeholder collaboration, county mental health services
MeSH terms
Attitude of Health Personnel, California, Child Abuse, Child Health Services, Child Welfare, Child, Preschool, Health Personnel, Humans, Infant, Infant, Newborn, Leadership, Mental Health Services, Public Health, Program Evaluation, Caregivers, Focus Groups, Qualitative Research, Evidence-Based Practice

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review