CoChat
Sign Up
Cited by 73Mobile Health and mHealth ApplicationsDigital Mental Health InterventionsSubstance Abuse Treatment and Outcomes

Successful Organizational Strategies to Sustain Use of A-CHESS: A Mobile Intervention for Individuals With Alcohol Use Disorders

Open access
Ford II, James H, Alagoz, Esra, Dinauer, Susan, Johnson, Kimberly A, Pe-Romashko, Klaren, Gustafson, David H·2015·Journal of Medical Internet Research
DOI 10.2196/jmir.3965PMC PMC4642385
Open this paper in CoChat — free accountOpen PDF

Abstract

BACKGROUND: Mobile health (mHealth) services are growing in importance in health care research with the advancement of wireless networks, tablets, and mobile phone technologies. These technologies offer a wide range of applications that cover the spectrum of health care delivery. Although preliminary experiments in mHealth demonstrate promising results, more robust real-world evidence is needed for widespread adoption and sustainment of these technologies. OBJECTIVE: Our aim was to identify the problems/challenges associated with sustained use of an mHealth addiction recovery support app and to determine strategies used by agencies that successfully sustained client use of A-CHESS. METHODS: Qualitative inquiry assessed staff perceptions about organizational attributes and strategies associated with sustained use of the mobile app, A-CHESS. A total of 73 interviews of clinicians and administrators were conducted. The initial interviews (n=36) occurred at the implementation of A-CHESS. Follow-up interviews (n=37) occurred approximately 12 and 24 months later. A coding scheme was developed and Multiuser NVivo was used to manage and analyze the blinded interview data. RESULTS: Successful strategies used by treatment providers to sustain A-CHESS included (1) strong leadership support, (2) use of client feedback reports to follow up on non-engaged clients, (3) identify passionate staff and incorporate A-CHESS discussions in weekly meetings, (4) develop A-CHESS guidelines related to client use, (5) establish internal work groups to engage clients, and (6) establish a financial strategy to sustain A-CHESS use. The study also identified attributes of A-CHESS that enhanced as well as inhibited its sustainability. CONCLUSIONS: Mobile apps can play an important role in health care delivery. However, providers will need to develop strategies for engaging both staff and patients in ongoing use of the apps. They will also need to rework business processes to accommodate the changes in communication frequency and style, learn to use app data for decision making, and identify financing mechanisms for supporting these changes.

AI snapshot

Written from the paper’s full text.

TL;DR

This qualitative study examined how addiction treatment agencies sustained use of the A-CHESS mobile recovery app after a consortium ended. It identifies leadership support, staff/client engagement routines, monitoring/feedback, and financial business-model strategies as central to short-term sustainability.

Key findings

  • Of 14 agencies, 3 (#5, #7, #13) continued A-CHESS as an integrated part of service delivery after the Consortium ended in December 2014; 3 agencies (#8, #9, #10) dropped out before the 12-month interview.
  • Sustaining agencies used strategies including strong leadership support, regular review of client-use reports to follow up with non-engaged clients, identifying interested staff, incorporating A-CHESS into weekly staff meetings, developing A-CHESS guidelines/contracts, establishing internal work groups, and seeking funding such as mini-grants or donations; Agency 13 used state Medicaid billing codes.
  • Across 73 interviews with 44 unique individuals from 14 agencies, top innovation enhancers included the discussion group, addressing client needs/staff support, and rapid response; inhibitors included phone availability/cost, concerns about being tracked, technical failures, and client smartphone skill/comfort.
  • Non-sustaining agencies often recognized A-CHESS benefits but did not systematize client engagement or develop a financial business model; one respondent said, “Funding is always the top barrier.”

Methods

  • Qualitative inquiry using 36 baseline and 37 follow-up interviews (total 73) with staff and administrators from 14 treatment agencies in the CHESS Health Education Consortium; follow-ups occurred approximately 12 and 24 months later.
  • Coding scheme based on implementation/sustainability frameworks yielded 9 main analytic themes with 42 nested codes; Multiuser NVivo was used, 4 researchers coded the data, and interrater reliability was 92%.
  • Sustainability was assessed at the organizational level as whether the agency continued to use A-CHESS (or another mHealth intervention) after the Consortium ended in December 2014.

Limitations

  • The paper states providers were a self-selected consortium sample and may not represent other treatment providers; findings concern a specific recovery-oriented app and may differ for other mHealth apps.
  • The paper states staff turnover meant different staff were interviewed at baseline and follow-up, interview timing was inconsistent (initial interviews 3-16 months after joining; follow-ups sometimes 1-5 months outside windows), and patients using A-CHESS were not interviewed.
  • The paper states external funding strategies are specific to the US substance abuse treatment financing system; evident from the text, only 3 agencies sustained A-CHESS, limiting generalizable comparison.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.2196/jmir.3965

Cites

49 of 49 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 45 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Three agencies (#5, #7, and #13) continued to utilize A-CHESS as an integrated part of their service delivery after the Consortium ended.

    “Three agencies (#5, #7, and #13) continued to utilize A-CHESS as an integrated part of their service delivery after the Consortium ended.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Staff and leadership interview responses accurately reflect the organizational strategies and sustainability of A-CHESS.
  • Findingown data

    The study identified attributes of A-CHESS that enhanced as well as inhibited its sustainability.

    “The study also identified attributes of A-CHESS that enhanced as well as inhibited its sustainability.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Staff and leadership interview responses accurately reflect the organizational strategies and sustainability of A-CHESS.
  • Findingown data

    Developing A-CHESS guidelines related to client use was a successful strategy used by treatment providers to sustain A-CHESS.

    “we work with patients more at the beginning to sustain interest, help them use the phone…[and]…developed some guidelines”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Staff and leadership interview responses accurately reflect the organizational strategies and sustainability of A-CHESS.
  • Findingown data

    The remaining agencies were not able to sustain A-CHESS after the Consortium ended.

    “Although the remaining agencies were not able to sustain A-CHESS after the Consortium ended, their efforts were also reviewed to help identify unsuccessful sustainability strategies.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Staff and leadership interview responses accurately reflect the organizational strategies and sustainability of A-CHESS.
  • Findingown data

    Use of client feedback reports to follow up on non-engaged clients was a successful strategy used by treatment providers to sustain A-CHESS.

    “One strategy involved the regular review of reports about client use of A-CHESS. This provided staff with additional opportunities to identify clients at risk and to initiate follow-up efforts to re-engage them.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Staff and leadership interview responses accurately reflect the organizational strategies and sustainability of A-CHESS.
  • The quote only partly states this claim. 1
  • May combine two claims. 28
  • Classified as probably not a substantial claim. 25

  • The quote only partly states this claim.

    Findingown data

    Strong leadership support was a successful strategy used by treatment providers to sustain A-CHESS.

    “Strong leadership support was present across multiple agencies.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Staff and leadership interview responses accurately reflect the organizational strategies and sustainability of A-CHESS.

  • May combine two claims.

    Findingown data

    Identifying passionate staff and incorporating A-CHESS discussions in weekly meetings was a successful strategy used by treatment providers to sustain A-CHESS.

    “Another staff engagement strategy involved integrating a discussion of A-CHESS into weekly staff meetings.”
    • Also stated in: § Staff Engagement · plain
      “The final staff engagement strategy these three agencies used involved identifying staff who expressed an interest in working with A-CHESS.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Staff and leadership interview responses accurately reflect the organizational strategies and sustainability of A-CHESS.

  • May combine two claims.

    Findingown data

    Agency 5 leadership uses the presentations to communicate their message to potential donors or foundations, which has helped brand the agency as a leader in the field.

    “Agency leadership uses the presentations to communicate their message to potential donors or foundations, which has helped brand the agency as a leader in the field.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Staff at agencies that did not sustain A-CHESS identified a lack of communication as a barrier to sustainability but did not have a solution to systematize the feedback process or had to determine if reports were a better source of data than internal tracking systems.

    “In contrast, staff at agencies who did not sustain A-CHESS identified a lack of communication as a barrier to sustainability but did not have a solution to systematize the feedback process or had to determine if these reports were a better source of data than internal tracking systems.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Technological challenges, phone availability and costs, concerns about being tracked, and the skills of the users were identified as potential innovation inhibitors.

    “Technological challenges, phone availability and costs, concerns about being tracked, and the skills of the users were identified as potential innovation inhibitors.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Attributes of an innovation influence whether it is sustained within an organization.

  • May combine two claims.

    Findingown data

    Agency 5 established an A-CHESS service line and developed presentations about A-CHESS impact on client outcomes.

    “For example, Agency 5 established an A-CHESS service line and developed presentations about A-CHESS impact on client outcomes.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The external environment (eg, reimbursement from insurance providers or Medicaid) is not keeping up with innovations in using technology to engage clients in treatment.

    “However, the external environment (eg, reimbursement from insurance providers or Medicaid) is not keeping up with the innovations.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Staff and leadership interview responses accurately reflect the organizational strategies and sustainability of A-CHESS.

  • Classified as probably not a substantial claim.

    Findingown data

    Staff believed that A-CHESS allowed clients to create a sense of community in order to help each other in a time of need.

    “Staff believed that A-CHESS allowed the clients to create a sense of community in order to help each other in a time of need.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Leadership in three non-sustaining agencies (#1, #2, and #11) took active steps to reduce costs and explored additional funding including switching carriers, initiating conversations with the state about Medicaid coverage, and using A-CHESS reports to highlight effectiveness to the community.

    “Leadership in three of those agencies (#1, #2, and #11) took active steps to reduce costs and explored the possibility of additional funding including switching carriers, initiating conversations with the state about Medicaid coverage, and using A-CHESS reports to highlight the effectiveness of A-CHESS to the community.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    Leadership in one sustaining agency recognized that external funding to support A-CHESS would need to come from private donors and/or foundations.

    “As a result, leadership in this agency recognized that external funding to support A-CHESS would need to come from private donors and/or foundations.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Staff and leadership interview responses accurately reflect the organizational strategies and sustainability of A-CHESS.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    In another agency that sustained A-CHESS, leadership understood that using technology to engage clients in treatment is innovative.

    “In another agency that sustained A-CHESS, leadership understood that using technology to engage clients in treatment is innovative.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Staff and leadership interview responses accurately reflect the organizational strategies and sustainability of A-CHESS.

  • May combine two claims.

    Findingown data

    Agency 1 used the protocol strategy and feedback reports but did not sustain A-CHESS after the Consortium ended.

    “Agency 1 also used this strategy and the feedback reports but did not sustain A-CHESS after the Consortium ended.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    As part of Agency 5's business model strategy, A-CHESS was offered only to clients with a smartphone and associated data plan, thus avoiding the variable expense associated with buying phones and providing the data service plan.

    “As part of the business model strategy for this agency, A-CHESS was offered only to clients with a smartphone and associated data plan, thus avoiding the variable expense associated with buying phones and providing the data service plan.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Staff in agencies that sustained A-CHESS used available reports to identify clients who were not using the system and created processes requiring follow-up with the client.

    “Staff in these agencies utilized the available reports to identify clients who were not using the system and created processes that required follow-up with the client.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Agencies that sustained A-CHESS (#5, #7, and #13) took active steps to reduce costs and explored the possibility of additional funding.

    “Agencies that sustained A-CHESS (#5, #7, and #13) took active steps to reduce costs (eg, provided to clients with a phone) and explored the possibility of additional funding.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingcitation

    Leaders in agencies that successfully sustained A-CHESS took steps to ensure that their strategic decisions and plans for A-CHESS created a coherent vision for organizational transformation; empowered employees to participate in the change process, including providing training; and allocated resources, change policies, procedures, and job descriptions.

    “These leaders took steps to ensure that their strategic decisions and plans for A-CHESS created a coherent vision for organizational transformation; empowered employees to participate in the change process, including providing training; and allocated resources, change policies, procedures, and job descriptions [31,41,42].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [31] source

      Not checked in the free run.

    • cites [41] source

      Not checked in the free run.

    • cites [42] source

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    Agency 13 worked closely with the MCOs in the state who are engaged in A-CHESS promotion and are devoting funds to pilot projects for A-CHESS implementation with their patient population.

    “This agency also worked closely with the MCOs in the state who “are engaged in A-CHESS promotion and are devoting funds to ‘pilot projects’ for A-CHESS implementation with their patient population”.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    While other strategies were applied to promote ongoing client engagement, agencies that sustained A-CHESS did not use them.

    “While other strategies were applied to promote ongoing client engagement, agencies that sustained A-CHESS did not use them.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Sustainability relies more on the identification of strategies that encourage ongoing client use of the intervention than on strategies for initial engagement.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    One challenge to sustaining A-CHESS was the cost of phones and clients not using A-CHESS as much the longer they are in treatment.

    “Another respondent indicated that one of the challenges to sustaining A-CHESS was the “cost of phones; clients aren’t really using [A-CHESS] that much the longer they are in treatment”.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The top three innovation enhancers mentioned by staff were the discussion group, the ability of A-CHESS to address client needs and staff support, and rapid response to client concerns.

    “The top three innovation enhancers mentioned by staff were the discussion group, the ability of A-CHESS to address client needs and staff support, and rapid response to client concerns.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Attributes of an innovation influence whether it is sustained within an organization.

  • May combine two claims.

    Findingown data

    Agency 1 utilized A-CHESS reports to provide information to staff about client usage, but unlike Agencies 5, 7, and 13, that information was not shared with staff on a regular basis.

    “Agency 1 utilized the A-CHESS reports to provide information to staff about client usage; however, unlike Agencies 5, 7, and 13, that information was not shared with staff on a regular basis.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Staff and leadership interview responses accurately reflect the organizational strategies and sustainability of A-CHESS.

  • May combine two claims.

    Findingown data

    The strategies of agencies sustaining A-CHESS integrated monitoring efforts into weekly meetings and established policies and procedures to follow up with unengaged clients.

    “Their strategies integrated monitoring efforts into weekly meetings and established policies and procedures to follow up with unengaged clients.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Successful agencies established A-CHESS implementation teams and utilized clients and peer counselors to train new users.

    “For example, successful agencies established A-CHESS implementation teams and utilized clients and peer counselors to train new users.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Existing sustainability frameworks suggest that sustainability is attributable to internal organizational supports, contextual influences of the external environment, and characteristics of the intervention.

    “Existing sustainability frameworks suggest that sustainability is attributable to the (1) internal organizational supports, (2) contextual influences of the external environment, and (3) characteristics of the intervention [31-34].”
    Quote in the paperSubstantial claim0 of 4 sources checked
    • cites [31] source

      Not checked in the free run.

    • cites [32] source

      Not checked in the free run.

    • cites [33] source

      Not checked in the free run.

    • cites [34] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The interrater reliability among coders was 92%.

    “The interrater reliability among coders was 92%.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Internal organizational supports relevant to sustainability include champion roles, leadership, resources, policies and procedures, and expert support (coaching).

    “Internal organizational supports include champion roles, leadership, resources, policies and procedures, and expert support (ie, coaching).”
    Quote in the paperSubstantial claim0 of 4 sources checked
    • cites [31] source

      Not checked in the free run.

    • cites [32] source

      Not checked in the free run.

    • cites [33] source

      Not checked in the free run.

    • cites [34] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    A-CHESS has been previously tested in a randomized trial involving 349 study participants and found to reduce risky drinking days and improve abstinence.

    “The A-CHESS mobile phone app has been previously tested in a randomized trial involving 349 study participants and found to reduce risky drinking days and improve abstinence [19].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [19] source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Innovation characteristics relevant to sustainability include stakeholder relationships, innovation integrity, effectiveness, quality of initial implementation efforts, and alignment between innovation and stakeholder needs.

    “Innovation characteristics include stakeholder relationships, innovation integrity, effectiveness, and quality of initial implementation efforts, and alignment between innovation and stakeholder (eg, agency staff or patients) needs.”
    Quote in the paperSubstantial claim0 of 4 sources checked
    • cites [31] source

      Not checked in the free run.

    • cites [32] source

      Not checked in the free run.

    • cites [33] source

      Not checked in the free run.

    • cites [34] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The environment context of sustainability frameworks explores the effects of financial, regulatory, and policy changes.

    “The environment context explores the effects of financial, regulatory, and policy changes.”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [31] source

      Not checked in the free run.

    • cites [32] source

      Not checked in the free run.

    • cites [33] source

      Not checked in the free run.

    • cites [34] source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisereasoning

    Whether A-CHESS can be implemented and sustained in standard clinical practice remains unanswered despite its efficacy in a clinical trial.

    “Even though the intervention was efficacious, the question of whether such a technology could be implemented and sustained in a standard clinical practice remains unanswered.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Changes in working with patients more at the beginning to sustain interest and help them use the phone, and developing guidelines, helped patients learn more about how the phone app works.

    “These changes helped patients learn more about how the phone app works.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Staff and leadership interview responses accurately reflect the organizational strategies and sustainability of A-CHESS.

  • May combine two claims.

    Premisecitation

    Effectively engaging staff, and in our case, clients, establishes a sense of empowerment that they are a part of the change process, enhances staff motivation to participate in the change process, and leads to more actionable steps by staff to promote sustainment of the change.

    “Effectively engaging staff, and in our case, clients, establishes a sense of empowerment that they are a part of the change process, enhances staff motivation to participate in the change process, and leads to more actionable steps by staff to promote sustainment of the change [40,44-48].”
    Quote in the paperSubstantial claim0 of 6 sources checked
    • cites [40] source

      Not checked in the free run.

    • cites [44] Davies, B, 2010, Evaluating the Impact of Implementing Evidence Based Practice

      Not checked in the free run.

    • cites [45] source

      Not checked in the free run.

    • cites [46] source

      Not checked in the free run.

    • cites [47] source

      Not checked in the free run.

    • cites [48] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    Mobile apps can play an important role in health care delivery.

    “Mobile apps can play an important role in health care delivery.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Our results indicate that leadership can support efforts for mHealth financing by establishing service lines, leveraging billing codes, and marketing the impact of mHealth on clients to perspective payers.

    “Our results indicate that leadership can support efforts for mHealth financing by establishing service lines, leveraging billing codes, and marketing the impact of mHealth on clients to perspective payers.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Fiduciary responsibility is a leadership task.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Efforts to secure ongoing funding are an important issue in mHealth implementation.

    “Efforts to secure ongoing funding are an important issue in mHealth implementation [25,43].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [25] source

      Not checked in the free run.

    • cites [43] source

      Not checked in the free run.

    • Rests on an assumption: Fiduciary responsibility is a leadership task.

  • May combine two claims.

    Premisecitation

    Efforts to successfully sustain an intervention like A-CHESS require support from leadership and a project champion, an understanding of how the innovation fits within the organizational mission, training for all staff involved in the implementation, seeking continued financial resources, integrating a new practice into organization policies and procedures, and continuous monitoring.

    “Efforts to successfully sustain such an intervention require support from leadership and a project champion, an understanding of how the innovation fits within the organizational mission, training for all staff involved in the implementation, seeking continued financial resources, integrating a new practice into organization policies and procedures, and continuous monitoring [37].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [37] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Strategies that might influence sustainability may be related to intervention types.

    “Strategies that might influence sustainability may be related to intervention types [37].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [37] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    Successful organizations will develop adoption plans that address staff and patient engagement, organizational procedures, and business model.

    “Successful organizations will develop adoption plans that address staff and patient engagement, organizational procedures, and business model.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    The organizational strategies might be transferable to non-US-based substance abuse treatment providers or other types of health care systems.

    “The organizational strategies might be transferable to non–US-based substance abuse treatment providers or other types of health care systems.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Providers that participated in the research consortium (a self-selected sample) may not be representative of other treatment providers that might have implemented other mHealth apps.

    “Providers that participated in the research consortium (a self-selected sample) may not be representative of other treatment providers that might have implemented other mHealth apps.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The qualitative process relied on a sample of interviews from the participating agencies.

    “The qualitative process relied on a sample of interviews from the participating agencies.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    The use of mHealth apps for disease management will require a series of changes to the operations in health care provider organizations.

    “As with any technology adoption, the use of mHealth apps for disease management will require a series of changes to the operations in health care provider organizations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Staff turnover meant that different staff members were interviewed for the baseline and follow-up surveys.

    “Staff turnover meant that different staff members were interviewed for the baseline and follow-up surveys.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    The interview process did not include patients using A-CHESS, and their responses about strategies to sustain A-CHESS may have differed from staff.

    “In addition, the interview process did not include patients using A-CHESS. Their responses about strategies to sustain A-CHESS may have differed from staff.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    Efforts to schedule follow-up interviews led to some interviews being scheduled as much as 1-5 months outside of the respective 12-month and 24-month timeframe.

    “Efforts to schedule the interviews with staff led to, in some instances, the interviews being scheduled as much as 1-5 months outside of the respective 12-month and 24-month timeframe.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    It has been suggested that such systems have four components: (1) continually and effectively monitor progress of change, (2) keep the organization informed about success, (3) identify new areas for improvement, and (4) establish protocols to communicate and act on the results.

    “It has been suggested that such systems have four components: (1) continually and effectively monitor progress of change, (2) keep the organization informed about success, (3) identify new areas for improvement, and (4) establish protocols to communicate and act on the results [55].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [55] MaherLGustafsonDEvansASustainability Model and Guide20042015-08-13Leceister, EnglandBritish National Health Service Modernization Agencyhttp://www.qihub.scot.nhs.uk/media/162236/sustainability_model.pdf6akpB7hr6

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    The initial interview was not conducted until after the plans were developed, with a range of 3-16 months between when the organization joined the consortium and when the initial interview was conducted.

    “As such, the initial interview was not conducted until after the plans were developed with a range of 3-16 months between when the organization joined the consortium and when we conducted the initial interview.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The strategies associated with implementing and sustaining the A-CHESS app may differ from other mHealth apps.

    “The strategies associated with implementing and sustaining the A-CHESS app may differ from other mHealth apps.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    The full implementation of the Affordable Care Act in the United States or the actual structure of the payment system in another country may require changes to existing strategies or the identification of new strategies for sustaining an mHealth app in general or A-CHESS in particular within their respective implementation environments.

    “However, the full implementation of the Affordable Care Act in the United States or the actual structure of the payment system in another country may require changes to existing strategies or the identification of new strategies for sustaining an mHealth app in general or A-CHESS in particular within their respective implementation environments.”
    Quote in the paperNot a substantial claim
49 of 49

  • The paper relies on this without support.

    Staff and leadership interview responses accurately reflect the organizational strategies and sustainability of A-CHESS.

    “Input from staff and leadership provided different perspectives about the strategies employed.”

  • The paper relies on this without support.

    Attributes of an innovation influence whether it is sustained within an organization.

    “Attributes of an innovation influence whether it is sustained within an organization.”

  • The paper relies on this without support.

    Sustainability relies more on the identification of strategies that encourage ongoing client use of the intervention than on strategies for initial engagement.

    “While both are important, sustainability relies more on the identification of strategies that encourage ongoing client use of the intervention.”

  • The paper relies on this without support.

    Fiduciary responsibility is a leadership task.

    “Fiduciary responsibility is a leadership task.”
4 of 4

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The ability of an initiative to share resources with partners and other organisations, seek out alternative and supplemental resources and/or uncover multiple funding sources were highlighted across some approaches as important to overall sustainability.

    “The ability of an initiative to share resources with partners and other organisations [41, 46], seek out alternative and supplemental resources [18, 47–49] and/or uncover multiple funding sources [8, 36, 49, 50] were also highlighted across some approaches as important to overall sustainability.”

    From Navigating the sustainability landscape: a systematic review of sustainability approaches in healthcare (2018) · cites this paper as [47]

  • Approaches to monitoring included diverse areas to assess including having appropriate data to document progress, having a management or monitoring system in place, and having regular reporting and feedback mechanisms.

    “Approaches to monitoring included diverse areas to assess including having appropriate data to document progress [64, 65], having a management or monitoring system in place [15, 53, 61, 66, 67], and having regular reporting and feedback mechanisms [46, 47, 52, 54, 68, 69].”

    From Navigating the sustainability landscape: a systematic review of sustainability approaches in healthcare (2018) · cites this paper as [47]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Administrator male count differs between text and Table 3

    Minor

    The text reports 6 male administrators out of 10, whereas Table 3 reports 12 out of 20. Both give 60%, but the underlying counts conflict.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 3

  1. Larimer, ME, 1999, Alcohol Res Health, 23, 151
  2. Bazeley, P, 2013, Qualitative data analysis with NVivo, 1
  3. Davies, B, 2010, Evaluating the Impact of Implementing Evidence Based Practice, 165

Cited by

Sorted by citations

Authors
Ford II, J. H., Alagoz, E., Dinauer, S., Johnson, K. A., Pe-Romashko, K., & Gustafson, D. H.
Affiliations
University of Wisconsin–Madison (US)
ORCID
Venue
Journal of Medical Internet Research
Publisher
JMIR Publications Inc.
Published
2015-08-18
Volume
17
Issue
8
Pages
e201
ISSN
1438-8871 (ISSN-L)
DOI
10.2196/jmir.3965
PMC
PMC4642385
OpenAlex
W1500637256
Semantic Scholar
22e10cf2f4eb3e612b570514595479b2989e9c9f
Cited by
73
References
49
Citations by year
2026: 3 · 2025: 6 · 2024: 3 · 2023: 16 · 2022: 8 · 2021: 5 · 2020: 5 · 2019: 4 · 2018: 15 · 2017: 4 · 2016: 3
FWCI
13.38
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Journal of Medical Internet Research (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • National Institute on Alcohol Abuse and Alcoholism (R01AA017192)
  • National Institute on Drug Abuse (R01DA034279, R21DA036700, R01DA040449)
Topics
  • Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
  • Health Sciences › Medicine › Epidemiology › Substance Abuse Treatment and Outcomes
Keywords
mobile health, alcoholism, organizational strategy, customer engagement
MeSH terms
Administrative Personnel, Adult, Delivery of Health Care, Female, Health Personnel, Health Services Research, Humans, Male, Middle Aged, Program Evaluation, Telemedicine, Alcohol-Related Disorders, Qualitative Research, Young Adult, Mobile Applications

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review