CoChat
Sign Up
Cited by 92Electronic Health Records SystemsTelemedicine and Telehealth ImplementationMobile Health and mHealth Applications

Lessons Learned From a Living Lab on the Broad Adoption of eHealth in Primary Health Care

Open access
Swinkels, Ilse Catharina Sophia, Huygens, Martine Wilhelmina Johanna, Schoenmakers, Tim M, Oude Nijeweme-D'Hollosy, Wendy, van Velsen, Lex, Vermeulen, Joan, Schoone-Harmsen, Marian, Jansen, Yvonne JFM +3 more·2018·Journal of Medical Internet Research
DOI 10.2196/jmir.9110PMC PMC5897624

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

BACKGROUND: Electronic health (eHealth) solutions are considered to relieve current and future pressure on the sustainability of primary health care systems. However, evidence of the effectiveness of eHealth in daily practice is missing. Furthermore, eHealth solutions are often not implemented structurally after a pilot phase, even if successful during this phase. Although many studies on barriers and facilitators were published in recent years, eHealth implementation still progresses only slowly. To further unravel the slow implementation process in primary health care and accelerate the implementation of eHealth, a 3-year Living Lab project was set up. In the Living Lab, called eLabEL, patients, health care professionals, small- and medium-sized enterprises (SMEs), and research institutes collaborated to select and integrate fully mature eHealth technologies for implementation in primary health care. Seven primary health care centers, 10 SMEs, and 4 research institutes participated. OBJECTIVE: This viewpoint paper aims to show the process of adoption of eHealth in primary care from the perspective of different stakeholders in a qualitative way. We provide a real-world view on how such a process occurs, including successes and failures related to the different perspectives. METHODS: Reflective and process-based notes from all meetings of the project partners, interview data, and data of focus groups were analyzed systematically using four theoretical models to study the adoption of eHealth in primary care. RESULTS: The results showed that large-scale implementation of eHealth depends on the efforts of and interaction and collaboration among 4 groups of stakeholders: patients, health care professionals, SMEs, and those responsible for health care policy (health care insurers and policy makers). These stakeholders are all acting within their own contexts and with their own values and expectations. We experienced that patients reported expected benefits regarding the use of eHealth for self-management purposes, and health care professionals stressed the potential benefits of eHealth and were interested in using eHealth to distinguish themselves from other care organizations. In addition, eHealth entrepreneurs valued the collaboration among SMEs as they were not big enough to enter the health care market on their own and valued the collaboration with research institutes. Furthermore, health care insurers and policy makers shared the ambition and need for the development and implementation of an integrated eHealth infrastructure. CONCLUSIONS: For optimal and sustainable use of eHealth, patients should be actively involved, primary health care professionals need to be reinforced in their management, entrepreneurs should work closely with health care professionals and patients, and the government needs to focus on new health care models stimulating innovations. Only when all these parties act together, starting in local communities with a small range of eHealth tools, the potential of eHealth will be enforced.

AI snapshot

Written from the paper’s full text.

TL;DR

This viewpoint/process evaluation reports lessons from the 3-year Dutch eLabEL Living Lab, in which 7 primary health care centers, 10 SMEs, and 4 research institutes collaborated to select and integrate mature eHealth tools into primary care. It concludes that broad eHealth adoption requires coordinated action by patients, health care professionals, entrepreneurs, and health care insurers/policy makers, but the project could not achieve implementation at that time because of misaligned incentives, limited readiness, and unresolved business models.

Key findings

  • Large-scale implementation of eHealth depends on the efforts of and interaction and collaboration among 4 stakeholder groups: patients, health care professionals, SMEs, and those responsible for health care policy (health care insurers and policy makers); each acts within its own context, values, and expectations. Despite collaboration, the authors conclude it was not possible to implement eHealth in these Living Labs at that moment.
  • Patients: 30 patients with a chronic disease participated in 5 semistructured focus groups (mean age 68 years; 73% [22/30] male). They reported expected benefits of eHealth for self-management, but many did not feel a need to use it, and frequent monitoring could make them feel more like a patient. Active patient involvement was achieved only to a minor extent, and recruitment by health care professionals was difficult.
  • Health care professionals: 30 health care professionals were interviewed (9 general practitioners, 8 physical therapists, 8 nurse practitioners, and 5 supporting staff members), and 8 care managers were interviewed in 2016. They stressed potential benefits and were interested in using eHealth to distinguish themselves from other care organizations, but needed instruction material, a helpdesk, time, workflow/role changes, integration into electronic medical records or protocols, support motivating patients, and funding; they also described a “wrong pocket problem” and high time pressure.
  • SMEs and insurers/policy makers: 10 SMEs valued collaboration but lacked technical knowledge for a sustainable interoperable ICT infrastructure, and their applications were neither fully mature nor evidence based; they struggled to create a shared vision and a positive business case. Health care insurers and policy makers shared the ambition for an integrated eHealth infrastructure, but insurers were reserved and needed a business case and insight into cost-effectiveness before reimbursement or investment, which the project could not achieve.

Methods

  • Qualitative design: the authors systematically analyzed reflective and process-based notes from meetings with health care professionals, scientific project members, management team members, societal organizations, health care insurers, and enterprises, plus interview and focus group data on needs, expectations, adoption, and implementation.
  • Sample and setting: the eLabEL project ran from September 2013 until December 2016 in the Netherlands with 7 primary health care centers, 10 SMEs, and 4 research institutes. Data included 30 patients with diabetes, COPD, or a cardiovascular condition in 5 focus groups; 30 health care professionals interviewed in the first months; and 8 care managers interviewed in 2016.
  • Analysis: a coding scheme based on four theoretical models (Wagemakers 2010; Nystrom 2014; Geels 2002; Fleuren et al 2004) was used. A qualitative summative process evaluation was performed 21 months after the start and at the end, with member checking by SMEs and health care professionals and observation by an action researcher.

Limitations

  • Stated: the main limitation was the involvement of patients, health care insurers, and policy makers. Patients were actively involved at the start, but slow progress in development and implementation changed the project focus and resulted in less patient involvement; health care insurers and policy makers were not part of the Living Lab and were therefore less involved.
  • Stated: the authors did not aim to test relationships and interactions between different factors and stakeholders, but rather to show the adoption process qualitatively from different stakeholder perspectives.
  • Evident from the text: the study was a single Dutch Living Lab with 7 primary care centers recruited via research partners’ networks or a recruitment call, and the project could not produce the cost-effectiveness evidence insurers requested; these features limit generalizability and leave the financial/business-case impasse unresolved.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.2196/jmir.9110

Cites

30 of 30 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 35 cited sources checked · 0 supported · 0 flagged

  • Findingnone given

    Active patient involvement was only achieved to a minor extent.

    “However, active patient involvement was only achieved to a minor extent.”
    Quote in the paperSubstantial claim
  • Findingown data

    eHealth was not integrated into the electronic medical records or protocols.

    “Moreover, eHealth was not integrated into the electronic medical records or protocols.”
    Quote in the paperSubstantial claim
  • Findingreasoning

    Throughout the project, actively involving patients to incorporate the patient perspective became of secondary importance.

    “Furthermore, throughout the project, actively involving patients to incorporate the patient perspective in the project became of secondary importance.”
    Quote in the paperSubstantial claim
  • Findingown data

    Health care insurers and policy makers shared the ambition and need for the development and implementation of an integrated eHealth infrastructure.

    “Furthermore, health care insurers and policy makers shared the ambition and need for the development and implementation of an integrated eHealth infrastructure.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Health care professionals expected and experienced problems regarding motivating patients to use eHealth.

    “Furthermore, health care professionals expected and experienced problems regarding motivating patients to use eHealth.”
    Quote in the paperSubstantial claim
  • May combine two claims. 33
  • Classified as probably not a substantial claim. 39

  • May combine two claims.

    Findingreasoning

    The focus of eLabEL shifted toward the development of an integrated eHealth structure and the investigation of barriers for its slow development and implementation.

    “The focus of eLabEL shifted toward the development of an integrated eHealth structure and the investigation of barriers for its slow development and implementation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    eHealth entrepreneurs valued the collaboration among SMEs as they were not big enough to enter the health care market on their own and valued the collaboration with research institutes.

    “eHealth entrepreneurs valued the collaboration among SMEs as they were not big enough to enter the health care market on their own and valued the collaboration with research institutes.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Health care professionals stressed the potential benefits of eHealth and were interested in using eHealth to distinguish themselves from other care organizations.

    “health care professionals stressed the potential benefits of eHealth and were interested in using eHealth to distinguish themselves from other care organizations.”
    • Also stated in: § Health Care Professionals · plain
      “The participating health care professionals stressed the potential benefits of eHealth.”
    • Also stated in: § Health Care Professionals · hedged
      “In addition, care professionals indicated that they were interested in using eHealth to distinguish themselves from other care organizations.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Health care professionals needed support for eHealth usage, including clear instruction material, a helpdesk, and time to gain experience with eHealth.

    “First, health care professionals needed support for eHealth usage, including clear instruction material, a helpdesk, and, most importantly, time to gain experience with eHealth, as they had not worked with the selected eHealth applications previously.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Health care insurers and policy makers agreed that it was necessary to work on an integrated infrastructure for eHealth applications to transform traditional primary health care into technology-supported health care.

    “Time after time it was clear that we had a shared ambition: health care insurers and policy makers agreed that it was necessary to work on an integrated infrastructure for eHealth applications to transform traditional primary health care into technology-supported health care.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    SMEs were willing to collaborate and to invest because they were ready to step into a new market.

    “Nevertheless, SMEs were willing to collaborate and to invest as they were ready to step into a new market.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Health care professionals believed that by using eHealth for people with mild conditions, they could save time and provide extra time to those with more severe conditions.

    “Health care professionals believed that by using eHealth for people with mild conditions, they could save time and provide extra time to those with more severe conditions.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingown data

    In the current financial model, professionals had to pay the costs (time and money) for eHealth implementation, while the health insurer would receive the proposed benefits in terms of cost reductions (wrong pocket problem).

    “Professionals mentioned that in the current financial model, they had to pay the costs (time and money) for eHealth implementation, while the health insurer would receive the proposed benefits in terms of cost reductions (also known as the wrong pocket problem).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Funding for support activities was needed but was not covered by the budget for the project.

    “For these support activities funding was needed, which was not covered by the budget for the project.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Entrepreneurs valued intensive collaboration among SMEs, resulting in small alliances of 2 or 3 SMEs, and the experiences of participating in the project as a whole.

    “They valued the intensive collaboration among the SMEs resulting in small alliances of 2 or 3 SMEs, as well as the experiences of participating in the project as a whole.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The project could not achieve a business case and cost-effectiveness insight, and therefore health care insurers did not want to support the project.

    “We could not achieve this in the project and therefore, they did not want to support the project.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The main reason for struggles in cooperation among SMEs was that SMEs differed in their convictions of future revenues because of uncertainties in the financial market and that it was not possible to make a positive business case.

    “The main reason for the struggles experienced in the cooperation among SMEs was that the SMEs differed in their convictions of future revenues because of uncertainties in the financial market and that it was not possible to make a positive business case.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Each SME considered an individual investment in the eHealth infrastructure unwise and too risky, and therefore they opted for a joint investment.

    “An individual investment in the eHealth infrastructure was considered as unwise and too risky by each SME, and therefore, they opted for a joint investment.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingown data

    The eLabEL project gathered in-depth information about the complexity of innovations in primary health care that can help researchers, entrepreneurs, and policy makers in setting up next initiatives.

    “However, we gathered in-depth information about the complexity of innovations in primary health care that can help many researchers, entrepreneurs, and policy makers in setting up the next initiatives on this topic.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The entrepreneurs' eHealth applications were neither fully mature nor evidence based.

    “and that their eHealth applications were neither fully mature nor evidence based.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Providing extramural care, monitoring patients at a distance, empowering and supporting self-management of patients, providing more intensive care in less time, providing care during out-of-office hours, and increasing the quality of care, were frequently mentioned anticipated benefits of eHealth.

    “Providing extramural care, monitoring patients at a distance, empowering and supporting self-management of patients, providing more intensive care in less time, providing care during out-of-office hours, and increasing the quality of care, were frequently mentioned anticipated benefits of eHealth.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Health care professionals experienced high time pressure in regular care processes and in keeping up with bureaucratic and legal changes, resulting in a lack of time to adopt eHealth.

    “In addition, health care professionals already experienced a high time pressure in regular care processes and in keeping up with bureaucratic and legal changes, resulting in a lack of time to adopt eHealth.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Professionals identified the rising development of eHealth technologies, the emergence of different eHealth initiatives, and their opportunities for better health care.

    “Professionals identified the rising development of eHealth technologies, the emergence of different eHealth initiatives, and their opportunities for better health care.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingnone given

    Implementation of eHealth is not yet a fully recognized aspect of primary health care organizations, which makes it difficult to fit eHealth locally.

    “The implementation of eHealth is not yet a fully recognized aspect of primary health care organizations, which makes it difficult to fit eHealth locally.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Successful implementation of eHealth asks for a complex innovation approach.

    “It is suggested that successful implementation of eHealth asks for a complex innovation approach [6].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [6] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    eHealth solutions are expected to empower patients and maintain or improve health outcomes, while generating cost-effective gains and lowering primary health care professionals' workload.

    “Electronic health (eHealth) solutions are expected to empower patients and maintain or improve health outcomes, while generating cost-effective gains and lowering primary health care professionals’ workload [1,2].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [1] Henderson, C, 2013, Br Med J source

      Not checked in the free run.

    • cites [2] source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Evaluations of eHealth should not focus exclusively on measuring outcome variables (via randomized controlled trials) but should also include in-depth process data concerning the usage of eHealth.

    “Van Gemert-Pijnen et al (2011) suggest that evaluations should not focus exclusively on measuring outcome variables (via randomized controlled trials) but should also include in-depth process data concerning the usage of eHealth [7].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [7] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Patient involvement does not always guarantee usage of specific eHealth technologies on a broad scale because not every patient seems willing to use eHealth.

    “In addition, we found that patient involvement does not always guarantee usage of specific eHealth technologies on a broad scale because not every patient seems willing to use eHealth.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Numerous factors are related to the success of eHealth implementation, including characteristics of the end users, the function and usability of the intervention, the technical infrastructure, change management of health care organizations, the health care system, and financial business models.

    “Numerous factors are related to its success, including characteristics of the end users, the function and usability of the intervention, the technical infrastructure, change management of health care organizations, the health care system, and financial business models [2,4,6,7].”
    Quote in the paperSubstantial claim0 of 4 sources checked
    • cites [2] source

      Not checked in the free run.

    • cites [4] source

      Not checked in the free run.

    • cites [6] source

      Not checked in the free run.

    • cites [7] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The model of Geels describes new technologies as arising and maturing within existing technology systems.

    “The model of Geels (2002) describes new technologies as arising and maturing within existing technology systems [18].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [18] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The success rate of an innovation is dependent on the level of the innovation itself, end users, organization, and the social-political context.

    “Fleuren et al (2004) state that the success rate of an innovation is dependent on the level of the innovation itself, end users, organization, and the social-political context [19].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [19] source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Relevant stakeholders should collaborate, and research should consist of qualitative and quantitative elements.

    “Van Gemert-Pijnen et al (2011) stated that relevant stakeholders should collaborate, and research should consist of qualitative and quantitative elements [7].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [7] source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    A multidisciplinary development approach is necessary for eHealth.

    “Furthermore, according to Van Velsen and colleagues (2013), a multidisciplinary development approach is necessary [11].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [11] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    The organization of primary health care was inadequate and not sufficiently equipped for health care professionals to implement and use eHealth without the help of the research team.

    “However, we found that the organization of primary health care was inadequate and not sufficiently equipped for doing so as we explain in the upcoming section.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Wagemakers' model focuses on collaboration among multidisciplinary organizations in health care.

    “Wagemakers’ model (2010) focuses on collaboration among multidisciplinary organizations in health care [16].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [16] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Four groups of stakeholders are responsible for a successful implementation of eHealth solutions: patients, health care professionals, entrepreneurs, and those responsible for health care policy (policy makers and health care insurers).

    “It can be stated that 4 groups of stakeholders are responsible for a successful implementation of eHealth solutions: patients, health care professionals, entrepreneurs, and those responsible for health care policy (policy makers and health care insurers) [8].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [8] source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Nystrom's model focuses on different role approaches within a collaboration.

    “Nystrom’s model (2014) was used because of its focus on different role approaches within a collaboration [17].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [17] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationnone given

    Whether patients will actually use eHealth is personal and differs among patients.

    “However, whether patients will actually use eHealth will always be personal and differ among patients.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Incentives for care practices seem to be needed to recruit patients for participation in eHealth projects.

    “In addition, incentives for care practices seem to be needed to recruit patients for participation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The eLabEL project had unrealistic expectations and goals that were not specific enough.

    “we realize we had unrealistic expectations and our goals were not specific enough.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationreasoning

    Our paper serves as an illustration that underlines the importance of including all 4 stakeholders, having a shared vision statement, and enabling all partners to invest time or money, as only then can the expected potential of eHealth solutions be reached.

    “Our paper serves as an illustration that underlines the importance of including all 4 stakeholders, having a shared vision statement, and enabling all partners to invest time or money, as only then can the expected potential of eHealth solutions be reached.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Without commitment of health care centers or insurers, no positive business model could be created, and SMEs could not invest in the eLabEL infrastructure.

    “In fact, it was those factors that led to an impasse: without commitment of health care centers or insurers, no positive business model could be created, and SMEs could not invest in the eLabEL infrastructure.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Only when all parties (patients, health care professionals, entrepreneurs, government) act together, starting in local communities with a small range of eHealth tools, the potential of eHealth will be enforced.

    “Only when all these parties act together, starting in local communities with a small range of eHealth tools, the potential of eHealth will be enforced.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Collaboration between all 4 groups of stakeholders-patients, health care professionals, entrepreneurs, and health care insurers or policy makers-is essential for eHealth.

    “we still believe that collaboration between all 4 groups of stakeholders, that is, patients, health care professionals, entrepreneurs, and health care insurers or policy makers is essential.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning

    Health care insurers and policy makers were not part of the Living Lab and were therefore less involved in the project.

    “Health care insurers and policy makers were not part of the Living Lab and were therefore less involved in the project.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Implementing eHealth requires it to be a fundamental part of the mission and vision of the health care organization.

    “Implementing eHealth requires it to be a fundamental part of the mission and vision of the health care organization.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    Greater knowledge of essential adjustments in health care provider workflow, roles, and responsibilities is needed.

    “Moreover, it is shown that a greater knowledge of essential adjustments in health care provider workflow, roles, and responsibilities is needed [29].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [29] source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductown data

    The study systematically analyzed reflective and process-based notes from meetings, interview data, and focus group data using four theoretical models.

    “Reflective and process-based notes from all meetings of the project partners, interview data, and data of focus groups were analyzed systematically using four theoretical models to study the adoption of eHealth in primary care.”
    • Also stated in: § Methods · plain
      “We systematically analyzed all reflective and process-based notes from meetings with health care professionals, the scientific project members, members of the management team of the Centre for Care Technology research, members of societal organizations, health care insurers, and enterprises.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Dutch regulations and legislation seemed to act inconsistently: they argue to stimulate eHealth on one side but require cost-effectiveness studies first on the other side.

    “In actual practice, the Dutch regulations and legislation seemed to act inconsistently: they argue to stimulate eHealth on one side but require cost-effectiveness studies first on the other side.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Wildevuur et al (2017) published 4 preconditions for enhancing partnership in ICT-enabled person-centered care.

    “Wildevuur et al (2017) recently published 4 preconditions for enhancing the partnership in ICT-enabled person-centered care [21].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [21] source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationown data

    Successful use of eHealth depends on the efforts of all stakeholders, their willingness to invest time or money, and shared vision statements.

    “Our experiences in eLabEL taught us that successful use of eHealth needs more than enthusiastic partners. Successful use also depends on the efforts of all stakeholders, their willingness to invest time or money, and shared vision statements.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    A coding scheme based on four theoretical models was used to structure the findings.

    “For the analyses, we used a coding scheme based on four theoretical models to initially structure our findings.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationnone given

    For a change to technology-supported health care, not only a technological innovation but also a societal innovation is essential.

    “For such a change, not only a technological innovation but also a societal innovation is essential.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Lau et al (2016) found that implementing any type of change in primary care is likely to be complex and that relevant barriers and facilitators are dynamic and likely to change over time.

    “Lau et al (2016) stated in a recent systematic review of reviews that implementing any type of change in primary care is likely to be complex and that relevant barriers and facilitators are dynamic and likely to change over time [25].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [25] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Investments by health care organizations in eHealth will lead to lower costs for health care insurers but also lower income for health care practices.

    “Actually, investments by the health care organizations will lead to lower costs for the health care insurers but also lower income for the health care practices.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The Dutch health care system is pay-for-performance-based.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    A summary of the findings of both rounds of analyses was shared with the SMEs and health care professionals for their confirmation (member check procedure).

    “A summary of the findings of both rounds of analyses was shared with the SMEs and health care professionals for their confirmation (member check procedure).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Each summary was checked by a researcher of another research institute.

    “Each summary was then checked by 1 researcher of another research institute.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Implementation of innovations, including eHealth, is difficult and progresses only slowly.

    “Nowadays, it is common knowledge that implementation of innovations, including eHealth, is difficult and progresses only slowly [25-27].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [25] source

      Not checked in the free run.

    • cites [26] source

      Not checked in the free run.

    • cites [27] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    YJ observed the project as action researcher.

    “Furthermore, YJ has observed the project as action researcher.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisenone given

    Use of eHealth goes beyond the own practice borders because primary health care professionals often operate close to other health care professionals in their region.

    “The use of eHealth goes beyond the own practice borders as primary health care professionals often operate close to other health care professionals in their region.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Primary health care professionals often operate close to other health care professionals in their region.

  • Classified as probably not a substantial claim.

    Premisecitation

    Theoretical models show that the innovation process or implementation infrastructure are important parts of implementation, next to intervention characteristics, organizational structure, context, and individuals.

    “Theoretical models show that the innovation process or implementation infrastructure are important parts of implementation, next to the intervention characteristics, the organizational structure, the context, and the individuals [8,28].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [8] source

      Not checked in the free run.

    • cites [28] Feldstein, A, 2008, Jt Comm J Qual Patient Saf source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    To remain competitive within a fast-moving market, it is important to develop quickly.

    “To remain competitive within a fast-moving market, it is important to develop quickly [31].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [31] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationcitation

    Broad-scale implementation of eHealth will need strategies that focus not only on financial models that stimulate innovation but also on requirements needed for societal innovations.

    “Broad-scale implementation will need strategies that not only focus on financial models that stimulate innovation but also on requirements needed for societal innovations [33,34].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [33] Janssen, M, 2016, Situated novelty: a study on healthcare innovation and its governance: Health Care Governance (HCG) PhD dissertation

      Not checked in the free run.

    • cites [34] Krijgsman, J, 2016, eHealth-monitor 2016

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Experiments allowing standard regulations to be partly neglected can only be arranged when policy makers and health care insurers are involved from the beginning of the project.

    “This can only be arranged when policy makers and health care insurers are involved from the beginning of the project.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Active user involvement is a time-consuming process.

    “Active user involvement is a time-consuming process.”
    Quote in the paperNot a substantial claim
60 of 60

  • The paper relies on this without support.

    Literature combining the views of the different fields (patients, health care professionals, entrepreneurs, policy makers) and describing their challenges systematically is scarce.

    “However, literature combining the views of these different fields and describing their challenges systematically is scarce.”

  • The paper relies on this without support.

    There was very little knowledge about what it takes to bring eHealth technologies into practice.

    “Simultaneously, there was very little knowledge about what it takes to bring such technologies into practice.”

  • The authors state this as an assumption.

    Incorporating eHealth into daily practice is essential for optimal effects on quality and efficiency of health care.

    “We believed, and still believe, that incorporating eHealth into daily practice is essential for optimal effects on quality and efficiency of health care.”

  • The paper relies on this without support.

    Use of eHealth services often stops when research projects are finished, even when successful.

    “Often, use of eHealth services stops when research projects are finished, even when successful.”

  • The paper relies on this without support.

    Current evidence on eHealth and care technologies is mainly based on clinical trials and isolated eHealth applications.

    “Current evidence on eHealth and care technologies is mainly based on clinical trials and isolated eHealth applications.”

  • The paper relies on this without support.

    The Dutch health care system is pay-for-performance-based.

    “However, in the Dutch pay-for-performance-based health care system, the use of eHealth applications that lower the number of consults will also lower the health care professionals’ revenues.”

  • The paper relies on this without support.

    The primary health care market is new and relatively unknown for SMEs.

    “as the primary health care market was a new and therefore relatively unknown market for the SMEs.”

  • The paper relies on this without support.

    Primary health care professionals often operate close to other health care professionals in their region.

    “as primary health care professionals often operate close to other health care professionals in their region.”

  • The paper relies on this without support.

    Dutch and European requirements on data safety, data exchange, and data privacy are different.

    “when facing different Dutch and European requirements on data safety, data exchange, and data privacy.”
9 of 9

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Research on the experiences of stakeholders in a policy role is scarce, as is knowledge about the overlap between policy and public (service user) perspectives.

    “Although several recent studies have identified factors that influence the perceptions of stakeholders in different populations [10-12], research on the experiences of stakeholders in a policy role is scarce, as is knowledge about the overlap between policy and public (service user) perspectives.”

    From Perspectives of Policy Makers and Service Users Concerning the Implementation of eHealth in Sweden: Interview Study (2022) · cites this paper as [11]

Cites

Flagged first, then by citations

Research works printed but not linked: 3

  1. VermeulenJHuygensMde WitteLOude Nijeweme-d'HollosyWvan VelsenLSwinkelsIJansenYJeLabEL: Technology-supported Living Labs in Primary CareeTELEMED 2015: The Seventh International Conference on eHealth, Telemedicine, and Social Medicine2015eTelemed22-27 februaryLisbon PortugaleTELEMED 2015IARIA
  2. Janssen, M, 2016, Situated novelty: a study on healthcare innovation and its governance: Health Care Governance (HCG) PhD dissertation
  3. Krijgsman, J, 2016, eHealth-monitor 2016

Cited by

Sorted by citations

Related

Authors
Swinkels, I. C. S., Huygens, M. W. J., Schoenmakers, T. M., Oude Nijeweme-D'Hollosy, W., van Velsen, L., Vermeulen, J., Schoone-Harmsen, M., Jansen, Y. J., van Schayck, O. C., Friele, R., & de Witte, L.
Affiliations
  • Netherlands Institute for Health Services Research (NL)
  • Maastricht University (NL)
  • Netherlands Organisation for Applied Scientific Research (NL)
  • Tilburg University (NL)
Corresponding author
Swinkels, Ilse Catharina Sophia
ORCID
Venue
Journal of Medical Internet Research
Publisher
JMIR Publications Inc.
Published
2018-03-29
Volume
20
Issue
3
Pages
e83
ISSN
1438-8871 (ISSN-L)
DOI
10.2196/jmir.9110
PMC
PMC5897624
OpenAlex
W2794553894
Semantic Scholar
c5c2e1bca8c74cfd19656125728ded6a3c0bd9b0
Cited by
90
References
30
Citations by year
2026: 10 · 2025: 8 · 2024: 6 · 2023: 12 · 2022: 10 · 2021: 14 · 2020: 14 · 2019: 10 · 2018: 6
FWCI
13.88
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Journal of Medical Internet Research (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • Research portal (Tilburg University) (publishedVersion, other-oa)
  • PubMed (publishedVersion)
  • Research Publications (Maastricht University) (publishedVersion, cc-by)
  • University of Twente Research Information (publishedVersion, cc-by)
  • White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
  • UvA-DARE (University of Amsterdam) (submittedVersion, cc-by)
  • TNO Repository (submittedVersion, cc-by)
  • Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
  • Repository hosted by TU Delft Library (TU Delft) (submittedVersion)
  • Research Publications (Maastricht University) (submittedVersion, cc-by)
  • University of Twente Research Information (submittedVersion, cc-by)
Topics
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Telemedicine and Telehealth Implementation
  • Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
Keywords
digital health adoption, primary care, eHealth implementation, living labs, stakeholder collaboration, patient involvement, healthcare workers, small and medium-sized enterprises, health policy, self-management, implementation barriers and facilitators, research institute collaboration
MeSH terms
Delivery of Health Care, Humans, Laboratories, Primary Health Care, Telemedicine

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review