Evidence-Based Evaluation of eHealth Interventions: Systematic Literature Review
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Until now, the use of technology in health care was driven mostly by the assumptions about the benefits of electronic health (eHealth) rather than its evidence. It is noticeable that the magnitude of evidence of effectiveness and efficiency of eHealth is not proportionate to the number of interventions that are regularly conducted. Reliable evidence generated through comprehensive evaluation of eHealth interventions may accelerate the growth of eHealth for long-term successful implementation and help to experience eHealth benefits in an enhanced way. OBJECTIVE: This study aimed to understand how the evidence of effectiveness and efficiency of eHealth can be generated through evaluation. Hence, we aim to discern (1) how evaluation is conducted in distinct eHealth intervention phases, (2) the aspects of effectiveness and efficiency that are typically evaluated during eHealth interventions, and (3) how eHealth interventions are evaluated in practice. METHODS: A systematic literature review was conducted to explore the evaluation methods for eHealth interventions. Preferred reporting items for systematic reviews and meta-analyses (PRISMA) guidelines were followed. We searched Google Scholar and Scopus for the published papers that addressed the evaluation of eHealth or described an eHealth intervention study. A qualitative analysis of the selected papers was conducted in several steps. RESULTS: We intended to see how the process of evaluation unfolds in distinct phases of an eHealth intervention. We revealed that in practice and in several conceptual papers, evaluation is performed at the end of the intervention. There are some studies that discuss the importance of conducting evaluation throughout the intervention; however, in practice, we found no case study that followed this. For our second research question, we discovered aspects of efficiency and effectiveness that are proposed to be assessed during interventions. The aspects that were recurrent in the conceptual papers include clinical, human and social, organizational, technological, cost, ethical and legal, and transferability. However, the case studies reviewed only evaluate the clinical and human and social aspects. At the end of the paper, we discussed a novel approach to look into the evaluation. Our intention was to stir up a discussion around this approach with the hope that it might be able to gather evidence in a comprehensive and credible way. CONCLUSIONS: The importance of evidence in eHealth has not been discussed as rigorously as have the diverse evaluation approaches and evaluation frameworks. Further research directed toward evidence-based evaluation can not only improve the quality of intervention studies but also facilitate successful long-term implementation of eHealth in general. We conclude that the development of more robust and comprehensive evaluation of eHealth studies or an improved validation of evaluation methods could ease the transferability of results among similar studies. Thus, the resources can be used for supplementary research in eHealth.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Evaluating the public health impact of health promotion interventions: the RE-AIM framework.
Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999) · American Journal of Public Health · doi:10.2105/ajph.89.9.1322 · Cited by 7,217
Realist review - a new method of systematic review designed for complex policy interventions
Pawson, R., Greenhalgh, T., Harvey, G., & Walshe, K. (2005) · Journal of Health Services Research & Policy · doi:10.1258/1355819054308530 · Cited by 3,190 · cited 1×
“Evaluation of eHealth interventions is complex because of several reasons (eg, the need for multidisciplinary collaboration [8], context dependency [9], and differences in epistemological beliefs considering the interventions in clinical studies or including social aspects as well [10-14]).”
BackgroundEysenbach, G. (2001) · Journal of Medical Internet Research · doi:10.2196/jmir.3.2.e20 · Cited by 2,449 · cited 5×
“According to Eysenbach [5], “e-health is an emerging field in the intersection of medical informatics, public health and business, referring to health services and information delivered or enhanced through the Internet and related technologies” (p.”
BackgroundInfluentialThe Impact of eHealth on the Quality and Safety of Health Care: A Systematic Overview
Black, A. D., Car, J., Pagliari, C., Anandan, C., Cresswell, K., Bokun, T., McKinstry, B., Procter, R., Majeed, A., & Sheikh, A. (2011) · PLoS Medicine · doi:10.1371/journal.pmed.1000387 · Cited by 1,442
What Is eHealth (3): A Systematic Review of Published Definitions
Oh, H., Jadad, A., Rizo, C., Enkin, M., Powell, J., & Pagliari, C. (2005) · Journal of Medical Internet Research · doi:10.2196/jmir.7.1.e1 · Cited by 1,015 · cited 2×
“The term eHealth was introduced in the 1990s [4]; however, it was hardly in use until 1999 [5].”
BackgroundYusof, M. M., Kuljis, J., Papazafeiropoulou, A., & Stergioulas, L. K. (2008) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2007.08.011 · Cited by 740 · cited 3×
“Acceptance and usability satisfaction of the technology used in the intervention [30,31,33,36,38,39,41] where the user can be physicians, nurses and other staff, and patients, depending on the type of the participants in the intervention [41]; system use: volume of use, who is using, purpose of use, and motivation to…”
BackgroundHarnessing context sensing to develop a mobile intervention for depression
Burns, M. N., Begale, M., Duffecy, J., Gergle, D., Karr, C. J., Giangrande, E., & Mohr, D. C. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1838 · Cited by 646 · cited 2×
“Multimedia Appendix 2 provides the result of the analysis [20,42-65].”
BackgroundEvaluating eHealth Interventions: The Need for Continuous Systemic Evaluation
Catwell, L., & Sheikh, A. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000126 · Cited by 307
Design and Evaluation in eHealth: Challenges and Implications for an Interdisciplinary Field
Pagliari, C. (2007) · Journal of Medical Internet Research · doi:10.2196/jmir.9.2.e15 · Cited by 275
Why Do Evaluations of eHealth Programs Fail? An Alternative Set of Guiding Principles
Greenhalgh, T., & Russell, J. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000360 · Cited by 273 · cited 1×
“Evaluation of eHealth interventions is complex because of several reasons (eg, the need for multidisciplinary collaboration [8], context dependency [9], and differences in epistemological beliefs considering the interventions in clinical studies or including social aspects as well [10-14]).”
Background
Cited by
Timpel, P., Oswald, S., Schwarz, P. E. H., & Harst, L. (2020) · Journal of Medical Internet Research · doi:10.2196/16791 · Cited by 243
Fernández, C. E., Maturana, C. A., Coloma, S. I., Carrasco-Labra, A., & Giacaman, R. A. (2021) · Journal of Dental Research · doi:10.1177/00220345211003828 · Cited by 152
Digital Self-Management in Support of Patients Living With Chronic Pain: Feasibility Pilot Study
Bostrøm, K., Børøsund, E., Varsi, C., Eide, H., Flakk Nordang, E., Schreurs, K. M., Waxenberg, L. B., Weiss, K. E., Morrison, E. J., Cvancarova Småstuen, M., et al. (2020) · JMIR Formative Research · doi:10.2196/23893 · Cited by 34
“As suggested by scientific literature reviews examining status and limitations of eHealth interventions [20,32,34], the EPIO intervention program was developed based on existing evidence, in close collaboration among scientists, user representatives, and health care providers [26], which may have contributed to the po…”
Backgroundvan der Kamp, M., Reimering Hartgerink, P., Driessen, J., Thio, B., Hermens, H., & Tabak, M. (2021) · JMIR Formative Research · doi:10.2196/24634 · Cited by 19
Pros and cons of eHealth: A systematic review of the literature and observations in Denmark
Svendsen, M. T., Tiedemann, S. N., & Andersen, K. E. (2021) · SAGE Open Medicine · doi:10.1177/20503121211016179 · Cited by 20
Metadata
- Authors
- Enam, A., Torres-Bonilla, J., & Eriksson, H.
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2018-11-23
- Volume
- 20
- Issue
- 11
- Pages
- e10971
- ISSN
- 1438-8871 (ISSN-L)
- OpenAlex
- W2887033287
- Cited by
- 151
- References
- 25
- FWCI
- 3.15
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Evidence-Based Evaluation of eHealth Interventions: Systematic Literature Review. https://app.cochat.ai/papers/10.2196/10971
- The paper
- Enam, A., Torres-Bonilla, J., & Eriksson, H. (2018). Evidence-Based Evaluation of eHealth Interventions: Systematic Literature Review. Journal of Medical Internet Research. https://doi.org/10.2196/10971
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.