Evidence-Based Evaluation of eHealth Interventions: Systematic Literature Review
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Until now, the use of technology in health care was driven mostly by the assumptions about the benefits of electronic health (eHealth) rather than its evidence. It is noticeable that the magnitude of evidence of effectiveness and efficiency of eHealth is not proportionate to the number of interventions that are regularly conducted. Reliable evidence generated through comprehensive evaluation of eHealth interventions may accelerate the growth of eHealth for long-term successful implementation and help to experience eHealth benefits in an enhanced way. OBJECTIVE: This study aimed to understand how the evidence of effectiveness and efficiency of eHealth can be generated through evaluation. Hence, we aim to discern (1) how evaluation is conducted in distinct eHealth intervention phases, (2) the aspects of effectiveness and efficiency that are typically evaluated during eHealth interventions, and (3) how eHealth interventions are evaluated in practice. METHODS: A systematic literature review was conducted to explore the evaluation methods for eHealth interventions. Preferred reporting items for systematic reviews and meta-analyses (PRISMA) guidelines were followed. We searched Google Scholar and Scopus for the published papers that addressed the evaluation of eHealth or described an eHealth intervention study. A qualitative analysis of the selected papers was conducted in several steps. RESULTS: We intended to see how the process of evaluation unfolds in distinct phases of an eHealth intervention. We revealed that in practice and in several conceptual papers, evaluation is performed at the end of the intervention. There are some studies that discuss the importance of conducting evaluation throughout the intervention; however, in practice, we found no case study that followed this. For our second research question, we discovered aspects of efficiency and effectiveness that are proposed to be assessed during interventions. The aspects that were recurrent in the conceptual papers include clinical, human and social, organizational, technological, cost, ethical and legal, and transferability. However, the case studies reviewed only evaluate the clinical and human and social aspects. At the end of the paper, we discussed a novel approach to look into the evaluation. Our intention was to stir up a discussion around this approach with the hope that it might be able to gather evidence in a comprehensive and credible way. CONCLUSIONS: The importance of evidence in eHealth has not been discussed as rigorously as have the diverse evaluation approaches and evaluation frameworks. Further research directed toward evidence-based evaluation can not only improve the quality of intervention studies but also facilitate successful long-term implementation of eHealth in general. We conclude that the development of more robust and comprehensive evaluation of eHealth studies or an improved validation of evaluation methods could ease the transferability of results among similar studies. Thus, the resources can be used for supplementary research in eHealth.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Evaluating the public health impact of health promotion interventions: the RE-AIM framework.
Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999) · American Journal of Public Health · doi:10.2105/ajph.89.9.1322 · Cited by 7,217
Realist review - a new method of systematic review designed for complex policy interventions
Pawson, R., Greenhalgh, T., Harvey, G., & Walshe, K. (2005) · Journal of Health Services Research & Policy · doi:10.1258/1355819054308530 · Cited by 3,190 · cited 1×
“Evaluation of eHealth interventions is complex because of several reasons (eg, the need for multidisciplinary collaboration [8], context dependency [9], and differences in epistemological beliefs considering the interventions in clinical studies or including social aspects as well [10-14]).”
BackgroundEysenbach, G. (2001) · Journal of Medical Internet Research · doi:10.2196/jmir.3.2.e20 · Cited by 2,449 · cited 5×
“According to Eysenbach [5], “e-health is an emerging field in the intersection of medical informatics, public health and business, referring to health services and information delivered or enhanced through the Internet and related technologies” (p.”
BackgroundInfluentialThe Impact of eHealth on the Quality and Safety of Health Care: A Systematic Overview
Black, A. D., Car, J., Pagliari, C., Anandan, C., Cresswell, K., Bokun, T., McKinstry, B., Procter, R., Majeed, A., & Sheikh, A. (2011) · PLoS Medicine · doi:10.1371/journal.pmed.1000387 · Cited by 1,442
What Is eHealth (3): A Systematic Review of Published Definitions
Oh, H., Jadad, A., Rizo, C., Enkin, M., Powell, J., & Pagliari, C. (2005) · Journal of Medical Internet Research · doi:10.2196/jmir.7.1.e1 · Cited by 1,015 · cited 2×
“The term eHealth was introduced in the 1990s [4]; however, it was hardly in use until 1999 [5].”
BackgroundYusof, M. M., Kuljis, J., Papazafeiropoulou, A., & Stergioulas, L. K. (2008) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2007.08.011 · Cited by 740 · cited 3×
“Acceptance and usability satisfaction of the technology used in the intervention [30,31,33,36,38,39,41] where the user can be physicians, nurses and other staff, and patients, depending on the type of the participants in the intervention [41]; system use: volume of use, who is using, purpose of use, and motivation to…”
BackgroundHarnessing context sensing to develop a mobile intervention for depression
Burns, M. N., Begale, M., Duffecy, J., Gergle, D., Karr, C. J., Giangrande, E., & Mohr, D. C. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1838 · Cited by 646 · cited 2×
“Multimedia Appendix 2 provides the result of the analysis [20,42-65].”
BackgroundEvaluating eHealth Interventions: The Need for Continuous Systemic Evaluation
Catwell, L., & Sheikh, A. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000126 · Cited by 307
Design and Evaluation in eHealth: Challenges and Implications for an Interdisciplinary Field
Pagliari, C. (2007) · Journal of Medical Internet Research · doi:10.2196/jmir.9.2.e15 · Cited by 275
Why Do Evaluations of eHealth Programs Fail? An Alternative Set of Guiding Principles
Greenhalgh, T., & Russell, J. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000360 · Cited by 273 · cited 1×
“Evaluation of eHealth interventions is complex because of several reasons (eg, the need for multidisciplinary collaboration [8], context dependency [9], and differences in epistemological beliefs considering the interventions in clinical studies or including social aspects as well [10-14]).”
BackgroundMobile mental health: Review of the emerging field and proof of concept study
Harrison, V., Proudfoot, J., Wee, P. P., Parker, G., Pavlovic, D. H., & Manicavasagar, V. (2011) · Journal of Mental Health · doi:10.3109/09638237.2011.608746 · Cited by 267
A Development and Evaluation Process for mHealth Interventions: Examples From New Zealand
Whittaker, R., Merry, S., Dorey, E., & Maddison, R. (2012) · Journal of Health Communication · doi:10.1080/10810730.2011.649103 · Cited by 263 · cited 7×
“Administer the intervention to a larger group of participants [21,23,26] with fewer eligibility restrictions [26] Execution Pragmatic trial phase”
BackgroundMethodInfluentialDesigning eHealth that Matters via a Multidisciplinary Requirements Development Approach
Van Velsen, L., Wentzel, J., & Van Gemert-Pijnen, J. E. (2013) · JMIR Research Protocols · doi:10.2196/resprot.2547 · Cited by 262
Leon, N., Schneider, H., & Daviaud, E. (2012) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-12-123 · Cited by 200
Moderated online social therapy: Designing and evaluating technology for mental health
Lederman, R., Wadley, G., Gleeson, J., Bendall, S., & Álvarez-Jiménez, M. (2014) · ACM Transactions on Computer-Human Interaction · doi:10.1145/2513179 · Cited by 151
Glozier, N., Christensen, H., Naismith, S., Cockayne, N., Donkin, L., Neal, B., Mackinnon, A., & Hickie, I. (2013) · PLoS ONE · doi:10.1371/journal.pone.0059139 · Cited by 129 · cited 1×
“[60] CONSORT 12 weeks 562 Qn: Survey with standardized questionnaire (PHQ-9, MOS, GAD-7, IPAQ, WHODAS-II).”
MethodBurton, C., Szentagotai Tatar, A., McKinstry, B., Matheson, C., Matu, S., Moldovan, R., Macnab, M., Farrow, E., David, D., Pagliari, C., et al. (2016) · Journal of Telemedicine and Telecare · doi:10.1177/1357633x15609793 · Cited by 125 · cited 1×
“[62] No 4 weeks 27 Qn: Survey with standardized questionnaire (BDI-2, QIDS-SR, EQ-5D-5L, DAS-SF12).”
MethodKlein, B., Meyer, D., Austin, D. W., & Kyrios, M. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1918 · Cited by 118
Guideline for good evaluation practice in health informatics (GEP-HI)
Nykänen, P., Brender, J., Talmon, J., de Keizer, N., Rigby, M., Beuscart-Zephir, M.-C., & Ammenwerth, E. (2011) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2011.08.004 · Cited by 105
Pham, Q., Khatib, Y., Stansfeld, S., Fox, S., & Green, T. (2016) · Games for Health Journal · doi:10.1089/g4h.2015.0033 · Cited by 104 · cited 1×
“Multimedia Appendix 2 provides the result of the analysis [20,42-65].”
BackgroundEvaluating eHealth: How to Make Evaluation More Methodologically Robust
Lilford, R. J., Foster, J., & Pringle, M. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000186 · Cited by 79
Feasibility of an eHealth Service to Support Collaborative Depression Care: Results of a Pilot Study
Meglic, M., Furlan, M., Kuzmanic, M., Kozel, D., Baraga, D., Kuhar, I., Kosir, B., Iljaz, R., Novak Sarotar, B., Dernovsek, M. Z., et al. (2010) · Journal of Medical Internet Research · doi:10.2196/jmir.1510 · Cited by 71
Lovejoy, T. I., Demireva, P. D., Grayson, J. L., & McNamara, J. R. (2009) · Psychotherapy: Theory, Research, Practice, Training · doi:10.1037/a0015153 · Cited by 64 · cited 1×
“Ensure trust [38], effectiveness, and contribution of quality of care [30,36] of the technology implemented; system performance: hardware and software requirements, correct functioning of the components [29,38], and system capability to meet users’ needs and fit the work patterns of the health care system’ professiona…”
BackgroundExperience with a structure, process and outcome framework for evaluating an information system
Cornford, T., Doukidis, G., & Forster, D. (1994) · Omega · doi:10.1016/0305-0483(94)90030-2 · Cited by 60 · cited 2×
“Organizational setting where the intervention is taking place; it can differ depending on the scale of the intervention (eg, health center, region, and country) [29]; all type of individuals or groups in the health care system that participate in the eHealth intervention, their characteristics, and expectations [30];…”
BackgroundTakian, A., Petrakaki, D., Cornford, T., Sheikh, A., & Barber, N. (2012) · BMC Health Services Research · doi:10.1186/1472-6963-12-105 · Cited by 51
Cited by
Timpel, P., Oswald, S., Schwarz, P. E. H., & Harst, L. (2020) · Journal of Medical Internet Research · doi:10.2196/16791 · Cited by 243
Fernández, C. E., Maturana, C. A., Coloma, S. I., Carrasco-Labra, A., & Giacaman, R. A. (2021) · Journal of Dental Research · doi:10.1177/00220345211003828 · Cited by 152
Digital Self-Management in Support of Patients Living With Chronic Pain: Feasibility Pilot Study
Bostrøm, K., Børøsund, E., Varsi, C., Eide, H., Flakk Nordang, E., Schreurs, K. M., Waxenberg, L. B., Weiss, K. E., Morrison, E. J., Cvancarova Småstuen, M., et al. (2020) · JMIR Formative Research · doi:10.2196/23893 · Cited by 34
“As suggested by scientific literature reviews examining status and limitations of eHealth interventions [20,32,34], the EPIO intervention program was developed based on existing evidence, in close collaboration among scientists, user representatives, and health care providers [26], which may have contributed to the po…”
Backgroundvan der Kamp, M., Reimering Hartgerink, P., Driessen, J., Thio, B., Hermens, H., & Tabak, M. (2021) · JMIR Formative Research · doi:10.2196/24634 · Cited by 19
Pros and cons of eHealth: A systematic review of the literature and observations in Denmark
Svendsen, M. T., Tiedemann, S. N., & Andersen, K. E. (2021) · SAGE Open Medicine · doi:10.1177/20503121211016179 · Cited by 20
Metadata
- Authors
- Enam, A., Torres-Bonilla, J., & Eriksson, H.
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2018-11-23
- Volume
- 20
- Issue
- 11
- Pages
- e10971
- ISSN
- 1438-8871 (ISSN-L)
- OpenAlex
- W2887033287
- Cited by
- 151
- References
- 25
- FWCI
- 3.15
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.