Methods to Evaluate the Effects of Internet-Based Digital Health Interventions for Citizens: Systematic Review of Reviews
Open accessPublished version of a preprint
The DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Digital health can empower citizens to manage their health and address health care system problems including poor access, uncoordinated care and increasing costs. Digital health interventions are typically complex interventions. Therefore, evaluations present methodological challenges. OBJECTIVE: The objective of this study was to provide a systematic overview of the methods used to evaluate the effects of internet-based digital health interventions for citizens. Three research questions were addressed to explore methods regarding approaches (study design), effects and indicators. METHODS: We conducted a systematic review of reviews of the methods used to measure the effects of internet-based digital health interventions for citizens. The protocol was developed a priori according to Preferred Reporting Items for Systematic review and Meta-Analysis Protocols and the Cochrane Collaboration methodology for overviews of reviews. Qualitative, mixed-method, and quantitative reviews published in English or French from January 2010 to October 2016 were included. We searched for published reviews in PubMed, EMBASE, The Cochrane Database of Systematic Reviews, CINHAL and Epistemonikos. We categorized the findings based on a thematic analysis of the reviews structured around study designs, indicators, types of interventions, effects and perspectives. RESULTS: A total of 20 unique reviews were included. The most common digital health interventions for citizens were patient portals and patients' access to electronic health records, covered by 10/20 (50%) and 6/20 (30%) reviews, respectively. Quantitative approaches to study design included observational study (15/20 reviews, 75%), randomized controlled trial (13/20 reviews, 65%), quasi-experimental design (9/20 reviews, 45%), and pre-post studies (6/20 reviews, 30%). Qualitative studies or mixed methods were reported in 13/20 (65%) reviews. Five main categories of effects were identified: (1) health and clinical outcomes, (2) psychological and behavioral outcomes, (3) health care utilization, (4) system adoption and use, and (5) system attributes. Health and clinical outcomes were measured with both general indicators and disease-specific indicators and reported in 11/20 (55%) reviews. Patient-provider communication and patient satisfaction were the most investigated psychological and behavioral outcomes, reported in 13/20 (65%) and 12/20 (60%) reviews, respectively. Evaluation of health care utilization was included in 8/20 (40%) reviews, most of which focused on the economic effects on the health care system. CONCLUSIONS: Although observational studies and surveys have provided evidence of benefits and satisfaction for patients, there is still little reliable evidence from randomized controlled trials of improved health outcomes. Future evaluations of digital health interventions for citizens should focus on specific populations or chronic conditions which are more likely to achieve clinically meaningful benefits and use high-quality approaches such as randomized controlled trials. Implementation research methods should also be considered. We identified a wide range of effects and indicators, most of which focused on patients as main end users. Implications for providers and the health system should also be included in evaluations or monitoring of digital health interventions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“However, these critical reflections consistently stop at that point, providing no concrete, actionable solutions to address the limitations they revealed [54-57].”
From Use of the Principles of Design Thinking to Address Limitations of Digital Mental Health Interventions for Youth: Viewpoint (2019) · cites this paper as [57]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
PRISMA-P Group, Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., & Stewart, L. A. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-1 · Cited by 26,223 · cited 2×
“A systematic review of reviews [18] focused on the methods used to measure the effects of internet-based digital health interventions for citizens was conducted in accordance with the Preferred Reporting Items for Systematic review and Meta-Analysis Protocols (PRISMA-P) [19].”
MethodShamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L. A., & the PRISMA-P Group (2015) · BMJ · doi:10.1136/bmj.g7647 · Cited by 13,120 · cited 1×
“This includes a checklist of recommended items to be addressed in a systematic review protocol [20].”
BackgroundA typology of reviews: an analysis of 14 review types and associated methodologies
Grant, M. J., & Booth, A. (2009) · Health Information & Libraries Journal · doi:10.1111/j.1471-1842.2009.00848.x · Cited by 10,817 · cited 1×
“The general characteristics of the studies, including the type of review [45], population, intervention, setting, and quality, were summarized by the information extracted from the analysis of the full-text articles (Table 1).”
MethodDevelopment of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews
Shea, B. J., Grimshaw, J. M., Wells, G. A., Boers, M., Andersson, N., Hamel, C., Porter, A. C., Tugwell, P., Moher, D., & Bouter, L. M. (2007) · BMC Medical Research Methodology · doi:10.1186/1471-2288-7-10 · Cited by 4,374
Eysenbach, G., & CONSORT-EHEALTH Group (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1923 · Cited by 1,863
Free, C., Phillips, G., Galli, L., Watson, L., Felix, L., Edwards, P., Patel, V., & Haines, A. (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001362 · Cited by 1,766 · cited 1×
“Patient portals [4], mobile health applications [5] and patients’ access to electronic health records (EHR) are common examples of digital health interventions.”
Murray, E., Treweek, S., Pope, C., MacFarlane, A., Ballini, L., Dowrick, C., Finch, T., Kennedy, A., Mair, F., O'Donnell, C., et al. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-63 · Cited by 1,478 · cited 1×
“[48].”
Mays, N., Pope, C., & Popay, J. (2005) · Journal of Health Services Research & Policy · doi:10.1258/1355819054308576 · Cited by 1,375
Methodology in conducting a systematic review of systematic reviews of healthcare interventions
Smith, V., Devane, D., Begley, C. M., & Clarke, M. (2011) · BMC Medical Research Methodology · doi:10.1186/1471-2288-11-15 · Cited by 1,227 · cited 1×
“A systematic review of reviews [18] focused on the methods used to measure the effects of internet-based digital health interventions for citizens was conducted in accordance with the Preferred Reporting Items for Systematic review and Meta-Analysis Protocols (PRISMA-P) [19].”
MethodPeters, D. H., Adam, T., Alonge, O., Agyepong, I. A., & Tran, N. (2014) · British Journal of Sports Medicine · doi:10.1136/bmj.f6753 · Cited by 1,109
Research works printed but not linked: 4
- Mold, F, 2012, Inform Prim Care, 20, 271
- 2016, Monitoring and evaluating digital health interventions: a practical guide to conducting research and assessment
- Lau, F, 2007, Healthc Q, 10, 112
- Ammenwerth, E, 2011, Stud Health Technol Inform, 165, 63
Cited by
Barteit, S., Lanfermann, L., Bärnighausen, T., Neuhann, F., & Beiersmann, C. (2021) · JMIR Serious Games · doi:10.2196/29080 · Cited by 547
The State of Evidence in Patient Portals: Umbrella Review
Antonio, M. G., Petrovskaya, O., & Lau, F. (2020) · Journal of Medical Internet Research · doi:10.2196/23851 · Cited by 146
“In addition to focusing on portal technology, both meta-level reviews [19,20] and several SRs [21-23] reported combined findings on a variety of eHealth tools, which may limit the reader’s ability to discern portal-specific effects.”
BackgroundInfluentialScholten, H., & Granic, I. (2019) · Journal of Medical Internet Research · doi:10.2196/11528 · Cited by 138
“However, these critical reflections consistently stop at that point, providing no concrete, actionable solutions to address the limitations they revealed [54-57].”
Marthick, M., McGregor, D., Alison, J. A., Cheema, B. S., Dhillon, H. M., & Shaw, T. J. (2021) · Journal of Medical Internet Research · doi:10.2196/24722 · Cited by 139
“Digital health has demonstrated potential in engaging people in their care [15], including as a tool for the treatment and self-management training of chronically ill patients [16,17].”
Zanaboni, P., Kummervold, P. E., Sørensen, T., & Johansen, M. A. (2020) · Journal of Medical Internet Research · doi:10.2196/16144 · Cited by 101
“Future evaluations of PAEHRs should focus on specific populations or chronic conditions that are more likely to achieve clinically meaningful benefits and use randomized controlled trials or implementation research methods [37,47].”
Self-citation
Metadata
- Authors
- Zanaboni, P., Ngangue, P., Mbemba, G. I. C., Schopf, T. R., Bergmo, T. S., & Gagnon, M.-P.
- Affiliations
- University Hospital of North Norway (NO)
- Université du Québec en Abitibi-Témiscamingue (CA)
- Université Laval (CA)
- Corresponding author
- Zanaboni, Paolo
- ORCID
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2018-06-07
- Volume
- 20
- Issue
- 6
- Pages
- e10202
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC6013714
- OpenAlex
- W2804205035
- Cited by
- 79
- References
- 51
- Citations by year
- 2026: 6 · 2025: 9 · 2024: 14 · 2023: 3 · 2022: 9 · 2021: 16 · 2020: 11 · 2019: 10 · 2018: 1
- FWCI
- 1.73
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Keywords
- digital health interventions, online intervention, assessment methods, patient portals, electronic health records, research design, health outcomes, health care utilization, patient satisfaction, patient-provider communication, system adoption, complex interventions, health system challenges, health care access, care coordination, health care costs, patient empowerment, evaluation metrics, evaluation perspectives
- MeSH terms
- Delivery of Health Care, Humans, Patient Satisfaction, Internet, Qualitative Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
