Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.

Methods to Evaluate the Effects of Internet-Based Digital Health Interventions for Citizens: Systematic Review of Reviews

Open access
Zanaboni, Paolo, Ngangue, Patrice, Mbemba, Gisele Irène Claudine, Schopf, Thomas Roger, Bergmo, Trine Strand, Gagnon, Marie-Pierre · 2018 · Journal of Medical Internet Research

Published version of a preprint

Abstract

BACKGROUND: Digital health can empower citizens to manage their health and address health care system problems including poor access, uncoordinated care and increasing costs. Digital health interventions are typically complex interventions. Therefore, evaluations present methodological challenges. OBJECTIVE: The objective of this study was to provide a systematic overview of the methods used to evaluate the effects of internet-based digital health interventions for citizens. Three research questions were addressed to explore methods regarding approaches (study design), effects and indicators. METHODS: We conducted a systematic review of reviews of the methods used to measure the effects of internet-based digital health interventions for citizens. The protocol was developed a priori according to Preferred Reporting Items for Systematic review and Meta-Analysis Protocols and the Cochrane Collaboration methodology for overviews of reviews. Qualitative, mixed-method, and quantitative reviews published in English or French from January 2010 to October 2016 were included. We searched for published reviews in PubMed, EMBASE, The Cochrane Database of Systematic Reviews, CINHAL and Epistemonikos. We categorized the findings based on a thematic analysis of the reviews structured around study designs, indicators, types of interventions, effects and perspectives. RESULTS: A total of 20 unique reviews were included. The most common digital health interventions for citizens were patient portals and patients' access to electronic health records, covered by 10/20 (50%) and 6/20 (30%) reviews, respectively. Quantitative approaches to study design included observational study (15/20 reviews, 75%), randomized controlled trial (13/20 reviews, 65%), quasi-experimental design (9/20 reviews, 45%), and pre-post studies (6/20 reviews, 30%). Qualitative studies or mixed methods were reported in 13/20 (65%) reviews. Five main categories of effects were identified: (1) health and clinical outcomes, (2) psychological and behavioral outcomes, (3) health care utilization, (4) system adoption and use, and (5) system attributes. Health and clinical outcomes were measured with both general indicators and disease-specific indicators and reported in 11/20 (55%) reviews. Patient-provider communication and patient satisfaction were the most investigated psychological and behavioral outcomes, reported in 13/20 (65%) and 12/20 (60%) reviews, respectively. Evaluation of health care utilization was included in 8/20 (40%) reviews, most of which focused on the economic effects on the health care system. CONCLUSIONS: Although observational studies and surveys have provided evidence of benefits and satisfaction for patients, there is still little reliable evidence from randomized controlled trials of improved health outcomes. Future evaluations of digital health interventions for citizens should focus on specific populations or chronic conditions which are more likely to achieve clinically meaningful benefits and use high-quality approaches such as randomized controlled trials. Implementation research methods should also be considered. We identified a wide range of effects and indicators, most of which focused on patients as main end users. Implications for providers and the health system should also be included in evaluations or monitoring of digital health interventions.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.2196/10202

Cites

51 of 51 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Preferred Reporting Items for Systematic review and Meta-Analysis (PRISMA) study flow diagram.
Figure 1 Preferred Reporting Items for Systematic review and Meta-Analysis (PRISMA) study flow diagram.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 51 references

Research works printed but not linked: 4

  1. Mold, F, 2012, Inform Prim Care, 20, 271
  2. 2016, Monitoring and evaluating digital health interventions: a practical guide to conducting research and assessment
  3. Lau, F, 2007, Healthc Q, 10, 112
  4. Ammenwerth, E, 2011, Stud Health Technol Inform, 165, 63

Cited by

See all 86

Metadata

Authors
Zanaboni, P., Ngangue, P., Mbemba, G. I. C., Schopf, T. R., Bergmo, T. S., & Gagnon, M.-P.
Affiliations
  • University Hospital of North Norway (NO)
  • Université du Québec en Abitibi-Témiscamingue (CA)
  • Université Laval (CA)
Corresponding author
Zanaboni, Paolo
Venue
Journal of Medical Internet Research
Published
2018-06-07
Volume
20
Issue
6
Pages
e10202
ISSN
1438-8871 (ISSN-L)
OpenAlex
W2804205035
Cited by
79
References
51
Citations by year
2026: 6 · 2025: 9 · 2024: 14 · 2023: 3 · 2022: 9 · 2021: 16 · 2020: 11 · 2019: 10 · 2018: 1
FWCI
1.73
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Journal of Medical Internet Research (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Topics
  • Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
Keywords
digital health interventions, online intervention, assessment methods, patient portals, electronic health records, research design, health outcomes, health care utilization, patient satisfaction, patient-provider communication, system adoption, complex interventions, health system challenges, health care access, care coordination, health care costs, patient empowerment, evaluation metrics, evaluation perspectives
MeSH terms
Delivery of Health Care, Humans, Patient Satisfaction, Internet, Qualitative Research

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Methods to Evaluate the Effects of Internet-Based Digital Health Interventions for Citizens: Systematic Review of Reviews. https://app.cochat.ai/papers/10.2196/10202
The paper
Zanaboni, P., Ngangue, P., Mbemba, G. I. C., Schopf, T. R., Bergmo, T. S., & Gagnon, M.-P. (2018). Methods to Evaluate the Effects of Internet-Based Digital Health Interventions for Citizens: Systematic Review of Reviews. Journal of Medical Internet Research. https://doi.org/10.2196/10202

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.