Methods to Evaluate the Effects of Internet-Based Digital Health Interventions for Citizens: Systematic Review of Reviews
Open accessPublished version of a preprint
The DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Digital health can empower citizens to manage their health and address health care system problems including poor access, uncoordinated care and increasing costs. Digital health interventions are typically complex interventions. Therefore, evaluations present methodological challenges. OBJECTIVE: The objective of this study was to provide a systematic overview of the methods used to evaluate the effects of internet-based digital health interventions for citizens. Three research questions were addressed to explore methods regarding approaches (study design), effects and indicators. METHODS: We conducted a systematic review of reviews of the methods used to measure the effects of internet-based digital health interventions for citizens. The protocol was developed a priori according to Preferred Reporting Items for Systematic review and Meta-Analysis Protocols and the Cochrane Collaboration methodology for overviews of reviews. Qualitative, mixed-method, and quantitative reviews published in English or French from January 2010 to October 2016 were included. We searched for published reviews in PubMed, EMBASE, The Cochrane Database of Systematic Reviews, CINHAL and Epistemonikos. We categorized the findings based on a thematic analysis of the reviews structured around study designs, indicators, types of interventions, effects and perspectives. RESULTS: A total of 20 unique reviews were included. The most common digital health interventions for citizens were patient portals and patients' access to electronic health records, covered by 10/20 (50%) and 6/20 (30%) reviews, respectively. Quantitative approaches to study design included observational study (15/20 reviews, 75%), randomized controlled trial (13/20 reviews, 65%), quasi-experimental design (9/20 reviews, 45%), and pre-post studies (6/20 reviews, 30%). Qualitative studies or mixed methods were reported in 13/20 (65%) reviews. Five main categories of effects were identified: (1) health and clinical outcomes, (2) psychological and behavioral outcomes, (3) health care utilization, (4) system adoption and use, and (5) system attributes. Health and clinical outcomes were measured with both general indicators and disease-specific indicators and reported in 11/20 (55%) reviews. Patient-provider communication and patient satisfaction were the most investigated psychological and behavioral outcomes, reported in 13/20 (65%) and 12/20 (60%) reviews, respectively. Evaluation of health care utilization was included in 8/20 (40%) reviews, most of which focused on the economic effects on the health care system. CONCLUSIONS: Although observational studies and surveys have provided evidence of benefits and satisfaction for patients, there is still little reliable evidence from randomized controlled trials of improved health outcomes. Future evaluations of digital health interventions for citizens should focus on specific populations or chronic conditions which are more likely to achieve clinically meaningful benefits and use high-quality approaches such as randomized controlled trials. Implementation research methods should also be considered. We identified a wide range of effects and indicators, most of which focused on patients as main end users. Implications for providers and the health system should also be included in evaluations or monitoring of digital health interventions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“However, these critical reflections consistently stop at that point, providing no concrete, actionable solutions to address the limitations they revealed [54-57].”
From Use of the Principles of Design Thinking to Address Limitations of Digital Mental Health Interventions for Youth: Viewpoint (2019) · cites this paper as [57]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
PRISMA-P Group, Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., & Stewart, L. A. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-1 · Cited by 26,223 · cited 2×
“A systematic review of reviews [18] focused on the methods used to measure the effects of internet-based digital health interventions for citizens was conducted in accordance with the Preferred Reporting Items for Systematic review and Meta-Analysis Protocols (PRISMA-P) [19].”
MethodShamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L. A., & the PRISMA-P Group (2015) · BMJ · doi:10.1136/bmj.g7647 · Cited by 13,120 · cited 1×
“This includes a checklist of recommended items to be addressed in a systematic review protocol [20].”
BackgroundA typology of reviews: an analysis of 14 review types and associated methodologies
Grant, M. J., & Booth, A. (2009) · Health Information & Libraries Journal · doi:10.1111/j.1471-1842.2009.00848.x · Cited by 10,817 · cited 1×
“The general characteristics of the studies, including the type of review [45], population, intervention, setting, and quality, were summarized by the information extracted from the analysis of the full-text articles (Table 1).”
MethodDevelopment of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews
Shea, B. J., Grimshaw, J. M., Wells, G. A., Boers, M., Andersson, N., Hamel, C., Porter, A. C., Tugwell, P., Moher, D., & Bouter, L. M. (2007) · BMC Medical Research Methodology · doi:10.1186/1471-2288-7-10 · Cited by 4,374
Eysenbach, G., & CONSORT-EHEALTH Group (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1923 · Cited by 1,863
Free, C., Phillips, G., Galli, L., Watson, L., Felix, L., Edwards, P., Patel, V., & Haines, A. (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001362 · Cited by 1,766 · cited 1×
“Patient portals [4], mobile health applications [5] and patients’ access to electronic health records (EHR) are common examples of digital health interventions.”
Murray, E., Treweek, S., Pope, C., MacFarlane, A., Ballini, L., Dowrick, C., Finch, T., Kennedy, A., Mair, F., O'Donnell, C., et al. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-63 · Cited by 1,478 · cited 1×
“[48].”
Mays, N., Pope, C., & Popay, J. (2005) · Journal of Health Services Research & Policy · doi:10.1258/1355819054308576 · Cited by 1,375
Methodology in conducting a systematic review of systematic reviews of healthcare interventions
Smith, V., Devane, D., Begley, C. M., & Clarke, M. (2011) · BMC Medical Research Methodology · doi:10.1186/1471-2288-11-15 · Cited by 1,227 · cited 1×
“A systematic review of reviews [18] focused on the methods used to measure the effects of internet-based digital health interventions for citizens was conducted in accordance with the Preferred Reporting Items for Systematic review and Meta-Analysis Protocols (PRISMA-P) [19].”
MethodPeters, D. H., Adam, T., Alonge, O., Agyepong, I. A., & Tran, N. (2014) · British Journal of Sports Medicine · doi:10.1136/bmj.f6753 · Cited by 1,109
Evaluating Digital Health Interventions
Murray, E., Hekler, E. B., Andersson, G., Collins, L. M., Doherty, A., Hollis, C., Rivera, D. E., West, R., & Wyatt, J. C. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.06.008 · Cited by 847 · cited 8×
“RCTs remain an essential method for determining the impact of digital health interventions concerning efficacy and cost-effectiveness but are best undertaken when the services are highly likely to lead to clinically meaningful benefits [1].”
BackgroundMethodInfluentialPatient Portals and Patient Engagement: A State of the Science Review
Irizarry, T., DeVito Dabbs, A., & Curran, C. R. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4255 · Cited by 732 · cited 9×
“6 No restrictions Patient portals Patients Literature review 120 2014 Irizarry et al [4]”
BackgroundInfluentialAgarwal, S., LeFevre, A. E., Lee, J., L’Engle, K., Mehl, G., Sinha, C., & Labrique, A. (2016) · BMJ · doi:10.1136/bmj.i1174 · Cited by 639 · cited 1×
“The term mHealth refers to the use of mobile and wireless technologies for health [50] and includes interventions that are delivered through mobile devices or the new generation of tablet computers [51].”
BackgroundO’Connor, S., Hanlon, P., O’Donnell, C. A., Garcia, S., Glanville, J., & Mair, F. S. (2016) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-016-0359-3 · Cited by 636
Personal health records: a scoping review
Archer, N., Fevrier-Thomas, U., Lokker, C., McKibbon, K. A., & Straus, S. E. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000105 · Cited by 573 · cited 3×
“Although there is some evidence of benefits from observational studies and surveys, with many studies reporting value in patients having access to more information through internet-based digital health interventions, there is still little reliable evidence from experimental studies of proven effectiveness in improved…”
BackgroundInfluentialMair, F. S., May, C., O’Donnell, C., Finch, T., Sullivan, F., & Murray, E. (2012) · Bulletin of the World Health Organization · doi:10.2471/blt.11.099424 · Cited by 553 · cited 1×
“RCTs have predefined protocols and strict inclusion criteria that can often mask wider implementation issues [14].”
BackgroundGoldzweig, C. L., Orshansky, G., Paige, N. M., Towfigh, A. A., Haggstrom, D. A., Miake-Lye, I., Beroes, J. M., & Shekelle, P. G. (2013) · Annals of Internal Medicine · doi:10.7326/0003-4819-159-10-201311190-00006 · Cited by 468 · cited 8×
“[34,40] [4,29,31,32,35,42,44] [28,30,33,36,39,43] Cross-sectional or surveys”
BackgroundInfluentialVimalananda, V. G., Gupte, G., Seraj, S. M., Orlander, J., Berlowitz, D., Fincke, B. G., & Simon, S. R. (2015) · Journal of Telemedicine and Telecare · doi:10.1177/1357633x15582108 · Cited by 437 · cited 7×
“N/A [7,33,38,39,44] N/A [28] Outpatient or clinic visits”
BackgroundInfluentialKaplan, B. (2001) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(01)00184-8 · Cited by 380 · cited 2×
“However, their applicability to evaluate the complex multifaceted nature of digital health interventions has been widely debated [13].”
BackgroundKruse, C. S., Bolton, K., & Freriks, G. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.3171 · Cited by 379 · cited 8×
“[34,40] [4,29,31,32,35,42,44] [28,30,33,36,39,43] Cross-sectional or surveys”
BackgroundInfluentialThe Impact of Electronic Patient Portals on Patient Care: A Systematic Review of Controlled Trials
Ammenwerth, E., Schnell-Inderst, P., & Hoerbst, A. (2012) · Journal of Medical Internet Research · doi:10.2196/jmir.2238 · Cited by 364 · cited 9×
“Interventions can also include other online platforms [6] such as medication refills, appointment scheduling, access to general medical information, or secure messaging between a patient and an institution [7].”
BackgroundInfluentialKruse, C. S., Argueta, D. A., Lopez, L., & Nair, A. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.3703 · Cited by 327 · cited 1×
“Of the 23 papers included, there were 3 publications [25,26,27] which were part of another main study [7,28,29], leading to a total of 20 unique reviews [4,7,23,28-44].”
BackgroundWeb 2.0 chronic disease self-management for older adults: a systematic review
Stellefson, M., Chaney, B., Barry, A. E., Chavarria, E., Tennant, B., Walsh-Childers, K., Sriram, P. S., & Zagora, J. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2439 · Cited by 315 · cited 10×
“[34,40] [4,29,31,32,35,42,44] [28,30,33,36,39,43] Cross-sectional or surveys”
BackgroundInfluentialDoctors' health: taking the lifecycle approach
Peters, M., Hasan, O., Puddester, D., Garelick, A., Holliday, C., Rapanakis, T., & Ryan, A. L. (2013) · BMJ · doi:10.1136/bmj.f7086 · Cited by 245
Rouleau, G., Gagnon, M.-P., Côté, J., Payne-Gagnon, J., Hudson, E., & Dubois, C.-A. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.6686 · Cited by 244 · cited 1×
“Non-systematic reviews generally had a lower quality score on the AMSTAR scale because this tool was specifically designed to assess systematic reviews of quantitative studies, and there is no equivalent tool for other types of reviews [46].”
MethodSelf-citationHandbook of eHealth Evaluation: An Evidence-based Approach
Lau, F., & Kuziemsky, C. E. (2016) · Project Muse (Johns Hopkins University) · Cited by 234
Alkureishi, M. A., Lee, W. W., Lyons, M., Press, V. G., Imam, S., Nkansah-Amankra, A., Werner, D., & Arora, V. M. (2016) · Journal of General Internal Medicine · doi:10.1007/s11606-015-3582-1 · Cited by 230 · cited 5×
“[34,40] [4,29,31,32,35,42,44] [28,30,33,36,39,43] Cross-sectional or surveys”
BackgroundInfluentialGrand Challenges in Digital Health
Kostkova, P. (2015) · Frontiers in Public Health · doi:10.3389/fpubh.2015.00134 · Cited by 227
de Lusignan, S., Mold, F., Sheikh, A., Majeed, A., Wyatt, J. C., Quinn, T., Cavill, M., Gronlund, T. A., Franco, C., Chauhan, U., et al. (2014) · BMJ Open · doi:10.1136/bmjopen-2014-006021 · Cited by 212
Georgsson, M., & Staggers, N. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv099 · Cited by 200 · cited 1×
“Patients seek information from the internet to learn more about their symptoms, diagnoses, and treatments and use a range of digital health interventions to manage their illness at home and support independent living and self-care [11].”
BackgroundPatient Web Portals to Improve Diabetes Outcomes: A Systematic Review
Osborn, C. Y., Mayberry, L. S., Mulvaney, S. A., & Hess, R. (2010) · Current Diabetes Reports · doi:10.1007/s11892-010-0151-1 · Cited by 176 · cited 8×
“6 Primary care Patient portals Patients with diabetes Systematic review 26 2010 Osborn et al [37]”
BackgroundInfluentialMold, F., de Lusignan, S., Sheikh, A., Majeed, A., Wyatt, J. C., Quinn, T., Cavill, M., Franco, C., Chauhan, U., Blakey, H., et al. (2015) · British Journal of General Practice · doi:10.3399/bjgp15x683941 · Cited by 168 · cited 9×
“N/A [7,33,38,39,44] N/A [28] Outpatient or clinic visits”
BackgroundInfluentialPatient access to medical records and healthcare outcomes: a systematic review
Giardina, T. D., Menon, S., Parrish, D. E., Sittig, D. F., & Singh, H. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-002239 · Cited by 167 · cited 10×
“N/A [7,33,38,39,44] N/A [28] Outpatient or clinic visits”
BackgroundInfluentialHow outcomes are achieved through patient portals: a realist review
Otte-Trojel, T., de Bont, A., Rundall, T. G., & van de Klundert, J. (2014) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-002501 · Cited by 159 · cited 9×
“4 Outpatient and inpatient Patient portals Patients Realist review 32 2014 Otte-Trojel et al [38]”
BackgroundResultInfluentialDavis, M. M., Freeman, M., Kaye, J. A., Vuckovic, N. H., & Buckley, D. (2014) · Telemedicine Journal and e-Health · doi:10.1089/tmj.2013.0166 · Cited by 116 · cited 7×
“[34,40] [4,29,31,32,35,42,44] [28,30,33,36,39,43] Cross-sectional or surveys”
BackgroundInfluentialConditions potentially sensitive to a Personal Health Record (PHR) intervention, a systematic review
Price, M., Bellwood, P., Kitson, N., Davies, I., Weber, J. H., & Lau, F. (2015) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-015-0159-1 · Cited by 114 · cited 10×
“N/A N/A N/A [39] Patient access to information”
BackgroundInfluentialA Systematic Review of Electronic Portal Usage Among Patients with Diabetes
Amante, D. J., Hogan, T. P., Pagoto, S. L., & English, T. M. (2014) · Diabetes Technology & Therapeutics · doi:10.1089/dia.2014.0078 · Cited by 108 · cited 6×
“Facilitators to adoption and use of patient portals were encouragement and assistance from providers and family members as well as self-engagement, while barriers included lack of skills, desire, or knowledge, technical difficulties, and lack of potential benefits [31].”
BackgroundInfluentialDigital Health Care—The Convergence of Health Care and the Internet
Frank, S. R. (2000) · Journal of Ambulatory Care Management · doi:10.1097/00004479-200004000-00003 · Cited by 100 · cited 1×
“They also provide a means for information exchange and support, and manage demand for health services, lowering direct medical costs [8].”
BackgroundBouma, G., Admiraal, J. M., de Vries, E. G. E., Schröder, C. P., Walenkamp, A. M. E., & Reyners, A. K. L. (2015) · Critical Reviews in Oncology/Hematology · doi:10.1016/j.critrevonc.2015.01.011 · Cited by 90 · cited 8×
“8 No restrictions Internet-based support programs Cancer patients Literature review 16 2015 Bouma et al [23]”
BackgroundInfluentialPatient satisfaction with electronic medical/health record: a systematic review
Liu, J., Luo, L., Zhang, R., & Huang, T. (2012) · Scandinavian Journal of Caring Sciences · doi:10.1111/scs.12015 · Cited by 70 · cited 4×
“Of the 23 papers included, there were 3 publications [25,26,27] which were part of another main study [7,28,29], leading to a total of 20 unique reviews [4,7,23,28-44].”
BackgroundInfluentialTao, D., & Or, C. K. (2013) · Journal of Telemedicine and Telecare · doi:10.1177/1357633x13479701 · Cited by 68 · cited 6×
“A meta-analysis of RCTs showed that the use of health information technology was associated with an improved glycaemic control in patients with diabetes [41].”
BackgroundInfluentialA proposed benefits evaluation framework for health information systems in Canada.
Lau, F., Hagens, S., & Muttitt, S. C. (2007) · PubMed · Cited by 68
Digital health and patient safety
Agboola, S. O., Bates, D. W., & Kvedar, J. C. (2016) · JAMA · doi:10.1001/jama.2016.2402 · Cited by 64 · cited 2×
“Recently, there has been an increasing public interest in digital health [9].”
BackgroundBush, R. A., Connelly, C. D., Fuller, M. G., & Pérez, A. (2015) · Telemedicine Journal and e-Health · doi:10.1089/tmj.2015.0033 · Cited by 52 · cited 8×
“However, overall evidence of improvement in quality of care was limited [7], and there was a lack of studies investigating long-term health outcomes [32].”
BackgroundInfluentialAmmenwerth, E., Schnell‐Inderst, P., & Hoerbst, A. (2011) · Studies in health technology and informatics · doi:10.3233/978-1-60750-735-2-63 · Cited by 37 · cited 1×
“Of the 23 papers included, there were 3 publications [25,26,27] which were part of another main study [7,28,29], leading to a total of 20 unique reviews [4,7,23,28-44].”
BackgroundTulu, B., Trudel, J. D., Strong, D. M., Johnson, S. A., Sundaresan, D., & Garber, L. (2015) · CHEST Journal · doi:10.1378/chest.14-2559 · Cited by 34
O’Connor, S., Hanlon, P., O’Donnell, C. A., Gonzalez-Moral, S. G., Glanville, J., & Mair, F. S. (2016) · BMJ Open · doi:10.1136/bmjopen-2015-010895 · Cited by 29 · cited 1×
“Many challenges only emerge when technologies are scaled up and implemented in ‘real-world’ complex health systems [15].”
BackgroundA Legal Framework to Support Development and Assessment of Digital Health Services
Garell, C., Svedberg, P., & Nygren, J. (2016) · JMIR Medical Informatics · doi:10.2196/medinform.5401 · Cited by 28 · cited 1×
“Digital health interventions can empower citizens to track, manage, and improve their health and quality of life while providing a more personalized health care delivery, at a lower cost and with higher efficiency and availability [10].”
BackgroundAdmiraal, J. M., Bouma, G., Schroeder, C. P., de Vries, E. G. E., Walenkamp, A., & Reyners, A. K. L. (2014) · University of Groningen research database (University of Groningen / Centre for Information Technology) · Cited by 22
Mold, F., Ellis, B., de Lusignan, S., Sheikh, A., Wyatt, J. C., Cavill, M., Michalakidis, G., Barker, F., Majeed, A., Quinn, T., et al. (2013) · Journal of Innovation in Health Informatics · doi:10.14236/jhi.v20i4.17 · Cited by 18 · cited 1×
“Interventions can also include other online platforms [6] such as medication refills, appointment scheduling, access to general medical information, or secure messaging between a patient and an institution [7].”
BackgroundPersonal health records for people living with HIV: a review
Turner, K. L., Klaman, S. L., & Shea, C. M. (2016) · AIDS Care · doi:10.1080/09540121.2016.1153594 · Cited by 11 · cited 10×
“[34,40] [4,29,31,32,35,42,44] [28,30,33,36,39,43] Cross-sectional or surveys”
BackgroundInfluential
Research works printed but not linked: 4
- Mold, F, 2012, Inform Prim Care, 20, 271
- 2016, Monitoring and evaluating digital health interventions: a practical guide to conducting research and assessment
- Lau, F, 2007, Healthc Q, 10, 112
- Ammenwerth, E, 2011, Stud Health Technol Inform, 165, 63
Cited by
Barteit, S., Lanfermann, L., Bärnighausen, T., Neuhann, F., & Beiersmann, C. (2021) · JMIR Serious Games · doi:10.2196/29080 · Cited by 547
The State of Evidence in Patient Portals: Umbrella Review
Antonio, M. G., Petrovskaya, O., & Lau, F. (2020) · Journal of Medical Internet Research · doi:10.2196/23851 · Cited by 146
“In addition to focusing on portal technology, both meta-level reviews [19,20] and several SRs [21-23] reported combined findings on a variety of eHealth tools, which may limit the reader’s ability to discern portal-specific effects.”
BackgroundInfluentialScholten, H., & Granic, I. (2019) · Journal of Medical Internet Research · doi:10.2196/11528 · Cited by 138
“However, these critical reflections consistently stop at that point, providing no concrete, actionable solutions to address the limitations they revealed [54-57].”
Marthick, M., McGregor, D., Alison, J. A., Cheema, B. S., Dhillon, H. M., & Shaw, T. J. (2021) · Journal of Medical Internet Research · doi:10.2196/24722 · Cited by 139
“Digital health has demonstrated potential in engaging people in their care [15], including as a tool for the treatment and self-management training of chronically ill patients [16,17].”
Zanaboni, P., Kummervold, P. E., Sørensen, T., & Johansen, M. A. (2020) · Journal of Medical Internet Research · doi:10.2196/16144 · Cited by 101
“Future evaluations of PAEHRs should focus on specific populations or chronic conditions that are more likely to achieve clinically meaningful benefits and use randomized controlled trials or implementation research methods [37,47].”
Self-citation
Metadata
- Authors
- Zanaboni, P., Ngangue, P., Mbemba, G. I. C., Schopf, T. R., Bergmo, T. S., & Gagnon, M.-P.
- Affiliations
- University Hospital of North Norway (NO)
- Université du Québec en Abitibi-Témiscamingue (CA)
- Université Laval (CA)
- Corresponding author
- Zanaboni, Paolo
- ORCID
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2018-06-07
- Volume
- 20
- Issue
- 6
- Pages
- e10202
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC6013714
- OpenAlex
- W2804205035
- Cited by
- 79
- References
- 51
- Citations by year
- 2026: 6 · 2025: 9 · 2024: 14 · 2023: 3 · 2022: 9 · 2021: 16 · 2020: 11 · 2019: 10 · 2018: 1
- FWCI
- 1.73
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Keywords
- digital health interventions, online intervention, assessment methods, patient portals, electronic health records, research design, health outcomes, health care utilization, patient satisfaction, patient-provider communication, system adoption, complex interventions, health system challenges, health care access, care coordination, health care costs, patient empowerment, evaluation metrics, evaluation perspectives
- MeSH terms
- Delivery of Health Care, Humans, Patient Satisfaction, Internet, Qualitative Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
