CoChat

An Assessment Framework for e-Mental Health Apps in Canada: Results of a Modified Delphi Process

Open access
Zelmer, Jennifer, van Hoof, Krystle, Notarianni, MaryAnn, van Mierlo, Trevor, Schellenberg, Megan, Tannenbaum, Cara · 2018 · JMIR mHealth and uHealth

Published version of a preprint

The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.

Abstract

BACKGROUND: The number of e-mental health apps is increasing rapidly. Studies have shown that the use of some apps is beneficial, whereas others are ineffective or do not meet users' privacy expectations. Individuals and organizations that curate, recommend, host, use, or pay for apps have an interest in categorizing apps according to the consensus criteria of usability and effectiveness. Others have previously published recommendations for assessing health-related apps; however, the extent to which these recommendations can be generalized across different population groups (eg, culture, gender, and language) remains unclear. This study describes an attempt by Canadian stakeholders to develop an e-mental health assessment framework that responds to the unique needs of people living in Canada in an evidence-based manner. OBJECTIVE: The objective of our study was to achieve consensus from a broad group of Canadian stakeholders on guiding principles and criteria for a framework to assess e-mental health apps in Canada. METHODS: We developed an initial set of guiding principles and criteria from a rapid review and environmental scan of pre-existing app assessment frameworks. The initial list was refined through a two-round modified Delphi process. Participants (N=25) included app developers and users, health care providers, mental health advocates, people with lived experience of a mental health problem or mental illness, policy makers, and researchers. Consensus on each guideline or criterion was defined a priori as at least 70% agreement. The first round of voting was conducted electronically. Prior to Round 2 voting, in-person presentations from experts and a persona empathy mapping process were used to explore the perspectives of diverse stakeholders. RESULTS: Of all respondents, 68% (17/25) in Round 1 and 100% (13/13) in Round 2 agreed that a framework for evaluating health apps is needed to help Canadian consumers identify high-quality apps. Consensus was reached on 9 guiding principles: evidence based, gender responsive, culturally appropriate, user centered, risk based, internationally aligned, enabling innovation, transparent and fair, and based on ethical norms. In addition, 15 informative and evaluative criteria were defined to assess the effectiveness, functionality, clinical applicability, interoperability, usability, transparency regarding security and privacy, security or privacy standards, supported platforms, targeted users, developers' transparency, funding transparency, price, user desirability, user inclusion, and meaningful inclusion of a diverse range of communities. CONCLUSIONS: Canadian mental health stakeholders reached the consensus on a framework of 9 guiding principles and 15 criteria important in assessing e-mental health apps. What differentiates the Canadian framework from other scales is explicit attention to user inclusion at all stages of the development, gender responsiveness, and cultural appropriateness. Furthermore, an empathy mapping process markedly influenced the development of the framework. This framework may be used to inform future mental health policies and programs.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.2196/10016

Cites

31 of 31 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 The modified Delphi process.
Figure 1 The modified Delphi process.CC BY · PubMed Central
Figure 2 Rating of the initial guiding principles in Round 1.
Figure 2 Rating of the initial guiding principles in Round 1.CC BY · PubMed Central
Figure 3 Round 1 rating of criteria for an e-mental health evaluation framework.
Figure 3 Round 1 rating of criteria for an e-mental health evaluation framework.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 6

  1. Sunderland, A, 2013, Health Rep, 24, 3
  2. Fiore, P, 2017, Building Capacity for Health Informatics in the Future, 109
  3. Singh, K, 2016, Issue Brief (Commonw Fund), 5, 1
  4. Mobasheri, M, 2016, E-mail
  5. World, Health Organization, 2011, mHealth New horizons for health through mobile technologies: Based on the findings of the second global survey on eHealth - Global Observatory for
  6. 10.2196/mhealth.3422 · DOI · Paper page

Cited by

See all 69

Metadata

Authors
Zelmer, J., van Hoof, K., Notarianni, M., van Mierlo, T., Schellenberg, M., & Tannenbaum, C.
Affiliations
  • University of Victoria (CA)
  • Mental Health Commission of Canada (CA)
  • Institut Universitaire de Gériatrie de Montréal (CA)
Venue
JMIR mHealth and uHealth
Published
2018-07-09
Volume
6
Issue
7
Pages
e10016
ISSN
2291-5222 (ISSN-L)
OpenAlex
W2808871401
Cited by
62
References
31
Citations by year
2026: 2 · 2025: 6 · 2024: 9 · 2023: 9 · 2022: 9 · 2021: 10 · 2020: 11 · 2019: 6
FWCI
8.51
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JMIR mhealth and uhealth (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
  • Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
  • Social Sciences › Sociology and Political Science › Delphi Technique in Research
Keywords
mental health apps, stakeholder consensus, usability, effectiveness, privacy and security, cultural appropriateness, gender responsiveness, user-centered design, lived experience of mental illness, interoperability, moral norms, user inclusion, empathy map, Delphi consensus

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.