An Assessment Framework for e-Mental Health Apps in Canada: Results of a Modified Delphi Process
Open accessPublished version of a preprint
The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
Abstract
BACKGROUND: The number of e-mental health apps is increasing rapidly. Studies have shown that the use of some apps is beneficial, whereas others are ineffective or do not meet users' privacy expectations. Individuals and organizations that curate, recommend, host, use, or pay for apps have an interest in categorizing apps according to the consensus criteria of usability and effectiveness. Others have previously published recommendations for assessing health-related apps; however, the extent to which these recommendations can be generalized across different population groups (eg, culture, gender, and language) remains unclear. This study describes an attempt by Canadian stakeholders to develop an e-mental health assessment framework that responds to the unique needs of people living in Canada in an evidence-based manner. OBJECTIVE: The objective of our study was to achieve consensus from a broad group of Canadian stakeholders on guiding principles and criteria for a framework to assess e-mental health apps in Canada. METHODS: We developed an initial set of guiding principles and criteria from a rapid review and environmental scan of pre-existing app assessment frameworks. The initial list was refined through a two-round modified Delphi process. Participants (N=25) included app developers and users, health care providers, mental health advocates, people with lived experience of a mental health problem or mental illness, policy makers, and researchers. Consensus on each guideline or criterion was defined a priori as at least 70% agreement. The first round of voting was conducted electronically. Prior to Round 2 voting, in-person presentations from experts and a persona empathy mapping process were used to explore the perspectives of diverse stakeholders. RESULTS: Of all respondents, 68% (17/25) in Round 1 and 100% (13/13) in Round 2 agreed that a framework for evaluating health apps is needed to help Canadian consumers identify high-quality apps. Consensus was reached on 9 guiding principles: evidence based, gender responsive, culturally appropriate, user centered, risk based, internationally aligned, enabling innovation, transparent and fair, and based on ethical norms. In addition, 15 informative and evaluative criteria were defined to assess the effectiveness, functionality, clinical applicability, interoperability, usability, transparency regarding security and privacy, security or privacy standards, supported platforms, targeted users, developers' transparency, funding transparency, price, user desirability, user inclusion, and meaningful inclusion of a diverse range of communities. CONCLUSIONS: Canadian mental health stakeholders reached the consensus on a framework of 9 guiding principles and 15 criteria important in assessing e-mental health apps. What differentiates the Canadian framework from other scales is explicit attention to user inclusion at all stages of the development, gender responsiveness, and cultural appropriateness. Furthermore, an empathy mapping process markedly influenced the development of the framework. This framework may be used to inform future mental health policies and programs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Most of the papers in this category 70% (7/10) presented new frameworks developed by the study authors, and the methods adopted in their creation varied: from reviews of related literature and pre-existing app evaluation frameworks [84], [86], [89] to more empirical methods such as engaging an expert panel of key stakeholders [87] and analysis of mHealth applications [85].”
From Security and Privacy of mHealth Applications: A Scoping Review (2020) · cites this paper as [84]
In the citing paper
“In a Canadian study where participants were asked to rate mHealth apps objectively, results were highly variable, and 28% of reviewers were not even sure about the overall quality of the health product [57].”
From Understanding Mobile Health and Youth Mental Health: Scoping Review (2023) · cites this paper as [57]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An Experimental Application of the DELPHI Method to the Use of Experts
Dalkey, N., & Helmer, O. (1963) · Management Science · doi:10.1287/mnsc.9.3.458 · Cited by 6,061 · cited 1×
“Unlike the open-ended initial phase of a traditional Delphi process [30,31], a modified Delphi process provides a starting point for discussion.”
MethodResearch guidelines for the Delphi survey technique
Hasson, F. (2000) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.2000.01567.x · Cited by 5,951 · cited 1×
“For Round 2 of the modified Delphi process, consensus was defined as 70% agreement, consistent with other studies of this type [41-43].”
ResultMobile app rating scale: a new tool for assessing the quality of health mobile apps
Stoyanov, S. R., Hides, L., Kavanagh, D. J., Zelenko, O., Tjondronegoro, D., & Mani, M. (2015) · JMIR mHealth and uHealth · doi:10.2196/mhealth.3422 · Cited by 2,233 · cited 2×
“The Mobile App Rating Scale (MARS), developed in Australia and one of the most developed examples of an app evaluation tool, weighs all criteria equally [27].”
BackgroundMethodConsensus methods: characteristics and guidelines for use
Fink, A., Kosecoff, J., Chassin, M., & Brook, R. H. (1984) · American Journal of Public Health · doi:10.2105/ajph.74.9.979 · Cited by 2,089
Boulkedid, R., Abdoul, H., Loustau, M., Sibony, O., & Alberti, C. (2011) · PLoS ONE · doi:10.1371/journal.pone.0020476 · Cited by 2,045 · cited 1×
“Methods and results are reported in alignment with the guidance for reporting results from Delphi processes developed by Boulkedid et al [29].”
MethodSmartphones for smarter delivery of mental health programs: A systematic review
Donker, T., Petrie, K., Proudfoot, J., Clarke, J., Birch, M.-R., & Christensen, H. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2791 · Cited by 1,136 · cited 2×
“Studies indicate that the use of some apps can be beneficial to mental health and many improve accessibility to mental health services [3].”
BackgroundFirth, J., Torous, J., Nicholas, J., Carney, R., Pratap, A., Rosenbaum, S., & Sarris, J. (2017) · World Psychiatry · doi:10.1002/wps.20472 · Cited by 1,097 · cited 1×
“According to a recent meta-analysis, mobile phone-based mental health apps can have positive effects on depressive symptoms, anxiety, and other mental health conditions [4].”
MethodMental Health Smartphone Apps: Review and Evidence-Based Recommendations for Future Developments
Bakker, D., Kazantzis, N., Rickwood, D., & Rickard, N. (2016) · JMIR Mental Health · doi:10.2196/mental.4984 · Cited by 886 · cited 3×
“Several formal development strategies, rating scales, or assessment frameworks have been published to help raise standards on app quality, and efforts are underway to develop others [1,14,20,23-28].”
BackgroundMobile Apps for Bipolar Disorder: A Systematic Review of Features and Content Quality
Nicholas, J., Larsen, M. E., Proudfoot, J., & Christensen, H. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4581 · Cited by 438 · cited 1×
“However, research suggests that some apps are unsafe, ineffective, poorly documented, or do not meet users’ privacy and security expectations [6-8].”
Huckvale, K., Prieto, J. T., Tilney, M., Benghozi, P.-J., & Car, J. (2015) · BMC Medicine · doi:10.1186/s12916-015-0444-y · Cited by 376 · cited 1×
“The National Health Service in the United Kingdom established a curated Apps Library, with apps assessed against a defined set of criteria, which was later rolled back following research that showed privacy and security gaps in a large proportion of the included apps [18].”
BackgroundInfluentialGustafson, D. H., Shukla, R. K., Delbecq, A., & Walster, G. W. (1973) · Organizational Behavior and Human Performance · doi:10.1016/0030-5073(73)90052-4 · Cited by 376 · cited 1×
“The mini-Delphi or estimate-talk-estimate approach adapts this technique for face-to-face meetings, allowing experts to interact between iterations of anonymous votes [32].”
MethodA systematic assessment of smartphone tools for suicide prevention
Larsen, M. E., Nicholas, J., & Christensen, H. (2016) · PLoS ONE · doi:10.1371/journal.pone.0152285 · Cited by 289 · cited 1×
“However, research suggests that some apps are unsafe, ineffective, poorly documented, or do not meet users’ privacy and security expectations [6-8].”
BackgroundFinding a Depression App: A Review and Content Analysis of the Depression App Marketplace
Shen, N., Levitan, M.-J., Johnson, A., Bender, J. L., Hamilton-Page, M., Jadad, A. (. R., & Wiljer, D. (2015) · JMIR mHealth and uHealth · doi:10.2196/mhealth.3713 · Cited by 282 · cited 1×
“Currently, it can be time-consuming and difficult for potential users to assess the quality, safety, and evidence base of available health apps [3,14,15].”
BackgroundSunderland, A., & Findlay, L. (2013) · PubMed · Cited by 218
Grindrod, K. A., Li, M., & Gates, A. (2014) · JMIR mhealth and uhealth · doi:10.2196/mhealth.3048 · Cited by 204 · cited 2×
“Several formal development strategies, rating scales, or assessment frameworks have been published to help raise standards on app quality, and efforts are underway to develop others [1,14,20,23-28].”
Kleynen, M., Braun, S. M., Bleijlevens, M. H., Lexis, M. A., Rasquin, S. M., Halfens, J., Wilson, M. R., Beurskens, A. J., & Masters, R. S. W. (2014) · PLoS ONE · doi:10.1371/journal.pone.0100227 · Cited by 173 · cited 1×
“For Round 2 of the modified Delphi process, consensus was defined as 70% agreement, consistent with other studies of this type [41-43].”
ResultTowards a Framework for Evaluating Mobile Mental Health Apps
Chan, S., Torous, J., Hinton, L., & Yellowlees, P. (2015) · Telemedicine and e-Health · doi:10.1089/tmj.2015.0002 · Cited by 163
Challenges in Assessing Mobile Health App Quality
Grundy, Q. H., Wang, Z., & Bero, L. A. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.07.009 · Cited by 154 · cited 3×
“…review of methods used to assess mobile health apps found that researchers evaluated the app quality in 6 domains: scientific or clinical basis, functionality, usability, accountability, impact, and popularity; 80% (73/91) participants used measures drawn from one or more of these domains [14].”
BackgroundMethodDesigning Personas with Empathy Map
Ferreira, B., Silva, W., Oliveira, E., & Conte, T. (2015) · International Conferences on Software Engineering and Knowledge Engineering · doi:10.18293/seke2015-152 · Cited by 151 · cited 1×
“We used a facilitated persona empathy mapping exercise to ensure that a broad range of end users’ needs and perspectives were considered in the process [39,40].”
MethodSingh, K., & Bates, D. W. (2016) · PubMed · doi:10.26099/g1ws-c172 · Cited by 138
Privacy and security issues in mobile health: Current research and future directions
Bhuyan, S. S., Kim, H., Isehunwa, O. O., Kumar, N., Bhatt, J., Wyant, D. K., Kedia, S., Chang, C. F., & Dasgupta, D. (2017) · Health Policy and Technology · doi:10.1016/j.hlpt.2017.01.004 · Cited by 103 · cited 1×
“However, research suggests that some apps are unsafe, ineffective, poorly documented, or do not meet users’ privacy and security expectations [6-8].”
BackgroundTransparency of Health-Apps for Trust and Decision Making
Albrecht, U.-V. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2981 · Cited by 90
Consensus-based standards for best supportive care in clinical trials in advanced cancer
Zafar, S. Y., Currow, D. C., Cherny, N. I., Strasser, F., Fowler, R., & Abernethy, A. P. (2012) · The Lancet Oncology · doi:10.1016/s1470-2045(11)70215-7 · Cited by 89 · cited 1×
“For Round 2 of the modified Delphi process, consensus was defined as 70% agreement, consistent with other studies of this type [41-43].”
ResultGuha, M. (2005) · Reference Reviews · doi:10.1108/09504120510622715 · Cited by 52
A Collaborative Rapid Persona-Building Workshop: Creating Design Personas with Health Researchers
Williams, I., Brereton, M. F., Donovan, J. W., McDonald, K., Millard, T. N., Tam, A., & Elliott, J. (2014) · International Journal of Sociotechnology and Knowledge Development · doi:10.4018/ijskd.2014040102 · Cited by 40 · cited 1×
“We used a facilitated persona empathy mapping exercise to ensure that a broad range of end users’ needs and perspectives were considered in the process [39,40].”
MethodLessons Learned From Using Focus Groups to Refine Digital Interventions
Avis, J. L. S., van Mierlo, T., Fournier, R., & Ball, G. D. C. (2015) · JMIR Research Protocols · doi:10.2196/resprot.4404 · Cited by 34 · cited 1×
“Researchers in Canada and beyond are testing techniques to engage app users in the development process [20,21].”
BackgroundAlbrecht, U., Pramann, O., & von Jan, U. (2014) · Advances in healthcare information systems and administration book series · doi:10.4018/978-1-4666-6150-9.ch007 · Cited by 19
Health Apps by Design: A Reference Architecture for Mobile Engagement
Chindalo, P., Karim, A., Brahmbhatt, R., Saha, N., & Keshavjee, K. (2016) · International Journal of Handheld Computing Research · doi:10.4018/ijhcr.2016040103 · Cited by 15 · cited 1×
“Several formal development strategies, rating scales, or assessment frameworks have been published to help raise standards on app quality, and efforts are underway to develop others [1,14,20,23-28].”
BackgroundHow to Evaluate Mobile Health Applications: A Scoping Review
Fiore, P. (2017) · Studies in health technology and informatics · doi:10.3233/978-1-61499-742-9-109 · Cited by 15 · cited 1×
“Several formal development strategies, rating scales, or assessment frameworks have been published to help raise standards on app quality, and efforts are underway to develop others [1,14,20,23-28].”
BackgroundEvaluating Mobile Medical Applications eReport
(2014) · ASHP eBooks · doi:10.37573/9781585284597 · Cited by 15
Tannenbaum, C., Voss, P., El-Gabalawy, H. S., & Joanette, Y. (2016) · Canadian Journal on Aging / La Revue canadienne du vieillissement · doi:10.1017/s0714980816000416 · Cited by 4 · cited 1×
“…the Canadian Institutes of Health Research, which aims to support evidence-based interventions that foster healthy, meaningful, and productive work for all workers, recognizes e-mental health as a potential direction for improving the wellness of workers across the mental health continuum [12,13].”
Self-citation
Research works printed but not linked: 6
- Sunderland, A, 2013, Health Rep, 24, 3
- Fiore, P, 2017, Building Capacity for Health Informatics in the Future, 109
- Singh, K, 2016, Issue Brief (Commonw Fund), 5, 1
- Mobasheri, M, 2016, E-mail
- World, Health Organization, 2011, mHealth New horizons for health through mobile technologies: Based on the findings of the second global survey on eHealth - Global Observatory for
- 10.2196/mhealth.3422 · DOI · Paper page
Cited by
Lam, R. W., Kennedy, S. H., Adams, C., Bahji, A., Beaulieu, S., Bhat, V., Blier, P., Blumberger, D. M., Brietzke, E., Chakrabarty, T., et al. (2024) · The Canadian Journal of Psychiatry · doi:10.1177/07067437241245384 · Cited by 369
Artificial intelligence technologies and compassion in healthcare: A systematic scoping review
Morrow, E., Zidaru, T., Ross, F., Mason, C., Patel, K. D., Ream, M., & Stockley, R. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.971044 · Cited by 250
Mobile Apps for Medication Management: Review and Analysis
Tabi, K., Randhawa, A. S., Choi, F., Mithani, Z., Albers, F., Schnieder, M., Nikoo, M., Vigo, D., Jang, K., Demlova, R., et al. (2019) · JMIR mHealth and uHealth · doi:10.2196/13608 · Cited by 165
“…could follow some of the standardized frameworks for quality assessment of mobile apps, for example, Mobile App Rating Scale [49], suitable for a wide range of health apps; a recently published assessment framework specifically for mental health apps [50]; or other published frameworks [33,51-54].”
BackgroundMobile health app usability and quality rating scales: a systematic review
Azad-Khaneghah, P., Neubauer, N., Miguel Cruz, A., & Liu, L. (2021) · Disability and Rehabilitation: Assistive Technology · doi:10.1080/17483107.2019.1701103 · Cited by 115
“illness, policy makers, and researchers [116].”
Security and Privacy of mHealth Applications: A Scoping Review
Nurgalieva, L., O'Callaghan, D., & Doherty, G. (2020) · IEEE Access · doi:10.1109/access.2020.2999934 · Cited by 132
“This includes evaluation of apps targeting specific user groups, for instance, adolescents [86] or people with mental health problems [84], or based on user input such as user reviews [64].”
BackgroundMethodInfluential
Metadata
- Authors
- Zelmer, J., van Hoof, K., Notarianni, M., van Mierlo, T., Schellenberg, M., & Tannenbaum, C.
- Affiliations
- University of Victoria (CA)
- Mental Health Commission of Canada (CA)
- Institut Universitaire de Gériatrie de Montréal (CA)
- ORCID
- Venue
- JMIR mHealth and uHealth
- Publisher
- JMIR Publications Inc.
- Published
- 2018-07-09
- Volume
- 6
- Issue
- 7
- Pages
- e10016
- ISSN
- 2291-5222 (ISSN-L)
- PMC
- PMC6056739
- OpenAlex
- W2808871401
- Cited by
- 62
- References
- 31
- Citations by year
- 2026: 2 · 2025: 6 · 2024: 9 · 2023: 9 · 2022: 9 · 2021: 10 · 2020: 11 · 2019: 6
- FWCI
- 8.51
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR mhealth and uhealth (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- mental health apps, stakeholder consensus, usability, effectiveness, privacy and security, cultural appropriateness, gender responsiveness, user-centered design, lived experience of mental illness, interoperability, moral norms, user inclusion, empathy map, Delphi consensus
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.


