An Assessment Framework for e-Mental Health Apps in Canada: Results of a Modified Delphi Process
Open accessPublished version of a preprint
The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
Abstract
BACKGROUND: The number of e-mental health apps is increasing rapidly. Studies have shown that the use of some apps is beneficial, whereas others are ineffective or do not meet users' privacy expectations. Individuals and organizations that curate, recommend, host, use, or pay for apps have an interest in categorizing apps according to the consensus criteria of usability and effectiveness. Others have previously published recommendations for assessing health-related apps; however, the extent to which these recommendations can be generalized across different population groups (eg, culture, gender, and language) remains unclear. This study describes an attempt by Canadian stakeholders to develop an e-mental health assessment framework that responds to the unique needs of people living in Canada in an evidence-based manner. OBJECTIVE: The objective of our study was to achieve consensus from a broad group of Canadian stakeholders on guiding principles and criteria for a framework to assess e-mental health apps in Canada. METHODS: We developed an initial set of guiding principles and criteria from a rapid review and environmental scan of pre-existing app assessment frameworks. The initial list was refined through a two-round modified Delphi process. Participants (N=25) included app developers and users, health care providers, mental health advocates, people with lived experience of a mental health problem or mental illness, policy makers, and researchers. Consensus on each guideline or criterion was defined a priori as at least 70% agreement. The first round of voting was conducted electronically. Prior to Round 2 voting, in-person presentations from experts and a persona empathy mapping process were used to explore the perspectives of diverse stakeholders. RESULTS: Of all respondents, 68% (17/25) in Round 1 and 100% (13/13) in Round 2 agreed that a framework for evaluating health apps is needed to help Canadian consumers identify high-quality apps. Consensus was reached on 9 guiding principles: evidence based, gender responsive, culturally appropriate, user centered, risk based, internationally aligned, enabling innovation, transparent and fair, and based on ethical norms. In addition, 15 informative and evaluative criteria were defined to assess the effectiveness, functionality, clinical applicability, interoperability, usability, transparency regarding security and privacy, security or privacy standards, supported platforms, targeted users, developers' transparency, funding transparency, price, user desirability, user inclusion, and meaningful inclusion of a diverse range of communities. CONCLUSIONS: Canadian mental health stakeholders reached the consensus on a framework of 9 guiding principles and 15 criteria important in assessing e-mental health apps. What differentiates the Canadian framework from other scales is explicit attention to user inclusion at all stages of the development, gender responsiveness, and cultural appropriateness. Furthermore, an empathy mapping process markedly influenced the development of the framework. This framework may be used to inform future mental health policies and programs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Most of the papers in this category 70% (7/10) presented new frameworks developed by the study authors, and the methods adopted in their creation varied: from reviews of related literature and pre-existing app evaluation frameworks [84], [86], [89] to more empirical methods such as engaging an expert panel of key stakeholders [87] and analysis of mHealth applications [85].”
From Security and Privacy of mHealth Applications: A Scoping Review (2020) · cites this paper as [84]
In the citing paper
“In a Canadian study where participants were asked to rate mHealth apps objectively, results were highly variable, and 28% of reviewers were not even sure about the overall quality of the health product [57].”
From Understanding Mobile Health and Youth Mental Health: Scoping Review (2023) · cites this paper as [57]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An Experimental Application of the DELPHI Method to the Use of Experts
Dalkey, N., & Helmer, O. (1963) · Management Science · doi:10.1287/mnsc.9.3.458 · Cited by 6,061 · cited 1×
“Unlike the open-ended initial phase of a traditional Delphi process [30,31], a modified Delphi process provides a starting point for discussion.”
MethodResearch guidelines for the Delphi survey technique
Hasson, F. (2000) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.2000.01567.x · Cited by 5,951 · cited 1×
“For Round 2 of the modified Delphi process, consensus was defined as 70% agreement, consistent with other studies of this type [41-43].”
ResultMobile app rating scale: a new tool for assessing the quality of health mobile apps
Stoyanov, S. R., Hides, L., Kavanagh, D. J., Zelenko, O., Tjondronegoro, D., & Mani, M. (2015) · JMIR mHealth and uHealth · doi:10.2196/mhealth.3422 · Cited by 2,233 · cited 2×
“The Mobile App Rating Scale (MARS), developed in Australia and one of the most developed examples of an app evaluation tool, weighs all criteria equally [27].”
BackgroundMethodConsensus methods: characteristics and guidelines for use
Fink, A., Kosecoff, J., Chassin, M., & Brook, R. H. (1984) · American Journal of Public Health · doi:10.2105/ajph.74.9.979 · Cited by 2,089
Boulkedid, R., Abdoul, H., Loustau, M., Sibony, O., & Alberti, C. (2011) · PLoS ONE · doi:10.1371/journal.pone.0020476 · Cited by 2,045 · cited 1×
“Methods and results are reported in alignment with the guidance for reporting results from Delphi processes developed by Boulkedid et al [29].”
MethodSmartphones for smarter delivery of mental health programs: A systematic review
Donker, T., Petrie, K., Proudfoot, J., Clarke, J., Birch, M.-R., & Christensen, H. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2791 · Cited by 1,136 · cited 2×
“Studies indicate that the use of some apps can be beneficial to mental health and many improve accessibility to mental health services [3].”
BackgroundFirth, J., Torous, J., Nicholas, J., Carney, R., Pratap, A., Rosenbaum, S., & Sarris, J. (2017) · World Psychiatry · doi:10.1002/wps.20472 · Cited by 1,097 · cited 1×
“According to a recent meta-analysis, mobile phone-based mental health apps can have positive effects on depressive symptoms, anxiety, and other mental health conditions [4].”
MethodMental Health Smartphone Apps: Review and Evidence-Based Recommendations for Future Developments
Bakker, D., Kazantzis, N., Rickwood, D., & Rickard, N. (2016) · JMIR Mental Health · doi:10.2196/mental.4984 · Cited by 886 · cited 3×
“Several formal development strategies, rating scales, or assessment frameworks have been published to help raise standards on app quality, and efforts are underway to develop others [1,14,20,23-28].”
BackgroundMobile Apps for Bipolar Disorder: A Systematic Review of Features and Content Quality
Nicholas, J., Larsen, M. E., Proudfoot, J., & Christensen, H. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4581 · Cited by 438 · cited 1×
“However, research suggests that some apps are unsafe, ineffective, poorly documented, or do not meet users’ privacy and security expectations [6-8].”
Huckvale, K., Prieto, J. T., Tilney, M., Benghozi, P.-J., & Car, J. (2015) · BMC Medicine · doi:10.1186/s12916-015-0444-y · Cited by 376 · cited 1×
“The National Health Service in the United Kingdom established a curated Apps Library, with apps assessed against a defined set of criteria, which was later rolled back following research that showed privacy and security gaps in a large proportion of the included apps [18].”
BackgroundInfluential
Research works printed but not linked: 6
- Sunderland, A, 2013, Health Rep, 24, 3
- Fiore, P, 2017, Building Capacity for Health Informatics in the Future, 109
- Singh, K, 2016, Issue Brief (Commonw Fund), 5, 1
- Mobasheri, M, 2016, E-mail
- World, Health Organization, 2011, mHealth New horizons for health through mobile technologies: Based on the findings of the second global survey on eHealth - Global Observatory for
- 10.2196/mhealth.3422 · DOI · Paper page
Cited by
Lam, R. W., Kennedy, S. H., Adams, C., Bahji, A., Beaulieu, S., Bhat, V., Blier, P., Blumberger, D. M., Brietzke, E., Chakrabarty, T., et al. (2024) · The Canadian Journal of Psychiatry · doi:10.1177/07067437241245384 · Cited by 369
Artificial intelligence technologies and compassion in healthcare: A systematic scoping review
Morrow, E., Zidaru, T., Ross, F., Mason, C., Patel, K. D., Ream, M., & Stockley, R. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.971044 · Cited by 250
Mobile Apps for Medication Management: Review and Analysis
Tabi, K., Randhawa, A. S., Choi, F., Mithani, Z., Albers, F., Schnieder, M., Nikoo, M., Vigo, D., Jang, K., Demlova, R., et al. (2019) · JMIR mHealth and uHealth · doi:10.2196/13608 · Cited by 165
“…could follow some of the standardized frameworks for quality assessment of mobile apps, for example, Mobile App Rating Scale [49], suitable for a wide range of health apps; a recently published assessment framework specifically for mental health apps [50]; or other published frameworks [33,51-54].”
BackgroundMobile health app usability and quality rating scales: a systematic review
Azad-Khaneghah, P., Neubauer, N., Miguel Cruz, A., & Liu, L. (2021) · Disability and Rehabilitation: Assistive Technology · doi:10.1080/17483107.2019.1701103 · Cited by 115
“illness, policy makers, and researchers [116].”
Security and Privacy of mHealth Applications: A Scoping Review
Nurgalieva, L., O'Callaghan, D., & Doherty, G. (2020) · IEEE Access · doi:10.1109/access.2020.2999934 · Cited by 132
“This includes evaluation of apps targeting specific user groups, for instance, adolescents [86] or people with mental health problems [84], or based on user input such as user reviews [64].”
BackgroundMethodInfluentialRamos, G., Ponting, C., Labao, J. P., & Sobowale, K. (2021) · Behaviour Research and Therapy · doi:10.1016/j.brat.2021.103990 · Cited by 117
Strudwick, G., Sockalingam, S., Kassam, I., Sequeira, L., Bonato, S., Youssef, A., Mehta, R., Green, N., Agic, B., Soklaridis, S., et al. (2021) · JMIR Mental Health · doi:10.2196/26550 · Cited by 110
“For example the American Psychiatric Association and the MHCC both have tools to support the selection and assessment of mobile apps [26,52].”
MethodUnderstanding the quality, effectiveness and attributes of top-rated smartphone health apps
Wisniewski, H., Liu, G., Henson, P., Vaidyam, A., Hajratalli, N. K., Onnela, J.-P., & Torous, J. (2019) · Evidence Based Mental Health · doi:10.1136/ebmental-2018-300069 · Cited by 112
“21 Prior efforts to create repositories of higher quality apps have failed at the national level 22 when closely scrutinised and other efforts to score apps by features have also been found to be unreliable 23 when closely examined.”
Digi-HTA: Health technology assessment framework for digital healthcare services
Haverinen, J., Keränen, N., Falkenbach, P., Maijala, A., Kolehmainen, T. T., & Reponen, J. (2019) · Finnish Journal of eHealth and eWelfare · doi:10.23996/fjhw.82538 · Cited by 109
“parency, funding transparency, and price [43].”
BackgroundWykes, T., & Schueller, S. M. (2019) · Journal of Medical Internet Research · doi:10.2196/12390 · Cited by 105
“Their views do not necessarily coincide with the expert groups’ views, as shown in the Delphi exercise by Zelmer et al [33].”
Charles, A., Nixdorf, R., Ibrahim, N., Meir, L. G., Mpango, R. S., Ngakongwa, F., Nudds, H., Pathare, S., Ryan, G. K., Repper, J., et al. (2021) · JMIR Mental Health · doi:10.2196/25528 · Cited by 63
“%) compared with other Delphi studies [46].”
Wong, H. W., Lo, B., Shi, J., Hollenberg, E., Abi-Jaoudé, A., Johnson, A., Chaim, G., Cleverley, K., Henderson, J. L., LEVINSON, A. J., et al. (2021) · JMIR Mental Health · doi:10.2196/23447 · Cited by 53
“Mobile mental health assessment frameworks, such as those developed by Chan et al [40], Zelmer et al [41], and Stoyanov et al [42], help researchers evaluate apps and guide developers in building high-quality, safe, and effective tools.”
Ortiz, A., Maslej, M. M., Husain, M. I., Daskalakis, Z. J., & Mulsant, B. H. (2021) · Journal of Affective Disorders · doi:10.1016/j.jad.2021.08.140 · Cited by 48
Jacob, C., Lindeque, J. P., Klein, A., Ivory, C. J., Heuss, S., & Peter, M. K. (2023) · JMIR Human Factors · doi:10.2196/45143 · Cited by 62
Bowie-DaBreo, D., Sas, C., Iles‐Smith, H., & Sünram‐Lea, S. I. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517498 · Cited by 49
“This is related to ethical issues in areas of risks and safety [50, 53, 57, 68, 71, 85, 97, 110, 112, 114, 125, 126], and benefits and evidence [45, 50, 53, 57, 68, 70, 71, 85, 97, 110, 112, 114, 125-127].”
User Perspectives on a Resilience-Building App (JoyPop): Qualitative Study
Mushquash, A. R., Pearson, E. S., Waddington, K., MacIsaac, A., Mohammed, S., Grassia, E., Smith, S., & Wekerle, C. (2021) · JMIR mhealth and uhealth · doi:10.2196/28677 · Cited by 37
Using apps for bipolar disorder – An online survey of healthcare provider perspectives and practices
Morton, E., Torous, J., Murray, G., & Michalak, E. E. (2021) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2021.02.047 · Cited by 36
The Mobile App Development and Assessment Guide (MAG): Delphi-Based Validity Study
Llorens-Vernet, P., & Miró, J. (2020) · JMIR mhealth and uhealth · doi:10.2196/17760 · Cited by 40
“…indicators [33]; predictors of chronic pain and disability in children [34]; predictors of chronic pain in adults with cancer [35]; the needs of adolescents and young adult cancer survivors [36]; and, in the mHealth field, to develop an assessment framework for electronic mental health apps [37].”
BackgroundConnected Mental Health: Systematic Mapping Study
(2020) · doi:10.2196/19950 · Cited by 33
Evans, K., Donelan, J. F., Rennick‐Egglestone, S., Cox, S., & Kuipers, Y. (2021) · Journal of Medical Internet Research · doi:10.2196/31831 · Cited by 34
“Zelmer et al [63] sought consensus from a broad group of stakeholders on guiding principles and criteria for a framework to assess e-mental health apps.”
BackgroundBowie-DaBreo, D., Sünram‐Lea, S. I., Sas, C., & Iles‐Smith, H. (2020) · JMIR Formative Research · doi:10.2196/14988 · Cited by 34
“Reviews of the ethics of mobile mental health [33,34] have also raised concerns with acceptance [35], risks and safety of apps [36-47], and the poverty of evidence regarding benefits and outcomes [36-40,42-46,48,49].”
BackgroundUnderstanding Mobile Health and Youth Mental Health: Scoping Review
Ding, X., Wuerth, K., Sakakibara, B., Schmidt, J., Parde, N., Holsti, L., & Barbic, S. (2023) · JMIR mHealth and uHealth · doi:10.2196/44951 · Cited by 30
“In a Canadian study where participants were asked to rate mHealth apps objectively, results were highly variable, and 28% of reviewers were not even sure about the overall quality of the health product [57].”
Rezaee, R., Khashayar, M., Saeedinezhad, S., Nasiri, M., & Zare, S. (2023) · JMIR mhealth and uhealth · doi:10.2196/39055 · Cited by 29
Smartphone psychology: New approaches towards safe and efficacious mobile mental health apps.
Marshall, J. M., Dunstan, D. A., & Bartik, W. (2020) · Professional Psychology: Research and Practice · doi:10.1037/pro0000278 · Cited by 25
Mental Health Mobile Apps in the French App Store: Assessment Study of Functionality and Quality
Carrouel, F., de Vigneulles, B. D. S., Bourgeois, D. M., Kabuth, B., Baltenneck, N., Nusbaum, F., Burge, V., Roy, S., Buchheit, S., Carrion-Martinaud, M.-L., et al. (2022) · JMIR mhealth and uhealth · doi:10.2196/41282 · Cited by 25
“They have positive effects on depressive symptoms, anxiety, and other mental health problems [65] and help overcome some of the barriers that make it difficult to access mental health services (stigma, cost, capacity, and geography) [66].”
BackgroundBatterham, P. J., Calear, A. L., O’Dea, B., Larsen, M. E., J Kavanagh, D., Titov, N., March, S., Hickie, I., Teesson, M., Dear, B. F., et al. (2019) · DIGITAL HEALTH · doi:10.1177/2055207619878069 · Cited by 21
“However, there are broader considerations that may be important in assisting potential users to determine the most appropriate intervention.(14) Accreditation systems for digital interventions need to account for multiple factors in identifying interventions that are appropriate for public use and providing recommenda…”
BackgroundComparing Professional and Consumer Ratings of Mental Health Apps: Mixed Methods Study
Hudson, G., Negbenose, E., Neary, M., Jansli, S. M., Schueller, S. M., Wykes, T., & Jilka, S. (2022) · JMIR Formative Research · doi:10.2196/39813 · Cited by 21
Jilka, S., Simblett, S. K., Odoi, C. M., van Bilsen, J., Wieczorek, A., Erturk, S., Wilson, E., Mutepua, M., & Wykes, T. (2021) · Internet Interventions · doi:10.1016/j.invent.2021.100433 · Cited by 23
Perowne, R., Rowe, S. L., & Gutman, L. M. (2024) · Health Expectations · doi:10.1111/hex.14102 · Cited by 20
Jacob, C., Lindeque, J. P., Müller, R., Klein, A., Metcalfe, T., Connolly, S. L., Koerber, F., Maguire, R., Denis, F., Heuss, S., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00982-w · Cited by 20
“She then used snowball sampling to minimize potential selection bias and expand the sample through the network of other participants 66 , in alignment with previous studies that had similar objectives and used similar sampling methods 69,70 .”
Evaluation Criteria for Weight Management Apps: Validation Using a Modified Delphi Process
Robles, N., Puigdomènech, E., Gómez-Calderón, C., Saigí-Rubió, F., Cuatrecasas, G., Zamora, A., Moharra, M., Paluzié, G., Balfegó, M., & Carrión, C. (2020) · JMIR mhealth and uhealth · doi:10.2196/16899 · Cited by 18
Treating Psychological Trauma in the Midst of COVID-19: The Role of Smartphone Apps
Marshall, J. M., Dunstan, D. A., & Bartik, W. J. (2020) · Frontiers in Public Health · doi:10.3389/fpubh.2020.00402 · Cited by 13
Malik, I., Perez, A. J., Toombs, E., SCHMIDT, F. H., Olthuis, J. V., Charlton, J. K., Grassia, E., Squier, C., Stasiuk, K., Bobinski, T., et al. (2023) · Frontiers in Digital Health · doi:10.3389/fdgth.2023.1197362 · Cited by 11
Frey, A.-L., Phillips, B., Baines, R., McCabe, A., Elmes, E., Yeardsley-Pierce, E., Wall, R., Parry, J., Vose, A., Hewitt, J., et al. (2024) · Health and Technology · doi:10.1007/s12553-024-00934-8 · Cited by 11
Help Is Just a Message Away: Online Counselling Chat Services Bridging Gaps in Youth Mental Health?
Dewaele, A., Denayer, E., Cabello, M., Higuera-Lozano, I., Pitkänen, T., Felvinczi, K., Kaló, Z., Soininvaara, S., & Goossens, L. (2025) · European Journal of Investigation in Health Psychology and Education · doi:10.3390/ejihpe15120257 · Cited by 9
“Training and support for providers are essential, as are meaningful stakeholder engagement and referral pathways, including youth voices and stigmatized communities (Zelmer et al., 2018; Ramos et al., 2021).”
Sleep well, worry less: A co-design study for the development of the SMILE app
Cormier, M., Orr, M., Käser, A., MacDonald, H., Chorney, J. M. L., & Meier, S. M. (2024) · Digital Health · doi:10.1177/20552076241283242 · Cited by 8
“13,49 Recognizing the signi fi - cance of these perspectives in tailoring apps to individual needs and stipulating the acceptance and uptake of mental health apps, the objective of the current study was to collaborate with youths in the design of a novel mental health app leveraging mobile sensing…”
Privacy and security in digital therapeutics
Nurgalieva, L., & Doherty, G. J. (2022) · Elsevier eBooks · doi:10.1016/b978-0-323-90045-4.00002-2 · Cited by 9
Lai, L., Sanatkar, S., Mackinnon, A. J., Deady, M., Petrie, K., Lipscomb, R., Counson, I., Francis‐Taylor, R., Dean, K., & Harvey, S. B. (2024) · JMIR Research Protocols · doi:10.2196/58288 · Cited by 6
“However, the intervention must undergo rigorous trials, given that many digital interventions currently available for mental health are recommended without testing and may be ineffective [33].”
BackgroundMindel, C., Mainstone-Cotton, L., García, S. D. O., Sefi, A., Sugarman, G., Salhi, L., Brick, H., Jackson, K., & Hanley, T. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.872404 · Cited by 7
“The Delphi technique is frequently used in healthcare research and has previously been used to modify a social responsiveness scale (36), while the Estimate-TalkEstimate method variation has been used in developing a framework for mental health apps (37).”
MethodEvaluation of Chinese HIV Mobile Apps by Researchers and Patients With HIV: Quality Evaluation Study
Liu, P., Wang, L., & Wang, F. (2024) · JMIR mhealth and uhealth · doi:10.2196/52573 · Cited by 6
“Developed by Zelmer et al [35], the MARS scale provides a standardized set of criteria for evaluating mobile apps in terms of engagement, functionality, esthetics, and information quality.”
MethodStakeholders' views and opinions on existing guidelines on “How to Choose Mental Health Apps”
Khan, W., Jebanesan, B., Ahmed, S., Trimmer, C., Agic, B., Safa, F., Ashraf, A., Tuck, A. C., Kavic, K., Wadhawan, S., et al. (2023) · Frontiers in Public Health · doi:10.3389/fpubh.2023.1251050 · Cited by 6
“In terms of AETs, Zelmer et al. (31), highlight that organizations or individuals recommend Apps based on various factors; however, it is uncertain if these factors are generalizable to different groups based on culture, gender, and language (31).”
BackgroundBuitenweg, D. C., van de Mheen, D., Grund, J.-P., van Oers, H. V., & van Nieuwenhuizen, C. (2020) · JMIR Mental Health · doi:10.2196/19593 · Cited by 5
LionVu 2.0 Usability Assessment for Pennsylvania, United States
Geyer, N. R., Kessler, F. C., & Lengerich, E. J. (2020) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi9110619 · Cited by 6
“[2,3], discordance or tension [4], qualitative weight and sum method [5], mixed methods [6], focus groups [7], or surveys [8].”
MethodZheng, T. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.785643 · Cited by 6
“, 2004), accelerate the development and deployment of new products (Yadav et al., 2007; Nadkarni and Chen, 2014), and maintain relationships with stakeholders (Flammer and Bansal, 2017).”
BackgroundAlberta Rating Index for Apps: Study of Reliability and Validity
Azad‐Khaneghah, P., Roberts, M. R., & Liu, L. (2022) · Canadian Journal of Occupational Therapy · doi:10.1177/00084174221085451 · Cited by 7
Essential Design Principles for a Family Digital Mental Health Intervention: A Delphi Study
Welsh, E. T., Cloud, Z. C. G., Boyd, J., & McIntosh, J. E. (2025) · Journal of Marital and Family Therapy · doi:10.1111/jmft.70045 · Cited by 4
“The Delphi method has been used in prior studies addressing issues of engagement in digital mental health (Kodish et al. 2023) and where the expert panel comprised both professional and consumer expertise (Bailey et al. 2024; Lo et al. 2022; Murphy et al. 2020; Zelmer et al. 2018).”
van Mierlo, T., Fournier, R., Yeung, S. K., & Lahutina, S. (2025) · JMIR Research Protocols · doi:10.2196/73773 · Cited by 5
“Following best practice [63,91-93], the Ukrainian version of the Shared Strength course is undergoing cultural adaptation.”
Alsaadi, B., & Alahmadi, D. H. (2022) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2022.2144125 · Cited by 5
“…round in the traditional Delphi contains open-ended questions as a starting point for discussions (Skulmoski et al., 2007), while the first round in the modified Delphi starts with voting questions, as the discussion points are pre-identified (Zelmer et al., 2018), usually from the literature.”
BackgroundLeventhal, E. L., Toner, E. R., Davidson, B., Boukhechba, M., Barnes, L. E., & Teachman, B. A. (2024) · Journal of Technology in Behavioral Science · doi:10.1007/s41347-024-00445-3 · Cited by 5
“Emphasizing the program rationale has not been a key feature in many previous mobile mental health assessment frameworks (e.g., Aryana et al., (2018); Chan et al., (2015); Zelmer et al. (2018); and Stoyanov et al. (2015)).”
Karthikeyan, R., Al-Shamaa, N., Kelly, E. J., Henn, P., Shiely, F., Divala, T. H., Fadahunsi, K. P., & O’Donoghue, J. (2024) · BMJ Open · doi:10.1136/bmjopen-2023-077148 · Cited by 5
“Based on other frameworks reviewed, 16 17 the MST has been modified to include the following additional criteria: ► Type of data collection tool.”
MethodOzolcer, M., Yang, Y., Kate, S., Samant, P., & Bae, S. W. (2024) · Conference on systems engineering research series · doi:10.1007/978-3-031-62554-1_37 · Cited by 3
Wanderlust and Motherhood: Navigating Pregnancy While Travelling
Fatima, Z., Fatima, S., Reddy, C. K. K., & Lippert, K. J. (2025) · IGI Global eBooks · doi:10.4018/979-8-3693-8774-0.ch027 · Cited by 1
Navigating E-Mental Health: How to Choose an Effective and Trustable App?
Song, M. J., Tabi, K., & Krausz, M. R. (2020) · Springer eBooks · doi:10.1007/978-3-319-70134-9_106-1 · Cited by 1
Digital interventions for mental health
Hariman, K. W. (2022) · Oxford University Press eBooks · doi:10.1093/med/9780198861478.003.0055 · Cited by 1
Evaluating performance of routine health information system for reproductive health in Tshwane
Moloko, S. M. (2021) · Unisa Institutional Repository (University of South Africa) · Cited by 1
Ding, X. (2025) · cIRcle (University of British Columbia) · doi:10.14288/1.0448344 · Cited by 3
Evaluating the Requirements of Digital Stress Management Systems: A Modified Delphi Study
Blankenhagel, K. J., Linker, M., & Zarnekow, R. (2020) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-030-49289-2_18
Buitenweg, D. C. (2020) · Tilburg University Research Portal
Navigating E-Mental Health: How to Choose an Effective and Trustable App?
Song, M. J., Tabi, K., & Krausz, M. R. (2021) · Springer eBooks · doi:10.1007/978-3-030-57296-9_106
Confidence and mastery in mothering occupations
Odgers, S. (2023) · doi:10.34074/thes.6056 · Cited by 1
Hunter, J. F., Schueller, S. M., Walsh, L. C., & Chan, C. (2023) · Frontiers research topics · doi:10.3389/978-2-8325-3790-9
Vallely, P. (2023) · Open Access Victoria University of Wellington | Te Herenga Waka (Figshare) · doi:10.26686/wgtn.24647856
Future Trends in Healthcare AI Section
Selvakumar, P., Mamatha, K. R., Justilin, J. R., Periyasamy, P. N., & Athavale, V. A. (2024) · Advances in healthcare information systems and administration book series · doi:10.4018/979-8-3693-6294-5.ch017
Validation and selection of criteria for evaluating apps for managing depression: a Delphi study
Robles, N., Fuster‐Casanovas, A., Melero, S. M., Duarte‐Díaz, A., Gelabert, E., Padilla-Ruiz, M., Gómez-Calderón, C., Pérez-Navarro, A., Vidal‐Alaball, J., Subirà, S., et al. (2025) · BMJ Open · doi:10.1136/bmjopen-2025-101302
A transdisciplinary framework for ethical mobile mental health
Bowie-DaBreo, D. (2024) · Lancaster EPrints (Lancaster University) · doi:10.17635/lancaster/thesis/2433
Moyne, A., Pérez, T., van Mierlo, T., Kebbe, M., & Drouin, O. (2026) · JMIR Pediatrics and Parenting · doi:10.2196/84304
“Active stakeholder involvement is essential in developing screening and intervention tools [39-41].”
Tong, F., Antoniou, S., & Larsen, M. (2026) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocag154
Environmental Scan of Mental Health App Libraries
CDA-AMC (2026) · Canadian Journal of Health Technologies · doi:10.51731/cjht.2026.1539
Metadata
- Authors
- Zelmer, J., van Hoof, K., Notarianni, M., van Mierlo, T., Schellenberg, M., & Tannenbaum, C.
- Affiliations
- University of Victoria (CA)
- Mental Health Commission of Canada (CA)
- Institut Universitaire de Gériatrie de Montréal (CA)
- ORCID
- Venue
- JMIR mHealth and uHealth
- Publisher
- JMIR Publications Inc.
- Published
- 2018-07-09
- Volume
- 6
- Issue
- 7
- Pages
- e10016
- ISSN
- 2291-5222 (ISSN-L)
- PMC
- PMC6056739
- OpenAlex
- W2808871401
- Cited by
- 62
- References
- 31
- Citations by year
- 2026: 2 · 2025: 6 · 2024: 9 · 2023: 9 · 2022: 9 · 2021: 10 · 2020: 11 · 2019: 6
- FWCI
- 8.51
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR mhealth and uhealth (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- mental health apps, stakeholder consensus, usability, effectiveness, privacy and security, cultural appropriateness, gender responsiveness, user-centered design, lived experience of mental illness, interoperability, moral norms, user inclusion, empathy map, Delphi consensus
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.


