Considerations of diversity, equity, and inclusion in mental health apps: A scoping review of evaluation frameworks
Open accessThe DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Abstract
Mental health applications or apps are a promising approach to reduce the pervasive unmet mental health need observed among marginalized groups. However, these groups face unique challenges to engage in and benefit from these interventions, and apps that consider issues of diversity, equity, and inclusion (DEI) may be better positioned to reach them. App evaluation frameworks emerge as crucial tools for researchers, clinicians, and users to select evidence-based apps. Nevertheless, it is unclear the extent to which existing assessment tools capture DEI factors. The present scoping review identified 68 studies that employed one or more assessment tools to evaluate a mental health app, leading to 44 unique app evaluation frameworks. Results showed that most frameworks were developed after 2015, and only 58% of them considered at least one DEI criterion. Frameworks that performed exceptionally well were those that combined multiple standardized and validated measures. This lack of consideration for DEI variables may limit the ability of app-based interventions to serve marginalized communities, or even worse, create new disparities. Therefore, we provide recommendations to improve current app evaluation frameworks' cultural robustness and clinical utility, maximizing their effectiveness when working with individuals from marginalized communities.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation
Tricco, A. C., Lillie, E., Zarin, W., O'Brien, K. K., Colquhoun, H., Levac, D., Moher, D., Peters, M. D. J., Horsley, T., Weeks, L., et al. (2018) · Annals of Internal Medicine · doi:10.7326/m18-0850 · Cited by 43,565
PRISMA-P Group, Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., & Stewart, L. A. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-1 · Cited by 26,223
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092
Munn, Z., Peters, M. D. J., Stern, C., Tufanaru, C., McArthur, A., & Aromataris, E. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0611-x · Cited by 11,385
A scoping review of scoping reviews: advancing the approach and enhancing the consistency
Pham, M. T., Rajić, A., Greig, J. D., Sargeant, J. M., Papadopoulos, A., & McEwen, S. A. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1123 · Cited by 3,078
Mobile app rating scale: a new tool for assessing the quality of health mobile apps
Stoyanov, S. R., Hides, L., Kavanagh, D. J., Zelenko, O., Tjondronegoro, D., & Mani, M. (2015) · JMIR mHealth and uHealth · doi:10.2196/mhealth.3422 · Cited by 2,233
Barriers to mental health treatment: results from the WHO World Mental Health surveys
Andrade, L. H., Alonso, J., Mneimneh, Z., Wells, J. E., Al-Hamzawi, A., Borges, G., Bromet, E., Bruffaerts, R., de Girolamo, G., de Graaf, R., et al. (2014) · Psychological Medicine · doi:10.1017/s0033291713001943 · Cited by 1,289
Persons with disabilities as an unrecognized health disparity population
Krahn, G. L., Walker, D. K., & Correa-De-Araujo, R. (2015) · American Journal of Public Health · doi:10.2105/ajph.2014.302182 · Cited by 1,080
Borghouts, J., Eikey, E., Mark, G., De Leon, C., Schueller, S. M., Schneider, M., Stadnick, N., Zheng, K., Mukamel, D., & Sorkin, D. H. (2021) · Journal of Medical Internet Research · doi:10.2196/24387 · Cited by 1,069
Barriers to mental health treatment: results from the National Comorbidity Survey Replication
Mojtabai, R., Olfson, M., Sampson, N. A., Jin, R., Druss, B., Wang, P. S., Wells, K. B., Pincus, H. A., & Kessler, R. C. (2011) · Psychological Medicine · doi:10.1017/s0033291710002291 · Cited by 951
Evans-Lacko, S., Aguilar-Gaxiola, S., Al-Hamzawi, A., Alonso, J., Benjet, C., Bruffaerts, R., Chiu, W. T., Florescu, S., de Girolamo, G., Gureje, O., et al. (2018) · Psychological Medicine · doi:10.1017/s0033291717003336 · Cited by 781
Good intentions are not enough: how informatics interventions can worsen inequality
Veinot, T. C., Mitchell, H., & Ancker, J. S. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy052 · Cited by 704
Linardon, J., & Fuller-Tyszkiewicz, M. (2020) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000459 · Cited by 573
Trends in Racial-Ethnic Disparities in Access to Mental Health Care, 2004–2012
Cook, B. L., Trinh, N.-H., Li, Z., Hou, S. S.-Y., & Progovac, A. M. (2017) · Psychiatric Services · doi:10.1176/appi.ps.201500453 · Cited by 546
Mobile Health Apps to Facilitate Self-Care: A Qualitative Study of User Experiences
Anderson, K., Burford, O., & Emmerton, L. (2016) · PLOS ONE · doi:10.1371/journal.pone.0156164 · Cited by 502
Waller, R., & Gilbody, S. (2009) · Psychological Medicine · doi:10.1017/s0033291708004224 · Cited by 393
Back to the Future: Achieving Health Equity Through Health Informatics and Digital Health
Brewer, L. C., Fortuna, K. L., Jones, C., Walker, R., Hayes, S. N., Patten, C. A., & Cooper, L. A. (2020) · JMIR mHealth and uHealth · doi:10.2196/14512 · Cited by 367
Kazdin, A. E. (2019) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.12937 · Cited by 274
Use of Digital Mental Health for Marginalized and Underserved Populations
Schueller, S. M., Hunter, J. F., Figueroa, C., & Aguilera, A. (2019) · Current Treatment Options in Psychiatry · doi:10.1007/s40501-019-00181-z · Cited by 268
State of the Field of Mental Health Apps
Neary, M., & Schueller, S. M. (2018) · Cognitive and Behavioral Practice · doi:10.1016/j.cbpra.2018.01.002 · Cited by 247
Criteria for assessing the quality of mHealth apps: a systematic review
Nouri, R., R Niakan Kalhori, S., Ghazisaeedi, M., Marchand, G., & Yasini, M. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy050 · Cited by 231
Torous, J., Chan, S. R., Yee-Marie Tan, S., Behrens, J., Mathew, I., Conrad, E. J., Hinton, L., Yellowlees, P., & Keshavan, M. (2014) · JMIR Mental Health · doi:10.2196/mental.4004 · Cited by 228
Evidence-based apps? A review of mental health mobile applications in a psychotherapy context
Lui, J. H. L., Marcus, D. K., & Barry, C. T. (2017) · Professional Psychology Research and Practice · doi:10.1037/pro0000122 · Cited by 217
Rathbone, J., Carter, M., Hoffmann, T., & Glasziou, P. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-6 · Cited by 206
Baumel, A., Faber, K., Mathur, N., Kane, J. M., & Muench, F. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7270 · Cited by 206
Geographic access to specialty mental health care across high- and low-income US communities
Cummings, J. R., Allen, L., Clennon, J., Ji, X., & Druss, B. G. (2017) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2017.0303 · Cited by 190
Pospos, S., Young, I. T., Downs, N., Iglewicz, A., Depp, C., Chen, J. Y., Newton, I., Lee, K., Light, G. A., & Zisook, S. (2018) · Academic Psychiatry · doi:10.1007/s40596-017-0868-0 · Cited by 180
Schueller, S. M., Neary, M., O'Loughlin, K., & Adkins, E. C. (2018) · Journal of Medical Internet Research · doi:10.2196/10141 · Cited by 167
Towards a Framework for Evaluating Mobile Mental Health Apps
Chan, S., Torous, J., Hinton, L., & Yellowlees, P. (2015) · Telemedicine and e-Health · doi:10.1089/tmj.2015.0002 · Cited by 163
Ramos, G., & Chavira, D. A. (2022) · Cognitive and Behavioral Practice · doi:10.1016/j.cbpra.2019.10.004 · Cited by 143
Self-citationRepresentative Literature Reviews Using Constrained Snowball Sampling and Citation Network Analysis
Lecy, J. D., & Beatty, K. E. (2012) · doi:10.2139/ssrn.1992601 · Cited by 133
Vilsaint, C. L., NeMoyer, A., Fillbrunn, M., Sadikova, E., Kessler, R. C., Sampson, N. A., Alvarez, K., Green, J. G., McLaughlin, K. A., Chen, R., et al. (2019) · Comprehensive Psychiatry · doi:10.1016/j.comppsych.2018.12.008 · Cited by 127
Evaluating evaluation frameworks: a scoping review of frameworks for assessing health apps
Lagan, S., Sandler, L., & Torous, J. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-047001 · Cited by 108
Text4baby: Development and Implementation of a National Text Messaging Health Information Service
Whittaker, R., Matoff-Stepp, S., Meehan, J., Kendrick, J., Jordan, E., Stange, P., Cash, A., Meyer, P., Baitty, J., Johnson, P., et al. (2012) · American Journal of Public Health · doi:10.2105/ajph.2012.300736 · Cited by 99
Kazdin, A. E. (2015) · Cognitive and Behavioral Practice · doi:10.1016/j.cbpra.2015.04.004 · Cited by 86
Terhorst, Y., Rathner, E.-M., Baumeister, H., & Sander, L. (2018) · Verhaltenstherapie · doi:10.1159/000481692 · Cited by 84
An Assessment Framework for e-Mental Health Apps in Canada: Results of a Modified Delphi Process
Zelmer, J., van Hoof, K., Notarianni, M., van Mierlo, T., Schellenberg, M., & Tannenbaum, C. (2018) · JMIR mHealth and uHealth · doi:10.2196/10016 · Cited by 62
What makes a good clinical app? Introducing the RCP Health Informatics Unit checklist
Wyatt, J. C., Thimbleby, H., Rastall, P., Hoogewerf, J., Wooldridge, D., & Williams, J. (2015) · Clinical Medicine · doi:10.7861/clinmedicine.15-6-519 · Cited by 52
Quality of Smartphone Apps Related to Panic Disorder
Van Singer, M., Chatton, A., & Khazaal, Y. (2015) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2015.00096 · Cited by 43
Ramos, G., Brookman-Frazee, L., Kodish, T., Rodriguez, A., & Lau, A. S. (2021) · Cultural Diversity & Ethnic Minority Psychology · doi:10.1037/cdp0000357 · Cited by 30
Self-citationPlaza García, I., Sánchez, C. M., Espílez, Á. S., García-Magariño, I., Guillén, G. A., & García-Campayo, J. (2017) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2017.05.018 · Cited by 25
Mobile App Tools for Identifying and Managing Mental Health Disorders in Primary Care
Magee, J. C., Adut, S., Brazill, K., & Warnick, S. (2018) · Current Treatment Options in Psychiatry · doi:10.1007/s40501-018-0154-0 · Cited by 25
Cited by
Lam, R. W., Kennedy, S. H., Adams, C., Bahji, A., Beaulieu, S., Bhat, V., Blier, P., Blumberger, D. M., Brietzke, E., Chakrabarty, T., et al. (2024) · The Canadian Journal of Psychiatry · doi:10.1177/07067437241245384 · Cited by 369
Timmons, A. C., Duong, J. B., Simo Fiallo, N., Lee, T., Vo, H. P. Q., Ahle, M. W., Comer, J. S., Brewer, L. C., Frazier, S. L., & Chaspari, T. (2023) · Perspectives on Psychological Science · doi:10.1177/17456916221134490 · Cited by 263
“One central goal of psychological science in the coming years will be to develop mental health interventions that are accessible and effective for all people (Gee et al., 2021; Gruber et al., 2021; Ramos et al., 2021).”
BackgroundSobowale, K., & Humphrey, D. K. (2025) · JMIR Formative Research · doi:10.2196/65605 · Cited by 15
“…users can easily navigate to begin conversing, whether chatbot outputs are at or below a 6th-grade reading level to aid comprehension [33], whether chatbots converse in multiple languages [33], whether the use of the chatbot is free [16,33], and whether chatbots can be used on mobile devices [33].”
Self-citationLorenzo-Luaces, L., Wasil, A., Kacmarek, C. N., & DeRubeis, R. (2024) · JMIR Formative Research · doi:10.2196/49780 · Cited by 13
“These findings reiterate how costs may be a barrier to mental health treatment and support other calls to enhance the accessibility of DMHIs for racial and ethnic minoritized individuals [14,47].”
BackgroundRamos, G., Aguilera, A., Montoya, A., Lau, A., Wen, C. Y., Cruz Torres, V., & Chavira, D. (2022) · JMIR Research Protocols · doi:10.2196/35196 · Cited by 12
“This line of research is necessary to guarantee that app-based interventions for POC attend to issues of diversity, equity, and inclusion, reducing inequities already present in brick-and-mortar mental health services [43].”
BackgroundSelf-citation
Metadata
- Authors
- Ramos, G., Ponting, C., Labao, J. P., & Sobowale, K.
- Venue
- Behaviour Research and Therapy
- Publisher
- Elsevier BV
- Published
- 2021-12
- Volume
- 147
- Pages
- 103990
- ISSN
- 0005-7967 (ISSN-L)
- OpenAlex
- W3207868655
- Cited by
- 117
- References
- 43
- FWCI
- 15.24
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.