Challenges in Assessing Mobile Health App Quality
Abstract
Context Mobile apps are poised to become a major source of health guidance; however, concerns about their quality and safety remain. Governments, clinicians, and health researchers are struggling to determine how to adequately evaluate the content and function of mobile health apps to guide consumers toward apps that will effectively and safely support their health. Researchers are both adapting existing research methods, such as those for systematic reviews, and exploring novel methods toward this aim. The purpose of this systematic review was to describe prevalent and emerging methods for searching, data extraction, and analysis in the context of mobile health apps and commercial app stores.Evidence acquisition A systematic search was conducted on April 6, 2016, according to Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines, for published studies that evaluated health-related apps targeted at consumers in a commercial app store, and a descriptive content analysis of their methods was performed.Evidence synthesis Although the majority of studies self-identified as "reviews" or "systematic reviews," their claims to have performed an exhaustive, replicable, and systematic search and data extraction are unsupported, given the nature of commercial app stores' search engines and personalized app content. Currently, studies of health-related app content are largely descriptive, and most use surrogate and one-dimensional outcome measures for app content quality.Conclusions This field is rapidly developing, and innovative methods for systematic searches, multidimensional quality assessment, and contextualizing content assessment are emerging. Areas for future methodologic development are highlighted, including expanding definitions of app "content" and assessing risk of bias.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Search results are personalized based on user characteristics.
“Search results are personalized based on user characteristics (Grundy et al. 2016)”
From Evaluating the quality, safety, and functionality of commonly used smartphone apps for bipolar disorder mood and sleep self-management (2022) · cites this paper as Grundy et al. 2016
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823 · see citing works
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000097 · Cited by 64,042 · see citing works
Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement.
(2009) · PubMed Central · Cited by 44,859 · see citing works
Mobile app rating scale: a new tool for assessing the quality of health mobile apps
Stoyanov, S. R., Hides, L., Kavanagh, D. J., Zelenko, O., Tjondronegoro, D., & Mani, M. (2015) · JMIR mHealth and uHealth · doi:10.2196/mhealth.3422 · Cited by 2,233 · see citing works
Just a Fad? Gamification in Health and Fitness Apps
Lister, C., West, J. H., Cannon, B., Sax, T., & Brodegard, D. (2014) · JMIR Serious Games · doi:10.2196/games.3413 · Cited by 489 · see citing works
Mobile Apps for Bipolar Disorder: A Systematic Review of Features and Content Quality
Nicholas, J., Larsen, M. E., Proudfoot, J., & Christensen, H. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4581 · Cited by 438 · see citing works
Huckvale, K., Prieto, J. T., Tilney, M., Benghozi, P.-J., & Car, J. (2015) · BMC Medicine · doi:10.1186/s12916-015-0444-y · Cited by 376 · see citing works
Bardus, M., van Beurden, S. B., Smith, J. R., & Abraham, C. (2016) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-016-0359-9 · Cited by 350 · see citing works
Mobile Applications for Weight Management
Azar, K. M. J., Lesser, L. I., Laing, B. Y., Stephens, J., Aurora, M. S., Burke, L. E., & Palaniappan, L. P. (2013) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2013.07.005 · Cited by 334 · see citing works
Direito, A., Dale, L. P., Shields, E., Dobson, R., Whittaker, R., & Maddison, R. (2014) · BMC Public Health · doi:10.1186/1471-2458-14-646 · Cited by 308 · see citing works
Medical apps for smartphones: lack of evidence undermines quality and safety
Buijink, A. W. G., Visser, B. J., & Marshall, L. (2013) · Evidence Based Medicine · doi:10.1136/eb-2012-100885 · Cited by 303 · see citing works
Cowan, L. T., Van Wagenen, S. A., Brown, B. A., Hedin, R. J., Seino-Stephan, Y., Hall, P. C., & West, J. H. (2013) · Health Education & Behavior · doi:10.1177/1090198112452126 · Cited by 264 · see citing works
Lalloo, C., Jibb, L. A., Rivera, J., Agarwal, A., & Stinson, J. N. (2015) · The Clinical Journal of Pain · doi:10.1097/ajp.0000000000000171 · Cited by 257 · see citing works
‘It's like having a physician in your pocket!’ A critical analysis of self-diagnosis smartphone apps
Lupton, D. A., & Jutel, A. (2015) · Social Science & Medicine · doi:10.1016/j.socscimed.2015.04.004 · Cited by 253 · see citing works
Apps as Artefacts: Towards a Critical Perspective on Mobile Health and Medical Apps
Lupton, D. A. (2014) · Societies · doi:10.3390/soc4040606 · Cited by 251 · see citing works
The mobile revolution—using smartphone apps to prevent cardiovascular disease
Neubeck, L., Lowres, N., Benjamin, E. J., Freedman, S. B., Coorey, G., & Redfern, J. (2015) · Nature Reviews Cardiology · doi:10.1038/nrcardio.2015.34 · Cited by 232 · see citing works
Mindfulness-Based Mobile Applications: Literature Review and Analysis of Current Features
Plaza, I., Demarzo, M., Herrera‐Mercadal, P., & García‐Campayo, J. (2013) · JMIR mhealth and uhealth · doi:10.2196/mhealth.2733 · Cited by 205 · see citing works
Smartphone apps for calculating insulin dose: a systematic assessment
Huckvale, K., Adomaviciute, S., Prieto, J. T., Leow, M. K., & Car, J. (2015) · BMC Medicine · doi:10.1186/s12916-015-0314-7 · Cited by 200 · see citing works
Assessing the quality and usability of smartphone apps for pain self-management
Reynoldson, C., Stones, C., Allsop, M., Gardner, P., Bennett, M. I., Closs, S. J., Jones, R., & Knapp, P. (2014) · Pain Medicine · doi:10.1111/pme.12327 · Cited by 178 · see citing works
mHealth: A Strategic Field without a Solid Scientific Soul. A Systematic Review of Pain-Related Apps
de la Vega, R., & Miró, J. (2014) · PLoS ONE · doi:10.1371/journal.pone.0101312 · Cited by 177 · see citing works
The evolution of mobile apps for asthma: an updated systematic assessment of content and tools
Huckvale, K., Morrison, C., Ouyang, J., Ghaghda, A., & Car, J. (2015) · BMC Medicine · doi:10.1186/s12916-015-0303-x · Cited by 166 · see citing works
How Smart are Smartphone Apps for Smoking Cessation? A Content Analysis
Hoeppner, B. B., Hoeppner, S. S., Seaboyer, L. M., Schick, M. R., Wu, G. W. Y., Bergman, B. G., & Kelly, J. F. (2015) · Nicotine & Tobacco Research · doi:10.1093/ntr/ntv117 · Cited by 148 · see citing works
A Systematic Review of Quality Assessment Methods for Smartphone Health Apps
BinDhim, N. F., Hawkey, A., & Trevena, L. (2015) · Telemedicine and e-Health · doi:10.1089/tmj.2014.0088 · Cited by 142 · see citing works
Tricco, A. C., Soobiah, C., Blondal, E., Veroniki, A. A., Khan, P. A., Vafaei, A., Ivory, J. D., Strifler, L., Ashoor, H. M., MacDonald, H. R., et al. (2015) · BMC Medicine · doi:10.1186/s12916-015-0371-y · Cited by 141 · see citing works
Smartphone breast applications – What's the evidence?
Mobasheri, M. H., Johnston, M., King, D., Leff, D., Thiruchelvam, P., & Darzi, A. (2014) · The Breast · doi:10.1016/j.breast.2014.07.006 · Cited by 137 · see citing works
mHealthApps: A Repository and Database of Mobile Health Apps
Xu, W., & Liu, Y. (2015) · JMIR mhealth and uhealth · doi:10.2196/mhealth.4026 · Cited by 129 · see citing works
Knight, E., Stuckey, M. I., Prapavessis, H., & Petrella, R. J. (2015) · JMIR mHealth and uHealth · doi:10.2196/mhealth.4003 · Cited by 112 · see citing works
Commercially Available Mobile Phone Headache Diary Apps: A Systematic Review
Hundert, A. S., Huguet, A., McGrath, P. J., Stinson, J., & Wheaton, M. (2014) · JMIR mhealth and uhealth · doi:10.2196/mhealth.3452 · Cited by 103 · see citing works
Digitizing diagnosis: a review of mobile applications in the diagnostic process
Jutel, A., & Lupton, D. A. (2015) · Diagnosis · doi:10.1515/dx-2014-0068 · Cited by 72 · see citing works
Comparative analysis of smoking cessation smartphone applications available in 2012 versus 2014
Ubhi, H. K., Kotz, D., Michie, S., van Schayck, O. C. P., Sheard, D., Selladurai, A., & West, R. (2016) · Addictive Behaviors · doi:10.1016/j.addbeh.2016.02.026 · Cited by 60 · see citing works
How trustworthy are apps for maternal and child health?
Scott, K. M., Gome, G. A., Richards, D., & Caldwell, P. H. Y. (2015) · Health and Technology · doi:10.1007/s12553-015-0099-x · Cited by 58 · see citing works
Smartphone applications in burns
Wurzer, P., Parvizi, D., Lumenta, D. B., Giretzlehner, M., Branski, L. K., Finnerty, C. C., Herndon, D. N., Tuca, A., Rappl, T., Smolle, C., et al. (2015) · Burns · doi:10.1016/j.burns.2014.11.010 · Cited by 56 · see citing works
Lee, H., Sullivan, S. J., Schneiders, A. G., Ahmed, O. H., Balasundaram, A. P., Williams, D. D., Meeuwisse, W. H., & McCrory, P. R. (2014) · British Journal of Sports Medicine · doi:10.1136/bjsports-2013-092930 · Cited by 54 · see citing works
Compliance of Blood Donation Apps with Mobile OS Usability Guidelines
Ouhbi, S., Fernández‐Alemán, J. L., Pozo, J. R., Bajta, M. E., Toval, A., & Idri, A. (2015) · Journal of Medical Systems · doi:10.1007/s10916-015-0243-1 · Cited by 53 · see citing works
Popular Glucose Tracking Apps and Use of mHealth by Latinos With Diabetes: Review
Williams, J. P., & Schroeder, D. G. (2015) · JMIR mhealth and uhealth · doi:10.2196/mhealth.3986 · Cited by 49 · see citing works
van Mechelen, D. M., van Mechelen, W., & Verhagen, E. A. L. M. (2013) · British Journal of Sports Medicine · doi:10.1136/bjsports-2012-092136 · Cited by 43 · see citing works
A scientific audit of smartphone applications for the management of obesity
Gan, K. O., & Allman-Farinelli, M. A. (2011) · Australian and New Zealand Journal of Public Health · doi:10.1111/j.1753-6405.2011.00707.x · Cited by 37 · see citing works
Medical Apps: Public and Academic Perspectives
Krieger, W. H. (2013) · Perspectives in biology and medicine · doi:10.1353/pbm.2013.0013 · Cited by 30 · see citing works
Content Analysis of Diet-Related Mobile Apps: A Self-Regulation Perspective
Zahry, N. R., Cheng, Y., & Peng, W. (2016) · Health Communication · doi:10.1080/10410236.2015.1072123 · Cited by 29 · see citing works
Health Behavior Theory in Popular Calorie Counting Apps: A Content Analysis
Davis, S. F., Marisa Ellsworth, A., Payne, H., Hall, S. M., West, J. H., & Nordhagen, A. L. (2016) · JMIR mhealth and uhealth · doi:10.2196/mhealth.4177 · Cited by 27 · see citing works
Advising patients on selecting trustful apps for diabetes self-care
Basilico, A., Marceglia, S. R. F., Bonacina, S., & Pinciroli, F. (2016) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2016.02.005 · Cited by 26 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Criteria for assessing the quality of mHealth apps: a systematic review
Nouri, R., R Niakan Kalhori, S., Ghazisaeedi, M., Marchand, G., & Yasini, M. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy050 · Cited by 231 · see citing works
“More recently, Grundy et al.(60) conducted a systematic review to find and explain emerging and common methods for evaluating the quality of mHealth apps.”
BackgroundSmartphone apps for the self-management of low back pain: A systematic review
Machado, G. C., Pinheiro, M. B., Lee, H., Ahmed, O. H., Hendrick, P. A., Williams, C. M., & Kamper, S. J. (2016) · Best Practice & Research Clinical Rheumatology · doi:10.1016/j.berh.2017.04.002 · Cited by 171 · see citing works
“assessing the quality and ‘fitness-for-purpose’ of apps is by using validated scales, and benchmarking app content against current best-practice guidelines [10].”
MethodMobile Apps for Medication Management: Review and Analysis
Tabi, K., Randhawa, A. S., Choi, F., Mithani, Z., Albers, F., Schnieder, M., Nikoo, M., Vigo, D., Jang, K. L., Demlová, R., et al. (2019) · JMIR mhealth and uhealth · doi:10.2196/13608 · Cited by 170 · see citing works
“The professional reviews could follow some of the standardized frameworks for quality assessment of mobile apps, for example, Mobile App Rating Scale [49], suitable for a wide range of health apps; a recently published assessment framework specifically for mental health apps [50]; or other published frameworks [33,51-…”
BackgroundBroglia, E., Millings, A., & Barkham, M. (2018) · British Journal of Guidance and Counselling · doi:10.1080/03069885.2017.1370695 · Cited by 141 · see citing works
A Review of the Quality and Impact of Mobile Health Apps
Grundy, Q. (2021) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-052020-103738 · Cited by 153 · see citing works
Self-citationUnsworth, H., Dillon, B., Collinson, L., Powell, H., Salmon, M., Oladapo, T., Ayiku, L., Shield, G., Holden, J., Patel, N., et al. (2021) · Digital Health · doi:10.1177/20552076211018617 · Cited by 137 · see citing works
Tofighi, B., Chemi, C., Ruiz-Valcarcel, J., Hein, P. N., & Hu, L. (2018) · JMIR mhealth and uhealth · doi:10.2196/11831 · Cited by 129 · see citing works
“Studies assessing app content are generally descriptive and limited to one-dimensional outcome quality measures [17].”
BackgroundMethodScott, I. A., Scuffham, P., Gupta, D., Harch, T. M., Borchi, J., & Richards, B. (2018) · Australian Health Review · doi:10.1071/ah18064 · Cited by 120 · see citing works
Mobile health app usability and quality rating scales: a systematic review
Azad‐Khaneghah, P., Neubauer, N., Cruz, A. M., & Liu, L. (2020) · Disability and Rehabilitation Assistive Technology · doi:10.1080/17483107.2019.1701103 · Cited by 119 · see citing works
Toward a Better Understanding of the Intention to Use mHealth Apps: Exploratory Study
(2021) · doi:10.2196/27021 · Cited by 108 · see citing works
“Furthermore, Grundy et al [17] studied the use of high-quality mHealth apps with”
BackgroundMobile Health in Oncology: A Patient Survey About App-Assisted Cancer Care
Kessel, K. A., Vogel, M. M. E., Kessel, C., Bier, H., Biedermann, T., Friess, H., Herschbach, P., von Eisenhart‐Rothe, R., Meyer, B., Kiechle, M. B., et al. (2017) · JMIR mhealth and uhealth · doi:10.2196/mhealth.7689 · Cited by 104 · see citing works
“However, a standardized validation process must be implemented for medical apps to guarantee safety for the patient [8,15,16].”
Beyond Dr. Google: the evidence on consumer-facing digital tools for diagnosis
Millenson, M. L., Baldwin, J. L., Zipperer, L., & Singh, H. (2018) · Diagnosis · doi:10.1515/dx-2018-0009 · Cited by 91 · see citing works
Hilty, D. M., Chan, S. R., Torous, J. B., Luo, J. S., & BOLAND, R. J. (2019) · JMIR mhealth and uhealth · doi:10.2196/12229 · Cited by 101 · see citing works
Users’ acceptability of a mobile application for persons with type 2 diabetes: a qualitative study
Torbjørnsen, A., Ribu, L., Rønnevig, M., Grøttland, A., & Helseth, S. (2019) · BMC Health Services Research · doi:10.1186/s12913-019-4486-2 · Cited by 101 · see citing works
“and for diabetes care in particular [3, 4].”
Richardson, B., Dol, J., Rutledge, K., Monaghan, J., Orovec, A., Howie, K., Boates, T., Smit, M., & Campbell-Yeo, M. (2018) · JMIR mhealth and uhealth · doi:10.2196/11620 · Cited by 84 · see citing works
“This is consistent with other studies reviewing mobile apps [35].”
MethodResultFurlong, L., Morris, M., Serry, T., & Erickson, S. (2018) · PLOS ONE · doi:10.1371/journal.pone.0201513 · Cited by 87 · see citing works
Wyatt, J. C. (2018) · BMC Medicine · doi:10.1186/s12916-018-1211-7 · Cited by 61 · see citing works
An Assessment Framework for e-Mental Health Apps in Canada: Results of a Modified Delphi Process
Zelmer, J., van Hoof, K., Notarianni, M., van Mierlo, T., Schellenberg, M., & Tannenbaum, C. (2018) · JMIR mHealth and uHealth · doi:10.2196/10016 · Cited by 62 · see citing works
“…review of methods used to assess mobile health apps found that researchers evaluated the app quality in 6 domains: scientific or clinical basis, functionality, usability, accountability, impact, and popularity; 80% (73/91) participants used measures drawn from one or more of these domains [14].”
BackgroundMethodAn overview on the emerging area of identification, characterization, and assessment of health apps
Paglialonga, A., Lugo, A., & Santoro, E. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2018.05.017 · Cited by 77 · see citing works
Vokinger, K. N., Nittas, V., Witt, C. M., Fabrikant, S. I., & von Wyl, V. (2020) · Swiss Medical Weekly · doi:10.4414/smw.2020.20282 · Cited by 82 · see citing works
Moshi, M. R., Tooher, R. L., & Merlin, T. (2018) · International Journal of Technology Assessment in Health Care · doi:10.1017/s026646231800051x · Cited by 74 · see citing works
“Only seven frameworks evaluated the harms of the app itself (e.g., adverse events) (14;18;19;26;32;36;37).”
BackgroundVijande, M. L. S., Gómez, M., Molina, A., & Davison, R. M. (2022) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2022.102956 · Cited by 82 · see citing works
Mobile Apps for Older Adults: Systematic Search and Evaluation Within Online Stores
Portenhauser, A. A., Terhorst, Y., Schultchen, D., Sander, L., Denkinger, M. D., Stach, M., Waldherr, N., Dallmeier, D., Baumeister, H., & Meßner, E.-M. (2021) · JMIR Aging · doi:10.2196/23313 · Cited by 77 · see citing works
A health app developer’s guide to law and policy: a multi-sector policy analysis
Parker, L., Karliychuk, T., Gillies, D., Mintzes, B., Raven, M., & Grundy, Q. (2017) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-017-0535-0 · Cited by 71 · see citing works
“There is, however, a particular lack of research on health app content and policy in countries outside of the Western, English speaking nations, and we would advocate for further work here [36].”
BackgroundMethodSelf-citationYang, X., Ma, L., Zhao, X., & Kankanhalli, A. (2019) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2019.104039 · Cited by 69 · see citing works
A Critical Appraisal of Fertility and Menstrual Tracking Apps for the iPhone
Zwingerman, R., Chaikof, M., & Jones, C. A. (2019) · Journal of Obstetrics and Gynaecology Canada · doi:10.1016/j.jogc.2019.09.023 · Cited by 68 · see citing works
Hilty, D. M., Armstrong, C. M., Luxton, D. D., Gentry, M. T., & Krupinski, E. A. (2021) · Journal of Technology in Behavioral Science · doi:10.1007/s41347-021-00199-2 · Cited by 68 · see citing works
The relevance of health literacy to mHealth
Kreps, G. L. (2017) · Information Services and Use · doi:10.3233/isu-170828 · Cited by 62 · see citing works
“Yet, there are serious communication challenges that must be addressed concerning the best uses of mHealth for health promotion, including assuring the appropriateness and effectiveness of mHealth messaging for audiences with different communication competencies, styles, and health literacy levels, to ensure that mHea…”
Carrasco-Hernandez, L., Jódar-Sánchez, F., Núñez-Benjumea, F., Moreno Conde, J., Mesa González, M., Civit-Balcells, A., Hors-Fraile, S., Parra-Calderón, C. L., Bamidis, P. D., & Ortega-Ruiz, F. (2020) · JMIR mHealth and uHealth · doi:10.2196/17530 · Cited by 65 · see citing works
Gulsrud, N. M., Raymond, C. M., Rutt, R. L., Olafsson, A. S., Plieninger, T., Sandberg, M., Beery, T. H., & Jonsson, K. (2018) · Landscape and Urban Planning · doi:10.1016/j.landurbplan.2018.08.012 · Cited by 62 · see citing works
Chen, A. T., Wu, S., Tomasino, K. N., Lattie, E. G., & Mohr, D. C. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103187 · Cited by 58 · see citing works
Mental Health Messages in Prominent Mental Health Apps
Parker, L. M., Bero, L. A., Gillies, D. M., Raven, M. K., Mintzes, B. J., Jureidini, J. N., & Grundy, Q. (2018) · The Annals of Family Medicine · doi:10.1370/afm.2260 · Cited by 58 · see citing works
Self-citationArsad, A. Z., Zuhdi, A. W. M., Ibrahim, N. B., & Hannan, M. A. (2023) · Applied Sciences · doi:10.3390/app13021218 · Cited by 57 · see citing works
“Numerous databases, including PubMed, Web of Science, Scopus, Web of Science, Google Scholar, and Baidu Scholar can be searched for articles containing strategy in research [52–54].”
BackgroundNightingale, R., Hall, A. G., Gelder, C., Friedl, S., Brennan, E., & Swallow, V. M. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7760 · Cited by 56 · see citing works
“However, to our knowledge, there are few coproduced, rigorously developed, evaluated apps providing tailored, CKD-specific, practical advice on day-to-day care management [13-20] and development processes for those that do exist are not systematic.”
BackgroundHouseh, M. S., Grainger, R., Petersen, C. C., Bamidis, P. D., & Merolli, M. (2018) · Yearbook of Medical Informatics · doi:10.1055/s-0038-1641197 · Cited by 55 · see citing works
“Adherence to medical devices/software certification and conformance with safety, security, and privacy issues should become standardized criteria for review of medical apps [41].”
Digital Technology and Mobile Health in Behavioral Migraine Therapy: a Narrative Review
Stubberud, A., & Linde, M. (2018) · Current Pain and Headache Reports · doi:10.1007/s11916-018-0718-0 · Cited by 49 · see citing works
Supporting Self-Management in Bipolar Disorder: Mixed-Methods Knowledge Translation Study
Michalak, E. E., Morton, E. E., Barnes, S. J., Hole, R., CREST.BD, & Murray, G. (2019) · JMIR Mental Health · doi:10.2196/13493 · Cited by 48 · see citing works
“Thus, there are significant barriers for people facing mental health challenges in identifying safe, evidence-informed apps [56,57].”
BackgroundJacob, C., Lindeque, J. P., Klein, A., Ivory, C. J., Heuss, S., & Peter, M. K. (2023) · JMIR Human Factors · doi:10.2196/45143 · Cited by 43 · see citing works
The Arabic Version of the Mobile App Rating Scale: Development and Validation Study
Bardus, M., Awada, N., Ghandour, L. A., Fares, E.-J., Gherbal, T., Al-Zanati, T., & Stoyanov, S. R. (2019) · JMIR mhealth and uhealth · doi:10.2196/16956 · Cited by 46 · see citing works
“Future studies may compare MARS to other instruments of app quality [23,30], even though they might not be equivalent.”
BackgroundTracing the Potential Flow of Consumer Data: A Network Analysis of Prominent Health and Fitness Apps
Grundy, Q., Held, F. P., & Bero, L. A. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7347 · Cited by 47 · see citing works
“…localized and personalized nature of app store search algorithms and rapidly changing population of apps, we triangulated two sampling strategies [12]: (1) a crawling program that systematically sampled the top-ranked 100 apps from the iTunes and Google Play app stores in the United States,…”
Self-citationBardus, M., Ali, A., Demachkieh, F., & Hamadeh, G. (2019) · JMIR mHealth and uHealth · doi:10.2196/mhealth.9836 · Cited by 44 · see citing works
“On one side, researchers focus on aspects related to theoretical and evidence-based content [38,39].”
Lambert, K., Mullan, J., Mansfield, K. J., & Owen, P. (2017) · Journal of Renal Nutrition · doi:10.1053/j.jrn.2017.06.007 · Cited by 43 · see citing works
“Importantly, up to one-third of health apps are designed to assist individuals with their chronic disease self-management.(4) Self-management of a chronic disease such as chronic kidney disease (CKD) is complex and requires an individual to balance complex dietary, lifestyle and medication regimens, as well as monitor…”
BackgroundFunnell, E. L., Spadaro, B., Martin-Key, N. A., Metcalfe, T., & Bahn, S. (2022) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2022.857304 · Cited by 41 · see citing works
“, in-app screenshots, app keyword, app description) based on geographical locations (83, 84).”
BackgroundMobile Health (mHealth) apps focused on menopause: Are they any good?
Gkrozou, F., Tsonis, O., Godden, M., Siafaka, V., & Paschopoulos, M. E. (2019) · Post Reproductive Health · doi:10.1177/2053369119878978 · Cited by 43 · see citing works
Portz, J. D., Moore, S. L., & Bull, S. S. (2024) · Journal of Medical Internet Research · doi:10.2196/62790 · Cited by 53 · see citing works
“The scientific evaluation of effectiveness and safety has not kept pace with the rapid expansion of the commercial health-related app market, creating a disconnect between scientific literature and available apps [50,60].”
Adoption of Mobile Health Apps in Dietetic Practice: Case Study of Diyetkolik
Akdur, G., Aydın, M. N., & Akdur, G. (2020) · JMIR mhealth and uhealth · doi:10.2196/16911 · Cited by 44 · see citing works
Challenges for developing health-care knowledge in the digital age
Alrahbi, D., Khan, M., Gupta, S., Modgil, S., & Jabbour, C. J. C. (2020) · Journal of Knowledge Management · doi:10.1108/jkm-03-2020-0224 · Cited by 43 · see citing works
Social Support for Breastfeeding in the Era of mHealth: A Content Analysis
Schindler-Ruwisch, J. M., Roess, A., Robert, R. C., Napolitano, M. A., & Chiang, S. C. (2018) · Journal of Human Lactation · doi:10.1177/0890334418773302 · Cited by 37 · see citing works
From Ancient Contemplative Practice to the App Store
Lukoff, K., Lyngs, U., Gueorguieva, S., Dillman, E., Hiniker, A., & Munson, S. A. (2020) · Designing Interactive Systems Conference · doi:10.1145/3357236.3395444 · Cited by 37 · see citing works
“n-distress indicates that they are not receiving adequate support from wherever they learned to meditate” [1]. Overstated benefits and understated risks have been reported in other health-related apps [35] and our results suggest this issue may be prevalent in mindfulness-related apps, too. Designers of technologies fo…”
BackgroundMobile Health, Smartphone/Device, and Apps for Psychiatry and Medicine
Hilty, D. M., Chan, S., Torous, J., Luo, J., & Boland, R. J. (2019) · Psychiatric Clinics of North America · doi:10.1016/j.psc.2019.05.007 · Cited by 38 · see citing works
Spadaro, B., Martin-Key, N. A., Funnell, E., & Bahn, S. (2022) · JMIR mhealth and uhealth · doi:10.2196/30724 · Cited by 34 · see citing works
“Unlike electronic journal databases, app stores are not designed for systematic search and export of data [69].”
BackgroundMarketing Innovations in Industry 4.0 and Their Impacts on Current Enterprises
Ungerman, O., & Dědková, J. (2019) · Applied Sciences · doi:10.3390/app9183685 · Cited by 34 · see citing works
The impact of mobile applications on medication adherence: a systematic review
Ng, R., Carter, S. R., & El‐Den, S. (2019) · Translational Behavioral Medicine · doi:10.1093/tbm/ibz125 · Cited by 31 · see citing works
Edwards, J., Waite-Jones, J., Schwarz, T., & Swallow, V. M. (2021) · Children · doi:10.3390/children8121203 · Cited by 31 · see citing works
Abbasi, E. (2025) · Environmental Toxicology and Chemistry · doi:10.1093/etojnl/vgaf209 · Cited by 29 · see citing works
Kalhori, S. R. N., Hemmat, M., Noori, T., Heydarian, S., & Katigari, M. R. (2020) · Current Diabetes Reviews · doi:10.2174/1573399816666200703181438 · Cited by 28 · see citing works
Beyond the virus: a first look at coronavirus-themed Android malware
Wang, L., Ren, H., Wang, H., Xia, P., Li, Y., Wu, L., Zhou, Y., Luo, X., Sui, Y., Guo, Y., et al. (2021) · Empirical Software Engineering · doi:10.1007/s10664-021-09974-4 · Cited by 28 · see citing works
Issues in reporting of systematic review methods in health app-focused reviews: A scoping review
Grainger, R., Devan, H., Sangelaji, B., & Hay-Smith, J. E. (2020) · Health Informatics Journal · doi:10.1177/1460458220952917 · Cited by 27 · see citing works
Mobile Health Apps for Breast Cancer: Content Analysis and Quality Assessment
Yang, S., Bui, C. N., & Park, K. (2023) · JMIR mhealth and uhealth · doi:10.2196/43522 · Cited by 31 · see citing works
“Therefore, methods for the systematic search and analysis of apps have been developed [9].”
MethodRECOVER-E – a mobile app for patients undergoing total knee or hip replacement: study protocol
Stauber, A., Schüßler, N., Palmdorf, S., Schürholz, N., Bruns, D., Osterbrink, J., & Nestler, N. (2020) · BMC Musculoskeletal Disorders · doi:10.1186/s12891-020-3090-2 · Cited by 24 · see citing works
How Do Infant Feeding Apps in China Measure Up? A Content Quality Assessment
Zhao, J., Freeman, B., & Li, M. (2017) · JMIR mhealth and uhealth · doi:10.2196/mhealth.8764 · Cited by 25 · see citing works
“Notably, studies assessing health-related apps were concentrated in English-speaking countries, whereas countries where apps were downloaded in large numbers such as China, Brazil, and Mexico were relatively neglected [16].”
BackgroundMethodGlattacker, M., Boeker, M., Anger, R., Reichenbach, F., Tassoni, A., Bredenkamp, R., & Giesler, J. M. (2020) · JMIR mhealth and uhealth · doi:10.2196/15514 · Cited by 26 · see citing works
“Therefore, health apps might increasingly become a “major source of health guidance” ([12] pg 1051) and have the potential to change existing health care delivery pathways [13].”
BackgroundPhillips, S. A., Ali, M. M., Modrich, C., Oke, S., Elokda, A. S., Laddu, D., & Bond, S. (2018) · Progress in Cardiovascular Diseases · doi:10.1016/j.pcad.2018.12.007 · Cited by 26 · see citing works
Management of cardiovascular disease using an mHealth tool: a randomized clinical trial
Kang, S., Baek, H., Cho, J.-H., Kim, S., Hwang, H. Y., Lee, W., Park, J. J., Yoon, Y. E., Yoon, C., Cho, Y., et al. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00535-z · Cited by 24 · see citing works
Ahmed, M., Oh, A., Vanderlee, L., Franco‐Arellano, B., Schermel, A., Lou, W. Y. W., & L’Abbé, M. R. (2020) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-020-0923-1 · Cited by 24 · see citing works
“, tailored feedback), iterative development process and incorporation of behaviour change theories to increase app engagement and retention rate (long-term use of consumer nutrition mobile apps) [58, 59].”
MethodSchellong, J., Lorenz, P., & Weidner, K. (2019) · European Journal of Psychotraumatology · doi:10.1080/20008198.2019.1611090 · Cited by 23 · see citing works
“If the app is to be qualified as a medical device, it is not only important to apply all the norms concerning such devices in development, but also to establish that the app has the desired effect (Grundy, Wang, & Bero, 2016).”
BackgroundRodríguez, S., Sanz, A. M., Llano, G., Andres Navarro, C., Parra‐Lara, L. G., Krystosik, A., & Rosso, F. (2020) · PLoS ONE · doi:10.1371/journal.pone.0233269 · Cited by 23 · see citing works
Beyond the Virus: A First Look at Coronavirus-themed Mobile MalwarePreprint
Wang, L., He, R., Wang, H., Xia, P., Li, Y., Wu, L., Zhou, Y., Luo, X., Sui, Y., Guo, Y., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.14619 · Cited by 23 · see citing works
“Mobile health apps have been studied by previous work [40], [41] and [29].”
BackgroundMorton, E., Nicholas, J., Yang, L., Lapadat, L., Barnes, S. J., Provencher, M. D., Depp, C., Chan, M., Kulur, R., & Michalak, E. E. (2022) · International Journal of Bipolar Disorders · doi:10.1186/s40345-022-00256-6 · Cited by 20 · see citing works
Albrecht, U., Hasenfuss, G. P., & von Jan, U. (2018) · JMIR mhealth and uhealth · doi:10.2196/11753 · Cited by 21 · see citing works
“In the context of mobile apps, the demand for methods to identify, characterize, and evaluate health-related apps has led to considerable research activity in recent years [4], be it in health-related fields or other areas of application.”
BackgroundFernández-Luque, L., Labarta, J. I., Palmer, E., & Koledova, E. B. (2020) · JMIR mhealth and uhealth · doi:10.2196/16208 · Cited by 21 · see citing works
“Complementary to the methods for the systematic search of apps in app stores [5] that have been published, this review also explores a semiautomated approach to extract mobile app data, which can be used to facilitate part of the process of the analysis of health apps.”
Feature Analysis: A Method for Analyzing the Role of Ideology in App Design
Hasinoff, A. A., & Bivens, R. (2021) · Journal of Digital Social Research · doi:10.33621/jdsr.v3i2.56 · Cited by 20 · see citing works
“One review of research on health apps found that 79% of studies assess apps’ clinical or scientific basis (Grundy, Wang, and Bero, 2016), and researchers taking this approach typically focus on how well apps’ design features fulfill established treatment guidelines.”
BackgroundBrew‐Sam, N., & Chib, A. I. (2019) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462319000163 · Cited by 20 · see citing works
“Tomitigate reported country differences in commercial app stores (73), an iPad (offering larger variety in search settings than an iPhone; iPad mini model ME276GP/A, iOS 9.1) with the latest operating system was registered anew with a dummy Singaporean account to ensure the inclusion of local…”
InfluentialBecker, E. R., Shegog, R., Savas, L. S., Frost, E. L., Coan, S. P., Healy, C. M., Spinner, S. W., & Vernon, S. W. (2021) · JMIR Pediatrics and Parenting · doi:10.2196/30340 · Cited by 18 · see citing works
“The most frequently evaluated domains in assessments of commercially available mHealth apps include the scientific and clinical basis, functionality, usability, accountability, impact, and popularity [23].”
BackgroundCarrilho, J. M., de Oliveira, I. J. R., Santos, D., Osanan, G. C., Cruz-Correia, R. J., & Reis, Z. S. N. (2018) · JMIR Formative Research · doi:10.2196/11374 · Cited by 17 · see citing works
Quality evaluation of smartphone applications for laboratory medicine
Jovičić, S. Ž., Siódmiak, J., & Watson, I. D. (2018) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2018-0710 · Cited by 15 · see citing works
Tomasella, F., & Morgan, H. M. (2021) · Digital Health · doi:10.1177/20552076211018366 · Cited by 15 · see citing works
“This is unsurprising, given the widely documented increase in these technologies’ popularity within society.(10,35,36) However, their proliferation has not increased HCPs’ experiences in dealing with patients enquiring about them.”
BackgroundeHealth for the prevention of healthcare-associated infections: a scoping review
Bentvelsen, R. G., Holten, E., Chavannes, N. H., & Veldkamp, K. E. (2021) · Journal of Hospital Infection · doi:10.1016/j.jhin.2021.04.029 · Cited by 14 · see citing works
A Social Network Analysis of the Financial Links Backing Health and Fitness Apps
Grundy, Q., Held, F. P., & Bero, L. A. (2017) · American Journal of Public Health · doi:10.2105/ajph.2017.303995 · Cited by 14 · see citing works
Self-citationShen, C., Jiang, B., Yang, Q., Wang, C., Lu, K., Gu, M., & Yuan, J. (2021) · JMIR mhealth and uhealth · doi:10.2196/26262 · Cited by 14 · see citing works
FAMAP: A Framework for Developing m-Health Apps
García‐Magariño, I., Bedía, M. G., & Palacios-Navarro, G. (2018) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-77703-0_83 · Cited by 13 · see citing works
Albrecht, U., Framke, T., & von Jan, U. (2019) · JMIR mhealth and uhealth · doi:10.2196/16442 · Cited by 13 · see citing works
Development of a Multidimensional App-Quality Assessment Tool for Health-Related Apps (AQUA)
O’Rourke, T., Pryss, R. C., Schlee, W., & Probst, T. (2020) · Digital Psychology · doi:10.24989/dp.v1i2.1816 · Cited by 13 · see citing works
“Assessing the quality of data security is therefore a crucial part of improving the use of health-related apps, and the development of methods to identify possible risks to personal privacy and data security is important (Grundy et al., 2016).”
BackgroundMobile healthcare platforms' sustainability: The perspective of health information quality
Sun, X., Zhou, W., & Feng, Y. (2023) · Frontiers in Public Health · doi:10.3389/fpubh.2022.1059252 · Cited by 12 · see citing works
“There is a general concern about the quality of health information available on mHealth platforms (12).”
BackgroundDescription of apps targeting stroke patients: A review of apps store
Cao, W., Kadir, A. A., Wang, Y., Wang, J., Dai, B., Zheng, Y., Mu, P., Hu, C., Chen, J., Luo, N., et al. (2023) · Digital Health · doi:10.1177/20552076231181473 · Cited by 11 · see citing works
Ford, K. L., Moore, S. L., Zhou, S., Gore, M. O., Portz, J., Zhang, X., Zane, R. D., Wiler, J. L., & Bull, S. S. (2019) · mHealth · doi:10.21037/mhealth.2019.08.08 · Cited by 12 · see citing works
Xu, Z., Luo, X., Shi, J., & Lai, Y. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0916-7 · Cited by 12 · see citing works
“However, when the Accountability score was ≥ 7, the total score of the app was higher (17.48 vs. 13.16, P 0.01), as was the Functionality and Usability score (4.03 vs. 2.53, P 0.01).”
MethodInfluentialVarshney, U., Singh, N., Bourgeois, A. G., & Dube, S. R. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab277 · Cited by 12 · see citing works
Rajeshkumar, L., Mishkind, M., Coleman, J. J., Pahwa, M., LaPreze, D., Solenske, S., & Wright, J. H. (2024) · Journal of Psychiatric Practice · doi:10.1097/pra.0000000000000815 · Cited by 7 · see citing works
Characteristics of Acute Childhood Illness Apps for Parents: Environmental Scan
Benoit, J. R. A., Hartling, L., Chan, M. C., & Scott, S. D. (2021) · Journal of Medical Internet Research · doi:10.2196/29441 · Cited by 12 · see citing works
“For example, the Apple and Google app stores differ in how apps are assessed for quality and how their search algorithms work, as well as in quality requirements for app developers [53].”
Palos‐Sánchez, P. R., Saura, J. R., & Álvarez‐García, J. (2018) · Innovation, technology and knowledge management · doi:10.1007/978-3-319-99590-8_7 · Cited by 10 · see citing works
Paglialonga, A., Schiavo, M., & Caiani, E. G. (2018) · American Journal of Audiology · doi:10.1044/2018_aja-imia3-18-0008 · Cited by 10 · see citing works
“…information to potential app users and thus to enable informed, aware adoption of health apps (Aungst, Clauson, Misra, Lewis, & Husain, 2014; Grundy, Wang, & Bero, 2016; Jungnickel, von Jan, & Albrecht, 2015; Paglialonga, Lugo, et al., 2018; Paglialonga, Pinciroli, Tognola, Barbieri, et…”
Importance-performance analysis in fitness apps. A study from the viewpoint of gender and age
Martín, F., Fernández, J. G., Valcarce-Torrente, M., García, A. B., Gálvez-Ruíz, P., & Sánchez, S. A. (2023) · Frontiers in Public Health · doi:10.3389/fpubh.2023.1226888 · Cited by 9 · see citing works
Alosaimi, H., Alshammari, M. K., Al-Otaibi, M., Alhejaili, S., Alshammari, B., Khalaf, A., Hajea, A., Saleh, R., Aldhafiri, F. J., AbuShahin, A., et al. (2025) · Digital Health · doi:10.1177/20552076241308993 · Cited by 10 · see citing works
Fu, S., Fang, J., Cai, Z., Lim, E. T. K., Tan, C., & Yang, H. (2022) · Internet Research · doi:10.1108/intr-09-2020-0485 · Cited by 9 · see citing works
Kabacińska, K., McLeod, K. E. R., MacKenzie, A., Vu, K., Cianfrone, M., Tugwell, A., & Robillard, J. M. (2022) · Digital Health · doi:10.1177/20552076221102775 · Cited by 10 · see citing works
“Although evidence-based mental health support apps do exist, there is little to no regulatory oversight of the content of published apps(9) and as a result, the vast majority of available applications are not grounded in empirical work.(11,12) When evidence-based apps have been clinically tested, results were promisin…”
BackgroundDigital Transformation in Indonesian Healthcare
Pitaloka, A. A., & Nugroho, A. P. (2021) · STI Policy and Management Journal · doi:10.14203/stipm.2021.301 · Cited by 8 · see citing works
Wang, T., Giunti, G., Goossens, R., & Melles, M. (2023) · Journal of Medical Internet Research · doi:10.2196/46308 · Cited by 8 · see citing works
Vogel, M. M. E., Eitz, K. A., & Combs, S. E. (2020) · JMIR mhealth and uhealth · doi:10.2196/19727 · Cited by 8 · see citing works
Jibb, L., Amoako, E., Heisey, M., Ren, L. Y., & Grundy, Q. (2022) · Archives of Disease in Childhood · doi:10.1136/archdischild-2021-323292 · Cited by 8 · see citing works
Self-citation
- Authors
- Grundy, Q. H., Wang, Z., & Bero, L. A.
- Affiliations
- The University of Sydney (AU)
- Corresponding author
- Grundy, Quinn H.
- Venue
- American Journal of Preventive Medicine
- Publisher
- Elsevier BV
- Published
- 2016-12
- Volume
- 51
- Issue
- 6
- Pages
- 1051-1059
- ISSN
- 0749-3797 (ISSN-L)
- OpenAlex
- W2521913084
- Cited by
- 154
- References
- 41
- Citations by year
- 2026: 4 · 2025: 8 · 2024: 8 · 2023: 17 · 2022: 11 · 2021: 24 · 2020: 26 · 2019: 20 · 2018: 26 · 2017: 9 · 2016: 1
- FWCI
- 3.57
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- American Journal of Preventive Medicine (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Keywords
- mobile health apps, app quality assessment, risk of bias
- MeSH terms
- Health Promotion, Mobile Applications
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
