The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Challenges in Assessing Mobile Health App Quality
Abstract
Context Mobile apps are poised to become a major source of health guidance; however, concerns about their quality and safety remain. Governments, clinicians, and health researchers are struggling to determine how to adequately evaluate the content and function of mobile health apps to guide consumers toward apps that will effectively and safely support their health. Researchers are both adapting existing research methods, such as those for systematic reviews, and exploring novel methods toward this aim. The purpose of this systematic review was to describe prevalent and emerging methods for searching, data extraction, and analysis in the context of mobile health apps and commercial app stores.Evidence acquisition A systematic search was conducted on April 6, 2016, according to Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines, for published studies that evaluated health-related apps targeted at consumers in a commercial app store, and a descriptive content analysis of their methods was performed.Evidence synthesis Although the majority of studies self-identified as "reviews" or "systematic reviews," their claims to have performed an exhaustive, replicable, and systematic search and data extraction are unsupported, given the nature of commercial app stores' search engines and personalized app content. Currently, studies of health-related app content are largely descriptive, and most use surrogate and one-dimensional outcome measures for app content quality.Conclusions This field is rapidly developing, and innovative methods for systematic searches, multidimensional quality assessment, and contextualizing content assessment are emerging. Areas for future methodologic development are highlighted, including expanding definitions of app "content" and assessing risk of bias.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Search results are personalized based on user characteristics (Grundy et al. 2016)”
From Evaluating the quality, safety, and functionality of commonly used smartphone apps for bipolar disorder mood and sleep self-management (2022) · cites this paper as Grundy et al. 2016
Citations
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000097 · Cited by 64,042
Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement.
(2009) · PubMed Central · Cited by 44,859
Mobile app rating scale: a new tool for assessing the quality of health mobile apps
Stoyanov, S. R., Hides, L., Kavanagh, D. J., Zelenko, O., Tjondronegoro, D., & Mani, M. (2015) · JMIR mHealth and uHealth · doi:10.2196/mhealth.3422 · Cited by 2,233
Just a Fad? Gamification in Health and Fitness Apps
Lister, C., West, J. H., Cannon, B., Sax, T., & Brodegard, D. (2014) · JMIR Serious Games · doi:10.2196/games.3413 · Cited by 489
Mobile Apps for Bipolar Disorder: A Systematic Review of Features and Content Quality
Nicholas, J., Larsen, M. E., Proudfoot, J., & Christensen, H. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4581 · Cited by 438
Huckvale, K., Prieto, J. T., Tilney, M., Benghozi, P.-J., & Car, J. (2015) · BMC Medicine · doi:10.1186/s12916-015-0444-y · Cited by 376
Bardus, M., van Beurden, S. B., Smith, J. R., & Abraham, C. (2016) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-016-0359-9 · Cited by 350
Mobile Applications for Weight Management
Azar, K. M. J., Lesser, L. I., Laing, B. Y., Stephens, J., Aurora, M. S., Burke, L. E., & Palaniappan, L. P. (2013) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2013.07.005 · Cited by 334
Direito, A., Dale, L. P., Shields, E., Dobson, R., Whittaker, R., & Maddison, R. (2014) · BMC Public Health · doi:10.1186/1471-2458-14-646 · Cited by 308
Medical apps for smartphones: lack of evidence undermines quality and safety
Buijink, A. W. G., Visser, B. J., & Marshall, L. (2013) · Evidence Based Medicine · doi:10.1136/eb-2012-100885 · Cited by 303
Apps as Artefacts: Towards a Critical Perspective on Mobile Health and Medical Apps
Lupton, D. A. (2014) · Societies · doi:10.3390/soc4040606 · Cited by 270
Cowan, L. T., Van Wagenen, S. A., Brown, B. A., Hedin, R. J., Seino-Stephan, Y., Hall, P. C., & West, J. H. (2013) · Health Education & Behavior · doi:10.1177/1090198112452126 · Cited by 264
Lalloo, C., Jibb, L. A., Rivera, J., Agarwal, A., & Stinson, J. N. (2015) · The Clinical Journal of Pain · doi:10.1097/ajp.0000000000000171 · Cited by 257
‘It's like having a physician in your pocket!’ A critical analysis of self-diagnosis smartphone apps
Lupton, D. A., & Jutel, A. (2015) · Social Science & Medicine · doi:10.1016/j.socscimed.2015.04.004 · Cited by 253
The mobile revolution—using smartphone apps to prevent cardiovascular disease
Neubeck, L., Lowres, N., Benjamin, E. J., Freedman, S. B., Coorey, G., & Redfern, J. (2015) · Nature Reviews Cardiology · doi:10.1038/nrcardio.2015.34 · Cited by 232
Mindfulness-Based Mobile Applications: Literature Review and Analysis of Current Features
Plaza, I., Demarzo, M., Herrera‐Mercadal, P., & García‐Campayo, J. (2013) · JMIR mhealth and uhealth · doi:10.2196/mhealth.2733 · Cited by 224
Smartphone apps for calculating insulin dose: a systematic assessment
Huckvale, K., Adomaviciute, S., Prieto, J. T., Leow, M. K., & Car, J. (2015) · BMC Medicine · doi:10.1186/s12916-015-0314-7 · Cited by 200
Assessing the quality and usability of smartphone apps for pain self-management
Reynoldson, C., Stones, C., Allsop, M., Gardner, P., Bennett, M. I., Closs, S. J., Jones, R., & Knapp, P. (2014) · Pain Medicine · doi:10.1111/pme.12327 · Cited by 178
mHealth: A Strategic Field without a Solid Scientific Soul. A Systematic Review of Pain-Related Apps
de la Vega, R., & Miró, J. (2014) · PLoS ONE · doi:10.1371/journal.pone.0101312 · Cited by 177
The evolution of mobile apps for asthma: an updated systematic assessment of content and tools
Huckvale, K., Morrison, C., Ouyang, J., Ghaghda, A., & Car, J. (2015) · BMC Medicine · doi:10.1186/s12916-015-0303-x · Cited by 166
How Smart are Smartphone Apps for Smoking Cessation? A Content Analysis
Hoeppner, B. B., Hoeppner, S. S., Seaboyer, L. M., Schick, M. R., Wu, G. W. Y., Bergman, B. G., & Kelly, J. F. (2015) · Nicotine & Tobacco Research · doi:10.1093/ntr/ntv117 · Cited by 148
A Systematic Review of Quality Assessment Methods for Smartphone Health Apps
BinDhim, N. F., Hawkey, A., & Trevena, L. (2015) · Telemedicine and e-Health · doi:10.1089/tmj.2014.0088 · Cited by 142
Tricco, A. C., Soobiah, C., Blondal, E., Veroniki, A. A., Khan, P. A., Vafaei, A., Ivory, J. D., Strifler, L., Ashoor, H. M., MacDonald, H. R., et al. (2015) · BMC Medicine · doi:10.1186/s12916-015-0371-y · Cited by 142
Smartphone breast applications – What's the evidence?
Mobasheri, M. H., Johnston, M., King, D., Leff, D., Thiruchelvam, P., & Darzi, A. (2014) · The Breast · doi:10.1016/j.breast.2014.07.006 · Cited by 137
mHealthApps: A Repository and Database of Mobile Health Apps
Xu, W., & Liu, Y. (2015) · JMIR mhealth and uhealth · doi:10.2196/mhealth.4026 · Cited by 129
Knight, E., Stuckey, M. I., Prapavessis, H., & Petrella, R. J. (2015) · JMIR mHealth and uHealth · doi:10.2196/mhealth.4003 · Cited by 112
Commercially Available Mobile Phone Headache Diary Apps: A Systematic Review
Hundert, A. S., Huguet, A., McGrath, P. J., Stinson, J., & Wheaton, M. (2014) · JMIR mhealth and uhealth · doi:10.2196/mhealth.3452 · Cited by 103
Digitizing diagnosis: a review of mobile applications in the diagnostic process
Jutel, A., & Lupton, D. A. (2015) · Diagnosis · doi:10.1515/dx-2014-0068 · Cited by 72
Comparative analysis of smoking cessation smartphone applications available in 2012 versus 2014
Ubhi, H. K., Kotz, D., Michie, S., van Schayck, O. C. P., Sheard, D., Selladurai, A., & West, R. (2016) · Addictive Behaviors · doi:10.1016/j.addbeh.2016.02.026 · Cited by 60
How trustworthy are apps for maternal and child health?
Scott, K. M., Gome, G. A., Richards, D., & Caldwell, P. H. Y. (2015) · Health and Technology · doi:10.1007/s12553-015-0099-x · Cited by 58
Smartphone applications in burns
Wurzer, P., Parvizi, D., Lumenta, D. B., Giretzlehner, M., Branski, L. K., Finnerty, C. C., Herndon, D. N., Tuca, A., Rappl, T., Smolle, C., et al. (2015) · Burns · doi:10.1016/j.burns.2014.11.010 · Cited by 56
Lee, H., Sullivan, S. J., Schneiders, A. G., Ahmed, O. H., Balasundaram, A. P., Williams, D. D., Meeuwisse, W. H., & McCrory, P. R. (2014) · British Journal of Sports Medicine · doi:10.1136/bjsports-2013-092930 · Cited by 54
Compliance of Blood Donation Apps with Mobile OS Usability Guidelines
Ouhbi, S., Fernández‐Alemán, J. L., Pozo, J. R., Bajta, M. E., Toval, A., & Idri, A. (2015) · Journal of Medical Systems · doi:10.1007/s10916-015-0243-1 · Cited by 53
Popular Glucose Tracking Apps and Use of mHealth by Latinos With Diabetes: Review
Williams, J. P., & Schroeder, D. G. (2015) · JMIR mhealth and uhealth · doi:10.2196/mhealth.3986 · Cited by 49
van Mechelen, D. M., van Mechelen, W., & Verhagen, E. A. L. M. (2013) · British Journal of Sports Medicine · doi:10.1136/bjsports-2012-092136 · Cited by 43
A scientific audit of smartphone applications for the management of obesity
Gan, K. O., & Allman-Farinelli, M. A. (2011) · Australian and New Zealand Journal of Public Health · doi:10.1111/j.1753-6405.2011.00707.x · Cited by 37
Medical Apps: Public and Academic Perspectives
Krieger, W. H. (2013) · Perspectives in biology and medicine · doi:10.1353/pbm.2013.0013 · Cited by 30
Content Analysis of Diet-Related Mobile Apps: A Self-Regulation Perspective
Zahry, N. R., Cheng, Y., & Peng, W. (2016) · Health Communication · doi:10.1080/10410236.2015.1072123 · Cited by 29
Health Behavior Theory in Popular Calorie Counting Apps: A Content Analysis
Davis, S. F., Marisa Ellsworth, A., Payne, H., Hall, S. M., West, J. H., & Nordhagen, A. L. (2016) · JMIR mhealth and uhealth · doi:10.2196/mhealth.4177 · Cited by 27
Advising patients on selecting trustful apps for diabetes self-care
Basilico, A., Marceglia, S. R. F., Bonacina, S., & Pinciroli, F. (2016) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2016.02.005 · Cited by 26
Cited by
Sorted by citations
Criteria for assessing the quality of mHealth apps: a systematic review
Nouri, R., R Niakan Kalhori, S., Ghazisaeedi, M., Marchand, G., & Yasini, M. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy050 · Cited by 231
“More recently, Grundy et al.(60) conducted a systematic review to find and explain emerging and common methods for evaluating the quality of mHealth apps.”
BackgroundSmartphone apps for the self-management of low back pain: A systematic review
Machado, G. C., Pinheiro, M. B., Lee, H., Ahmed, O. H., Hendrick, P. A., Williams, C. M., & Kamper, S. J. (2016) · Best Practice & Research Clinical Rheumatology · doi:10.1016/j.berh.2017.04.002 · Cited by 171
“assessing the quality and ‘fitness-for-purpose’ of apps is by using validated scales, and benchmarking app content against current best-practice guidelines [10].”
MethodMobile Apps for Medication Management: Review and Analysis
Tabi, K., Randhawa, A. S., Choi, F., Mithani, Z., Albers, F., Schnieder, M., Nikoo, M., Vigo, D., Jang, K. L., Demlová, R., et al. (2019) · JMIR mhealth and uhealth · doi:10.2196/13608 · Cited by 178
“The professional reviews could follow some of the standardized frameworks for quality assessment of mobile apps, for example, Mobile App Rating Scale [49], suitable for a wide range of health apps; a recently published assessment framework specifically for mental health apps [50]; or other published frameworks [33,51-…”
BackgroundBroglia, E., Millings, A., & Barkham, M. (2018) · British Journal of Guidance and Counselling · doi:10.1080/03069885.2017.1370695 · Cited by 141
A Review of the Quality and Impact of Mobile Health Apps
Grundy, Q. (2021) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-052020-103738 · Cited by 167
Self-citation
Metadata
- Authors
- Grundy, Q. H., Wang, Z., & Bero, L. A.
- Affiliations
- The University of Sydney (AU)
- Corresponding author
- Grundy, Quinn H.
- Venue
- American Journal of Preventive Medicine
- Publisher
- Elsevier BV
- Published
- 2016-12
- Volume
- 51
- Issue
- 6
- Pages
- 1051-1059
- ISSN
- 0749-3797 (ISSN-L)
- OpenAlex
- W2521913084
- Cited by
- 154
- References
- 41
- Citations by year
- 2026: 4 · 2025: 8 · 2024: 8 · 2023: 17 · 2022: 11 · 2021: 24 · 2020: 26 · 2019: 20 · 2018: 26 · 2017: 9 · 2016: 1
- FWCI
- 3.57
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- American Journal of Preventive Medicine (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Keywords
- mobile health apps, app quality assessment, risk of bias
- MeSH terms
- Health Promotion, Mobile Applications
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
