CoChat
Sign Up
Cited by 165Mobile Health and mHealth ApplicationsDigital Mental Health InterventionsHealth, Technology, Consumer Behavior

Methods for Evaluating the Content, Usability, and Efficacy of Commercial Mobile Health Apps

Open access
Jake-Schoffman, Danielle E, Silfee, Valerie J, Waring, Molly E, Boudreaux, Edwin D, Sadasivam, Rajani S, Mullen, Sean P, Carey, Jennifer L, Hayes, Rashelle B +3 more·2017·JMIR mHealth and uHealth
DOI 10.2196/mhealth.8758PMC PMC5748471
Open this paper in CoChat — free account

Abstract

Commercial mobile apps for health behavior change are flourishing in the marketplace, but little evidence exists to support their use. This paper summarizes methods for evaluating the content, usability, and efficacy of commercially available health apps. Content analyses can be used to compare app features with clinical guidelines, evidence-based protocols, and behavior change techniques. Usability testing can establish how well an app functions and serves its intended purpose for a target population. Observational studies can explore the association between use and clinical and behavioral outcomes. Finally, efficacy testing can establish whether a commercial app impacts an outcome of interest via a variety of study designs, including randomized trials, multiphase optimization studies, and N-of-1 studies. Evidence in all these forms would increase adoption of commercial apps in clinical practice, inform the development of the next generation of apps, and ultimately increase the impact of commercial apps.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.2196/mhealth.8758

Cites

23 of 23 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Many academic mHealth projects do not reach commercial marketplaces because research grants rarely cover ongoing customer/technical support costs.

    “research grants rarely cover the costs required for ongoing customer/technical support, and as a consequence many academic mHealth projects do not reach commercial marketplaces (Jake-Schoffman et al. 2017).”

    From Evaluating the quality, safety, and functionality of commonly used smartphone apps for bipolar disorder mood and sleep self-management (2022) · cites this paper as Jake-Schoffman et al. 2017

    The Introduction says public release requires ongoing hosting, updates, customer service, and technical support, and that many researchers lack these resources. It also says academic research may face long delays in reaching commercial markets. It does not say that research grants rarely cover support costs or establish that those costs cause many academic projects never to reach marketplaces. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 44

  1. [1]Istepanian R Laxminarayan S Pattichis C M-health M-health: emerging mobile health systems 2006 New York, NY Springer
  2. [6]Chan S Torous J Hinton L Yellowlees P Towards a framework for evaluating mobile mental health apps Telemed J E Health 2015 7 14 21 12 1038 41 10.1089/tmj.2015.0002 26171663 26171663 · DOI · Paper page
  3. [7]Baysari MT Westbrook JI Mobile applications for patient-centered care coordination: a review of human factors methods applied to their design, development, and evaluation Yearb Med Inform 2015 8 13 10 1 47 54 10.15265/IY-2015-011 26293851 26293851 PMC4587034 · DOI
  4. [8]Riley WT Glasgow RE Etheredge L Abernethy AP Rapid, responsive, relevant (R3) research: a call for a rapid learning health research enterprise Clin Transl Med 2013 2 1 10 10.1186/2001-1326-2-10 23663660 23663660 PMC3658895 · DOI · Paper page
  5. [9]Joorabchi ME Mesbah A Kruchten P Real challenges in mobile app development 2013 Empirical Software Engineering and Measurement, ACM/IEEE International Symposium on IEEE 2013 Baltimore, MD
  6. [12]Boudreaux ED Waring ME Hayes RB Sadasivam RS Mullen S Pagoto S Evaluating and selecting mobile health apps: strategies for healthcare providers and healthcare organizations Transl Behav Med 2014 12 4 4 363 71 10.1007/s13142-014-0293-9 25584085 25584085 PMC4286553 · DOI
  7. [13]Kumar S Nilsen WJ Abernethy A Atienza A Patrick K Pavel M Riley WT Shar A Spring B Spruijt-Metz D Hedeker D Honavar V Kravitz R Lefebvre RC Mohr DC Murphy SA Quinn C Shusterman V Swendeman D Mobile health technology evaluation: the mHealth evidence workshop Am J Prev Med 2013 8 45 2 228 36 10.1016/j.amepre.2013.03.017 23867031 23867031 PMC3803146 · DOI · Paper page
  8. [15]Duriau VJ Reger RK Pfarrer MD A content analysis of the content analysis literature in organization studies: research themes, data sources, and methodological refinements Organ Res Methods 2016 6 29 10 1 5 34 10.1177/1094428106289252 · DOI
  9. [19]Breland JY Yeh VM Yu J Adherence to evidence-based guidelines among diabetes self-management apps Transl Behav Med 2013 9 3 3 277 86 10.1007/s13142-013-0205-4 24073179 24073179 PMC3771007 · DOI · Paper page
  10. [20]Hoeppner BB Hoeppner SS Seaboyer L Schick MR Wu GW Bergman BG Kelly JF How smart are smartphone apps for smoking cessation? A content analysis Nicotine Tob Res 2015 6 4 18 5 1025 31 10.1093/ntr/ntv117 26045249 26045249 PMC5942604 · DOI
  11. [21]Schoffman DE Turner-McGrievy G Jones SJ Wilcox S Mobile apps for pediatric obesity prevention and treatment, healthy eating, and physical activity promotion: just fun and games? Transl Behav Med 2013 9 3 3 320 5 10.1007/s13142-013-0206-3 24073184 24073184 PMC3771006 · DOI
  12. [25]Conroy DE Yang CH Maher JP Behavior change techniques in top-ranked mobile apps for physical activity Am J Prev Med 2014 6 46 6 649 52 10.1016/j.amepre.2014.01.010 24842742 24842742 · DOI
  13. [26]Sarkar U Gourley GI Lyles CR Tieu L Clarity C Newmark L Singh K Bates DW Usability of commercially available mobile applications for diverse patients J Gen Intern Med 2016 12 31 12 1417 1426 10.1007/s11606-016-3771-6 27418347 27418347 PMC5130945 · DOI · Paper page
  14. [27]Demidowich AP Lu K Tamler R Bloomgarden Z An evaluation of diabetes self-management applications for Android smartphones J Telemed Telecare 2012 6 18 4 235 8 10.1258/jtt.2012.111002 22604278 22604278 · DOI
  15. [29]Martínez-Pérez B de la Torre-Díez I López-Coronado M Experiences and results of applying tools for assessing the quality of a mHealth app named Heartkeeper J Med Syst 2015 11 39 11 142 10.1007/s10916-015-0303-6 26345452 26345452 · DOI
  16. [30]Mendiola MF Kalnicki M Lindenauer S Valuable features in mobile health apps for patients and consumers: content analysis of apps and user ratings JMIR Mhealth Uhealth 2015 5 13 3 2 e40 10.2196/mhealth.4283 25972309 25972309 PMC4446515 · DOI
  17. [31]Carpenter J Crutchley P Zilca RD Schwartz HA Smith LK Cobb AM Parks AC Seeing the “big” picture: big data methods for exploring relationships between usage, language, and outcome in internet intervention data J Med Internet Res 2016 8 31 18 8 e241 10.2196/jmir.5725 27580524 27580524 PMC5023946 · DOI
  18. [33]Althoff T White RW Horvitz E Influence of Pokémon Go on physical activity: study and implications J Med Internet Res 2016 12 06 18 12 e315 10.2196/jmir.6759 27923778 27923778 PMC5174727 · DOI
  19. [34]Howe KB Suharlim C Ueda P Howe D Kawachi I Rimm EB Gotta catch'em all! Pokémon GO and physical activity among young adults: difference in differences study Br Med J 2016 12 13 355 i6270 27965211 27965211 PMC5154977
  20. [35]Xian Y Xu H Xu H Liang L Hernandez AF Wang TY Peterson ED An initial evaluation of the impact of Pokémon GO on physical activity J Am Heart Assoc 2017 5 16 6 5 pii: e005341 10.1161/JAHA.116.005341 28512111 PMC5524088 28512111 · DOI
  21. [36]Wharton CM Johnston CS Cunningham BK Sterner D Dietary self-monitoring, but not dietary quality, improves with use of smartphone app technology in an 8-week weight loss trial J Nutr Educ Behav 2014 10 46 5 440 4 10.1016/j.jneb.2014.04.291 25220777 25220777 · DOI
  22. [37]Laing BY Mangione CM Tseng CH Leng M Vaisberg E Mahida M Bholat M Glazier E Morisky DE Bell DS Effectiveness of a smartphone application for weight loss compared with usual care in overweight primary care patients: a randomized, controlled trial Ann Intern Med 2014 11 18 161 10 Suppl S5 12 10.7326/M13-3005 25402403 25402403 PMC4422872 · DOI · Paper page
  23. [39]Barlow SE Expert Committee Expert committee recommendations regarding the prevention, assessment, and treatment of child and adolescent overweight and obesity: summary report Pediatrics 2007 12 120 Suppl 4 S164 92 10.1542/peds.2007-2329C 18055651 18055651 · DOI · Paper page
  24. [40]AADE 7 self-care behaviors Diabetes Educ 2008 34 3 445 9 10.1177/0145721708316625 18535317 18535317 · DOI
  25. [41]Sacks FM Lichtenstein AH Wu JHY Appel LJ Creager MA Kris-Etherton PM Miller M Rimm EB Rudel LL Robinson JG Stone NJ Van Horn LV On behalf of the American Heart Association Dietary fats and cardiovascular disease: a presidential advisory from the American Heart Association Circulation 2017 7 18 136 3 e1 e23 10.1161/CIR.0000000000000510 28620111 28620111 · DOI · Paper page
  26. [46]Powell AC Torous J Chan S Raynor GS Shwarts E Shanahan M Landman AB Interrater reliability of mHealth app rating measures: analysis of top depression and smoking cessation apps JMIR Mhealth Uhealth 2016 4 1 e15 10.2196/mhealth.5176 26863986 26863986 PMC4766362 · DOI
  27. [48]Nassar V Common criteria for usability review Work 2012 41 Suppl 1 1053 7 10.3233/WOR-2012-0282-1053 22316859 22316859 · DOI
  28. [50]Zapata BC Fernández-Alemán JL Idri A Toval A Empirical studies on usability of mHealth apps: a systematic literature review J Med Syst 2015 2 39 2 1 10.1007/s10916-014-0182-2 25600193 25600193 · DOI · Paper page
  29. [51]Zhang D Adipat B Challenges, methodologies, and issues in the usability testing of mobile applications Int J Hum Comput Interact 2005 7 18 3 293 308 10.1207/s15327590ijhc1803_3 · DOI · Paper page
  30. [53]Moumane K Idri A Abran A Usability evaluation of mobile applications using ISO 9241 and ISO 25062 standards Springerplus 2016 5 548 10.1186/s40064-016-2171-z 27190747 27190747 PMC4851667 · DOI · Paper page
  31. [54]Nielsen CM Overgaard M Pedersen MB Stage J Stenild S It's worth the hassle!: the added value of evaluating the usability of mobile systems in the field 2006 4th Nordic conference on Human-computer interaction: changing roles 2006 Oslo, Norway 272 280
  32. [55]Montague K Rodrigues A Nicolau H Guerreiro T TinyBlackBox: Supporting Mobile In-The-Wild Studies 2015 17th International ACM SIGACCESS Conference on Computers & Accessibility 2015 Lisbon, Portugal 379 380
  33. [56]Pustozerov E von Jan U Albrecht UV Evaluation of mHealth applications quality based on user ratings Stud Health Technol Inform 2016 226 237 40 27350514 27350514
  34. [58]Mehl MR Conner TS Handbook of research methods for studying daily life 2012 New York, NY Guilford Press
  35. [59]Schwartz JE Stone AA Strategies for analyzing ecological momentary assessment data Health Psychol 1998 1 17 1 6 16 9459065 9459065
  36. [62]Collins LM Murphy SA Strecher V The multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART): new methods for more potent eHealth interventions Am J Prev Med 2007 5 32 5 Suppl S112 8 10.1016/j.amepre.2007.01.022 17466815 17466815 PMC2062525 · DOI · Paper page
  37. [63]Collins LM Nahum-Shani I Almirall D Optimization of behavioral dynamic treatment regimens based on the sequential, multiple assignment, randomized trial (SMART) Clin Trials 2014 6 5 11 4 426 434 10.1177/1740774514536795 24902922 24902922 PMC4257903 · DOI
  38. [65]Lillie EO Patay B Diamant J Issell B Topol EJ Schork NJ The n-of-1 clinical trial: the ultimate strategy for individualizing medicine? Per Med 2011 3 8 2 161 173 10.2217/pme.11.7 21695041 21695041 PMC3118090 · DOI · Paper page
  39. [66]Vieira R McDonald S Araújo-Soares V Sniehotta FF Henderson R Dynamic modelling of n-of-1 data: powerful and flexible data analytics applied to individualised studies Health Psychol Rev 2017 9 11 3 222 234 10.1080/17437199.2017.1343680 28629262 28629262 · DOI
  40. [67]Zucker DR Ruthazer R Schmid CH Individual (N-of-1) trials can be combined to give population comparative treatment effect estimates: methodologic considerations J Clin Epidemiol 2010 12 63 12 1312 23 10.1016/j.jclinepi.2010.04.020 20863658 20863658 PMC2963698 · DOI
  41. [70]Wicks P Chiauzzi E 'Trust but verify'--five approaches to ensure safe medical apps BMC Med 2015 13 205 10.1186/s12916-015-0451-z 26404791 26404791 PMC4583172 · DOI
  42. [71]Albrecht UV Transparency of health-apps for trust and decision making J Med Internet Res 2013 15 12 e277 10.2196/jmir.2981 24449711 24449711 PMC3958693 · DOI
  43. [72]Pustozerov E von Jan U Albrecht UV Evaluation of mHealth applications security based on application permissions Stud Health Technol Inform 2016 226 241 4 27350515 27350515
  44. [74]Torous J Nebeker C Navigating ethics in the digital age: introducing Connected and Open Research Ethics (CORE), a tool for researchers and institutional review boards J Med Internet Res 2017 2 08 19 2 e38 10.2196/jmir.6793 28179216 28179216 PMC5322198 · DOI

Cited by

Sorted by citations

Authors
Jake-Schoffman, D. E., Silfee, V. J., Waring, M. E., Boudreaux, E. D., Sadasivam, R. S., Mullen, S. P., Carey, J. L., Hayes, R. B., Ding, E. Y., Bennett, G. G., & Pagoto, S. L.
Affiliations
  • University of Massachusetts Chan Medical School (US)
  • University of Connecticut (US)
  • University of Illinois Urbana-Champaign (US)
  • Virginia Commonwealth University (US)
  • Duke University (US)
ORCID
Venue
JMIR mHealth and uHealth
Publisher
JMIR Publications Inc.
Published
2017-12-18
Volume
5
Issue
12
Pages
e190
ISSN
2291-5222 (ISSN-L)
DOI
10.2196/mhealth.8758
PMC
PMC5748471
OpenAlex
W2780043826
Cited by
165
References
23
Citations by year
2026: 9 · 2025: 16 · 2024: 16 · 2023: 19 · 2022: 25 · 2021: 17 · 2020: 28 · 2019: 25 · 2018: 10
FWCI
26.62
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JMIR mhealth and uhealth (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • eScholarship@UMassChan (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • National Institute on Aging (R01AG052707)
  • National Heart, Lung, and Blood Institute (K24HL124366, T32HL120823)
  • National Cancer Institute (K07CA172677, R25CA172009)
Topics
  • Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
  • Health Sciences › Health Professions › Health Information Management › Health, Technology, Consumer Behavior
Keywords
health behavior change, usability testing, app efficacy, behaviour change techniques, clinical guidelines, N-of-1 studies, observational studies, clinical adoption

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review