How can clinicians, specialty societies and others evaluate and improve the quality of apps for patient use?
Open accessAbstract
BACKGROUND: Health-related apps have great potential to enhance health and prevent disease globally, but their quality currently varies too much for clinicians to feel confident about recommending them to patients. The major quality concerns are dubious app content, loss of privacy associated with widespread sharing of the patient data they capture, inaccurate advice or risk estimates and the paucity of impact studies. This may explain why current evidence about app use by people with health-related conditions is scanty and inconsistent. MAIN TEXT: There are many concerns about health-related apps designed for use by patients, such as poor regulation and implicit trust in technology. However, there are several actions that various stakeholders, including users, developers, health professionals and app distributors, can take to tackle these concerns and thus improve app quality. This article focuses on the use of checklists that can be applied to apps, novel evaluation methods and suggestions for how clinical specialty organisations can develop a low-cost curated app repository with explicit risk and quality criteria. CONCLUSIONS: Clinicians and professional societies must act now to ensure they are using good quality apps, support patients in choosing between available apps and improve the quality of apps under development. Funders must also invest in research to answer important questions about apps, such as how clinicians and patients decide which apps to use and which app factors are associated with effectiveness.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
60% (6/10) of studies in this category focused on overall evaluation of mHealth apps, including quality assessment, review of official frameworks such as APA, and frameworks to support decision-making in choosing mHealth apps for research purposes.
“Most of the studies in this category 60% (6/10) focused on overall evaluation of mHealth apps, including quality assessment [86], [89], review of of cial frameworks such as APA [5], and frameworks to support decision-making in choosing mHealth apps for research purposes [87].”
From Security and Privacy of mHealth Applications: A Scoping Review (2020) · cites this paper as [89]
The App checklists section reviews a professional body's app-evaluation checklist—the UK Royal College of Physicians'—so the paper supports the general description of reviewing an established framework. Our copy does not identify or review an APA framework, which is the example attached to this citation. No page number is available in the indexed copy. (LLM, conf=0.84)
Experimental — accuracy not yet measuredPopularity on the app store often does not correlate with quality of evidence.
“Clinicians should evaluate the evidence for clinical outcomes for each digital therapeutic as popularity on the app store often does not correlate with quality of evidence.”
From What is the clinical value of mHealth for patients? (2020) · cites this paper as [24]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Mobile app rating scale: a new tool for assessing the quality of health mobile apps
Stoyanov, S. R., Hides, L., Kavanagh, D. J., Zelenko, O., Tjondronegoro, D., & Mani, M. (2015) · JMIR mHealth and uHealth · doi:10.2196/mhealth.3422 · Cited by 2,233 · see citing works
Automation bias: a systematic review of frequency, effect mediators, and mitigators
Goddard, K., Roudsari, A., & Wyatt, J. C. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000089 · Cited by 1,162 · see citing works
Self-citationCollins, L. M., Murphy, S. A., & Strecher, V. (2007) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2007.01.022 · Cited by 1,082 · see citing works
Evaluating Digital Health Interventions
Murray, E., Hekler, E. B., Andersson, G., Collins, L. M., Doherty, A., Hollis, C., Rivera, D. E., West, R., & Wyatt, J. C. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.06.008 · Cited by 847 · see citing works
Self-citationA meta-analysis of web-delivered tailored health behavior change interventions
Lustria, M. L. A., Noar, S. M., Cortese, J., Van Stee, S. K., Glueckauf, R. L., & Lee, J. (2013) · Journal of Health Communication · doi:10.1080/10810730.2013.768727 · Cited by 689 · see citing works
Mobile medical and health apps: state of the art, concerns, regulatory control and certification
Kamel Boulos, M. N., Brewer, A. C., Karimkhani, C., Buller, D. B., & Dellavalle, R. P. (2014) · Online Journal of Public Health Informatics · doi:10.5210/ojphi.v5i3.4814 · Cited by 659 · see citing works
Diagnostic Inaccuracy of Smartphone Applications for Melanoma Detection
Wolf, J. A., Moreau, J. F., Akilov, O., Patton, T., English, J. C., Ho, J., & Ferris, L. K. (2013) · JAMA Dermatology · doi:10.1001/jamadermatol.2013.2382 · Cited by 393 · see citing works
Huckvale, K., Prieto, J. T., Tilney, M., Benghozi, P.-J., & Car, J. (2015) · BMC Medicine · doi:10.1186/s12916-015-0444-y · Cited by 376 · see citing works
Commentary: Prognostic models: clinically useful or quickly forgotten?
Wyatt, J. C., & Altman, D. G. (1995) · BMJ · doi:10.1136/bmj.311.7019.1539 · Cited by 367 · see citing works
Self-citationA Content Analysis of Popular Smartphone Apps for Smoking Cessation
Abroms, L. C., Westmaas, J. L., Bontemps-Jones, J. E., Ramani, R., & Mellerson, J. L. (2013) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2013.07.008 · Cited by 325 · see citing works
Fleming, G. A., Petrie, J. R., Bergenstal, R. M., Holl, R. W., Peters, A. L., & Heinemann, L. (2020) · Diabetes Care · doi:10.2337/dci19-0062 · Cited by 306 · see citing works
mHealth and Mobile Medical Apps: A Framework to Assess Risk and Promote Safer Use
Lewis, T., & Wyatt, J. C. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.3133 · Cited by 281 · see citing works
Self-citationEvaluating medical expert systems: what to test and how?
Wyatt, J. C., & Spiegelhalter, D. J. (1990) · Medical Informatics · doi:10.3109/14639239009025268 · Cited by 196 · see citing works
Self-citationEvaluation Methods in Biomedical Informatics
Friedman, C. P., & Wyatt, J. C. (2006) · Computers in health care · doi:10.1007/0-387-30677-3 · Cited by 187 · see citing works
Self-citationFu, H., McMahon, S. K., Gross, C. R., Adam, T. J., & Wyman, J. F. (2017) · Diabetes Research and Clinical Practice · doi:10.1016/j.diabres.2017.06.016 · Cited by 173 · see citing works
The evolution of mobile apps for asthma: an updated systematic assessment of content and tools
Huckvale, K., Morrison, C., Ouyang, J., Ghaghda, A., & Car, J. (2015) · BMC Medicine · doi:10.1186/s12916-015-0303-x · Cited by 166 · see citing works
Automation bias: Empirical results assessing influencing factors
Goddard, K., Roudsari, A., & Wyatt, J. C. (2014) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2014.01.001 · Cited by 163 · see citing works
Self-citationBeyond the Randomized Controlled Trial: A Review of Alternatives in mHealth Clinical Trial Methods
Pham, Q., Wiljer, D., & Cafazzo, J. A. (2016) · JMIR mHealth and uHealth · doi:10.2196/mhealth.5720 · Cited by 162 · see citing works
Challenges in Assessing Mobile Health App Quality
Grundy, Q. H., Wang, Z., & Bero, L. A. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.07.009 · Cited by 154 · see citing works
‘Trust but verify’ – five approaches to ensure safe medical apps
Wicks, P. J., & Chiauzzi, E. J. (2015) · BMC Medicine · doi:10.1186/s12916-015-0451-z · Cited by 133 · see citing works
Accuracy of smartphone apps for heart rate measurement
Coppetti, T., Brauchlin, A. E., Müggler, S. A., Attinger‐Toller, A., Templin, C., Schoenrath, F., Hellermann, J. P., Lüscher, T. F., Biaggi, P., & Wyss, C. A. (2017) · European Journal of Preventive Cardiology · doi:10.1177/2047487317702044 · Cited by 130 · see citing works
Massachusetts Institute of Technology (MIT) (1865–)
Cesare, C. (2016) · The Bloomsbury Encyclopedia of Design · doi:10.5040/9781472596161-bed-m022 · Cited by 125 · see citing works
Weaver, E. R., Horyniak, D., Jenkinson, R., Dietze, P., & Lim, M. S. C. (2013) · JMIR mhealth and uhealth · doi:10.2196/mhealth.2709 · Cited by 107 · see citing works
A Comparison of the Reliability of Smartphone Apps for Opioid Conversion
Haffey, F., Brady, R. R. W., & Maxwell, S. R. J. (2013) · Drug Safety · doi:10.1007/s40264-013-0015-0 · Cited by 94 · see citing works
Boyle, L., Grainger, R., Hall, R. M., & Krebs, J. D. (2017) · JMIR mhealth and uhealth · doi:10.2196/mhealth.7263 · Cited by 86 · see citing works
The Quantified Patient in the Doctor's Office
West, P. A., Giordano, R., Kleek, M. G. V., & Shadbolt, N. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858445 · Cited by 81 · see citing works
Is digital medicine different?
Lancet, T. (2018) · The Lancet · doi:10.1016/s0140-6736(18)31562-9 · Cited by 73 · see citing works
The case for randomized controlled trials to assess the impact of clinical information systems
Liu, J. L. Y., & Wyatt, J. C. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.010306 · Cited by 66 · see citing works
Self-citationRubanovich, C. K., Mohr, D. C., & Schueller, S. M. (2017) · JMIR Mental Health · doi:10.2196/mental.7603 · Cited by 63 · see citing works
Scott, G., Shah, P., Wyatt, J. C., Makubate, B., & Cross, F. W. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000199 · Cited by 61 · see citing works
Self-citationA Digital NHS: An Introduction to the Digital Agenda and Plans for Implementation
Rooney, L. (2016) · Strathprints: The University of Strathclyde institutional repository (University of Strathclyde) · doi:10.17868/65658 · Cited by 54 · see citing works
What makes a good clinical app? Introducing the RCP Health Informatics Unit checklist
Wyatt, J. C., Thimbleby, H., Rastall, P., Hoogewerf, J., Wooldridge, D., & Williams, J. (2015) · Clinical Medicine · doi:10.7861/clinmedicine.15-6-519 · Cited by 52 · see citing works
Self-citationPublicly available apps for cancer survivors: a scoping review
Adam, R., McMichael, D., Powell, D., & Murchie, P. (2019) · BMJ Open · doi:10.1136/bmjopen-2019-032510 · Cited by 50 · see citing works
How standards and user involvement can improve app quality: A lifecycle approach
van Velthoven, M. H., Wyatt, J. C., Meinert, E., Brindley, D. A., & Wells, G. (2018) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2018.08.003 · Cited by 27 · see citing works
Self-citationThe Interactive Mobile App Review Toolkit (IMART): a Clinical Practice-Oriented System
Maheu, M. M., Nicolucci, V., Pulier, M. L., Wall, K. M., Frye, T. J., & Hudlická, E. (2016) · Journal of Technology in Behavioral Science · doi:10.1007/s41347-016-0005-z · Cited by 21 · see citing works
The need for quality assurance of health apps
McMillan, B., Hickey, E., Mitchell, C. A., et al. (2015) · BMJ · doi:10.1136/bmj.h5915 · Cited by 13 · see citing works
Phạm, Q., Shaw, J., Morita, P. P., Seto, E., Stinson, J., & Cafazzo, J. A. (2019) · Journal of Medical Internet Research · doi:10.2196/14849 · Cited by 10 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [24]Apple Developer 1.4 Physical harm App Store review guidelines 2018 Cupertino Apple Inc.
Cited by
Sorted by citations
What is the clinical value of mHealth for patients?
Rowland, S. P., Fitzgerald, J. E., Holme, T., Powell, J., & McGregor, A. (2020) · npj Digital Medicine · doi:10.1038/s41746-019-0206-x · Cited by 454 · see citing works
Fleming, G. A., Petrie, J. R., Bergenstal, R. M., Holl, R. W., Peters, A. L., & Heinemann, L. (2020) · Diabetes Care · doi:10.2337/dci19-0062 · Cited by 306 · see citing works
Security and Privacy of mHealth Applications: A Scoping Review
Nurgalieva, L., O'Callaghan, D., & Doherty, G. (2020) · IEEE Access · doi:10.1109/access.2020.2999934 · Cited by 132 · see citing works
“For clinicians and healthcare professionals: to establish a sustainable curated app repository, based on explicit quality and risk criteria [89], to assist them with evaluating and integrating mental health apps into practice [5], and on what apps to recommend [88].”
BackgroundMethodSmartphone and mobile health apps for tinnitus: Systematic identification, analysis, and assessment
Mehdi, M., Stach, M., Riha, C., Neff, P., Dode, A., Pryss, R., Schlee, W., Reichert, M., & Hauck, F. J. (2020) · JMIR mHealth and uHealth · doi:10.2196/21767 · Cited by 40 · see citing works
“Several mobile health (mHealth) app assessment tools and models are available [26-29], yet they either are focused on a particular health domain or are too time consuming and complex to employ in research.”
BackgroundDevan, H., Godfrey, H., Perry, M., Hempel, D., Saipe, B., Hale, L., & Grainger, R. (2019) · Journal of Pain Research · doi:10.2147/jpr.s206539 · Cited by 26 · see citing works
Jones, C., O'Toole, K., Jones, K., & Brémault-Phillips, S. (2020) · JMIR mHealth and uHealth · doi:10.2196/19807 · Cited by 11 · see citing works
- Authors
- Wyatt, J. C.
- Affiliations
- University of Southampton (GB)
- Corresponding author
- Wyatt, Jeremy C.
- ORCID
- Wyatt, Jeremy C.
- Venue
- BMC Medicine
- Published
- 2018-12
- Volume
- 16
- Issue
- 1
- Pages
- 225-225
- ISSN
- 1741-7015 (ISSN-L)
- PMC
- PMC6276222
- OpenAlex
- W2902737726
- Cited by
- 61
- References
- 37
- Citations by year
- 2026: 1 · 2025: 2 · 2024: 9 · 2023: 5 · 2022: 9 · 2021: 13 · 2020: 13 · 2019: 9
- FWCI
- 13.36
- Type
- journal article
- Language
- en
- Update notices
- correction (2019-07-20)
- Open access
- gold
- doi.org
- Licence
- Locations
- BMC Medicine (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- ePrints Soton (University of Southampton) (acceptedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Health Professions › Health Information Management › Health, Technology, Consumer Behavior
- Keywords
- mobile health, app quality, app evaluation, checklists, patient data privacy, app regulation, app developers, application effectiveness
- MeSH terms
- Humans, Quality Assurance, Health Care, Societies, Medical, Checklist, Mobile Applications
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
