Design and testing of a mobile health application rating tool
Open accessThe DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
Mobile health applications (“apps”) have rapidly proliferated, yet their ability to improve outcomes for patients remains unclear. A validated tool that addresses apps’ potentially important dimensions has not been available to patients and clinicians. The objective of this study was to develop and preliminarily assess a usable, valid, and open-source rating tool to objectively measure the risks and benefits of health apps. We accomplished this by using a Delphi process, where we constructed an app rating tool called THESIS that could promote informed app selection. We used a systematic process to select chronic disease apps with ≥4 stars and <4-stars and then rated them with THESIS to examine the tool’s interrater reliability and internal consistency. We rated 211 apps, finding they performed fair overall (3.02 out of 5 [95% CI, 2.96–3.09]), but especially poorly for privacy/security (2.21 out of 5 [95% CI, 2.11–2.32]), interoperability (1.75 [95% CI, 1.59–1.91]), and availability in multiple languages (1.43 out of 5 [95% CI, 1.30–1.56]). Ratings using THESIS had fair interrater reliability (κ = 0.3–0.6) and excellent scale reliability (ɑ = 0.85). Correlation with traditional star ratings was low (r = 0.24), suggesting THESIS captures issues beyond general user acceptance. Preliminary testing of THESIS suggests apps that serve patients with chronic disease could perform much better, particularly in privacy/security and interoperability. THESIS warrants further testing and may guide software and policymakers to further improve app performance, so apps can more consistently improve patient outcomes.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Mokdad (2018) · doi:10.1001/jama.2018.0158 · Cited by 1,517
The Global Burden of Chronic Diseases: Overcoming Impediments to Prevention and Control
Yach, D., Hawkes, C., Gould, C. L., & Hofman, K. J. (2004) · JAMA · doi:10.1001/jama.291.21.2616 · Cited by 1,369
Health App Use Among US Mobile Phone Owners: A National Survey
Krebs, P., & Duncan, D. T. (2015) · JMIR mHealth and uHealth · doi:10.2196/mhealth.4924 · Cited by 1,274
Schoeppe, S., Alley, S., Van Lippevelde, W., Bray, N. A., Williams, S. L., Duncan, M. J., & Vandelanotte, C. (2016) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-016-0454-y · Cited by 949
Development and validation of the user version of the Mobile application rating scale (uMARS)
Stoyanov, S. R., Hides, L., Kavanagh, D. J., & Wilson, H. (2016) · JMIR mhealth and uhealth · doi:10.2196/mhealth.5849 · Cited by 872
Whitehead, L., & Seaton, P. (2016) · Journal of Medical Internet Research · doi:10.2196/jmir.4883 · Cited by 743
Diagnostic Inaccuracy of Smartphone Applications for Melanoma Detection
Wolf, J. A., Moreau, J. F., Akilov, O., Patton, T., English, J. C., Ho, J., & Ferris, L. K. (2013) · JAMA Dermatology · doi:10.1001/jamadermatol.2013.2382 · Cited by 393
Miner, A. S., Milstein, A., Schueller, S., Hegde, R., Mangurian, C., & Linos, E. (2016) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2016.0400 · Cited by 368
Bonoto, B. C., de Araújo, V. E., Godói, I. P., de Lemos, L. L. P., Godman, B., Bennie, M., Diniz, L. M., & Junior, A. A. G. (2017) · JMIR mhealth and uhealth · doi:10.2196/mhealth.6309 · Cited by 294
Wu, Y., Yao, X., Vespasiani, G., Nicolucci, A., Dong, Y., Kwong, J., Li, L., Sun, X., Tian, H., & Li, S. (2017) · JMIR mHealth and uHealth · doi:10.2196/mhealth.6522 · Cited by 283
Cited by
Precision medicine in the era of artificial intelligence: implications in chronic disease management
Subramanian, M., Wojtusciszyn, A., Favre, L., Boughorbel, S., Shan, J., Letaief, K. B., Pitteloud, N., & Chouchane, L. (2020) · Journal of Translational Medicine · doi:10.1186/s12967-020-02658-5 · Cited by 362
Evaluating evaluation frameworks: a scoping review of frameworks for assessing health apps
Lagan, S., Sandler, L., & Torous, J. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-047001 · Cited by 108
Lagan, S., D'Mello, R., Vaidyam, A., Bilden, R., & Torous, J. (2021) · Acta Psychiatrica Scandinavica · doi:10.1111/acps.13306 · Cited by 65
Lagan, S., Ramakrishnan, A., Lamont, E., Ramakrishnan, A., Frye, M., & Torous, J. (2020) · International Journal of Bipolar Disorders · doi:10.1186/s40345-020-00202-4 · Cited by 50
Smartphone and mobile health apps for tinnitus: Systematic identification, analysis, and assessment
Mehdi, M., Stach, M., Riha, C., Neff, P., Dode, A., Pryss, R., Schlee, W., Reichert, M., & Hauck, F. J. (2020) · JMIR mHealth and uHealth · doi:10.2196/21767 · Cited by 40
“For example, recently, the THESIS app evaluation instrument was presented [64].”
Method
Metadata
- Authors
- Levine, D. M., Co, Z., Newmark, L. P., Groisser, A. R., Holmgren, A. J., Haas, J. S., & Bates, D. W.
- Venue
- npj Digital Medicine
- Published
- 2020-05-21
- Volume
- 3
- Issue
- 1
- Pages
- 74-74
- ISSN
- 2398-6352 (ISSN-L)
- OpenAlex
- W3027351805
- Cited by
- 74
- References
- 14
- FWCI
- 11.51
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Design and testing of a mobile health application rating tool. https://app.cochat.ai/papers/10.1038/s41746-020-0268-9
- The paper
- Levine, D. M., Co, Z., Newmark, L. P., Groisser, A. R., Holmgren, A. J., Haas, J. S., & Bates, D. W. (2020). Design and testing of a mobile health application rating tool. npj Digital Medicine. https://doi.org/10.1038/s41746-020-0268-9
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.