Actionable health app evaluation: translating expert frameworks into objective metrics
Open accessThe DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Abstract
As use and availability of mobile health apps have increased, so too has the need for a thorough, accessible framework for app evaluation. The American Psychiatric Association’s app evaluation model has emerged as a way to critically assess an app by considering accessibility, privacy and security, clinical foundation, engagement, and interoperability; however, there is no centralized database where users can view how various health apps perform when assessed via the APA model. In this perspective, we propose and outline our effort to translate the APA’s model for the evaluation of health apps into a set of objective metrics that can be published online, making the framework actionable and accessible to a broad audience. The questions from the APA model were operationalized into 105 objective questions that are either binary or numeric. These questions serve as the foundation of an online database, where app evaluation consists of answering these 105 questions and can be crowdsourced. While the database has yet to be published and crowdsourced, initial internal testing demonstrated excellent interrater reliability. The database proposed here introduces a public and interactive approach to data collection that is guided by the APA model. The published product enables users to sort through the many mobile health apps and filter them according to individual preferences and priorities, making the ever-growing health app market more navigable.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Mobile app rating scale: a new tool for assessing the quality of health mobile apps
Stoyanov, S. R., Hides, L., Kavanagh, D. J., Zelenko, O., Tjondronegoro, D., & Mani, M. (2015) · JMIR mHealth and uHealth · doi:10.2196/mhealth.3422 · Cited by 2,233
Global Burden of Disease and the Impact of Mental and Addictive Disorders
Rehm, J., & Shield, K. D. (2019) · Current Psychiatry Reports · doi:10.1007/s11920-019-0997-0 · Cited by 1,388
Firth, J., Torous, J., Nicholas, J., Carney, R., Pratap, A., Rosenbaum, S., & Sarris, J. (2017) · World Psychiatry · doi:10.1002/wps.20472 · Cited by 1,097
Self-citationObjective user engagement with mental health apps: systematic search and panel-based usage analysis
Baumel, A., Muench, F., Edan, S., & Kane, J. M. (2019) · Journal of Medical Internet Research · doi:10.2196/14567 · Cited by 965
Linardon, J., Cuijpers, P., Carlbring, P., Messer, M., & Fuller‐Tyszkiewicz, M. (2019) · World Psychiatry · doi:10.1002/wps.20673 · Cited by 860
Standalone smartphone apps for mental health—a systematic review and meta-analysis
Weisel, K. K., Fuhrmann, L. M., Berking, M., Baumeister, H., Cuijpers, P., & Ebert, D. D. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0188-8 · Cited by 447
Mobile Apps for Bipolar Disorder: A Systematic Review of Features and Content Quality
Nicholas, J., Larsen, M. E., Proudfoot, J., & Christensen, H. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4581 · Cited by 438
Torous, J., & Roberts, L. W. (2017) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2017.0262 · Cited by 362
Self-citationUsing science to sell apps: Evaluation of mental health app store quality claims
Larsen, M. E., Huckvale, K., Nicholas, J., Torous, J., Birrell, L., Li, E., & Reda, B. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0093-1 · Cited by 334
Self-citationHuckvale, K., Torous, J., & Larsen, M. E. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.2542 · Cited by 312
Self-citation
Cited by
Evaluating evaluation frameworks: a scoping review of frameworks for assessing health apps
Lagan, S., Sandler, L., & Torous, J. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-047001 · Cited by 108
Self-citationLagan, S., D'Mello, R., Vaidyam, A., Bilden, R., & Torous, J. (2021) · Acta Psychiatrica Scandinavica · doi:10.1111/acps.13306 · Cited by 65
King, D. R., Nanda, G., Stoddard, J., Dempsey, A., Hergert, S., Shore, J. H., & Torous, J. (2023) · Current Psychiatry Reports · doi:10.1007/s11920-023-01477-x · Cited by 63
Self-citationLagan, S., Ramakrishnan, A., Lamont, E., Ramakrishnan, A., Frye, M., & Torous, J. (2020) · International Journal of Bipolar Disorders · doi:10.1186/s40345-020-00202-4 · Cited by 50
Self-citationSpadaro, B., Martin-Key, N. A., & Bahn, S. (2021) · Journal of Medical Internet Research · doi:10.2196/27507 · Cited by 45
Metadata
- Authors
- Lagan, S., Aquino, P., Emerson, M. R., Fortuna, K., Walker, R., & Torous, J.
- Venue
- npj Digital Medicine
- Published
- 2020-07-30
- Volume
- 3
- Issue
- 1
- Pages
- 100-100
- ISSN
- 2398-6352 (ISSN-L)
- OpenAlex
- W3045841993
- Cited by
- 144
- References
- 16
- FWCI
- 22.30
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Actionable health app evaluation: translating expert frameworks into objective metrics. https://app.cochat.ai/papers/10.1038/s41746-020-00312-4
- The paper
- Lagan, S., Aquino, P., Emerson, M. R., Fortuna, K., Walker, R., & Torous, J. (2020). Actionable health app evaluation: translating expert frameworks into objective metrics. npj Digital Medicine. https://doi.org/10.1038/s41746-020-00312-4
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.