Heuristic Evaluation of Ehealth Interventions: Establishing Standards That Relate to the Therapeutic Process Perspective
Open accessThe DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
In recent years, the number of available eHealth interventions aimed at treating behavioral and mental health challenges has been growing. From the perspective of health care providers, there is a need for eHealth interventions to be evaluated prior to clinical trials and for the limited resources allocated to empirical research to be invested in the most promising products. Following a literature review, a gap was found in the availability of eHealth interventions evaluation principles related to the patient experience of the therapeutic process. This paper introduces principles and concepts for the evaluation of eHealth interventions developed as a first step in a process to outline general evaluation guidelines that relate to the clinical context from health care providers’ perspective. Our approach was to conduct a review of literature that relates to the examination of eHealth interventions. We identified the literature that was most relevant to our study and used it to define guidelines that relate to the clinical context. We then compiled a list of heuristics we found to be useful for the evaluation of eHealth intervention products’ suitability for empirical examination. Four heuristics were identified with respect to the therapeutic process: (1) the product’s ease of use (ie, usability), (2) the eHealth intervention’s compatibility with the clinical setting, (3) the presence of tools that make it easier for the user to engage in therapeutic activities, and (4) the provision of a feasible therapeutic pathway to growth. We then used this set of heuristics to conduct a detailed examination of MyFitnessPal. This line of work could help to set the bar higher for product developers and to inform health care providers about preferred eHealth intervention designs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A behavior model for persuasive design
Fogg, B. (2009) · Proceedings of the 4th International Conference on Persuasive Technology · doi:10.1145/1541948.1541999 · Cited by 2,328 · cited 6×
“Relating to the product’s clinical aim and how it is met is a crucial part of the eHealth product design process [5,6,8] and evaluation processes of eHealth interventions.”
InfluentialMobile app rating scale: a new tool for assessing the quality of health mobile apps
Stoyanov, S. R., Hides, L., Kavanagh, D. J., Zelenko, O., Tjondronegoro, D., & Mani, M. (2015) · JMIR mHealth and uHealth · doi:10.2196/mhealth.3422 · Cited by 2,233 · cited 4×
“Most recently, Stoyanov et al [18] developed the Mobile Apps Rating Scale (MARS) for health and well-being based on the quality rating criteria found in the research literature.”
InfluentialPersuasive Systems Design: Key Issues, Process Model, and System Features
Oinas-Kukkonen, H., & Harjumaa, M. (2009) · Communications of the Association for Information Systems · doi:10.17705/1cais.02428 · Cited by 1,663
Enhancing the explanatory power of usability heuristics
Nielsen, J. (1994) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/191666.191729 · Cited by 1,439
Mohr, D. C., Cuijpers, P., & Lehman, K. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1602 · Cited by 1,097 · cited 1×
“These features also add an element of supportive accountability [29], which is especially important with higher-burden behaviors, such as food logging, and any reactive data entry by the end user.”
Mobile Health Technology Evaluation
Kumar, S., Nilsen, W. J., Abernethy, A., Atienza, A., Patrick, K., Pavel, M., Riley, W. T., Shar, A., Spring, B., Spruijt-Metz, D., et al. (2013) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2013.03.017 · Cited by 900
mHealth for mental health: Integrating smartphone technology in behavioral healthcare.
Luxton, D. D., McCann, R. A., Bush, N. E., Mishkind, M. C., & Reger, G. M. (2011) · Professional Psychology: Research and Practice · doi:10.1037/a0024485 · Cited by 809 · cited 1×
“…to heuristics and principles reviewed within the study scope (Textbox 1) as a starting point for the process of evaluating the therapeutic potential, whereas a product that does not answer, for example, safety or quality of information concerns [21] cannot hold a strong therapeutic potential.”
A Behavior Change Model for Internet Interventions
Ritterband, L. M., Thorndike, F. P., Cox, D. J., Kovatchev, B. P., & Gonder-Frederick, L. A. (2009) · Annals of Behavioral Medicine · doi:10.1007/s12160-009-9133-4 · Cited by 591 · cited 1×
“There are several models that have informed the development of eHealth interventions [5-9] and established the fundamental principles of the design process.”
Hoge, C. W., Grossman, S. H., Auchterlonie, J. L., Riviere, L. A., Milliken, C. S., & Wilk, J. E. (2014) · Psychiatric Services · doi:10.1176/appi.ps.201300307 · Cited by 368 · cited 1×
“This notion is congruent with studies in the behavioral health domain demonstrating that availability and accessibility of services promotes their utilization [22-27].”
Evaluating eHealth Interventions: The Need for Continuous Systemic Evaluation
Catwell, L., & Sheikh, A. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000126 · Cited by 307
Cited by
Objective user engagement with mental health apps: systematic search and panel-based usage analysis
Baumel, A., Muench, F., Edan, S., & Kane, J. M. (2019) · Journal of Medical Internet Research · doi:10.2196/14567 · Cited by 965
“Although there are some recommender websites [27,29,37] and approaches to help users identify the right apps [38-41], a therapeutic framework that provides guidance to users about how to use the right app at the right time could be useful.”
BackgroundSelf-citationGranja, C., Janssen, W., & Johansen, M. A. (2018) · Journal of Medical Internet Research · doi:10.2196/10235 · Cited by 578
“Multimedia Appendix 1 [19-239] presents the outcomes.”
Baumel, A., Faber, K., Mathur, N., Kane, J. M., & Muench, F. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7270 · Cited by 206
“In facilitating the cost-effective evaluation of available eHealth interventions, such tools can enable stakeholders to discuss programs’ potential prior to empirical testing [4] and to provide information that supports user recommender systems [12-14].”
InfluentialSelf-citationBaumel, A., Birnbaum, M. L., & Sucala, M. (2017) · Journal of Medical Systems · doi:10.1007/s10916-017-0776-6 · Cited by 36
Batterham, P. J., Calear, A. L., O’Dea, B., Larsen, M. E., J Kavanagh, D., Titov, N., March, S., Hickie, I., Teesson, M., Dear, B. F., et al. (2019) · DIGITAL HEALTH · doi:10.1177/2055207619878069 · Cited by 21
Metadata
- Authors
- Baumel, A., & Muench, F.
- Venue
- JMIR Mental Health
- Publisher
- JMIR Publications Inc.
- Published
- 2016-01-13
- Volume
- 3
- Issue
- 1
- Pages
- e5
- ISSN
- 2368-7959 (ISSN-L)
- OpenAlex
- W2238647392
- Cited by
- 51
- References
- 10
- FWCI
- 9.95
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.