A systematic review and taxonomy of published quality criteria related to the evaluation of user-facing eHealth programs
Abstract
The purpose of this review was to identify and classify key criteria concepts related to the evaluation of user-facing eHealth programs. In line with the PRISMA statement methodology, computer searches of relevant databases were conducted for studies published between January 1, 2000 and March 1, 2016 that contained explicit quality criteria related to mHealth and eHealth products. Reference lists of included articles, review articles, and grey literature (e.g., books, websites) were searched for additional sources. A team of nine experts led by the first author was gathered to support the classification of these criteria. Identified criteria were extracted, grouped and organized using an inductive thematic analysis. Eighty-four sources - emanating from 26 different courtiers - were included in this review. The team extracted 454 criteria that were grouped into 11 quality domains, 58 criteria concepts and 134 concepts' sub-groups. Quality domains were: Usability, Visual Design, User Engagement, Content, Behavior Change/Persuasive Design, Influence of Social Presence, Therapeutic Alliance, Classification, Credibility/Accountability, and Privacy/Security. Findings suggest that authors around the globe agree on key criteria concepts when evaluating user-facing eHealth products. The high proportion of new published criteria in the second half of this review time-frame (2008-2016), and more specifically, the high proportion of criteria relating to persuasive design, therapeutic alliance and privacy/security within this time-frame, points to the advancements made in recent years within this field.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The scope of IMIs seems to be limited by a low uptake and intervention adherence of individuals.
“However, the scope of IMIs seems to be limited by a low uptake and intervention adherence of individuals (Baumel and Yom-Tov, 2018; Baumel et al., 2017).”
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414 · see citing works
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164 · see citing works
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00136 · Cited by 10,349 · see citing works
Charnock, D., Shepperd, S., Needham, G., & Gann, R. (1999) · Journal of Epidemiology and Community Health · doi:10.1136/jech.53.2.105 · Cited by 3,329 · see citing works
Webb, T. L., Joseph, J., Yardley, L., & Michie, S. (2010) · Journal of Medical Internet Research · doi:10.2196/jmir.1376 · Cited by 2,517 · see citing works
Persuasive System Design Does Matter: a Systematic Review of Adherence to Web-based Interventions
Kelders, S. M., Kok, R. N., Ossebaard, H. C., & Van Gemert-Pijnen, J. E. (2012) · Journal of Medical Internet Research · doi:10.2196/jmir.2104 · Cited by 1,308 · see citing works
The role of the therapeutic alliance in psychotherapy.
Horvath, A. O., & Luborsky, L. B. (1993) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.61.4.561 · Cited by 1,283 · see citing works
The role of the therapeutic alliance in psychotherapy.
Horvath, A. O., & Luborsky, L. (1993) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.61.4.561 · Cited by 1,279 · see citing works
Mobile Health Technology Evaluation
Kumar, S., Nilsen, W. J., Abernethy, A., Atienza, A., Patrick, K., Pavel, M., Riley, W. T., Shar, A., Spring, B., Spruijt-Metz, D., et al. (2013) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2013.03.017 · Cited by 900 · see citing works
The Delivery of Public Health Interventions via the Internet: Actualizing Their Potential
Bennett, G. G., & Glasgow, R. E. (2009) · Annual Review of Public Health · doi:10.1146/annurev.publhealth.031308.100235 · Cited by 800 · see citing works
Published criteria for evaluating health related web sites: review
Kim, P., Eng, T. R., Deering, M. J., & Maxfield, A. M. (1999) · BMJ · doi:10.1136/bmj.318.7184.647 · Cited by 498 · see citing works
Measuring Web application quality with WebQEM
Olsina, L. A., & Rossi, G. H. (2002) · IEEE Multimedia · doi:10.1109/mmul.2002.1041945 · Cited by 256 · see citing works
Powell, A. C., Landman, A., & Bates, D. W. (2014) · JAMA · doi:10.1001/jama.2014.2564 · Cited by 236 · see citing works
Baumel, A., Faber, K., Mathur, N., Kane, J. M., & Muench, F. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7270 · Cited by 206 · cited 1× · see citing works
“While prior work identified criteria to develop a suite of measures to be used by expert raters [15], it incorporated some criteria that were not focused on technological programs and did not produce criteria taxonomy.”
Muench, F. J., & Baumel, A. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7463 · Cited by 138 · see citing works
A Historical Overview of Health Disparities and the Potential of eHealth Solutions
Gibbons, M. C. (2005) · Journal of Medical Internet Research · doi:10.2196/jmir.7.5.e50 · Cited by 137 · see citing works
Provost, M., Koompalum, D., Dong, D., & Martin, B. C. (2005) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2005.07.034 · Cited by 79 · see citing works
Mobile Mental Health: Navigating New Rules and Regulations for Digital Tools
Armontrout, J. A., Torous, J. B., Fisher, M., Drogin, E. Y., & Gutheil, T. G. (2016) · Current Psychiatry Reports · doi:10.1007/s11920-016-0726-x · Cited by 61 · see citing works
Baumel, A., & Muench, F. J. (2016) · JMIR Mental Health · doi:10.2196/mental.4563 · Cited by 50 · see citing works
Making the Case for a Feasible Evaluation Method of Available E-Mental Health Products
Baumel, A. (2016) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-016-0764-z · Cited by 19 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Objective user engagement with mental health apps: systematic search and panel-based usage analysis
Baumel, A., Muench, F., Edan, S., & Kane, J. M. (2019) · Journal of Medical Internet Research · doi:10.2196/14567 · Cited by 965 · see citing works
“components of mental health apps [27,30], drawing on the thematic analysis method suggested by Braun and Clarke [31].”
MethodSelf-citationValidation of the Mobile Application Rating Scale (MARS)
Terhorst, Y., Philippi, P., Sander, L., Schultchen, D., Paganini, S., Bardus, M., Santo, K., Knitza, J., Machado, G. C., Schöeppe, S., et al. (2020) · PLoS ONE · doi:10.1371/journal.pone.0241480 · Cited by 276 · see citing works
“, persuasive design) [62, 63], user features (e.”
Baumel, A., & Kane, J. M. (2018) · Journal of Medical Internet Research · doi:10.2196/11491 · Cited by 131 · see citing works
“Enlight is a comprehensive suite of criteria-based measurements—completed by trained raters who review the eHealth program—that was developed by a multidisciplinary team through a rigorous process of content analysis, grouping, and classification of 476 identified criteria items [21,40].”
The role of mobile health technologies in allergy care: An EAACI position paper
Matricardi, P. M., Dramburg, S., Álvarez‐Perea, A., Antolín‐Amérigo, D., Apfelbacher, C., Atanasković‐Marković, M., Berger, U. E., Blaiss, M. S., Blank, S., Boni, E., et al. (2019) · Allergy · doi:10.1111/all.13953 · Cited by 152 · see citing works
Unsworth, H., Dillon, B., Collinson, L., Powell, H., Salmon, M., Oladapo, T., Ayiku, L., Shield, G., Holden, J., Patel, N., et al. (2021) · Digital Health · doi:10.1177/20552076211018617 · Cited by 137 · see citing works
Baumel, A., Fleming, T. M., & Schueller, S. M. (2020) · Journal of Medical Internet Research · doi:10.2196/20631 · Cited by 116 · see citing works
“The literature suggests the importance of a clear therapeutic pathway and rationale to the user’s therapeutic process and that relatability factors embedded within a digital intervention support a working alliance [50-53].”
BackgroundContreras-Somoza, L. M., Irazoki, E., Toribio-Guzmán, J. M., Díez, I. D. L. T., Diaz-Baquero, A. A., Parra−Vidales, E., Bartolomé, M. V. P., & Franco, M. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.636116 · Cited by 73 · see citing works
“Accordingly, the analysis and presentation of quality evidencebased information allows it to be adequately conveyed to those interested in mental health support technological programs, whose fast and growing development means that not all of them include the characteristics that would be desirable or achieve suitable…”
BackgroundDigital Patient Experience: Umbrella Systematic Review
Wang, T., Giunti, G., Melles, M., & Goossens, R. (2022) · Journal of Medical Internet Research · doi:10.2196/37952 · Cited by 57 · see citing works
“Some influencing factors may have a soft or indirect influence on digital PEx [44,76].”
BackgroundPersuasive E-Health Design for Behavior Change
Baumeister, H., Kraft, R., Baumel, A., Pryss, R., & Messner, E.-M. (2019) · Studies in Neuroscience, Psychology and Behavioral Economics · doi:10.1007/978-3-030-31620-4_17 · Cited by 53 · see citing works
Self-citationLemon, C. A., Huckvale, K., Carswell, K., & Torous, J. B. (2020) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462319003507 · Cited by 52 · see citing works
“Constructs relevant to UX include satisfaction, acceptability, feasibility, utility, likeability, learnability, credibility, and usability (21;25;26).”
BackgroundJacob, C., Lindeque, J. P., Klein, A., Ivory, C. J., Heuss, S., & Peter, M. K. (2023) · JMIR Human Factors · doi:10.2196/45143 · Cited by 43 · see citing works
Salisbury, J. P., Keshav, N. U., Sossong, A. D., & Sahin, N. T. (2018) · JMIR mhealth and uhealth · doi:10.2196/mhealth.8478 · Cited by 41 · see citing works
“accessible patient performance metrics [61,62].”
Baumel, A., Torous, J. B., Edan, S., & Kane, J. M. (2020) · Journal of Affective Disorders · doi:10.1016/j.jad.2020.05.011 · Cited by 40 · see citing works
Bults, M., van Leersum, C. M., Olthuis, T. J. J., Bekhuis, R. E. M., & den Ouden, M. E. M. (2022) · JMIR Diabetes · doi:10.2196/41076 · Cited by 38 · see citing works
“Furthermore, these 4 apps met mobile health app requirements [25], and providers were willing to participate in citizen science projects.”
BackgroundUlrich, S., Gantenbein, A. R., Zuber, V., Von Wyl, A., Kowatsch, T., & Künzli, H. (2024) · Journal of Medical Internet Research · doi:10.2196/50132 · Cited by 36 · see citing works
“In contrast, conventional mobile health (mHealth) interventions may lack a therapist relationship and suffer from noncommittal timing [61], may not be used as intended [62], and engagement may often be low [63-65].”
Baumel, A., & Muench, F. J. (2021) · Journal of Medical Internet Research · doi:10.2196/24905 · Cited by 30 · see citing works
“Developers can help to make therapeutic activities less effortful by reducing the cognitive and environmental effort required to successfully complete each activity [52-54].”
BackgroundZIEL: Internet-Based Self-Help for Adjustment Problems: Results of a Randomized Controlled Trial
Moser, C., Bachem, R., Berger, T., & Maercker, A. (2019) · Journal of Clinical Medicine · doi:10.3390/jcm8101655 · Cited by 31 · see citing works
Bendig, E., Erb, B., Meißner, E., Bauereiß, N., & Baumeister, H. (2021) · Internet Interventions · doi:10.1016/j.invent.2021.100377 · Cited by 24 · see citing works
“However, the scope of IMIs seems to be limited by a low uptake and intervention adherence of individuals (Baumel and Yom-Tov, 2018; Baumel et al., 2017).”
BackgroundGlattacker, M., Boeker, M., Anger, R., Reichenbach, F., Tassoni, A., Bredenkamp, R., & Giesler, J. M. (2020) · JMIR mhealth and uhealth · doi:10.2196/15514 · Cited by 26 · see citing works
“Usability, which is a critical factor for the continuous application and the effectiveness of health apps [43] and is therefore part of several taxonomies for assessing health apps [12,44], was rated as good or very good in the subgroup of study participants who were followed up.”
BackgroundBaumel, A. (2022) · Journal of Technology in Behavioral Science · doi:10.1007/s41347-022-00245-7 · Cited by 22 · see citing works
“…model, the literature points to other aspects of product design that are meant to directly provoke positive changes in users’ lives by embedding a persuasive system design focused on incorporating behavior change techniques (therapeutic persuasivness features) (Baumel et al., 2017a, b).”
BackgroundMethodInfluentialAn empirically sound telemedicine taxonomy – applying the CAFE methodology
Harst, L., Otto, L., Timpel, P., Richter, P., Lantzsch, H., Wollschlaeger, B., Winkler, K., & Schlieter, H. (2021) · Journal of Public Health · doi:10.1007/s10389-021-01558-2 · Cited by 21 · see citing works
“…in three taxonomies (Elasy et al. 2001; Fitch 2004; Krumholz et al. 2006), while technological specifications such as mode of data provision , technology used or the setting of the application were covered by all but three taxonomies (Valentijn et al. 2015; Pearce et al. 2016; Baumel et al. 2017).”
BackgroundPsychometric evaluation of the Mobile Application Rating Scale (MARS) (Preprint)Preprint
Terhorst, Y., Philippi, P., Sander, L., Schultchen, D., Paganini, S., Bardus, M., Santo, K., Knitza, J., Machado, G. C., Schöeppe, S., et al. (2020) · doi:10.2196/preprints.17689 · Cited by 19 · see citing works
Persuasive e-Health Design for Behavior Change
Baumeister, H., Kraft, R., Baumel, A., Pryss, R. C., & Meßner, E.-M. (2022) · Studies in neuroscience, psychology and behavioral economics · doi:10.1007/978-3-030-98546-2_20 · Cited by 15 · see citing works
Hind, D., Drabble, S. J., Arden, M. A., Mandefield, L., Waterhouse, S., Maguire, C., Cantrill, H., Robinson, L., Beever, D., Scott, A. J., et al. (2020) · BMJ Open · doi:10.1136/bmjopen-2020-039089 · Cited by 14 · see citing works
Lin, Y., & Chen, T. (2021) · Journal of theoretical and applied electronic commerce research · doi:10.3390/jtaer16070178 · Cited by 12 · see citing works
Quality Assessment Criteria for Mobile Health Apps: A Systematic Review
HONGSANUN, W., & Insuk, S. (2020) · Walailak Journal of Science and Technology (WJST) · doi:10.48048/wjst.2020.6482 · Cited by 14 · see citing works
“[68] that grouped “Ease of use” into “Usability” domain, we have decided to separate these 2 domains in this study.”
BackgroundMethodInfluentialWesterlinck, P., & Coucke, P. A. (2021) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2021.104564 · Cited by 10 · see citing works
Wang, T., Giunti, G., Goossens, R., & Melles, M. (2023) · Journal of Medical Internet Research · doi:10.2196/46308 · Cited by 8 · see citing works
“Thanks to the use of digital technologies [54,75], patients’illness experience and what they feel when participating in a health care intervention can be monitored.”
InfluentialChen, T., & Chiu, M.-C. (2019) · Computers & Industrial Engineering · doi:10.1016/j.cie.2019.05.009 · Cited by 8 · see citing works
Baumel, A. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1541676 · Cited by 7 · see citing works
Qualitätsindikatoren für Videosprechstunden in der hausärztlichen Versorgung – ein Scoping Review
Traulsen, P., Steinhäuser, J., & Waschkau, A. (2022) · Das Gesundheitswesen · doi:10.1055/a-1791-0479 · Cited by 4 · see citing works
“Hrynyschyn et al. (2020); Systematisches Review [19] Telekonsultation bei vaskulären und diabetisch bedingten chronischen Wunden Aktueller Forschungsstand für Telekonsultation bei chronischen Wunden wird untersucht.”
BackgroundInfluentialVilla, S., CRISTOFARO, R. D., Minno, G. D., Laratro, S., Peyvandi, F., Pippo, L., Villa, S., & de Belvis, A. G. (2024) · Health Services Management Research · doi:10.1177/09514848241231585 · Cited by 2 · see citing works
“57 The results of the semi-structured interviews con fi rm the relevance of considering, in the design of care, the need of coordination between the different actors and institutions involved across the full cycle of care.”
Baumel, A., & Tamir, S. (2022) · Journal of Technology in Human Services · doi:10.1080/15228835.2022.2036300 · Cited by 2 · see citing works
Digital Patient Experience: Umbrella Systematic Review (Preprint)Preprint
Wang, T., Giunti, G., Melles, M., & Goossens, R. (2022) · doi:10.2196/preprints.37952 · Cited by 2 · see citing works
Meßner, E.-M., Kraft, R., Baumel, A., Pryss, R. C., & Baumeister, H. (2023) · Springer eBooks · doi:10.1007/978-3-662-65816-1_26 · Cited by 1 · see citing works
Gutierrez-Rios, S., Clunie, C., & Vargas-Lombardo, M. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-48858-0_21
- Authors
- Baumel, A., Birnbaum, M. L., & Sucala, M.
- Affiliations
- Northwell Health (US)
- Babeș-Bolyai University (RO)
- Corresponding author
- Baumel, Amit
- Venue
- Journal of Medical Systems
- Published
- 2017-07-22
- Volume
- 41
- Issue
- 8
- Pages
- 128-128
- ISSN
- 0148-5598 (ISSN-L)
- 1573-689X
- OpenAlex
- W2738245146
- Cited by
- 36
- References
- 20
- Citations by year
- 2025: 1 · 2024: 1 · 2023: 5 · 2022: 7 · 2021: 7 · 2020: 8 · 2019: 5 · 2018: 2
- FWCI
- 0.72
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Medical Systems (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Health Professions › General Health Professions › Health Literacy and Information Accessibility
- Keywords
- quality standards, mobile health, usability, visual design, user engagement, persuasive design, behavior change, social presence, therapeutic relationship, credibility, privacy and security
- MeSH terms
- Telemedicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
