Framework to assess the quality of mHealth apps: a mixed-method international case study protocol
Open accessThe DOI matches its Crossref record; 17 of the 18 references it cites were found in a registry or an index.
Abstract
Introduction: Healthcare professionals (HCPs) often recommend their patients to use a specific mHealth app as part of health promotion, disease prevention and patient self-management. There has been a significant growth in the number of HCPs downloading and using mobile health (mHealth) apps. Most mHealth apps that are available in app stores employ a ‘star rating’ system. This is based on user feedback on an app, but is highly subjective. Thus, the identification of quality mHealth apps which are deemed fit for purpose can be a difficult task for HCPs. Currently, there is no unified, validated standard guidelines for assessment of mHealth apps for patient safety, which can be used by HCPs. The Modified Enlight Suite (MES) is a quality assessment framework designed to provide a means for HCPs to evaluate mHealth apps before they are recommended to patients. MES was adapted from the original Enlight Suite for international use through a Delphi method, followed by preliminary validation process among a population predominantly consisting of medical students. This study aims to evaluate the applicability and validity of the MES, by HCPs, in low, middle and high income country settings. Methods and analysis: MES will be evaluated through a mixed-method study, consisting of qualitative (focus group) and quantitative (survey instruments) research, in three target countries: Malaŵi (low income), South Africa (middle income) and Ireland (high income). The focus groups will be conducted through Microsoft Teams (Microsoft, Redmond, Washington, USA) and surveys will be conducted online using Qualtrics (Qualtrics International, Seattle, Washington, USA). Participants will be recruited through the help of national representatives in Malawi (Mzuzu University), South Africa (University of Fort Hare) and Ireland (University College Cork) by email invitation. Data analysis for the focus group will be by the means of thematic analysis. Data analysis for the survey will use descriptive statistics and use Cronbach alpha as an indicator of internal consistency of the MES. The construct validity of the mHealth app will be assessed by computing the confirmatory factor analysis using Amos. Ethics and dissemination: The study has received ethical approval from the Social Research Ethics Committee (SREC) SREC/SOM/03092021/1 at University College Cork, Ireland, Malaŵi Research Ethics Committee (MREC), Malaŵi MZUNIREC/DOR/21/59 and Inter-Faculty Research Ethics Committee (IFREC) of University of Fort Hare (REC-2 70 710-028-RA). The results of the study will be disseminated through the internet, peer-reviewed journals and conference presentations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“ORCHA assesses apps across multiple domains, including clinical effectiveness, data privacy, usability and accessibility, providing users with reliable information to make informed choices (53).”
From Improving health literacy using the power of digital communications to achieve better health outcomes for patients and practitioners. (2023) · cites this paper as [53]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Thematic analysis of qualitative data: AMEE Guide No. 131
Kiger, M. E., & Varpio, L. (2020) · Medical Teacher · doi:10.1080/0142159x.2020.1755030 · Cited by 3,787
Mobile app rating scale: a new tool for assessing the quality of health mobile apps
Stoyanov, S. R., Hides, L., Kavanagh, D. J., Zelenko, O., Tjondronegoro, D., & Mani, M. (2015) · JMIR mHealth and uHealth · doi:10.2196/mhealth.3422 · Cited by 2,233 · cited 2×
“While guidelines do exist, such as the Enlight Suite and MARS, past research has highlighted potential weaknesses associated with their use when applied in resource poor settings.”
MethodTsang, S., Royse, C., & Terkawi, A. (2017) · Saudi Journal of Anaesthesia · doi:10.4103/sja.sja_203_17 · Cited by 1,692
Mobile apps in trauma education
Research, N. (2023) · PubMed · doi:10.57945/manara.23966214 · Cited by 844
Wallace, S., Clark, M., & White, J. (2012) · BMJ Open · doi:10.1136/bmjopen-2012-001099 · Cited by 432
Akbar, S., Coiera, & Magrabi, F. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz175 · Cited by 245 · cited 3×
“11 For example, mHealth apps that were available on the Apple Appstore and Android Google Play Store used to estimate blood alcohol concentration levels were shown to be highly unreliable.”
BackgroundMethodInfluentialCriteria for assessing the quality of mHealth apps: a systematic review
Nouri, R., R Niakan Kalhori, S., Ghazisaeedi, M., Marchand, G., & Yasini, M. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy050 · Cited by 231 · cited 1×
“12 Until now, even though multiple evaluation frameworks exist, there has not been a standardised evaluation framework that can be used among HCPs to evaluate the quality of mHealth apps.”
BackgroundBaumel, A., Faber, K., Mathur, N., Kane, J. M., & Muench, F. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7270 · Cited by 206
Mickan, S., Tilson, J. K., Atherton, H., Roberts, N. W., & Heneghan, C. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2530 · Cited by 191 · cited 1×
“6 7–9 However, many HCPs are still cautious when adopting mHealth apps, 10 despite the added value such technology could bring to assist their workflow, at a lower overall cost, to care for their patients.”
BackgroundMedical Applications for Pharmacists Using Mobile Devices
Aungst, T. D. (2013) · Annals of Pharmacotherapy · doi:10.1345/aph.1s035 · Cited by 159
Research works printed but not linked: 1
- Ventola, 2014, Mobile devices and apps for health care professionals: uses and benefits, P T, 39, 356
Cited by
Fitzpatrick, P. J. (2023) · Frontiers in Digital Health · doi:10.3389/fdgth.2023.1264780 · Cited by 416
Wang, J., Fu, L., Huang, Z., Hu, K., Lin, Z., & Tang, Q. (2025) · Digital Health · doi:10.1177/20552076251358673 · Cited by 12
“11 Although these models have been applied to internet healthcare settings, their emphasized dimensions primarily focus on interpersonal interaction, and do not fully re fl ect the distinctive technological characteristics of AI-powered online medical services — such as system ef fi ciency, technical…”
Dobrina, R., Starec, A., Brunelli, L., Orzan, E., Vita, C. D., Bicego, L., Ronfani, L., de Castro, V. P. N., Rocco, P. D., Zanchiello, S., et al. (2024) · Healthcare · doi:10.3390/healthcare12040442 · Cited by 6
“However, authors in the literature warn that the quality and reliability of the mHealth apps currently available on the market should be evaluated with caution [36].”
BackgroundIntroducing the Chronic Disease mHealth App Quality Evaluation Framework (CHRONIQ)
Charlie, C., & Cilliers, L. (2024) · Conference on Information Communications Technology and Society (ICTAS) · doi:10.1109/ictas59620.2024.10507109 · Cited by 2
Self-citationPublic perceptions of digitalisation and patient safety: a cross-sectional survey in Germany
Amberger, O. A., Lemke, D., Müller, H., Luettel, D., Schwappach, D. L. B., Geraedts, M., & Müller, B. S. (2025) · BMJ Open · doi:10.1136/bmjopen-2025-100516 · Cited by 1
Metadata
- Authors
- Tan, Y. Y., Woulfe, F., Chirambo, G. B., Henn, P., Cilliers, L., Fadahunsi, K. P., Taylor-Robinson, S. D., & O'Donoghue, J.
- Affiliations
- University College Cork (IE)
- Mzuzu University (MW)
- University of Fort Hare (ZA)
- Imperial College London (GB)
- Malawi eHealth Research Centre, University College Cork, Cork, Ireland
- Corresponding author
- Taylor-Robinson, Simon D
- ORCID
- Venue
- BMJ Open
- Publisher
- BMJ
- Published
- 2022-10
- Volume
- 12
- Issue
- 10
- Pages
- e062909
- ISSN
- 2044-6055 (ISSN-L)
- PMC
- PMC9621190
- OpenAlex
- W4307468510
- Cited by
- 10
- References
- 18
- Citations by year
- 2026: 3 · 2025: 4 · 2024: 2 · 2023: 1
- FWCI
- 2.01
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- BMJ Open (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Spiral (Imperial College London) (submittedVersion, other-oa)
- Funders
- University College Cork
- University of Fort Hare
- Topics
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Health Sciences › Health Professions › Health Information Management › Health, Technology, Consumer Behavior
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- mobile health apps, app quality assessment, healthcare workers, patient safety, app ratings, construct validity, internal consistency, Delphi method, patient self-management, health promotion, disease prevention
- MeSH terms
- Self-Management, Animals, Humans, Surveys and Questionnaires, Telemedicine, Hares, Mobile Applications
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Framework to assess the quality of mHealth apps: a mixed-method international case study protocol. https://app.cochat.ai/papers/10.1136/bmjopen-2022-062909
- The paper
- Tan, Y. Y., Woulfe, F., Chirambo, G. B., Henn, P., Cilliers, L., Fadahunsi, K. P., Taylor-Robinson, S. D., & O'Donoghue, J. (2022). Framework to assess the quality of mHealth apps: a mixed-method international case study protocol. BMJ Open. https://doi.org/10.1136/bmjopen-2022-062909
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.