SUITABILITY OF CURRENT EVALUATION FRAMEWORKS FOR USE IN THE HEALTH TECHNOLOGY ASSESSMENT OF MOBILE MEDICAL APPLICATIONS: A SYSTEMATIC REVIEW
Open accessThe DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
Objectives: To identify and appraise existing evaluation frameworks for mobile medical applications (MMA) and determine their suitability for use in health technology assessment (HTA) of these technologies. Methods: Systematic searches were conducted of seven bibliographic databases to identify literature published between 2008 and 2016 on MMA evaluation frameworks. Frameworks were eligible if they were used to evaluate at least one of the HTA domains of effectiveness, safety, and/or cost and cost-effectiveness of an MMA. After inclusion, the frameworks were reviewed to determine the number and extent to which other elements of an HTA were addressed by the framework. Results: A total of forty-five frameworks were identified that assessed MMAs. All frameworks assessed whether the app was effective. Of the thirty-four frameworks that examined safety, only seven overtly evaluated potential harms from the MMA (e.g., the impact of inaccurate information). Only one framework explicitly considered a comparator. Technology specific domains were sporadically addressed. Conclusion: None of the evaluation frameworks could be used, unaltered, to guide the HTA of MMAs. To use these frameworks in HTA they would need to identify relevant comparators, improve assessments of harms and consider the ongoing effect of software updates on the safety and effectiveness of MMAs. Attention should also be paid to ethical issues, such as data privacy, and technology specific characteristics. Implications: Existing MMA evaluation frameworks are not suitable for use in HTA. Further research is needed before an MMA evaluation framework can be developed that will adequately inform policy makers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Mobile app rating scale: a new tool for assessing the quality of health mobile apps
Stoyanov, S. R., Hides, L., Kavanagh, D. J., Zelenko, O., Tjondronegoro, D., & Mani, M. (2015) · JMIR mHealth and uHealth · doi:10.2196/mhealth.3422 · Cited by 2,233 · cited 1×
“Four (17 percent) frameworks appraised equity (e.g., socioeconomic status, disability, language, and age) (13;18;27;43), and an additional four (17 percent) (19) all assessed six HTA domains, but none of these addressed all of the six core domains.”
BackgroundSmartphones for smarter delivery of mental health programs: A systematic review
Donker, T., Petrie, K., Proudfoot, J., Clarke, J., Birch, M.-R., & Christensen, H. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2791 · Cited by 1,136 · cited 1×
“Literature Search We searched PubMed (MEDLINE), EMBASE, CINAHL, PsychINFO, The Cochrane Library, Compendex, and Business Source Complete between January 1, 2008 (when the first publicly accessible online application store opened) and October 31, 2016 (9).”
MethodChomutare, T., Fernandez-Luque, L., Årsand, E., & Hartvigsen, G. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1874 · Cited by 451
Finding a Depression App: A Review and Content Analysis of the Depression App Marketplace
Shen, N., Levitan, M.-J., Johnson, A., Bender, J. L., Hamilton-Page, M., Jadad, A. (. R., & Wiljer, D. (2015) · JMIR mHealth and uHealth · doi:10.2196/mhealth.3713 · Cited by 282 · cited 1×
“However, this domain was partially addressed by (20), 2013 Anxiety and Depression Association of America (ADAA) (38), 2016 11 (24 percent) frameworks that reviewed the cost of MMAs in terms of the price to download the application or to undertake in-app purchases (13;15;22;25;27;41 – 46).”
BackgroundLalloo, C., Jibb, L. A., Rivera, J., Agarwal, A., & Stinson, J. N. (2015) · The Clinical Journal of Pain · doi:10.1097/ajp.0000000000000171 · Cited by 257 · cited 3×
“Six (13 percent) frameworks assessed how the MMA provides social support to the users (15;17;28;29;37;47), for example, whether the MMA provides psychosocial support, if the MMA can provide support through social media, or if access to social support is facilitated.”
BackgroundPandey, A., Hasan, S., Dubey, D., & Sarangi, S. (2013) · Journal of Cancer Education · doi:10.1007/s13187-012-0446-9 · Cited by 213 · cited 2×
“Six frameworks addressed whether the MMA had been trialed or tested and whether safety concerns had been identified during the process (15;16;20;22;38;39).”
BackgroundAssessing the quality and usability of smartphone apps for pain self-management
Reynoldson, C., Stones, C., Allsop, M., Gardner, P., Bennett, M. I., Closs, S. J., Jones, R., & Knapp, P. (2014) · Pain Medicine · doi:10.1111/pme.12327 · Cited by 178
The evolution of mobile apps for asthma: an updated systematic assessment of content and tools
Huckvale, K., Morrison, C., Ouyang, J., Ghaghda, A., & Car, J. (2015) · BMC Medicine · doi:10.1186/s12916-015-0303-x · Cited by 166
Towards a Framework for Evaluating Mobile Mental Health Apps
Chan, S., Torous, J., Hinton, L., & Yellowlees, P. (2015) · Telemedicine and e-Health · doi:10.1089/tmj.2015.0002 · Cited by 163
Challenges in Assessing Mobile Health App Quality
Grundy, Q. H., Wang, Z., & Bero, L. A. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.07.009 · Cited by 154 · cited 3×
“Only seven frameworks evaluated the harms of the app itself (e.g., adverse events) (14;18;19;26;32;36;37).”
Background
Cited by
By the numbers: ratings and utilization of behavioral health mobile applications
Carlo, A. D., Hosseini Ghomi, R., Renn, B. N., & Areán, P. A. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0129-6 · Cited by 164
Evaluating evaluation frameworks: a scoping review of frameworks for assessing health apps
Lagan, S., Sandler, L., & Torous, J. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-047001 · Cited by 108
Marshall, J. M., Dunstan, D. A., & Bartik, W. (2020) · JMIR Mental Health · doi:10.2196/16525 · Cited by 83
“It seems conceivable, however, that an app developed without a proven theoretical framework and method of intervention could have the potential to do harm to a user [18].”
BackgroundDistinguishing features in the assessment of mHealth apps
Tarricone, R., Petracca, F., Ciani, O., & Cucciniello, M. (2021) · Expert Review of Pharmacoeconomics & Outcomes Research · doi:10.1080/14737167.2021.1891883 · Cited by 62
“ous rating scales and frameworks that are the subject of a vivid scientific debate [46].”
BackgroundWright, A. (2021) · mHealth · doi:10.21037/mhealth-20-161 · Cited by 16
Metadata
- Authors
- Moshi, M. R., Tooher, R., & Merlin, T.
- Venue
- International Journal of Technology Assessment in Health Care
- Publisher
- Cambridge University Press (CUP)
- Published
- 2018
- Volume
- 34
- Issue
- 5
- Pages
- 464-475
- ISSN
- 0266-4623 (ISSN-L)
- 1471-6348
- OpenAlex
- W2891454610
- Cited by
- 74
- References
- 17
- FWCI
- 1.08
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: SUITABILITY OF CURRENT EVALUATION FRAMEWORKS FOR USE IN THE HEALTH TECHNOLOGY ASSESSMENT OF MOBILE MEDICAL APPLICATIONS: A SYSTEMATIC REVIEW. https://app.cochat.ai/papers/10.1017/s026646231800051x
- The paper
- Moshi, M. R., Tooher, R., & Merlin, T. (2018). SUITABILITY OF CURRENT EVALUATION FRAMEWORKS FOR USE IN THE HEALTH TECHNOLOGY ASSESSMENT OF MOBILE MEDICAL APPLICATIONS: A SYSTEMATIC REVIEW. International Journal of Technology Assessment in Health Care. https://doi.org/10.1017/s026646231800051x
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.