Proportionate methods for evaluating a simple digital mental health tool
Open accessAbstract
Background: Traditional evaluation methods are not keeping pace with rapid developments in mobile health. More flexible methodologies are needed to evaluate mHealth technologies, particularly simple, self-help tools. One approach is to combine a variety of methods and data to build a comprehensive picture of how a technology is used and its impact on users. Objective: This paper aims to demonstrate how analytical data and user feedback can be triangulated to provide a proportionate and practical approach to the evaluation of a mental well-being smartphone app (In Hand). Methods: A three-part process was used to collect data: (1) app analytics; (2) an online user survey and (3) interviews with users. Findings: Analytics showed that >50% of user sessions counted as ‘meaningful engagement’. User survey findings (n=108) revealed that In Hand was perceived to be helpful on several dimensions of mental well-being. Interviews (n=8) provided insight into how these self-reported positive effects were understood by users. Conclusions: This evaluation demonstrates how different methods can be combined to complete a real world, naturalistic evaluation of a self-help digital tool and provide insights into how and why an app is used and its impact on users’ well-being. Clinical implications: This triangulation approach to evaluation provides insight into how well-being apps are used and their perceived impact on users’ mental well-being. This approach is useful for mental healthcare professionals and commissioners who wish to recommend simple digital tools to their patients and evaluate their uptake, use and benefits.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Acceptability and engagement are often measured through the extent of use and user-reported subjective views and experiences.
“Acceptability and engagement are often measured through the extent of use (eg, app opens, minutes spent in the app), along with user-reported subjective views and experiences [21,22].”
From Feasibility and Acceptability of the Aboriginal and Islander Mental Health Initiative for Youth App: Nonrandomized Pilot With First Nations Young People (2023) · cites this paper as [22]
Methods for evaluating dMH tools need to be flexible and proportionate to the tools' complexity and the anticipated size of the effect on users' mental health.
“Given the speed at which technologies advance, the methods for evaluating dMH tools need to be flexible and proportionate to the tools’ complexity and the anticipated size of the effect on users’ mental health [21,22].”
From Feasibility and Acceptability of the Aboriginal and Islander Mental Health Initiative for Youth App: Nonrandomized Pilot With First Nations Young People (2023) · cites this paper as [22]
There is a need for better-quality early evaluation of new digital health products.
“There is a need for better-quality early evaluation of new digital health products, whether an initial summative evaluation of an established product to help decide whether it is worth adopting or a formative evaluation of an app during its development [22].”
From Feasibility and Acceptability of the Aboriginal and Islander Mental Health Initiative for Youth App: Nonrandomized Pilot With First Nations Young People (2023) · cites this paper as [22]
The time and resources required for planning and undertaking an RCT are not always appropriate for the assessment of dMH tools due to their rapid development and the accelerating time to the obsolescence of technological innovations.
“The time and resources required for planning and undertaking an RCT are not always appropriate for the assessment of dMH tools due to their rapid development and the accelerating time to the obsolescence of technological innovations [9,22,23].”
From Feasibility and Acceptability of the Aboriginal and Islander Mental Health Initiative for Youth App: Nonrandomized Pilot With First Nations Young People (2023) · cites this paper as [22]
Davies et al segmented sessions by those in the 30- to 60-second range.
“Davies et al also segmented sessions by those that were in the 30- to 60-second range [38].”
From A Library of Analytic Indicators to Evaluate Effective Engagement with Consumer mHealth Apps for Chronic Conditions: Scoping Review (2019) · cites this paper as [38]
Kuhn et al measured the number and content of pages visited by users in their PTSD self-management app, as did other researchers.
“Kuhn et al measured the number and content of pages visited by users in their app for PTSD self-management, as did other researchers [38,41,71].”
From A Library of Analytic Indicators to Evaluate Effective Engagement with Consumer mHealth Apps for Chronic Conditions: Scoping Review (2019) · cites this paper as [38]
Reviewed researchers measured time spent on unique sections, unique pages, length of a unique session, length between unique sessions, and total time spent on an app.
“Researchers measured the time spent on unique sections of an app [66], the time spent on unique pages [56], the length of a unique session [38,42,43,71], the length between unique sessions [72], and the total time spent on an app [62,68,73].”
From A Library of Analytic Indicators to Evaluate Effective Engagement with Consumer mHealth Apps for Chronic Conditions: Scoping Review (2019) · cites this paper as [38]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414 · see citing works
Stewart-Brown, S., Tennant, A., Tennant, R., Platt, S., Parkinson, J., & Weich, S. (2009) · Health and Quality of Life Outcomes · doi:10.1186/1477-7525-7-15 · Cited by 1,251 · see citing works
Perski, O., Blandford, A., West, R., & Michie, S. (2017) · Translational Behavioral Medicine · doi:10.1007/s13142-016-0453-1 · Cited by 1,193 · see citing works
Adherence in Internet Interventions for Anxiety and Depression
Christensen, H., Griffiths, K. M., & Farrer, L. (2009) · Journal of Medical Internet Research · doi:10.2196/jmir.1194 · Cited by 1,133 · see citing works
Hollis, C., Falconer, C. J., Martin, J. L., Whittington, C., Stockton, S., Glazebrook, C., & Davies, E. B. (2017) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.12663 · Cited by 846 · cited 2× · see citing works
“Three methods of data collection were used to gain insight into naturalistic use of In Hand: (1) sampling and analysis of quantitative mobile analytical data (eg, number of individual user sessions, number of interactions with each section of the app); (2) a user survey with questions adapted from a validated well-bei…”
MethodSelf-citationClarke, A., Friede, T., Putz, R., Ashdown, J., Martin, S., Blake, A., Adi, Y., Parkinson, J., Flynn, P., Platt, S., et al. (2011) · BMC Public Health · doi:10.1186/1471-2458-11-487 · Cited by 493 · see citing works
Gupta, J. K., Daniels, J. P., Middleton, L. J., Pattison, H. M., Prileszky, G., Roberts, T. E., Sanghera, S., Barton, P., Gray, R., Kai, J., et al. (2015) · Health Technology Assessment · doi:10.3310/hta19880 · Cited by 399 · see citing works
Technological innovations in mental healthcare: harnessing the digital revolution
Hollis, C., Morriss, R., Martin, J., Amani, S., Cotton, R., Denis, M., & Lewis, S. (2015) · British Journal of Psychiatry · doi:10.1192/bjp.bp.113.142612 · Cited by 375 · see citing works
Self-citationMohr, D. C., Weingardt, K. R., Reddy, M., & Schueller, S. M. (2017) · Psychiatric Services · doi:10.1176/appi.ps.201600541 · Cited by 365 · see citing works
Do online mental health services improve help-seeking for young people? A systematic review
Kauer, S. D., Mangan, C., & Sanci, L. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.3103 · Cited by 339 · see citing works
mHealth and Mobile Medical Apps: A Framework to Assess Risk and Promote Safer Use
Lewis, T., & Wyatt, J. C. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.3133 · Cited by 281 · see citing works
Van Ameringen, M., Turna, J., Khalesi, Z., Pullia, K., & Patterson, B. (2017) · Depression and Anxiety · doi:10.1002/da.22657 · Cited by 234 · see citing works
Hilvert-Bruce, Z., Rossouw, P. J., Wong, N., Sunderland, M., & Andrews, G. (2012) · Behaviour Research and Therapy · doi:10.1016/j.brat.2012.04.001 · Cited by 193 · see citing works
Reid, S. C., Kauer, S. D., Hearps, S. J., Crooke, A. H., Khor, A. S., Sanci, L. A., & Patton, G. C. (2011) · BMC Family Practice · doi:10.1186/1471-2296-12-131 · Cited by 192 · see citing works
Beyond the Randomized Controlled Trial: A Review of Alternatives in mHealth Clinical Trial Methods
Pham, Q., Wiljer, D., & Cafazzo, J. A. (2016) · JMIR mHealth and uHealth · doi:10.2196/mhealth.5720 · Cited by 162 · see citing works
Taki, S., Lymer, S., Russell, C. G., Campbell, K., Laws, R., Ong, K.-L., Elliott, R., & Denney-Wilson, E. (2017) · JMIR mHealth and uHealth · doi:10.2196/mhealth.7236 · Cited by 114 · see citing works
Littlewood, E., Duarte, A., Hewitt, C. E., Knowles, S. E., Palmer, S., Walker, S., Andersen, P., Araya, R., Barkham, M., Bower, P., et al. (2015) · Health Technology Assessment · doi:10.3310/hta191010 · Cited by 94 · see citing works
Feasibility of "CopeSmart": A Telemental Health App for Adolescents
Kenny, R., Dooley, B., & Fitzgerald, A. (2015) · JMIR Mental Health · doi:10.2196/mental.4370 · Cited by 79 · see citing works
Bidargaddi, N. P., Musiat, P., Winsall, M., Vogl, G., Blake, V., Quinn, S. J., Orlowski, S., Antezana, G., & Schrader, G. D. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.6775 · Cited by 59 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Pham, Q., Graham, G., Carrion, C., Morita, P. P., Seto, E., Stinson, J. N., & Cafazzo, J. A. (2019) · JMIR mHealth and uHealth · doi:10.2196/11941 · Cited by 131 · see citing works
Jones, C. H. D., Thornton, J. R., & Wyatt, J. C. (2023) · Medical Law Review · doi:10.1093/medlaw/fwad013 · Cited by 122 · see citing works
Mobile Health for Perinatal Depression and Anxiety: Scoping Review
Hussain-Shamsy, N., Shah, A., Vigod, S. N., Zaheer, J., & Seto, E. (2020) · Journal of Medical Internet Research · doi:10.2196/17011 · Cited by 119 · see citing works
Wykes, T., & Schueller, S. M. (2019) · Journal of Medical Internet Research · doi:10.2196/12390 · Cited by 95 · see citing works
Williams, J., & Pykett, J. (2022) · Social Science & Medicine · doi:10.1016/j.socscimed.2022.114802 · Cited by 49 · see citing works
Øksnebjerg, L., Woods, B. T., Ruth, K., Lauridsen, A., Kristiansen, S., Holst, H. D., & Waldemar, G. (2019) · JMIR mhealth and uhealth · doi:10.2196/14694 · Cited by 49 · see citing works
Connected Mental Health: Systematic Mapping Study
(2020) · doi:10.2196/19950 · Cited by 33 · see citing works
Alessa, T., Hawley, M., Alsulamy, N. A., & de Witte, L. (2020) · JMIR mhealth and uhealth · doi:10.2196/24177 · Cited by 31 · see citing works
Dingwall, K. M., Povey, J., Sweet, M., Friel, J., Shand, F., Titov, N., Wormer, J., Mirza, T., & Nagel, T. (2023) · JMIR Human Factors · doi:10.2196/40111 · Cited by 21 · see citing works
“Overwhelmingly, playing the games (P5, 9, 22, 23, 24, 25, 26, 29, 31,32, 33, 35, 38) was mentioned as a favorite part of the app, as was watching the videos (P9, 11, 22, 24, 26,27, 29, 38).”
InfluentialPotential of using visual imagery to revolutionise measurement of emotional health
Herring, A., Craven, M. P., Mughal, F., Rawsthorne, M., Rees, K., Walker, L., & Wolpert, M. (2020) · Archives of Disease in Childhood · doi:10.1136/archdischild-2019-317758 · Cited by 8 · see citing works
Self-citationBengoa, A. B., Martinez, J. B., Orue-Saiz, I., & Zorrilla, A. M. (2020) · IEEE International Symposium on Signal Processing and Information Technology (ISSPIT) · doi:10.1109/isspit51521.2020.9408775 · Cited by 8 · see citing works
Cushnan, J., Best, P., Daele, T. V., D’Arcy, J., Cafferty, P. M., Roche, R., & Elliott, A. (2025) · Cognitive and Behavioral Practice · doi:10.1016/j.cbpra.2025.08.003 · Cited by 4 · see citing works
Perspectives on Mental Health Digital Platforms: A Usability Study with Ergonomics Design StudentsPreprint
Ngabo-Woods, H., & Cuesta, S. A. (2023) · doi:10.33774/coe-2023-zb8xg · Cited by 1 · see citing works
Perspectives on Mental Health Digital Platforms: A Usability Study with Ergonomics Design StudentsPreprint
Ngabo-Woods, H. (2023) · doi:10.33774/coe-2023-zb8xg-v2 · Cited by 1 · see citing works
Mendoza, N. B., Buenconsejo, J. U., Ba, S., & Yan, Z. (2026) · Journal of Psychoeducational Assessment · doi:10.1177/07342829261423745
- Authors
- Davies, E. B., Craven, M. P., Martin, J. L., & Simons, L.
- Affiliations
- University of Nottingham (GB)
- Corresponding author
- Davies, E Bethan
- Venue
- Evidence Based Mental Health
- Publisher
- BMJ
- Published
- 2017-10-09
- Volume
- 20
- Issue
- 4
- Pages
- 112-117
- ISSN
- 1362-0347 (ISSN-L)
- 1468-960X
- PMC
- PMC5750412
- OpenAlex
- W2762886707
- Cited by
- 15
- References
- 19
- Citations by year
- 2026: 1 · 2025: 1 · 2023: 4 · 2022: 1 · 2020: 5 · 2019: 3
- FWCI
- 2.50
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc
- Locations
- Evidence-Based Mental Health (publishedVersion, cc-by-nc)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Nottingham ePrints (University of Nottingham) (submittedVersion, cc-by-nc)
- Repository@Nottingham (University of Nottingham) (submittedVersion, cc-by-nc)
- HighWire Press Open Archive (submittedVersion)
- Funders
- Comic Relief
- National Institute for Health and Care Research
- NIHR Nottingham Biomedical Research Centre
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Health Sciences › Health Professions › Health Information Management › Health, Technology, Consumer Behavior
- Keywords
- app analytics, user engagement, user feedback, naturalistic assessment, mental health technology
- MeSH terms
- Adolescent, Adult, Female, Health Services Research, Humans, Male, Mental Health, Middle Aged, Patient Acceptance of Health Care, Telemedicine, Young Adult, Mobile Applications
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
