Beyond clicks and downloads: a call for a more comprehensive approach to measuring mobile-health app engagement
Open accessAbstract
Downloading a mobile health (m-health) app on your smartphone does not mean you will ever use it. Telling another person about an app does not mean you like it. Using an online intervention does not mean it has had an impact on your well-being. Yet we consistently rely on downloads, clicks, ‘likes’ and other usage and popularity metrics to measure m-health app engagement. Doing so misses the complexity of how people perceive and use m-health apps in everyday life to manage mental health conditions. This article questions commonly used behavioural metrics of engagement in mental health research and care, and proposes a more comprehensive approach to measuring in-app engagement.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Individuals with BD may engage, disengage, and re-engage with apps according to the fluctuating course of illness.
“Individuals with BD may engage, disengage, and re-engage with apps according to the fluctuating course of illness (O'Brien et al. 2020).”
From Evaluating the quality, safety, and functionality of commonly used smartphone apps for bipolar disorder mood and sleep self-management (2022) · cites this paper as O'Brien et al. 2020
The (Re)Defining user engagement section describes engagement, disengagement and re-engagement, while Qualifying the quantification of m-health app usage says people with chronic conditions have fluctuating needs and discusses bipolar disorder as an example. The paper does not specifically say that people with bipolar disorder cycle through those engagement stages according to the course of their illness. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
What is user engagement? A conceptual framework for defining user engagement with technology
O'Brien, H. L., & Toms, E. G. (2008) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20801 · Cited by 1,397 · see citing works
Self-citationPerski, O., Blandford, A., West, R., & Michie, S. (2017) · Translational Behavioral Medicine · doi:10.1007/s13142-016-0453-1 · Cited by 1,193 · see citing works
(2005) · Texas ScholarWorks (Texas Digital Library) · Cited by 1,064 · see citing works
Fleming, T., Bavin, L., Lucassen, M., Stasiak, K., Hopkins, S., & Merry, S. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.9275 · Cited by 670 · see citing works
Torous, J., Lipschitz, J., Ng, M., & Firth, J. (2020) · Journal of Affective Disorders · doi:10.1016/j.jad.2019.11.167 · Cited by 546 · see citing works
Standalone smartphone apps for mental health—a systematic review and meta-analysis
Weisel, K. K., Fuhrmann, L. M., Berking, M., Baumeister, H., Cuijpers, P., & Ebert, D. D. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0188-8 · Cited by 447 · see citing works
User engagement in mental health apps: A review of measurement, reporting, and validity
Ng, M. M., Firth, J., Minen, M., & Torous, J. (2019) · Psychiatric Services · doi:10.1176/appi.ps.201800519 · Cited by 293 · see citing works
Nicholas, J., Fogarty, A. S., Boydell, K., & Christensen, H. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7273 · Cited by 158 · see citing works
Cole-Lewis, H., Ezeanochie, N., & Turgiss, J. (2019) · JMIR Formative Research · doi:10.2196/14052 · Cited by 155 · see citing works
Torous, J., Firth, J., Huckvale, K., Larsen, M. E., Cosco, T. D., Carney, R., Chan, S., Pratap, A., Yellowlees, P., Wykes, T., et al. (2018) · Journal of Nervous & Mental Disease · doi:10.1097/nmd.0000000000000864 · Cited by 101 · see citing works
Chen, A. T., Wu, S., Tomasino, K. N., Lattie, E. G., & Mohr, D. C. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103187 · Cited by 58 · see citing works
Morton, E. E., Hole, R., Murray, G., Buzwell, S., & Michalak, E. E. (2019) · JMIR Mental Health · doi:10.2196/16121 · Cited by 25 · see citing works
Self-citationFletcher, K., Foley, F., & Murray, G. (2018) · Journal of Medical Internet Research · doi:10.2196/11160 · Cited by 21 · see citing works
Allan, S., McLeod, H. J., Bradstreet, S., Beedie, S. A., Moir, B., Gleeson, J., Farhall, J., Morton, E. E., & Gumley, A. I. (2019) · JMIR Research Protocols · doi:10.2196/15634 · Cited by 16 · see citing works
Self-citation
Flagged first, then by citations
1 of the 15 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- [9]Jacques RD. The Nature of Engagement and its Role in Hypermedia Evaluation and Design (Doctoral dissertation). South Bank University, 1996.
Cited by
Sorted by citations
Lipschitz, J. M., Pike, C. K., Hogan, T. P., Murphy, S. A., & Burdick, K. E. (2023) · Current Treatment Options in Psychiatry · doi:10.1007/s40501-023-00297-3 · Cited by 230 · see citing works
Rethinking (Dis)engagement in human-computer interaction
O'Brien, H. L., Roll, I., Kampen, A., & Davoudi, N. (2021) · Computers in Human Behavior · doi:10.1016/j.chb.2021.107109 · Cited by 153 · see citing works
Self-citationSzinay, D., Forbes, C. C., Busse, H., DeSmet, A., Smit, E. S., & König, L. M. (2023) · Obesity Reviews · doi:10.1111/obr.13542 · Cited by 71 · see citing works
Flaherty, S. J., McCarthy, M. B., Collins, A., McCafferty, C., & McAuliffe, F. M. (2021) · European Journal of Marketing · doi:10.1108/ejm-06-2019-0531 · Cited by 35 · see citing works
Human-Centered Design to Enhance Implementation and Impact in Health
Lyon, A. R., Aung, T., Bruzios, K. E., & Munson, S. A. (2024) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-071823-122337 · Cited by 36 · see citing works
User engagement in mobile apps for people with schizophrenia: A scoping review
de Almeida, R. S., & Marques, A. J. (2023) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.1023592 · Cited by 29 · see citing works
Browne, J., Halverson, T. F., & Vilardaga, R. (2021) · Translational Behavioral Medicine · doi:10.1093/tbm/ibab100 · Cited by 27 · see citing works
Borghouts, J. W., Eikey, E. V., Leon, C. D., Schueller, S. M., Schneider, M. L., Stadnick, N. A., Zheng, K., Wilson, L., Caro, D., Mukamel, D. B., et al. (2022) · JMIR Formative Research · doi:10.2196/43192 · Cited by 24 · see citing works
Morton, E. E., Nicholas, J., Lapadat, L., O'Brien, H. L., Barnes, S. J., Poh, C., & Michalak, E. E. (2021) · Journal of Affective Disorders · doi:10.1016/j.jad.2021.08.132 · Cited by 21 · see citing works
Self-citationMorton, E., Nicholas, J., Yang, L., Lapadat, L., Barnes, S. J., Provencher, M. D., Depp, C., Chan, M., Kulur, R., & Michalak, E. E. (2022) · International Journal of Bipolar Disorders · doi:10.1186/s40345-022-00256-6 · Cited by 20 · see citing works
Self-citationPetrinec, A. B., Wilk, C., Hughes, J. W., Zullo, M. D., & George, R. L. (2023) · American Journal of Critical Care · doi:10.4037/ajcc2023800 · Cited by 20 · see citing works
Nardi, W. R., Roy, A., DUNSIGER, S. I., & Brewer, J. A. (2022) · Journal of Medical Internet Research · doi:10.2196/33696 · Cited by 20 · see citing works
Kleiman, E. M., Bentley, K. H., Wacha-Montes, A., Taylor, M. E., Lozy, O., Conti, K., & Reese Mayer, W. (2022) · Behavior Therapy · doi:10.1016/j.beth.2022.04.008 · Cited by 12 · see citing works
Collier, A. F., Hagemann, S., Trinidad, S. B., & Vigil-Hayes, M. (2023) · JMIR Formative Research · doi:10.2196/44926 · Cited by 12 · see citing works
How digital technology can steal your time
Černohorská, V., Schoetensack, C., Klegr, T., Witowska, J., Goncikowska, K., Giner-Domínguez, G., Papastamatelou, J., Chappuis, S., Boente, M. F., Meteier, Q., et al. (2025) · Computers in Human Behavior · doi:10.1016/j.chb.2025.108680 · Cited by 13 · see citing works
Goodman, W., Bagnall, A., Ashley, L., Azizoddin, D. R., Muehlensiepen, F., Blum, D., Bennett, M., & Allsop, M. J. (2022) · JMIR Cancer · doi:10.2196/33355 · Cited by 12 · see citing works
User Engagement in an Online Digital Health Intervention to Promote Problem Solving
O'Brien, H. L., Chen, A., Kaneshiro, J., & Zaslavsky, O. (2024) · Interacting with Computers · doi:10.1093/iwc/iwae030 · Cited by 11 · see citing works
Self-citationMichalak, E. E., Barnes, S. J., Morton, E. E., O'Brien, H. L., Murray, G., Hole, R., & Meyer, D. (2022) · JMIR Research Protocols · doi:10.2196/36213 · Cited by 9 · see citing works
Self-citationA look under the hood: analyzing engagement and usage data of a smartphone-based intervention
Siezenga, A. M., Mertens, E. C. A., & van Gelder, J. (2023) · BMC Digital Health · doi:10.1186/s44247-023-00048-7 · Cited by 9 · see citing works
Herrijgers, C., Verboon, P., Florence, É., Vandebosch, H., Poels, K., & Platteau, T. (2024) · JMIR Formative Research · doi:10.2196/56606 · Cited by 8 · see citing works
Morton, E. E., Hole, R., O'Brien, H. L., LI, L. C., Barnes, S. J., & Michalak, E. E. (2025) · Journal of Affective Disorders · doi:10.1016/j.jad.2025.04.107 · Cited by 7 · see citing works
Self-citationFarrand, P. A., Raue, P. J., Ward, E. C., Repper, D., Baker, J., & Areán, P. (2023) · JMIR mhealth and uhealth · doi:10.2196/47321 · Cited by 9 · see citing works
Application of the Supportive Accountability Model in Digital Health Interventions: Scoping Review
Kwok, G., Cheung, S., Duffecy, J. L., & Devine, K. A. (2025) · Journal of Medical Internet Research · doi:10.2196/72639 · Cited by 8 · see citing works
Larson, D. N., Henning, J., & Burgermaster, M. (2023) · Journal of Nutrition Education and Behavior · doi:10.1016/j.jneb.2023.05.247 · Cited by 9 · see citing works
Towards More Gender-Inclusive Q&As
Dubois, P. M. J., Maftouni, M., & Bunt, A. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555567 · Cited by 6 · see citing works
Rosman, M. R. M., Alias, N. R., Arshad, I. H., Hamid, H. A., & Idris, Z. (2022) · IEEE Engineering Management Review · doi:10.1109/emr.2022.3196929 · Cited by 6 · see citing works
Szinay, D., Forbes, C., Busse, H., DeSmet, A., Smit, E. S., & König, L. M. (2022) · doi:10.31234/osf.io/cq24y · Cited by 6 · see citing works
Ormeño-Arriagada, P., Márquez, G., Taramasco, C., Gatica, G., Vásconez, J. P., & Navarro, E. (2025) · Applied Sciences · doi:10.3390/app151910792 · Cited by 5 · see citing works
Gray, L., Marcynikola, N., Barnett, I., & Torous, J. B. (2024) · Journal of Integrative and Complementary Medicine · doi:10.1089/jicm.2023.0698 · Cited by 5 · see citing works
Morton, E. E., Hole, R., O'Brien, H. L., LI, L. C., Barnes, S. J., & Michalak, E. E. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0001017 · Cited by 3 · see citing works
Self-citationYan, X., Newman, M., Park, S. Y., Sander, A. M., Choi, S. W., Miner, J. A., Wu, Z., & Carlozzi, N. E. (2023) · JMIR Formative Research · doi:10.2196/47813 · Cited by 3 · see citing works
Farrand, P. A., Raue, P. J., Ward, E. C., Healey, S., & Baker, J. (2025) · Academia Mental Health and Well-Being · doi:10.20935/mhealthwellb7785 · Cited by 3 · see citing works
Günther, F., Wong, D., Elison, S., & Yau, C. (2023) · JAMIA Open · doi:10.1093/jamiaopen/ooad072 · Cited by 2 · see citing works
Suleman, Z. J., & Lemieux, V. L. (2023) · Future of business and finance · doi:10.1007/978-3-031-45339-7_6 · Cited by 2 · see citing works
Dawson, R. M., & Donevant, S. B. (2023) · Elsevier eBooks · doi:10.1016/b978-0-323-99271-8.00002-4 · Cited by 2 · see citing works
Effectiveness of a Just-In-Time Adaptive App to Increase Daily Steps: An RCT
Vos, A. L., de Bruijn, G., Klein, M. C. A., Boerman, S. C., Stuber, J. M., & Smit, E. G. (2024) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2024.09.010 · Cited by 1 · see citing works
O’Brien, H. (2025) · Synthesis lectures on information concepts, retrieval, and services · doi:10.1007/978-3-031-80916-3_1 · Cited by 1 · see citing works
Self-citationApproaches to Measuring User Engagement
O'Brien, H. L. (2025) · Synthesis lectures on information concepts, retrieval, and services · doi:10.1007/978-3-031-80916-3_5 · Cited by 1 · see citing works
Self-citationMurphy, J., Nguyen, V. C., Dang, L., Xie, H. W., Tran, T. H. T., Barbic, S. P., Chau, L. W., Samji, H., Minas, H., Nguyen, M. V., et al. (2025) · PLoS ONE · doi:10.1371/journal.pone.0332875 · Cited by 1 · see citing works
Self-citationDoğan, K., ten Klooster, I., Kip, H., Haeri, M. A., & Kelders, S. M. (2026) · Behaviour and Information Technology · doi:10.1080/0144929x.2026.2619659 · Cited by 1 · see citing works
Michalak, E. E., Barnes, S. J., Morton, E. E., O'Brien, H. L., Murray, G., Hole, R., & Meyer, D. (2022) · doi:10.2196/preprints.36213
Self-citationBorghouts, J. W., Eikey, E. V., Leon, C. D., Schueller, S. M., Schneider, M. L., Stadnick, N. A., Zheng, K., Mukamel, D. B., & Sorkin, D. H. (2023) · doi:10.2196/preprints.53161
Automated Influence and Value Collapse
White, D. J. (2024) · American Philosophical Quarterly · doi:10.5406/21521123.61.4.06
Mir, R., Poduval, S., & Hamilton, F. L. (2025) · doi:10.2196/preprints.71383
Conceptual Approaches to User Engagement
O'Brien, H. L. (2025) · Synthesis lectures on information concepts, retrieval, and services · doi:10.1007/978-3-031-80916-3_2
Self-citationOewel, B., Nambiar, K., Agapie, E., & Reddy, M. C. (2025) · doi:10.2196/preprints.89009
Durneva, P., & LeRouge, C. (2026) · European Journal of Information Systems · doi:10.1080/0960085x.2025.2605343 · Cited by 1 · see citing works
Danilo Moggia, N., Lutz, W., Bakker, D., Schwartz, B., & Kazantzis, N. (2026) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.70239
Mir, R., Poduval, S., Sheringham, J., & Hamilton, F. L. (2026) · JMIR Research Protocols · doi:10.2196/71383
Janbakhsh, M., Turnbull, E., Baker, J., Bacon, A., & Farrand, P. A. (2026) · BMJ Open · doi:10.1136/bmjopen-2025-112494
Hampole, S. R., McCaslin, S. E., Carr, V. A., Schuster, D. M., Reyes, S. I., Steinmetz, S. E., Ramsey, K. M., Owen, J. E., Skidmore, W. C., Street, A. E., et al. (2026) · Journal of Medical Internet Research · doi:10.2196/85098
Engagement in Digital Health Interventions: Open Questions for Research and Design
Kelders, S. M., ten Klooster, I., Bastoni, S., Cadoni, I., Doğan, K., Kip, H., Jong, N. B.-D., Gouveia, R., Schueller, S. M., Oewel, B., et al. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3778762
Doğan, K., ten Klooster, I., Kip, H., Haeri, M. A., & Kelders, S. M. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3799330
Kalhapure, P. J. R. M. A. M. V. V. (2026) · International Journal of Advanced Research in Science Communication and Technology · doi:10.48175/ijarsct-34984
Oewel, B., Nambiar, K., Agapie, E., & Reddy, M. C. (2026) · JMIR Formative Research · doi:10.2196/89009
O'Brien, H. L. (2026) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.70121
Self-citationWeingarden, H. M., Jonathan, G., Hoeppner, S. S., Velazquez, S., Jacoby, R. J., Greenberg, J. L., Harrison, O., & Wilhelm, S. (2026) · JMIR Formative Research · doi:10.2196/98129
- Authors
- O'Brien, H. L., Morton, E., Kampen, A., Barnes, S. J., & Michalak, E. E.
- Affiliations
- University of British Columbia (CA)
- Corresponding author
- Michalak, Erin E.
- Venue
- BJPsych Open
- Publisher
- Royal College of Psychiatrists
- Published
- 2020-08-11
- Volume
- 6
- Issue
- 5
- Pages
- e86-e86
- ISSN
- 2056-4724 (ISSN-L)
- PMC
- PMC7453800
- OpenAlex
- W3048418686
- Cited by
- 57
- References
- 15
- Citations by year
- 2026: 12 · 2025: 12 · 2024: 6 · 2023: 13 · 2022: 10 · 2021: 4
- FWCI
- 6.10
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- BJPsych Open (publishedVersion, cc-by-nc-sa)
- PubMed Central (publishedVersion, cc-by-nc-sa)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- Canadian Institutes of Health Research
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Health Sciences › Health Professions › Health Information Management › Health, Technology, Consumer Behavior
- Keywords
- mobile health apps, app engagement, engagement metrics, download metrics, mental health, app usage, digital mental health interventions
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
