Assessing User Engagement of an mHealth Intervention: Development and Implementation of the Growing Healthy App Engagement Index
Open accessAbstract
BACKGROUND: Childhood obesity is an ongoing problem in developed countries that needs targeted prevention in the youngest age groups. Children in socioeconomically disadvantaged families are most at risk. Mobile health (mHealth) interventions offer a potential route to target these families because of its relatively low cost and high reach. The Growing healthy program was developed to provide evidence-based information on infant feeding from birth to 9 months via app or website. Understanding user engagement with these media is vital to developing successful interventions. Engagement is a complex, multifactorial concept that needs to move beyond simple metrics. OBJECTIVE: The aim of our study was to describe the development of an engagement index (EI) to monitor participant interaction with the Growing healthy app. The index included a number of subindices and cut-points to categorize engagement. METHODS: The Growing program was a feasibility study in which 300 mother-infant dyads were provided with an app which included 3 push notifications that was sent each week. Growing healthy participants completed surveys at 3 time points: baseline (T1) (infant age ≤3 months), infant aged 6 months (T2), and infant aged 9 months (T3). In addition, app usage data were captured from the app. The EI was adapted from the Web Analytics Demystified visitor EI. Our EI included 5 subindices: (1) click depth, (2) loyalty, (3) interaction, (4) recency, and (5) feedback. The overall EI summarized the subindices from date of registration through to 39 weeks (9 months) from the infant's date of birth. Basic descriptive data analysis was performed on the metrics and components of the EI as well as the final EI score. Group comparisons used t tests, analysis of variance (ANOVA), Mann-Whitney, Kruskal-Wallis, and Spearman correlation tests as appropriate. Consideration of independent variables associated with the EI score were modeled using linear regression models. RESULTS: The overall EI mean score was 30.0% (SD 11.5%) with a range of 1.8% - 57.6%. The cut-points used for high engagement were scores greater than 37.1% and for poor engagement were scores less than 21.1%. Significant explanatory variables of the EI score included: parity (P=.005), system type including "app only" users or "both" app and email users (P<.001), recruitment method (P=.02), and baby age at recruitment (P=.005). CONCLUSIONS: The EI provided a comprehensive understanding of participant behavior with the app over the 9-month period of the Growing healthy program. The use of the EI in this study demonstrates that rich and useful data can be collected and used to inform assessments of the strengths and weaknesses of the app and in turn inform future interventions.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Taki et al combined session analytics with page analytics and measured the number of pages viewed per session in their obesity self-management app.
“Taki et al combined session analytics with page analytics and measured the number of pages viewed per session in their app for obesity self-management [72].”
From A Library of Analytic Indicators to Evaluate Effective Engagement with Consumer mHealth Apps for Chronic Conditions: Scoping Review (2019) · cites this paper as [72]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
(1990) · Choice Reviews Online · doi:10.5860/choice.27-3338 · Cited by 35,305 · see citing works
Developing and evaluating complex interventions: the new Medical Research Council guidance
Craig, P., Dieppe, P., Macintyre, S., Michie, S., Nazareth, I., & Petticrew, M. (2008) · BMJ · doi:10.1136/bmj.a1655 · Cited by 10,995 · see citing works
Webb, T. L., Joseph, J., Yardley, L., & Michie, S. (2010) · Journal of Medical Internet Research · doi:10.2196/jmir.1376 · Cited by 2,517 · see citing works
(2014) · Cited by 2,276 · see citing works
Eysenbach, G., Powell, J., Kuss, O., & Sa, E.-R. (2002) · JAMA · doi:10.1001/jama.287.20.2691 · Cited by 1,871 · see citing works
What is user engagement? A conceptual framework for defining user engagement with technology
O'Brien, H. L., & Toms, E. G. (2008) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20801 · Cited by 1,397 · see citing works
Perski, O., Blandford, A., West, R., & Michie, S. (2017) · Translational Behavioral Medicine · doi:10.1007/s13142-016-0453-1 · Cited by 1,193 · see citing works
Understanding and Promoting Effective Engagement With Digital Behavior Change Interventions
Yardley, L., Spring, B. J., Riper, H., Morrison, L. G., Crane, D. H., Curtis, K., Merchant, G. C., Naughton, F., & Blandford, A. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.06.015 · Cited by 1,118 · see citing works
Dennison, L., Morrison, L., Conway, G., & Yardley, L. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2583 · Cited by 895 · see citing works
Can Mobile Phone Apps Influence People’s Health Behavior Change? An Evidence Review
Zhao, J., Freeman, B., & Li, M. (2016) · Journal of Medical Internet Research · doi:10.2196/jmir.5692 · Cited by 618 · see citing works
Scott, A. J., Hosmer, D. W., & Lemeshow, S. A. (1991) · Biometrics · doi:10.2307/2532419 · Cited by 372 · see citing works
Measuring engagement in video game-based environments: Investigation of the User Engagement Scale
Wiebe, E., Lamb, A., Frankosky, M. H., & Sharek, D. J. (2014) · Computers in Human Behavior · doi:10.1016/j.chb.2013.12.001 · Cited by 309 · see citing works
MAINTAINING ENGAGEMENT IN LONG-TERM INTERVENTIONS WITH RELATIONAL AGENTS
Bickmore, T., Schulman, D., & Yin, L. (2010) · Applied Artificial Intelligence · doi:10.1080/08839514.2010.492259 · Cited by 246 · see citing works
Engagement and retention: measuring breadth and depth of participant use of an online intervention
Couper, M. P., Alexander, G. L., Zhang, N., Little, R. J., Maddy, N., Nowak, M. A., McClure, J. B., Calvi, J. J., Rolnick, S. J., Stopponi, M. A., et al. (2010) · Journal of Medical Internet Research · doi:10.2196/jmir.1430 · Cited by 244 · see citing works
Turner, T., Spruijt‐Metz, D., Wen, C. K. F., & Hingle, M. D. (2015) · Pediatric Obesity · doi:10.1111/ijpo.12002 · Cited by 172 · see citing works
Guertler, D., Vandelanotte, C., Kirwan, M., & Duncan, M. J. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4339 · Cited by 147 · see citing works
Partridge, S. R., McGeechan, K., Hebden, L., Balestracci, K., Wong, A. T. Y., Denney‐Wilson, E., Harris, M., Phongsavan, P., Bauman, A. E., & Allman-Farinelli, M. A. (2015) · JMIR mhealth and uhealth · doi:10.2196/mhealth.4530 · Cited by 147 · see citing works
Self-citationNeve, M. J., Collins, C. E., & Morgan, P. J. (2010) · Journal of Medical Internet Research · doi:10.2196/jmir.1640 · Cited by 137 · see citing works
Targeted Health Behavior Interventions Promoting Physical Activity
Morgan, P. J., Young, M. D., Smith, J. J., & Lubans, D. R. (2016) · Exercise and Sport Sciences Reviews · doi:10.1249/jes.0000000000000075 · Cited by 131 · see citing works
Allison, S. M., Bauermeister, J. A., Bull, S. S., Lightfoot, M. A., Mustanski, B. S., Shegog, R., & Levine, D. (2012) · Journal of Adolescent Health · doi:10.1016/j.jadohealth.2012.06.012 · Cited by 127 · see citing works
Danaher, B. G., Brendryen, H., Seeley, J. R., Tyler, M. S., & Woolley, T. (2015) · Internet Interventions · doi:10.1016/j.invent.2015.01.002 · Cited by 102 · see citing works
Parents, infants and health care: Utilization of health services in the first 12 months of life
Goldfeld, S. R., Wright, M., & Oberklaid, F. (2003) · Journal of Paediatrics and Child Health · doi:10.1046/j.1440-1754.2003.00146.x · Cited by 91 · see citing works
Exploring the Usability of a Mobile App for Adolescent Obesity Management
O’Malley, G., Dowdall, G., Burls, A., Perry, I. J., & Curran, N. (2014) · JMIR mhealth and uhealth · doi:10.2196/mhealth.3262 · Cited by 89 · see citing works
Infant Feeding Websites and Apps: A Systematic Assessment of Quality and Content
Taki, S., Campbell, K. J., Russell, C. G., Elliott, R. M., Laws, R., & Denney‐Wilson, E. (2015) · Interactive Journal of Medical Research · doi:10.2196/ijmr.4323 · Cited by 89 · see citing works
Self-citationWhite, B. K., Martin, A., White, J. A., Burns, S. K., Maycock, B. R., Giglia, R. C., & Scott, J. A. (2016) · JMIR mHealth and uHealth · doi:10.2196/mhealth.5652 · Cited by 86 · see citing works
Baltierra, N., Muessig, K. E., Pike, E. C., LeGrand, S., Bull, S. S., & Hightow‐Weidman, L. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.12.015 · Cited by 79 · see citing works
Laws, R., Litterbach, E., Denney‐Wilson, E., Russell, C. G., Taki, S., Ong, K., Elliott, R. M., Lymer, S. J., & Campbell, K. J. (2016) · Journal of Medical Internet Research · doi:10.2196/jmir.5691 · Cited by 68 · see citing works
Self-citationNational Health and Medical Research Council Infant Feeding Guidelines for Health Workers, 2003
(2005) · New South Wales Public Health Bulletin · doi:10.1071/nb05010 · Cited by 68 · see citing works
Davies, C., Corry, K., Itallie, A. K. V., Vandelanotte, C., Caperchione, C. M., & Mummery, W. K. (2012) · Journal of Medical Internet Research · doi:10.2196/jmir.1792 · Cited by 66 · see citing works
Crutzen, R., de Nooijer, J., Brouwer, W., Oenema, A., Brug, J., & de Vries, N. K. (2009) · Health Promotion International · doi:10.1093/heapro/dap018 · Cited by 60 · see citing works
Campbell, K. J., Hesketh, K. D., McNaughton, S. A., Ball, K., McCallum, Z., Lynch, J. W., & Crawford, D. A. (2016) · BMC Public Health · doi:10.1186/s12889-016-2836-0 · Cited by 60 · see citing works
Self-citationAskie, L. M., Baur, L. A., Campbell, K. J., Daniels, L., Hesketh, K. D., Magarey, A., Mihrshahi, S., Rissel, C. E., Simes, R. J., Taylor, B. J., et al. (2010) · BMC Public Health · doi:10.1186/1471-2458-10-728 · Cited by 55 · see citing works
Self-citationUnderstanding affect in design: an outline conceptual framework
Aboulafia, A., & Bannon, L. J. (2004) · Theoretical Issues in Ergonomics Science · doi:10.1080/1463922031000086708 · Cited by 50 · see citing works
Preventing obesity in infants: the Growing healthy feasibility trial protocol
Denney‐Wilson, E., Laws, R., Russell, C. G., Ong, K.-L., Taki, S., Elliot, R., Azadi, L., Lymer, S. J., Taylor, R. W., Lynch, J. W., et al. (2015) · BMJ Open · doi:10.1136/bmjopen-2015-009258 · Cited by 46 · see citing works
Self-citationSuccessful Enrollment in Text4Baby More Likely With Higher Health Literacy
Gazmararian, J. A., Yang, B., Elon, L., Graham, M., & Parker, R. M. (2012) · Journal of Health Communication · doi:10.1080/10810730.2012.712618 · Cited by 43 · see citing works
Influences on infant feeding decisions of first-time mothers in five European countries
Gage, H., Williams, P., Hoewel, J. V. R.-V., et al. (2012) · European Journal of Clinical Nutrition · doi:10.1038/ejcn.2012.56 · Cited by 39 · see citing works
Arden-Close, E. J., Smith, E., Bradbury, K., Morrison, L., Dennison, L., Michaelides, D., & Yardley, L. (2015) · JMIR Human Factors · doi:10.2196/humanfactors.4310 · Cited by 32 · see citing works
Li, Y., Wang, W., van Velthoven, M. H., Chen, L., Car, J., Rudan, I., Zhang, Y., Wu, Q., Du, X., & Scherpbier, R. W. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2906 · Cited by 28 · see citing works
Tools for teen moms to reduce infant obesity: a randomized clinical trial
Horodynski, M. A. O., Silk, K. J., Hsieh, G., Hoffman, A., & Robson, M. (2015) · BMC Public Health · doi:10.1186/s12889-015-1345-x · Cited by 27 · see citing works
Sources of feeding advice in the first year of life: who do parents value?
Gildea, A., Sloan, S., & Stewart, M. (2009) · PubMed · Cited by 25 · see citing works
Uesugi, K. H., Dattilo, A. M., Black, M. M., & Saavedra, J. M. (2016) · Journal of Obesity · doi:10.1155/2016/5067421 · Cited by 16 · see citing works
Obesity in Young Children: What Can We Do About?
Wen, L. M. (2015) · Journal of Obesity & Weight loss Therapy · doi:10.4172/2165-7904.1000286 · Cited by 4 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Short, C. E., DeSmet, A., Woods, C., Williams, S. L., Maher, C., Middelweerd, A., Müller, A. M., Wark, P. A., Vandelanotte, C., Poppe, L., et al. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.9397 · Cited by 400 · see citing works
Pham, Q., Graham, G., Carrion, C., Morita, P. P., Seto, E., Stinson, J. N., & Cafazzo, J. A. (2019) · JMIR mHealth and uHealth · doi:10.2196/11941 · Cited by 131 · see citing works
Carrasco-Hernandez, L., Jódar-Sánchez, F., Núñez-Benjumea, F., Moreno Conde, J., Mesa González, M., Civit-Balcells, A., Hors-Fraile, S., Parra-Calderón, C. L., Bamidis, P. D., & Ortega-Ruiz, F. (2020) · JMIR mHealth and uHealth · doi:10.2196/17530 · Cited by 65 · see citing works
Kim, M., Yang, J., Ahn, W.-Y., & Choi, H. J. (2021) · Journal of Medical Internet Research · doi:10.2196/27218 · Cited by 41 · see citing works
Taki, S., Russell, C. G., Lymer, S., Laws, R., Campbell, K., Appleton, J., Ong, K.-L., & Denney-Wilson, E. (2019) · Frontiers in Endocrinology · doi:10.3389/fendo.2019.00397 · Cited by 24 · see citing works
Self-citationProportionate methods for evaluating a simple digital mental health tool
Davies, E. B., Craven, M. P., Martin, J. L., & Simons, L. (2017) · Evidence Based Mental Health · doi:10.1136/eb-2017-102755 · Cited by 15 · see citing works
Verbiest, M., Borrell, S., Dalhousie, S., Tupa'i-Firestone, R., Funaki, T., Goodwin, D., Grey, J., Henry, A., Hughes, E., Humphrey, G., et al. (2018) · JMIR Research Protocols · doi:10.2196/10789 · Cited by 13 · see citing works
- Authors
- Taki, S., Lymer, S., Russell, C. G., Campbell, K., Laws, R., Ong, K.-L., Elliott, R., & Denney-Wilson, E.
- Affiliations
- University of Technology Sydney (AU)
- The University of Sydney (AU)
- Deakin University (AU)
- La Trobe University (AU)
- Corresponding author
- Taki, Sarah
- ORCID
- Venue
- JMIR mHealth and uHealth
- Publisher
- JMIR Publications Inc.
- Published
- 2017-06-29
- Volume
- 5
- Issue
- 6
- Pages
- e89
- ISSN
- 2291-5222 (ISSN-L)
- PMC
- PMC5509951
- OpenAlex
- W2730024606
- Cited by
- 114
- References
- 42
- Citations by year
- 2026: 4 · 2025: 13 · 2024: 10 · 2023: 11 · 2022: 15 · 2021: 13 · 2020: 13 · 2019: 13 · 2018: 18 · 2017: 4
- FWCI
- 21.28
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR mhealth and uhealth (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RMIT Research Repository (RMIT University Library) (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Deakin Research Online (Deakin University) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- UTS ePRESS (University of Technology Sydney) (submittedVersion)
- UNSWorks (University of New South Wales, Sydney, Australia) (submittedVersion, cc-by)
- Funders
- Australian Government
- Australian Primary Health Care Research Institute, Australian National University
- Topics
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Keywords
- user engagement, mobile health apps, engagement index, infant feeding, childhood obesity prevention, socioeconomic deprivation, mother-infant dyads, push notifications, parity, app retention, app abandonment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
