An Analytics Platform to Evaluate Effective Engagement With Pediatric Mobile Health Apps: Design, Development, and Formative Evaluation
Open accessPublished version of a preprint
Abstract
BACKGROUND: Mobile health (mHealth) apps for pediatric chronic conditions are growing in availability and challenge investigators to conduct rigorous evaluations that keep pace with mHealth innovation. Traditional research methods are poorly suited to operationalize the agile, iterative trials required to evidence and optimize these digitally mediated interventions. OBJECTIVE: We sought to contribute a resource to support the quantification, analysis, and visualization of analytic indicators of effective engagement with mHealth apps for chronic conditions. METHODS: We applied user-centered design methods to design and develop an Analytics Platform to Evaluate Effective Engagement (APEEE) with consumer mHealth apps for chronic conditions and implemented the platform to analyze both retrospective and prospective data generated from a smartphone-based pain self-management app called iCanCope for young people with chronic pain. RESULTS: Through APEEE, we were able to automate the process of defining, operationalizing, and evaluating effective engagement with iCanCope. Configuring the platform to integrate with the app was feasible and provided investigators with a resource to consolidate, analyze, and visualize engagement data generated by participants in real time. Preliminary efforts to evaluate APEEE showed that investigators perceived the platform to be an acceptable evaluative resource and were satisfied with its design, functionality, and performance. Investigators saw potential in APEEE to accelerate and augment evidence generation and expressed enthusiasm for adopting the platform to support their evaluative practice once fully implemented. CONCLUSIONS: Dynamic, real-time analytic platforms may provide investigators with a powerful means to characterize the breadth and depth of mHealth app engagement required to achieve intended health outcomes. Successful implementation of APEEE into evaluative practice may contribute to the realization of effective and evidence-based mHealth care.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
These digitally mediated mechanisms of action require more granular evaluations capable of analyzing multilevel, temporally dense engagement data.
“These digitally mediated mechanisms of action require more granular evaluations capable of analyzing multilevel, temporally dense engagement data [13].”
From A Library of Analytic Indicators to Evaluate Effective Engagement with Consumer mHealth Apps for Chronic Conditions: Scoping Review (2019) · cites this paper as [13]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Understanding and Promoting Effective Engagement With Digital Behavior Change Interventions
Yardley, L., Spring, B. J., Riper, H., Morrison, L. G., Crane, D. H., Curtis, K., Merchant, G. C., Naughton, F., & Blandford, A. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.06.015 · Cited by 1,118 · see citing works
Vega-Lite: A Grammar of Interactive Graphics
Satyanarayan, A., Moritz, D., Wongsuphasawat, K., & Heer, J. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2016.2599030 · Cited by 883 · see citing works
Michie, S., Yardley, L., West, R., Patrick, K., & Greaves, F. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7126 · Cited by 865 · see citing works
Microrandomized trials: An experimental design for developing just-in-time adaptive interventions.
Klasnja, P., Hekler, E. B., Shiffman, S., Boruvka, A., Almirall, D., Tewari, A., & Murphy, S. A. (2015) · Health Psychology · doi:10.1037/hea0000305 · Cited by 671 · see citing works
The mPower study, Parkinson disease mobile data collected using ResearchKit
Bot, B. M., Suver, C., Neto, E. C., Kellen, M., Klein, A., Bare, C., Doerr, M., Pratap, A., Wilbanks, J., Dorsey, E. R., et al. (2016) · Scientific Data · doi:10.1038/sdata.2016.11 · Cited by 630 · see citing works
<i>mHealth Consumer Apps</i> : The Case for User-Centered Design
McCurdie, T., Taneva, S., Casselman, M., Yeung, M., McDaniel, C., Ho, W., & Cafazzo, J. (2012) · Biomedical Instrumentation & Technology · doi:10.2345/0899-8205-46.s2.49 · Cited by 478 · see citing works
Self-citationRyu, S. (2012) · Healthcare Informatics Research · doi:10.4258/hir.2012.18.3.231 · Cited by 451 · see citing works
Rapid, responsive, relevant (R3) research: a call for a rapid learning health research enterprise
Riley, W. T., Glasgow, R. E., Etheredge, L., & Abernethy, A. P. (2013) · Clinical and Translational Medicine · doi:10.1186/2001-1326-2-10 · Cited by 267 · see citing works
Lalloo, C., Jibb, L. A., Rivera, J., Agarwal, A., & Stinson, J. N. (2015) · The Clinical Journal of Pain · doi:10.1097/ajp.0000000000000171 · Cited by 257 · see citing works
Majeed-Ariss, R., Baildam, E., Campbell, M., Chieng, A., Fallon, D., Hall, A., McDonagh, J. E., Stones, S. R., Thomson, W., & Swallow, V. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.5043 · Cited by 251 · see citing works
The Asthma Mobile Health Study, a large-scale clinical observational study using ResearchKit
Chan, Y.-F. Y., Wang, P., Rogers, L., Tignor, N., Zweig, M., Hershman, S. G., Genes, N., Scott, E. R., Krock, E., Badgeley, M., et al. (2017) · Nature Biotechnology · doi:10.1038/nbt.3826 · Cited by 204 · see citing works
Beyond “implementation”: digital health innovation and service design
Shaw, J., Agarwal, P., Desveaux, L., Palma, D. C., Stamenova, V., Jamieson, T., Yang, R., Bhatia, R. S., & Bhattacharyya, O. K. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0059-8 · Cited by 187 · see citing works
The Pace of Technologic Change
Patrick, K., Hekler, E. B., Estrin, D., Mohr, D. C., Riper, H., Crane, D., Godino, J., & Riley, W. T. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.05.001 · Cited by 169 · see citing works
Beyond the Randomized Controlled Trial: A Review of Alternatives in mHealth Clinical Trial Methods
Pham, Q., Wiljer, D., & Cafazzo, J. A. (2016) · JMIR mHealth and uHealth · doi:10.2196/mhealth.5720 · Cited by 162 · see citing works
Self-citationStinson, J. N., Lalloo, C., Harris, L., Isaac, L., Campbell, F., Brown, S., Ruskin, D., Gordon, A., Galonski, M., Pink, L. R., et al. (2014) · Pain Research and Management · doi:10.1155/2014/935278 · Cited by 149 · see citing works
Self-citationCrane, D., Garnett, C., Michie, S., West, R., & Brown, J. (2018) · Scientific Reports · doi:10.1038/s41598-018-22420-8 · Cited by 118 · see citing works
Current Issues and Future Directions for Research Into Digital Behavior Change Interventions
Yardley, L., Choudhury, T., Patrick, K., & Michie, S. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.07.019 · Cited by 112 · see citing works
Opening the Black Box of Electronic Health: Collecting, Analyzing, and Interpreting Log Data
Sieverink, F., Kelders, S., Poel, M., & van Gemert-Pijnen, L. (2017) · JMIR Research Protocols · doi:10.2196/resprot.6452 · Cited by 88 · see citing works
Apple Apps for the Management of Pediatric Pain and Pain-Related Stress
Smith, K. R., Iversen, C., Kossowsky, J., Odell, S., Gambhir, R., & Coakley, R. (2015) · Clinical Practice in Pediatric Psychology · doi:10.1037/cpp0000092 · Cited by 32 · see citing works
Clough, B. A., & Casey, L. M. (2015) · Professional Psychology Research and Practice · doi:10.1037/pro0000053 · Cited by 29 · see citing works
Challenges and Paradoxes of Human Factors in Health Technology Design
Morita, P. P., & Cafazzo, J. A. (2016) · JMIR Human Factors · doi:10.2196/humanfactors.4653 · Cited by 25 · see citing works
Self-citationNoser, A. E., Cushing, C. C., McGrady, M. E., Amaro, C. M., & Huffhines, L. P. (2017) · Clinical Practice in Pediatric Psychology · doi:10.1037/cpp0000185 · Cited by 23 · see citing works
METHODS FOR CLUSTERING TIME SERIES DATA ACQUIRED FROM MOBILE HEALTH APPS
Tignor, N. L., Wang, P., Genes, N. G., Rogers, L., Hershman, S. G., Scott, E. R., Zweig, M., Chan, Y. Y., & Schadt, E. E. (2016) · Pacific Symposium on Biocomputing (PSB) · doi:10.1142/9789813207813_0029 · Cited by 7 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [27]Riley WT A new era of clinical research methods in a data-rich environment Oncology Informatics 2016 United States Academic Press 55
Cited by
Sorted by citations
Pham, Q., Graham, G., Carrion, C., Morita, P. P., Seto, E., Stinson, J. N., & Cafazzo, J. A. (2019) · JMIR mHealth and uHealth · doi:10.2196/11941 · Cited by 131 · see citing works
Self-citation
- Authors
- Pham, Q., Graham, G., Lalloo, C., Morita, P. P., Seto, E., Stinson, J. N., & Cafazzo, J. A.
- Affiliations
- University Health Network (CA)
- University of Toronto (CA)
- University of Waterloo (CA)
- ORCID
- Venue
- JMIR mHealth and uHealth
- Publisher
- JMIR Publications Inc.
- Published
- 2018-12-21
- Volume
- 6
- Issue
- 12
- Pages
- e11447
- ISSN
- 2291-5222 (ISSN-L)
- PMC
- PMC6320392
- OpenAlex
- W2905952024
- Cited by
- 30
- References
- 23
- Citations by year
- 2026: 2 · 2025: 2 · 2024: 4 · 2023: 2 · 2022: 8 · 2021: 1 · 2020: 8 · 2019: 3
- FWCI
- 4.58
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR mhealth and uhealth (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Health Professions › Health Information Management › Health, Technology, Consumer Behavior
- Keywords
- mobile health apps, app engagement, data analysis platform, pediatric chronic conditions, chronic pain, pain self-management, user-centered design, app evaluation, youth, formative assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
