Estimating the impact of engagement with digital health interventions on patient outcomes in randomized trials
Open accessAbstract
Objective: Guidance is needed on studying engagement and treatment effects in digital health interventions, including levels required for benefit. We evaluated multiple analytic approaches for understanding the association between engagement and clinical outcomes. Materials and Methods: We defined engagement as intervention participants’ response rate to interactive text messages, and considered moderation, standard regression, mediation, and a modified instrumental variable (IV) analysis to investigate the relationship between engagement and clinical outcomes. We applied each approach to two randomized controlled trials featuring text message content in the intervention: REACH (Rapid Encouragement/Education and Communications for Health), which targeted diabetes, and VERB (Vanderbilt Emergency Room Bundle), which targeted hypertension. Results: In REACH, the treatment effect on hemoglobin A1c was estimated to be −0.73% (95% CI: [−1.29, −0.21]; P = 0.008), and in VERB, the treatment effect on systolic blood pressure was estimated to be −10.1 mmHg (95% CI: [−17.7, −2.8]; P = 0.007). Only the IV analyses suggested an effect of engagement on outcomes; the difference in treatment effects between engagers and non-engagers was −0.29% to −0.51% in the REACH study and −1.08 to −3.25 mmHg in the VERB study. Discussion: Standard regression and mediation have less power than a modified IV analysis, but the IV approach requires specification of assumptions. This is the first review of the strengths and limitations of various approaches to evaluating the impact of engagement on outcomes. Conclusions: Understanding the role of engagement in digital health interventions can help reveal when and how these interventions achieve desired outcomes.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
This finding is consistent with previous research highlighting the role of continuous engagement in the effectiveness of digital health interventions.
“This finding is consistent with previous research that has highlighted the role of continuous engagement in the effectiveness of digital health interventions25–27.”
From Impact of digital health interventions on glycemic control and weight management. (2025) · cites this paper as [26]
The Background and Significance section says users must be willing to engage for digital health interventions to be effective. It does not establish that engagement must be continuous: the Results found insufficient evidence of an association between text-message response rate and outcomes in either trial, and the paper emphasizes identifying how much engagement is needed rather than assuming more is better. Page numbers were not available in our copy. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Identification of Causal Effects Using Instrumental Variables
Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476902 · Cited by 4,230 · see citing works
Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction
Imbens, G. W., & Rubin, D. B. (2015) · doi:10.1017/cbo9781139025751 · Cited by 4,210 · see citing works
Process Analysis: Estimating Mediation in Treatment Evaluations
Judd, C. M., & Kenny, D. A. (1981) · Evaluation Review · doi:10.1177/0193841x8100500502 · Cited by 2,442 · see citing works
Perski, O., Blandford, A., West, R., & Michie, S. (2017) · Translational Behavioral Medicine · doi:10.1007/s13142-016-0453-1 · Cited by 1,193 · see citing works
The impact of mHealth interventions: systematic review of systematic reviews
Marcolino, M. S., Oliveira, J. A. Q., D'Agostino, M., Ribeiro, A. L., Alkmim, M. B. M., & Novillo-Ortiz, D. (2018) · JMIR mHealth and uHealth · doi:10.2196/mhealth.8873 · Cited by 1,163 · see citing works
Hamine, S., Gerth-Guyette, E., Faulx, D., Green, B. B., & Ginsburg, A. S. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.3951 · Cited by 1,149 · see citing works
Understanding and Promoting Effective Engagement With Digital Behavior Change Interventions
Yardley, L., Spring, B. J., Riper, H., Morrison, L. G., Crane, D. H., Curtis, K., Merchant, G. C., Naughton, F., & Blandford, A. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.06.015 · Cited by 1,118 · see citing works
A Systematic Review of the Impact of Adherence on the Effectiveness of e-Therapies
Donkin, L., Christensen, H., Naismith, S. L., Neal, B., Hickie, I. B., & Glozier, N. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1772 · Cited by 876 · see citing works
Evaluating Digital Health Interventions
Murray, E., Hekler, E. B., Andersson, G., Collins, L. M., Doherty, A., Hollis, C., Rivera, D. E., West, R., & Wyatt, J. C. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.06.008 · Cited by 847 · see citing works
Instrumental variable methods for causal inference
Baiocchi, M., Cheng, J., & Small, D. S. (2014) · Statistics in Medicine · doi:10.1002/sim.6128 · Cited by 687 · see citing works
Greenwood, D. A., Gee, P. M., Fatkin, K. J., & Peeples, M. (2017) · Journal of Diabetes Science and Technology · doi:10.1177/1932296817713506 · Cited by 526 · see citing works
Kripalani, S. B., Risser, J., Gatti, M. E., & Jacobson, T. A. (2008) · Value in Health · doi:10.1111/j.1524-4733.2008.00400.x · Cited by 510 · see citing works
Bootstrap inference when using multiple imputation
Schomaker, M., & Heumann, C. (2018) · Statistics in Medicine · doi:10.1002/sim.7654 · Cited by 466 · see citing works
Short, C. E., DeSmet, A., Woods, C., Williams, S. L., Maher, C., Middelweerd, A., Müller, A. M., Wark, P. A., Vandelanotte, C., Poppe, L., et al. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.9397 · Cited by 400 · see citing works
The Effectiveness of Prompts to Promote Engagement With Digital Interventions: A Systematic Review
Alkhaldi, G., Hamilton, F. L., Lau, R., Webster, R., Michie, S., & Murray, E. (2016) · Journal of Medical Internet Research · doi:10.2196/jmir.4790 · Cited by 273 · see citing works
Simblett, S., Greer, B., Matcham, F., Curtis, H., Polhemus, A., Ferrão, J., Gamble, P., & Wykes, T. (2018) · Journal of Medical Internet Research · doi:10.2196/10480 · Cited by 270 · see citing works
Li, R., Liang, N., Bu, F., & Hesketh, T. (2020) · JMIR mHealth and uHealth · doi:10.2196/17776 · Cited by 262 · see citing works
Psychometric Properties of the Perceived Diabetes Self-Management Scale (PDSMS)
Wallston, K. A., Rothman, R. L., & Cherrington, A. (2007) · Journal of Behavioral Medicine · doi:10.1007/s10865-007-9110-y · Cited by 248 · see citing works
Wei, Y., Zheng, P., Deng, H., Wang, X., Li, X., & Fu, H. (2020) · Journal of Medical Internet Research · doi:10.2196/21687 · Cited by 238 · see citing works
Liu, K., Xie, Z., & Or, C. K. (2020) · JMIR mHealth and uHealth · doi:10.2196/15779 · Cited by 181 · see citing works
Cole-Lewis, H., Ezeanochie, N., & Turgiss, J. (2019) · JMIR Formative Research · doi:10.2196/14052 · Cited by 155 · see citing works
Pal, K., Dack, C., Ross, J., Michie, S., May, C., Stevenson, F., Farmer, A., Yardley, L., Barnard, M., & Murray, E. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.8439 · Cited by 139 · see citing works
Kebede, M. M., Zeeb, H., Peters, M., Heise, T. L., & Pischke, C. R. (2018) · Diabetes Technology & Therapeutics · doi:10.1089/dia.2018.0216 · Cited by 131 · see citing works
The ARMS-D out performs the SDSCA, but both are reliable, valid, and predict glycemic control
Mayberry, L. S., Gonzalez, J. S., Wallston, K. A., Kripalani, S. B., & Osborn, C. Y. (2013) · Diabetes Research and Clinical Practice · doi:10.1016/j.diabres.2013.09.010 · Cited by 129 · see citing works
Self-citationYeager, C. M., & Benight, C. C. (2018) · mHealth · doi:10.21037/mhealth.2018.08.04 · Cited by 115 · see citing works
Mobile Phone-Based Video Messages for Diabetes Self-Care Support
Bell, A. M., Fonda, S. J., Walker, M. S., Schmidt, V., & Vigersky, R. A. (2012) · Journal of Diabetes Science and Technology · doi:10.1177/193229681200600214 · Cited by 112 · see citing works
Edney, S., Ryan, J. C., Olds, T., Monroe, C., Fraysse, F., Vandelanotte, C., Plotnikoff, R., Curtis, R., & Maher, C. (2019) · Journal of Medical Internet Research · doi:10.2196/14645 · Cited by 104 · see citing works
Engagement Promotes Abstinence in a Web-based Cessation Intervention: Cohort Study
Richardson, A. K., Graham, A. L., Cobb, N. K., Xiao, H., Mushro, A., Abrams, D. B., & Vallone, D. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2277 · Cited by 100 · see citing works
Alvarado, M. M., Kum, H.-C., Gonzalez Coronado, K., Foster, M. J., Ortega, P., & Lawley, M. A. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.6382 · Cited by 95 · see citing works
Old-fashioned technology in the era of “bling”: is there a future for text messaging in health care?
Willcox, J. C., Dobson, R., & Whittaker, R. (2019) · Journal of Medical Internet Research · doi:10.2196/16630 · Cited by 95 · see citing works
Nelson, L. A., Coston, T. D., Cherrington, A., & Osborn, C. Y. (2016) · Current Diabetes Reports · doi:10.1007/s11892-016-0755-1 · Cited by 83 · see citing works
Self-citationRay, P., Liaw, S.-T., & Lewis, J. (2016) · Yearbook of Medical Informatics · doi:10.15265/iy-2016-020 · Cited by 82 · see citing works
Nelson, L. A., Greevy, R. A., Spieker, A., Wallston, K. A., Elasy, T. A., Kripalani, S., Gentry, C., Bergner, E. M., LeStourgeon, L. M., Williamson, S. E., et al. (2021) · Diabetes Care · doi:10.2337/dc20-0961 · Cited by 74 · see citing works
Self-citationNouri, S. S., Adler-Milstein, J., Thao, C., Acharya, P., Barr-Walker, J., Sarkar, U., & Lyles, C. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa024 · Cited by 70 · see citing works
Miller, S., Ainsworth, B., Yardley, L., Milton, A., Weal, M. J., Smith, P. W. F., & Morrison, L. G. (2018) · Journal of Medical Internet Research · doi:10.2196/10966 · Cited by 66 · see citing works
The Assessment of User Engagement with eHealth Content: The eHealth Engagement Scale1
Lefebvre, R. C., Tada, Y., Hilfiker, S. W., & Baur, C. (2010) · Journal of Computer-Mediated Communication · doi:10.1111/j.1083-6101.2009.01514.x · Cited by 62 · see citing works
Quinn, C. C., Butler, E. C., Swasey, K. K., Shardell, M. D., Terrin, M. L., Barr, E. A., & Gruber‐Baldini, A. L. (2018) · JMIR mhealth and uhealth · doi:10.2196/mhealth.9265 · Cited by 61 · see citing works
Bachmann, J. M., Goggins, K. M., Nwosu, S., Schildcrout, J. S., Kripalani, S. B., & Wallston, K. A. (2016) · Patient Education and Counseling · doi:10.1016/j.pec.2016.07.020 · Cited by 54 · see citing works
Choi, W., Wang, S., Lee, Y., Oh, H., & Zheng, Z. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa029 · Cited by 52 · see citing works
One Drop | Mobile: An Evaluation of Hemoglobin A1c Improvement Linked to App Engagement
Osborn, C. Y., van Ginkel, J. R., Rodbard, D., Heyman, M., Marrero, D. G., Huddleston, B., & Dachis, J. (2017) · JMIR Diabetes · doi:10.2196/diabetes.8039 · Cited by 39 · see citing works
Zhang, S., Hamburger, E., Kahanda, S., Lyttle, M., Williams, R., & Jaser, S. S. (2018) · Diabetes Technology & Therapeutics · doi:10.1089/dia.2018.0015 · Cited by 31 · see citing works
Alshurafa, N. I., Jain, J., Alharbi, R., Iakovlev, G., Spring, B. J., & Pfammatter, A. F. (2018) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3287031 · Cited by 30 · see citing works
Perski, O., Lumsden, J., Garnett, C. V., BLANDFORD, A. E., West, R., & Michie, S. (2019) · Journal of Medical Internet Research · doi:10.2196/16197 · Cited by 23 · see citing works
Intervention engagement moderates the dose-response relationships in a dietary intervention
Lippke, S., Corbet, J. M., Lange, D., Parschau, L., & Schwarzer, R. (2016) · Dose-Response · doi:10.1177/1559325816637515 · Cited by 22 · see citing works
Nelson, L. A., Spieker, A. J., Kripalani, S. B., Rothman, R. L., Roumie, C. L., Coco, J., Fabbri, D., Levy, P. D., Collins, S. P., & McNaughton, C. D. (2021) · Patient Education and Counseling · doi:10.1016/j.pec.2021.10.011 · Cited by 13 · see citing works
Self-citationCohn, A. M., Zhou, Y., Cha, S., Perreras, L., & Graham, A. L. (2018) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2018.11.001 · Cited by 8 · see citing works
Spieker, A. J., Greevy, R. A., Nelson, L. A., & Mayberry, L. S. (2022) · The Annals of Applied Statistics · doi:10.1214/21-aoas1476 · Cited by 4 · see citing works
Self-citation
Flagged first, then by citations
1 of the 48 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- [33]33. Imbens GW, Rubin DB. Causal Inference in Statistics, Social, and Biomedical Sciences. Cambridge, England: Cambridge University Press, 2015. [Google Scholar]
Cited by
Sorted by citations
Impact of digital health interventions on glycemic control and weight management.
Veluvali, A., Dehghani Zahedani, A., Hosseinian, A., Aghaeepour, N., McLaughlin, T., Woodward, M., DiTullio, A., Hashemi, N., & Snyder, M. P. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01430-7 · Cited by 39 · see citing works
Bucher, A., Blazek, E. S., & Symons, C. T. (2024) · Mayo Clinic Proceedings Digital Health · doi:10.1016/j.mcpdig.2024.05.007 · Cited by 37 · see citing works
Eiselt, A.-K., Kirkendall, S., Xiong, E., Langner, D., & Goldfarb, M. B. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1713334 · Cited by 17 · see citing works
Fareed, N., Olvera, R. G., Wang, Y., Hayes, M. A., Larimore, E., Balvanz, P., Langley, R. E., Noel, C. A., Rock, P. J., Redmond, D., et al. (2024) · JMIR Human Factors · doi:10.2196/51525 · Cited by 13 · see citing works
User Engagement in an Online Digital Health Intervention to Promote Problem Solving
O'Brien, H. L., Chen, A., Kaneshiro, J., & Zaslavsky, O. (2024) · Interacting with Computers · doi:10.1093/iwc/iwae030 · Cited by 11 · see citing works
Fakhoury, H., Trochez, R. J., Kripalani, S. B., Choma, N. N., Blessinger, E. K., & Nelson, L. A. (2024) · Journal of Hospital Medicine · doi:10.1002/jhm.13334 · Cited by 7 · see citing works
Self-citationAlodhialah, A. M., Almutairi, A. A., & Almutairi, M. (2024) · Healthcare · doi:10.3390/healthcare12222257 · Cited by 8 · see citing works
Solar, C. A., Nansubuga, A., Murillo, J., Ranker, L. R., Borrelli, B., Bowen, D. J., Xuan, Z., Kane, J., Werntz, S., Crouter, S. E., et al. (2022) · Contemporary Clinical Trials · doi:10.1016/j.cct.2022.106836 · Cited by 6 · see citing works
Nelson, L. A., Spieker, A. J., LeStourgeon, L. M., Greevy, R. A., Molli, S., Roddy, M. K., & Mayberry, L. S. (2023) · JMIR mhealth and uhealth · doi:10.2196/47632 · Cited by 6 · see citing works
Self-citationThe preferences of breast cancer patients regarding a digital social care platform
Rafiei, S., Heidarpoor, P., Souri, S., Nejatifar, Z., & Amerzadeh, M. (2025) · BMC Women s Health · doi:10.1186/s12905-025-03792-2 · Cited by 3 · see citing works
Sidhu, A., Shegog, R., Rushing, S. C., Trevino, N., Singer, M., Jessen, C. M., Gorman, G., Simpson, S., Peskin, M. F., Hernandez, B., et al. (2025) · JMIR Formative Research · doi:10.2196/67885 · Cited by 2 · see citing works
Ionov, M., Yudina, Y., Kurapeev, D. I., Zvartau, N., & Konradi, A. (2024) · Russian Journal of Preventive Medicine · doi:10.17116/profmed20242712137 · Cited by 1 · see citing works
Mihail, S., Partida, M., Villanueva, L., Thompson, D. I., O’Connor, T. M., Musaad, S. M. A., REDONDO, M. J., & Soltero, E. G. (2026) · JMIR Formative Research · doi:10.2196/67800 · Cited by 1 · see citing works
The use of telemedicine in palliative care in low- and middle-income countries: A scoping review
Goodman, W., Bhattarai, M., Shahzad, H., Chen, S., Burke, S. M., & Allsop, M. J. (2026) · Palliative Medicine · doi:10.1177/02692163261441432 · Cited by 1 · see citing works
McNaughton, C. D., Austin, P. C., Jackevicius, C. A., Chu, A., Holodinsky, J. K., Hill, M. D., Norris, C. M., Kumar, M., Kamal, N. R., Lee, D. S., et al. (2025) · BMC Cardiovascular Disorders · doi:10.1186/s12872-025-04492-3 · Cited by 1 · see citing works
Self-citationEvia, Á. F. V. D. L. G., Eberle, J. W., Baee, S., Wolfe, E. C., Boukhechba, M., Funk, D., Teachman, B. A., & Barnes, L. E. (2024) · doi:10.31234/osf.io/yucqf · Cited by 1 · see citing works
The Goldilocks Dilemma in mHealth Interventions: Balancing User Response and Reflection (Preprint)Preprint
Nelson, L. A., Spieker, A. J., LeStourgeon, L. M., Greevy, R. A., Molli, S., Roddy, M. K., & Mayberry, L. S. (2023) · doi:10.2196/preprints.47632
Self-citationSidhu, A., Shegog, R., Rushing, S. C., Trevino, N., Singer, M., Jessen, C. M., Gorman, G., Simpson, S., Peskin, M. F., Hernandez, B., et al. (2024) · doi:10.2196/preprints.67885
Evia, Á. F. V. D. L. G., Eberle, J. W., Baee, S., Wolfe, E. C., Boukhechba, M., Funk, D. H., Teachman, B. A., & Barnes, L. E. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000945
Biedebach, L., Friðgeirsdóttir, K. Ý., Carpinelli, C., Ísberg, A. P., Helgadóttir, H., Arnardóttir, E. S., Saavedra, J. M., & Íslind, A. S. (2026) · JMIR Formative Research · doi:10.2196/75358
Maternal mHealth usability strengthened by trust, governance, and navigation
Azugbene, E. A. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1825544
- Authors
- Nelson, L. A., Spieker, A. J., Mayberry, L. S., McNaughton, C., & Greevy, R. A.
- Affiliations
- Vanderbilt University Medical Center (US)
- Vanderbilt University (US)
- VA Tennessee Valley Healthcare System (US)
- Corresponding author
- Nelson, Lyndsay A
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2021-11-22
- Volume
- 29
- Issue
- 1
- Pages
- 128-136
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC8714267
- OpenAlex
- W4200357219
- Cited by
- 23
- References
- 48
- Citations by year
- 2026: 6 · 2025: 6 · 2024: 8 · 2023: 2 · 2022: 1
- FWCI
- 3.08
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model (vor)
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Institutes of Health
- National Institute of Diabetes and Digestive and Kidney Diseases (R01-DK100694)
- Center for Diabetes Translation Research Pilot & Feasibility Award (P30-DK092986)
- National Heart, Lung, and Blood Institute (K12-HL137943)
- Topics
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- digital health interventions, patient engagement, text messaging, instrumental variables, mediation analysis, moderation, diabetes mellitus, HbA1c, hypertension, systolic blood pressure, REACH study, treatment effect estimation
- MeSH terms
- Blood Pressure, Diabetes Mellitus, Humans, Hypertension, Randomized Controlled Trials as Topic, Text Messaging
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
