Can a Single Variable Predict Early Dropout From Digital Health Interventions? Comparison of Predictive Models From Two Large Randomized Trials
Open accessThe DOI matches its Crossref record; 49 of the 50 references it cites were found in a registry or an index.
Abstract
Background: A single generalizable metric that accurately predicts early dropout from digital health interventions has the potential to readily inform intervention targets and treatment augmentations that could boost retention and intervention outcomes. We recently identified a type of early dropout from digital health interventions for smoking cessation, specifically, users who logged in during the first week of the intervention and had little to no activity thereafter. These users also had a substantially lower smoking cessation rate with our iCanQuit smoking cessation app compared with users who used the app for longer periods. Objective: This study aimed to explore whether log-in count data, using standard statistical methods, can precisely predict whether an individual will become an iCanQuit early dropout while validating the approach using other statistical methods and randomized trial data from 3 other digital interventions for smoking cessation (combined randomized N=4529). Methods: Standard logistic regression models were used to predict early dropouts for individuals receiving the iCanQuit smoking cessation intervention app, the National Cancer Institute QuitGuide smoking cessation intervention app, the WebQuit.org smoking cessation intervention website, and the Smokefree.gov smoking cessation intervention website. The main predictors were the number of times a participant logged in per day during the first 7 days following randomization. The area under the curve (AUC) assessed the performance of the logistic regression models, which were compared with decision trees, support vector machine, and neural network models. We also examined whether 13 baseline variables that included a variety of demographics (eg, race and ethnicity, gender, and age) and smoking characteristics (eg, use of e-cigarettes and confidence in being smoke free) might improve this prediction. Results: The AUC for each logistic regression model using only the first 7 days of log-in count variables was 0.94 (95% CI 0.90-0.97) for iCanQuit, 0.88 (95% CI 0.83-0.93) for QuitGuide, 0.85 (95% CI 0.80-0.88) for WebQuit.org, and 0.60 (95% CI 0.54-0.66) for Smokefree.gov. Replacing logistic regression models with more complex decision trees, support vector machines, or neural network models did not significantly increase the AUC, nor did including additional baseline variables as predictors. The sensitivity and specificity were generally good, and they were excellent for iCanQuit (ie, 0.91 and 0.85, respectively, at the 0.5 classification threshold). Conclusions: Logistic regression models using only the first 7 days of log-in count data were generally good at predicting early dropouts. These models performed well when using simple, automated, and readily available log-in count data, whereas including self-reported baseline variables did not improve the prediction. The results will inform the early identification of people at risk of early dropout from digital health interventions with the goal of intervening further by providing them with augmented treatments to increase their retention and, ultimately, their intervention outcomes.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Estimating the Dimension of a Model
Schwarz, G. (1978) · The Annals of Statistics · doi:10.1214/aos/1176344136 · Cited by 47,033
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
Modern Applied Statistics with S
Venables, W. N., & Ripley, B. D. (2002) · Statistics and Computing · doi:10.1007/978-0-387-21706-2 · Cited by 24,305
pROC: an open-source package for R and S+ to analyze and compare ROC curves
Robin, X., Turck, N., Hainard, A., Tiberti, N., Lisacek, F., Sanchez, J.-C., & Müller, M. (2011) · BMC Bioinformatics · doi:10.1186/1471-2105-12-77 · Cited by 13,697 · cited 1×
“AUC values were compared between models using the “roc.test” function in the R library pROC [47].”
Finding Groups in Data: An Introduction to Cluster Analysis
Kaufman, L., & Rousseeuw, P. J. (1990) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316801 · Cited by 7,344
Receiver Operating Characteristic Curve in Diagnostic Test Assessment
Mandrekar, J. N. (2010) · Journal of Thoracic Oncology · doi:10.1097/jto.0b013e3181ec173d · Cited by 4,506 · cited 1×
“In general, an AUC value between 0.7 and 0.8 is considered acceptable, 0.8 to 0.9 is considered excellent, and >0.9 is considered outstanding [46].”
InfluentialEysenbach, G. (2005) · Journal of Medical Internet Research · doi:10.2196/jmir.7.1.e11 · Cited by 2,524 · cited 2×
“An ongoing central challenge to the overall efficacy and broadly scalable implementation of digital interventions is the phenomenon of users dropping out early [10].”
Tu, J. V. (1996) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(96)00002-9 · Cited by 1,818 · cited 1×
“…model that best predicted early dropout was a standard logistic regression model, as opposed to more sophisticated machine learning models that require more advanced knowledge of machine learning as well as large amounts of data, have greater computational burden, and are prone to overfitting [50].”
A Clinical Practice Guideline for Treating Tobacco Use and Dependence: 2008 Update
(2008) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2008.04.009 · Cited by 1,410 · cited 2×
“…diverse sample of 2415 adult daily smokers from all 50 US states who were randomized 1:1 to receive access to an ACT-based smartphone app (iCanQuit; n=1214) or a United States Clinical Practice Guidelines (USCPG) approach–based smartphone app (QuitGuide; n=1201) for smoking cessation [27].”
Adherence in Internet Interventions for Anxiety and Depression
Christensen, H., Griffiths, K. M., & Farrer, L. (2009) · Journal of Medical Internet Research · doi:10.2196/jmir.1194 · Cited by 1,133 · cited 1×
“Although early drop out has been variously defined across studies (eg, completing only 1 module or stopping within the first week or the first few weeks) [10,11], the fraction of users who fall into this category is quite large, with some studies showing it as high as 82% [12-14].”
Hayes, S. C., Levin, M. E., Plumb-Vilardaga, J., Villatte, J. L., & Pistorello, J. (2013) · Behavior Therapy · doi:10.1016/j.beth.2009.08.002 · Cited by 1,120 · cited 1×
“The overall differences between the 4 interventions were that (1) iCanQuit and WebQuit taught skills based on the principles of ACT [29] for smoking cessation [30-34], whereas QuitGuide and Smokefree taught skills based on the USCPG for smoking cessation [27], and (2) iCanQuit and QuitGuide were…”
ROC Curves for Continuous Data
Krzanowski, W. J., & Hand, D. J. (2009) · Chapman & Hall/CRC eBooks · doi:10.1201/9781439800225 · Cited by 666 · cited 1×
“The receiver operating characteristic curve and corresponding area under the curve (AUC) were used to evaluate the performance of each predictive classification model [45].”
A Conceptual Framework for Adaptive Preventive Interventions
Collins, L. M., Murphy, S. A., & Bierman, K. L. (2004) · Prevention Science · doi:10.1023/b:prev.0000037641.26017.00 · Cited by 608 · cited 1×
“The Sequential Multiple Assignment Randomized Trial is an increasingly common type of factorial design that could be applied to the problem of early dropout from a digital intervention by experimentally manipulating the offer (vs not) of a specific treatment augmentation [53-55].”
Dropout from Internet‐based treatment for psychological disorders
Melville, K. M., Casey, L. M., & Kavanagh, D. J. (2010) · British Journal of Clinical Psychology · doi:10.1348/014466509x472138 · Cited by 597
Flores Mateo, G., Granado-Font, E., Ferré-Grau, C., & Montaña-Carreras, X. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4836 · Cited by 540
Predictors of dropout in weight loss interventions: a systematic review of the literature
Moroshko, I., Brennan, L., & O'Brien, P. (2011) · Obesity Reviews · doi:10.1111/j.1467-789x.2011.00915.x · Cited by 497
Almirall, D., Nahum-Shani, I., Sherwood, N. E., & Murphy, S. A. (2014) · Translational Behavioral Medicine · doi:10.1007/s13142-014-0265-0 · Cited by 453
Mobile phone text messaging and app-based interventions for smoking cessation
Whittaker, R., McRobbie, H., Bullen, C., Rodgers, A., Gu, Y., & Dobson, R. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006611.pub5 · Cited by 452 · cited 1×
“Digital interventions, including smartphone apps, websites, and SMS text messaging interventions, have proven efficacious for a wide variety of behavioral health outcomes, including mental health, diet, exercise, and smoking cessation [1-6].”
The cost-effectiveness of digital health interventions: A systematic review of the literature
Gentili, A., Failla, G., Melnyk, A., Puleo, V., Tanna, G. L. D., Ricciardi, W., & Cascini, F. (2022) · Frontiers in Public Health · doi:10.3389/fpubh.2022.787135 · Cited by 444 · cited 1×
“An important value of digital interventions for behavioral health is that they are accessible at a low cost, thereby increasing their potential public impact and reducing disparities in behavioral health care [7-9].”
Philippe, T. J., Sikder, N., Jackson, A., Koblanski, M. E., Liow, E., Pilarinos, A., & Vasarhelyi, K. (2022) · JMIR Mental Health · doi:10.2196/35159 · Cited by 436
Richardson, C. R., Buis, L. R., Janney, A. W., Goodrich, D. E., Sen, A., Hess, M. L., Mehari, K. S., Fortlage, L. A., Resnick, P. J., Zikmund-Fisher, B. J., et al. (2010) · Journal of Medical Internet Research · doi:10.2196/jmir.1338 · Cited by 239
Afshin, A., Babalola, D., Mclean, M., Yu, Z., Ma, W., Chen, C., Arabi, M., & Mozaffarian, D. (2016) · Journal of the American Heart Association · doi:10.1161/jaha.115.003058 · Cited by 235
A Randomized Trial of Text2Quit
Abroms, L. C., Boal, A. L., Simmens, S. J., Mendel, J. A., & Windsor, R. A. (2014) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2014.04.010 · Cited by 224
Efficacy of Smartphone Applications for Smoking Cessation: A Randomized Clinical Trial
Bricker, J. B., Watson, N. L., Mull, K. E., Sullivan, B. M., & Heffner, J. L. (2020) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2020.4055 · Cited by 207
Self-citationAdaptive design clinical trials: a review of the literature and ClinicalTrials.gov
Bothwell, L. E., Avorn, J., Khan, N. F., & Kesselheim, A. S. (2018) · BMJ Open · doi:10.1136/bmjopen-2017-018320 · Cited by 196
Predicting player churn in the wild
Hadiji, F., Sifa, R., Drachen, A., Thurau, C., Kersting, K., & Bauckhage, C. (2014) · 2014 IEEE Conference on Computational Intelligence and Games · doi:10.1109/cig.2014.6932876 · Cited by 171 · cited 1×
“A small body of research in the domain of gaming has examined the prediction of early dropout [51,52].”
Dropout from an eHealth intervention for adults with type 2 diabetes: a qualitative study
Lie, S. S., Karlsen, B., Oord, E. R., Graue, M., & Oftedal, B. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7479 · Cited by 136 · cited 3×
“…studies that have used a complex set of variables that are difficult to generalize and replicate (eg, specific intervention user journeys) or have relied on self-reporting, which requires extra resources to collect, has higher missing data rates, and has subjective measurement biases [12-20].”
Bricker, J. B., Bush, T., Zbikowski, S. M., Mercer, L. D., & Heffner, J. L. (2014) · Nicotine & Tobacco Research · doi:10.1093/ntr/ntu102 · Cited by 100
Self-citationBricker, J. B., Mull, K. E., McClure, J. B., Watson, N. L., & Heffner, J. L. (2018) · Addiction · doi:10.1111/add.14127 · Cited by 91
Self-citationPedersen, D. H., Mansourvar, M., Sortsø, C., & Schmidt, T. (2019) · Journal of Medical Internet Research · doi:10.2196/13617 · Cited by 75 · cited 4×
“As far as we are aware, this is one of the few studies to use log-in count data as a predictor of early dropout from digital health interventions [15,20].”
InfluentialChurn prediction of mobile and online casual games using play log data
Kim, S., Choi, D., Lee, E., & Rhee, W. (2017) · PLOS ONE · doi:10.1371/journal.pone.0180735 · Cited by 72 · cited 2×
“For example, in an analysis of casual gaming data, Kim et al [51] evaluated 5 different machine learning algorithms to predict game player churn (ie, dropout) in web-based and mobile games and found that, although prediction performance evaluated using AUC had little dependence on the choice of…”
Kannisto, K. A., Korhonen, J., Adams, C. E., Koivunen, M. H., Vahlberg, T., & Välimäki, M. A. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.6417 · Cited by 69
Kahler, C. W., LaChance, H. R., Strong, D. R., Ramsey, S. E., Monti, P. M., & Brown, R. A. (2007) · Addictive Behaviors · doi:10.1016/j.addbeh.2007.04.002 · Cited by 60 · cited 1×
“For iCanQuit and QuitGuide, confidence in being smoke free was a single-item self-report measure on a scale from 0 to 100; for WebQuit and Smokefree, confidence was measured using the Commitment to Quitting Smoking Scale ranging from 1 to 5 [36].”
The Cost-Effectiveness of Mobile Health (mHealth) Interventions for Older Adults: Systematic Review
Ghani, Z., Jarl, J., Sanmartin Berglund, J., Andersson, M., & Anderberg, P. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17155290 · Cited by 53 · cited 1×
“An important value of digital interventions for behavioral health is that they are accessible at a low cost, thereby increasing their potential public impact and reducing disparities in behavioral health care [7-9].”
Bricker, J. B., Mull, K. E., Santiago-Torres, M., Miao, Z., Perski, O., & Di, C. (2022) · Journal of Medical Internet Research · doi:10.2196/39208 · Cited by 44 · cited 4×
“Such differences in cessation rates are substantial, especially when considering that, across all 4 interventions, 49%-65% of all users were 1-week users and indeed were the largest log-in trajectory group [24,25].”
InfluentialSelf-citationAcceptance and commitment therapy (ACT) for smoking cessation: a synthesis
McCallion, E. A., & Zvolensky, M. J. (2015) · Current Opinion in Psychology · doi:10.1016/j.copsyc.2015.02.005 · Cited by 43 · cited 1×
“…differences between the 4 interventions were that (1) iCanQuit and WebQuit taught skills based on the principles of ACT [29] for smoking cessation [30-34], whereas QuitGuide and Smokefree taught skills based on the USCPG for smoking cessation [27], and (2) iCanQuit and QuitGuide were smartphone…”
Baseline Motivation Type as a Predictor of Dropout in a Healthy Eating Text Messaging Program
Coa, K. I., & Patrick, H. (2016) · JMIR mhealth and uhealth · doi:10.2196/mhealth.5992 · Cited by 39 · cited 1×
“Although early drop out has been variously defined across studies (eg, completing only 1 module or stopping within the first week or the first few weeks) [10,11], the fraction of users who fall into this category is quite large, with some studies showing it as high as 82% [12-14].”
von Brachel, R., Hötzel, K., Hirschfeld, G., Rieger, E., Schmidt, U., Kosfelder, J., Hechler, T., Schulte, D., & Vocks, S. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.3104 · Cited by 37
Vilardaga, R., Rizo, J., Ries, R. K., Kientz, J. A., Ziedonis, D. M., Hernandez, K., & McClernon, F. J. (2019) · Translational Behavioral Medicine · doi:10.1093/tbm/iby097 · Cited by 35 · cited 1×
“…differences between the 4 interventions were that (1) iCanQuit and WebQuit taught skills based on the principles of ACT [29] for smoking cessation [30-34], whereas QuitGuide and Smokefree taught skills based on the USCPG for smoking cessation [27], and (2) iCanQuit and QuitGuide were smartphone…”
Ramos, L. A., Blankers, M., van Wingen, G. A., de Bruijn, T., Pauws, S. C., & Goudriaan, A. E. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.734633 · Cited by 33 · cited 1×
“Statistical methods for predicting early dropout have ranged from standard logistic and proportional hazard regression models to decision trees, random forests, and other machine learning techniques [22].”
Ben‐Zeev, D., Razzano, L. A., Pashka, N. J., & Levin, C. (2021) · Psychiatric Services · doi:10.1176/appi.ps.202000349 · Cited by 32
Evans-Lacko, S., Hahn, J. S., Peter, L.-J., & Schomerus, G. (2022) · Current Opinion in Psychiatry · doi:10.1097/yco.0000000000000788 · Cited by 32 · cited 1×
“Digital interventions, including smartphone apps, websites, and SMS text messaging interventions, have proven efficacious for a wide variety of behavioral health outcomes, including mental health, diet, exercise, and smoking cessation [1-6].”
Bremer, V., Chow, P. I., Funk, B., Thorndike, F. P., & Ritterband, L. M. (2020) · doi:10.2196/preprints.17738 · Cited by 29 · cited 1×
“…objective measures of intervention engagement (eg, number of log-ins and proportion of content completed); and even sophisticated variables of user journeys, which encompass the myriad of sequences of interactions that a user takes to navigate through a digital intervention program [21].”
Leach, R., Carreiro, S., Shaffer, P. M., Gaba, A., & Smelson, D. A. (2022) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2021.794785 · Cited by 29
Linardon, J., Fuller‐Tyszkiewicz, M., Shatte, A. B. R., & Greenwood, C. J. (2022) · International Journal of Eating Disorders · doi:10.1002/eat.23733 · Cited by 26 · cited 2×
“…testing 7 machine learning models using 36 baseline variables found poor predictive performance of early dropout from an eating disorder digital intervention tested in 3 randomized trials (N=826), although adding multiple measures of intervention use from a subsample improved model prediction [15].”
Bricker, J. B., Sridharan, V., Zhu, Y., Mull, K. E., Heffner, J. L., Watson, N. L., McClure, J. B., & Di, C. (2018) · Journal of Medical Internet Research · doi:10.2196/10143 · Cited by 25 · cited 5×
“Recently, we published 2 studies identifying a type of early dropout from digital interventions that is highly predictive of treatment outcomes [24,25].”
InfluentialSelf-citationMak, Y. W., Leung, D. Y. P., & Loke, A. Y. (2020) · BMC Public Health · doi:10.1186/s12889-020-09820-0 · Cited by 24
Mak, Y. W., Loke, A. Y., & Leung, D. Y. P. (2021) · Journal of Clinical Medicine · doi:10.3390/jcm10194304 · Cited by 19
Predicting Attrition in a Text-Based Nutrition Education Program: Survival Analysis of Text2BHealthy
Grutzmacher, S. K., Munger, A., Speirs, K. E., Vafai, Y., Hilberg, E., Duru, E. B., Worthington, L., & Lachenmayr, L. (2018) · JMIR mhealth and uhealth · doi:10.2196/mhealth.9967 · Cited by 14 · cited 1×
“Other studies have similarly used a varying blend of baseline factors and use variables to calculate the probability of intervention attrition [23].”
Research works printed but not linked: 4
- 10.2196/17738 · doi:10.2196/17738 · Paper page
- Witten, IH, 2000, Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations
- Alpaydin, E, 2020, Introduction to Machine Learning. 4th edition
- Pedregosa, F, 2011, J Mach Learn Res, 12, 2825
Cited by
Zantvoort, K., Nacke, B., Görlich, D., Hornstein, S., Jacobi, C., & Funk, B. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01360-w · Cited by 116
“As pre-intervention questionnaire data has limited predictive power regarding dropout by itself 27,30,32,57 , it was used to inves-tigatealowpredictivepowersetting.”
Ardito, V., Golubev, G., Ciani, O., & Tarricone, R. (2023) · JMIR Cancer · doi:10.2196/42092 · Cited by 56
“Figure 1 describes the PRISMA flowchart [36,37].”
BackgroundWillingham, T. B., Stowell, J. F., Collier, G., & Backus, D. A. (2024) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph21010079 · Cited by 62
Krotter, A., Diego, G. A., Menéndez, A. G., González‐Roz, A., Secades‐Villa, R., & Pérez, Á. G. (2024) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2024.100773 · Cited by 37
Predicting early user churn in a public digital weight loss intervention
Jakob, R., Lepper, N., Fleisch, E., & Kowatsch, T. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642321 · Cited by 21
“As a baseline, we adopt a low-dimensional model informed by Bricker et al. (2023) that predicts churn based on the number of app logins per day of the first seven days of user interaction [7].”
BackgroundMethodResultInfluential
Metadata
- Authors
- Bricker, J., Miao, Z., Mull, K., Santiago-Torres, M., & Vock, D. M.
- Affiliations
- University of Washington (US)
- Fred Hutch Cancer Center (US)
- University of Minnesota (US)
- Corresponding author
- Bricker, Jonathan
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2023-01-20
- Volume
- 25
- Pages
- e43629
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC9898835
- OpenAlex
- W4313335732
- Cited by
- 22
- References
- 50
- Citations by year
- 2026: 3 · 2025: 8 · 2024: 10 · 2023: 1
- FWCI
- 2.94
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Cancer Institute (R01CA192849, R01CA247156, R01CA253975)
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Smoking Behavior and Cessation
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Keywords
- digital health interventions, smoking cessation apps, user retention, predictive modeling, decision tree, support vector machine, artificial neural networks, logistic regression, electronic cigarettes, baseline predictors, augmentation therapy, user engagement
- MeSH terms
- Humans, Randomized Controlled Trials as Topic, Smoking Cessation, Self Report, Mobile Applications, Electronic Nicotine Delivery Systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.