CoChat

Can a Single Variable Predict Early Dropout From Digital Health Interventions? Comparison of Predictive Models From Two Large Randomized Trials

Open access
Bricker, Jonathan, Miao, Zhen, Mull, Kristin, Santiago-Torres, Margarita, Vock, David M · 2023 · Journal of Medical Internet Research

The DOI matches its Crossref record; 49 of the 50 references it cites were found in a registry or an index.

Abstract

Background: A single generalizable metric that accurately predicts early dropout from digital health interventions has the potential to readily inform intervention targets and treatment augmentations that could boost retention and intervention outcomes. We recently identified a type of early dropout from digital health interventions for smoking cessation, specifically, users who logged in during the first week of the intervention and had little to no activity thereafter. These users also had a substantially lower smoking cessation rate with our iCanQuit smoking cessation app compared with users who used the app for longer periods. Objective: This study aimed to explore whether log-in count data, using standard statistical methods, can precisely predict whether an individual will become an iCanQuit early dropout while validating the approach using other statistical methods and randomized trial data from 3 other digital interventions for smoking cessation (combined randomized N=4529). Methods: Standard logistic regression models were used to predict early dropouts for individuals receiving the iCanQuit smoking cessation intervention app, the National Cancer Institute QuitGuide smoking cessation intervention app, the WebQuit.org smoking cessation intervention website, and the Smokefree.gov smoking cessation intervention website. The main predictors were the number of times a participant logged in per day during the first 7 days following randomization. The area under the curve (AUC) assessed the performance of the logistic regression models, which were compared with decision trees, support vector machine, and neural network models. We also examined whether 13 baseline variables that included a variety of demographics (eg, race and ethnicity, gender, and age) and smoking characteristics (eg, use of e-cigarettes and confidence in being smoke free) might improve this prediction. Results: The AUC for each logistic regression model using only the first 7 days of log-in count variables was 0.94 (95% CI 0.90-0.97) for iCanQuit, 0.88 (95% CI 0.83-0.93) for QuitGuide, 0.85 (95% CI 0.80-0.88) for WebQuit.org, and 0.60 (95% CI 0.54-0.66) for Smokefree.gov. Replacing logistic regression models with more complex decision trees, support vector machines, or neural network models did not significantly increase the AUC, nor did including additional baseline variables as predictors. The sensitivity and specificity were generally good, and they were excellent for iCanQuit (ie, 0.91 and 0.85, respectively, at the 0.5 classification threshold). Conclusions: Logistic regression models using only the first 7 days of log-in count data were generally good at predicting early dropouts. These models performed well when using simple, automated, and readily available log-in count data, whereas including self-reported baseline variables did not improve the prediction. The results will inform the early identification of people at risk of early dropout from digital health interventions with the goal of intervening further by providing them with augmented treatments to increase their retention and, ultimately, their intervention outcomes.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.2196/43629

Cites

49 of 49 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 49 references

Research works printed but not linked: 4

  1. 10.2196/17738 · doi:10.2196/17738 · Paper page
  2. Witten, IH, 2000, Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations
  3. Alpaydin, E, 2020, Introduction to Machine Learning. 4th edition
  4. Pedregosa, F, 2011, J Mach Learn Res, 12, 2825

Cited by

Metadata

Authors
Bricker, J., Miao, Z., Mull, K., Santiago-Torres, M., & Vock, D. M.
Affiliations
  • University of Washington (US)
  • Fred Hutch Cancer Center (US)
  • University of Minnesota (US)
Corresponding author
Bricker, Jonathan
Venue
Journal of Medical Internet Research
Published
2023-01-20
Volume
25
Pages
e43629
ISSN
1438-8871 (ISSN-L)
OpenAlex
W4313335732
Cited by
22
References
50
Citations by year
2026: 3 · 2025: 8 · 2024: 10 · 2023: 1
FWCI
2.94
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Journal of Medical Internet Research (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
National Cancer Institute (R01CA192849, R01CA247156, R01CA253975)
Topics
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Smoking Behavior and Cessation
  • Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
Keywords
digital health interventions, smoking cessation apps, user retention, predictive modeling, decision tree, support vector machine, artificial neural networks, logistic regression, electronic cigarettes, baseline predictors, augmentation therapy, user engagement
MeSH terms
Humans, Randomized Controlled Trials as Topic, Smoking Cessation, Self Report, Mobile Applications, Electronic Nicotine Delivery Systems

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.