Student or educator? Sign up with your school email and get 6 months of Pro free.

The DOI matches its Crossref record; 84 of the 84 references it cites were found in a registry or an index.

Indicators of retention in remote digital health studies: a cross-study evaluation of 100,000 participants

Open access

Abstract

Digital technologies such as smartphones are transforming the way scientists conduct biomedical research. Several remotely conducted studies have recruited thousands of participants over a span of a few months allowing researchers to collect real-world data at scale and at a fraction of the cost of traditional research. Unfortunately, remote studies have been hampered by substantial participant attrition, calling into question the representativeness of the collected data including generalizability of outcomes. We report the findings regarding recruitment and retention from eight remote digital health studies conducted between 2014–2019 that provided individual-level study-app usage data from more than 100,000 participants completing nearly 3.5 million remote health evaluations over cumulative participation of 850,000 days. Median participant retention across eight studies varied widely from 2–26 days (median across all studies = 5.5 days). Survival analysis revealed several factors significantly associated with increase in participant retention time, including (i) referral by a clinician to the study (increase of 40 days in median retention time); (ii) compensation for participation (increase of 22 days, 1 study); (iii) having the clinical condition of interest in the study (increase of 7 days compared with controls); and (iv) older age (increase of 4 days). Additionally, four distinct patterns of daily app usage behavior were identified by unsupervised clustering, which were also associated with participant demographics. Most studies were not able to recruit a sample that was representative of the race/ethnicity or geographical diversity of the US. Together these findings can help inform recruitment and retention strategies to enable equitable participation of populations in future digital health research.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41746-020-0224-8

Cites

84 of 84 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Geographical and race/ethnic diversity of the recruited participants. a Map of US showing the proportion (median across the studies) of recruited participants relative to state’s population proportion of the US and b Race/Ethnicity proportion of recruited participants compared to 2010 census data. The median value across the studies is shown by the black point with error bars indicating the interquartile range.
Fig. 1 Geographical and race/ethnic diversity of the recruited participants. a Map of US showing the proportion (median across the studies) of recruited participants relative to state’s population proportion of the US and b Race/Ethnicity proportion of recruited participants compared to 2010 census data. The median value across the studies is shown by the black point with error bars indicating the interquartile range.CC BY · PubMed Central
Fig. 2 Factors impacting participant retention in digital health studies. a Proportion of active tasks (N = 3.3 million) completed by participants based on their local time of day. The centerline of the boxplot shows the median value across the studies and upper and lower whisker corresponding to outlier point (1.5 times the interquartile range). b Kaplan Meir survival curve showing significant differences (P < 1e-16) in user retention across the apps. Brighten App where monetary incentives were given to participants showed the longest retention time(Median = 26 days, 95% CI = 17–33) followed by Asthma(Median = 12 days, 95% CI = 11–13), MyHeartCounts(Median = 9 days, 95% CI = 9–9), ElevateMS(Median = 7 days, 95% CI = 5–10), mPower(Median = 5 days, 95% CI = 4–5), Phendo(Median = 4 days, 95% CI = 3–4), Start(Median = 2 days, 95% CI = 2–2) and SleepHealth(Median = 2 days, 95% CI = 2–2), c Lift curve showing the change in median survival time (with 95% CI indicated by error bars) based on the minimum number of days(1–32) a subset of participants continued to use the study apps, Kaplan-Meier survival curve showing significant differences in user retention across d Age group, with 60 years and older using the apps for longest duration(Median = 7days, 95% CI = 6–8, P < 1e-16) followed by 50–59 years (Median = 4 days, 95% CI = 4–5) and 17–49 years (Median = 2–3 days, 95% CI = 2–3). e Disease status; participants reporting having a disease stayed active longer(N50 = 13days, 95% CI = 13–14) compared to people without disease(N50 = 6 days, 95% CI = 5–6) and finally f Clinical referral; Two studies (mPower and ElevateMS), had a subpopulation, that were referred to the study by clinicians and showed significantly (P < 1e-16) longer app usage period(Median = 44 days, 95% CI = 27–58) compared to self-referred participants with disease (N50 = 4 days, 95% CI = 4–4). For all survival curves the shaded region shows the 95% confidence limits based on the survival model fit.
Fig. 2 Factors impacting participant retention in digital health studies. a Proportion of active tasks (N = 3.3 million) completed by participants based on their local time of day. The centerline of the boxplot shows the median value across the studies and upper and lower whisker corresponding to outlier point (1.5 times the interquartile range). b Kaplan Meir survival curve showing significant differences (P < 1e-16) in user retention across the apps. Brighten App where monetary incentives were given to participants showed the longest retention time(Median = 26 days, 95% CI = 17–33) followed by Asthma(Median = 12 days, 95% CI = 11–13), MyHeartCounts(Median = 9 days, 95% CI = 9–9), ElevateMS(Median = 7 days, 95% CI = 5–10), mPower(Median = 5 days, 95% CI = 4–5), Phendo(Median = 4 days, 95% CI = 3–4), Start(Median = 2 days, 95% CI = 2–2) and SleepHealth(Median = 2 days, 95% CI = 2–2), c Lift curve showing the change in median survival time (with 95% CI indicated by error bars) based on the minimum number of days(1–32) a subset of participants continued to use the study apps, Kaplan-Meier survival curve showing significant differences in user retention across d Age group, with 60 years and older using the apps for longest duration(Median = 7days, 95% CI = 6–8, P < 1e-16) followed by 50–59 years (Median = 4 days, 95% CI = 4–5) and 17–49 years (Median = 2–3 days, 95% CI = 2–3). e Disease status; participants reporting having a disease stayed active longer(N50 = 13days, 95% CI = 13–14) compared to people without disease(N50 = 6 days, 95% CI = 5–6) and finally f Clinical referral; Two studies (mPower and ElevateMS), had a subpopulation, that were referred to the study by clinicians and showed significantly (P < 1e-16) longer app usage period(Median = 44 days, 95% CI = 27–58) compared to self-referred participants with disease (N50 = 4 days, 95% CI = 4–4). For all survival curves the shaded region shows the 95% confidence limits based on the survival model fit.CC BY · PubMed Central
Fig. 3 Daily participant engagement patterns in digital health studies. a Schematic representation of an individual’s in-app activity for the first 84 days. The participant app usage time is determined based on the number of days between the first and last day they perform an active task(indicated by the green box) in the app. Days active in the study is the total number of days a participant performs at least one active task (indicated by the number of green boxes). b Heatmaps showing participants in-app activity across the apps for the first 12 weeks (84 days), grouped into four broad clusters using unsupervised k-means clustering. The optimum number of clusters was determined by minimizing the within-cluster variation across different cluster sizes between 1–10. Seven out of eight studies indicated four clusters to be an optimum number using the elbow method. The heatmaps are arranged by the highest (C1) to the lowest user-engagement cluster (C4).
Fig. 3 Daily participant engagement patterns in digital health studies. a Schematic representation of an individual’s in-app activity for the first 84 days. The participant app usage time is determined based on the number of days between the first and last day they perform an active task(indicated by the green box) in the app. Days active in the study is the total number of days a participant performs at least one active task (indicated by the number of green boxes). b Heatmaps showing participants in-app activity across the apps for the first 12 weeks (84 days), grouped into four broad clusters using unsupervised k-means clustering. The optimum number of clusters was determined by minimizing the within-cluster variation across different cluster sizes between 1–10. Seven out of eight studies indicated four clusters to be an optimum number using the elbow method. The heatmaps are arranged by the highest (C1) to the lowest user-engagement cluster (C4).CC BY · PubMed Central
Fig. 4 Comparison of characteristics across participant engagement clusters. a Proportion of participants in each cluster across the study apps, b Participants total app usage duration(between 1–84 days) and the number of days participants completed tasks in the study apps, c Significant differences [F(4,163) = 18.5, P = 1.4e-12] in the age groups of participants across five clusters and d Significant differences [F(2,81) = 28.5, P = 4.1e-10] in proportion of minority population present in the five clusters. C5* cluster contains the participants that used the apps for less than a week and were removed from the clustering; however, they were added back for accurate proportional comparison of participants in each cluster. The centerline of the boxplots shows the median value across the studies for each cluster and upper and lower whisker corresponds to the outlier point that is at least 1.5 times the interquartile range.
Fig. 4 Comparison of characteristics across participant engagement clusters. a Proportion of participants in each cluster across the study apps, b Participants total app usage duration(between 1–84 days) and the number of days participants completed tasks in the study apps, c Significant differences [F(4,163) = 18.5, P = 1.4e-12] in the age groups of participants across five clusters and d Significant differences [F(2,81) = 28.5, P = 4.1e-10] in proportion of minority population present in the five clusters. C5* cluster contains the participants that used the apps for less than a week and were removed from the clustering; however, they were added back for accurate proportional comparison of participants in each cluster. The centerline of the boxplots shows the median value across the studies for each cluster and upper and lower whisker corresponds to the outlier point that is at least 1.5 times the interquartile range.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 84 references

Research works printed but not linked: 2

  1. Virtual Clinical Trials Challenges and Opportunities: Proceedings of a Workshop: Health and Medicine Division. (2019). http://www.nationalacademies.org/hmd/Reports/2019/virtual-clinical-trials-challenges-and-opportunities-pw.aspx.
  2. National Academies of Sciences, Engineering, and Medicine, Health and Medicine Division, Board on Health Sciences Policy & Committee on the Return of Individual-Specific Research Results Generated in Research Laboratories. Returning Individual Research Results to Participants: Guidance for a New Research Paradigm. (National Academies Press, 2018).

Cited by

See all 411

Metadata

Authors
Pratap, A., Neto, E. C., Snyder, P., Stepnowsky, C., Elhadad, N., Grant, D., Mohebbi, M. H., Mooney, S., Suver, C., Wilbanks, J., Mangravite, L., Heagerty, P. J., Areán, P., & Omberg, L.
Affiliations
  • Sage Bionetworks (US)
  • University of California San Diego (US)
  • Columbia University (US)
  • University of Washington (US)
Corresponding author
Pratap, Abhishek
Venue
npj Digital Medicine
Published
2020-02-17
Volume
3
Issue
1
Pages
21-21
ISSN
2398-6352 (ISSN-L)
OpenAlex
W2978342502
Cited by
410
References
84
Citations by year
2026: 65 · 2025: 71 · 2024: 75 · 2023: 56 · 2022: 81 · 2021: 38 · 2020: 24
FWCI
51.78
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University)
  • Synapse
Funders
  • U.S. Department of Health & Human Services | NIH | National Institute of Mental Health (MH100466)
  • Robert Wood Johnson Foundation (73205)
  • American Sleep Apnea Foundation, Washington, DC
  • U.S. Department of Health & Human Services | NIH | National Center for Advancing Translational Sciences (1UL1TR002319-01)
Topics
  • Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Ethics in Clinical Research
  • Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
Keywords
participant retention, participant attrition, smartphone app usage, participant recruitment, clinician referral, app usage patterns, demographic representativeness, health equity, real-world data

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Indicators of retention in remote digital health studies: a cross-study evaluation of 100,000 participants. https://app.cochat.ai/papers/10.1038/s41746-020-0224-8
The paper
Pratap, A., Neto, E. C., Snyder, P., Stepnowsky, C., Elhadad, N., Grant, D., Mohebbi, M. H., Mooney, S., Suver, C., Wilbanks, J., Mangravite, L., Heagerty, P. J., Areán, P., & Omberg, L. (2020). Indicators of retention in remote digital health studies: a cross-study evaluation of 100,000 participants. npj Digital Medicine. https://doi.org/10.1038/s41746-020-0224-8

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.