The DOI matches its Crossref record; 84 of the 84 references it cites were found in a registry or an index.
Indicators of retention in remote digital health studies: a cross-study evaluation of 100,000 participants
Open accessAbstract
Digital technologies such as smartphones are transforming the way scientists conduct biomedical research. Several remotely conducted studies have recruited thousands of participants over a span of a few months allowing researchers to collect real-world data at scale and at a fraction of the cost of traditional research. Unfortunately, remote studies have been hampered by substantial participant attrition, calling into question the representativeness of the collected data including generalizability of outcomes. We report the findings regarding recruitment and retention from eight remote digital health studies conducted between 2014–2019 that provided individual-level study-app usage data from more than 100,000 participants completing nearly 3.5 million remote health evaluations over cumulative participation of 850,000 days. Median participant retention across eight studies varied widely from 2–26 days (median across all studies = 5.5 days). Survival analysis revealed several factors significantly associated with increase in participant retention time, including (i) referral by a clinician to the study (increase of 40 days in median retention time); (ii) compensation for participation (increase of 22 days, 1 study); (iii) having the clinical condition of interest in the study (increase of 7 days compared with controls); and (iv) older age (increase of 4 days). Additionally, four distinct patterns of daily app usage behavior were identified by unsupervised clustering, which were also associated with participant demographics. Most studies were not able to recruit a sample that was representative of the race/ethnicity or geographical diversity of the US. Together these findings can help inform recruitment and retention strategies to enable equitable participation of populations in future digital health research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures



![Fig. 4 Comparison of characteristics across participant engagement clusters. a Proportion of participants in each cluster across the study apps, b Participants total app usage duration(between 1–84 days) and the number of days participants completed tasks in the study apps, c Significant differences [F(4,163) = 18.5, P = 1.4e-12] in the age groups of participants across five clusters and d Significant differences [F(2,81) = 28.5, P = 4.1e-10] in proportion of minority population present in the five clusters. C5* cluster contains the participants that used the apps for less than a week and were removed from the clustering; however, they were added back for accurate proportional comparison of participants in each cluster. The centerline of the boxplots shows the median value across the studies for each cluster and upper and lower whisker corresponds to the outlier point that is at least 1.5 times the interquartile range.](/paper-figures/ca0709c4d0ea43e5a0ef64eb0e7c07d0/indicators-of-retention-in-remote-digital-health-studies-a-cross-study-figure-4.jpg)
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This level of attrition is commensurate with other digital mental health solution trial attrition rates [47,57].”
From A Therapeutic Relational Agent for Reducing Problematic Substance Use (Woebot): Development and Usability Study (2021) · cites this paper as [57]
Experimental — accuracy not yet measuredIn the citing paper
“This is especially important given the rapid dropout of user engagement for some forms of mobile phone interventions (e.g., health apps generally and mental health apps specifically [56, 57]).”
From Mobile phone-based interventions for mental health: A systematic meta-review of 14 meta-analyses of randomized controlled trials (2022) · cites this paper as [57]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rogers, E. M., Singhal, A., & Quinlan, M. M. (2019) · An Integrated Approach to Communication Theory and Research · doi:10.4324/9780203710753-35 · Cited by 37,322
Miller, R. G. (1981) · Cited by 3,336
Johnson, C. O., Nguyen, M., Roth, G. A., Nichols, E., Alam, T., Abate, D., Abd-Allah, F., Abdelalim, A., Abraha, H. N., Abu-Rmeileh, N. M., et al. (2019) · The Lancet Neurology · doi:10.1016/s1474-4422(19)30034-1 · Cited by 3,264
Information bias in health research: definition, pitfalls, and adjustment methods
Althubaiti, A. (2016) · Journal of Multidisciplinary Healthcare · doi:10.2147/jmdh.s104807 · Cited by 3,018 · cited 1×
“Rather than retrospectively asking people to recall their health over the past week or month, researchers using mobile technologies can assess participants frequently including outside clinic and at important points in time without having to rely on recall that is known to have bias.(29) While these recent studies sho…”
Eysenbach, G. (2005) · Journal of Medical Internet Research · doi:10.2196/jmir.7.1.e11 · Cited by 2,524
The “All of Us” Research Program
The All of Us Research Program Investigators (2019) · New England Journal of Medicine · doi:10.1056/nejmsr1809937 · Cited by 2,440
Nahum-Shani, I., Smith, S. N., Spring, B. J., Collins, L. M., Witkiewitz, K., Tewari, A., & Murphy, S. A. (2018) · Annals of Behavioral Medicine · doi:10.1007/s12160-016-9830-8 · Cited by 2,171
Yardley, L., Morrison, L., Bradbury, K., & Muller, I. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4055 · Cited by 1,442
Fogel, D. B. (2018) · Contemporary Clinical Trials Communications · doi:10.1016/j.conctc.2018.08.001 · Cited by 1,035
An Integrated Approach to Communication Theory and Research
(2014) · Routledge eBooks · doi:10.4324/9780203887011 · Cited by 981
Research works printed but not linked: 2
- Virtual Clinical Trials Challenges and Opportunities: Proceedings of a Workshop: Health and Medicine Division. (2019). http://www.nationalacademies.org/hmd/Reports/2019/virtual-clinical-trials-challenges-and-opportunities-pw.aspx.
- National Academies of Sciences, Engineering, and Medicine, Health and Medicine Division, Board on Health Sciences Policy & Committee on the Return of Individual-Specific Research Results Generated in Research Laboratories. Returning Individual Research Results to Participants: Guidance for a New Research Paradigm. (National Academies Press, 2018).
Cited by
Torous, J., Bucci, S., Bell, I. H., Kessing, L. V., Faurholt‐Jepsen, M., Whelan, P., Carvalho, A. F., Keshavan, M., Linardon, J., & Firth, J. (2021) · World Psychiatry · doi:10.1002/wps.20883 · Cited by 1,199
Wearable sensor data and self-reported symptoms for COVID-19 detection
Quer, G., Radin, J. M., Gadaleta, M., Baca-Motes, K., Ariniello, L., Ramos, E., Kheterpal, V., Topol, E. J., & Steinhubl, S. R. (2021) · Nature Medicine · doi:10.1038/s41591-020-1123-x · Cited by 447
“This has proven to be extremely challenging, with a recent analysis of eight different digital research programs involving 100,000 participants having a median duration of retention of only 5.5 days 29 .”
InfluentialLubitz, S. A., Faranesh, A. Z., Selvaggi, C., Atlas, S. J., McManus, D. D., Singer, D. E., Pagoto, S., McConnell, M. V., Pantelopoulos, A., & Foulkes, A. S. (2022) · Circulation · doi:10.1161/circulationaha.122.060291 · Cited by 355
Challenges in Participant Engagement and Retention Using Mobile Health Apps: Literature Review
Amagai, S., Pila, S., Kaat, A. J., Nowinski, C. J., & Gershon, R. C. (2022) · Journal of Medical Internet Research · doi:10.2196/35120 · Cited by 343
“In their cross-study evaluation of 100,000 participants, Pratap et al [10] analyzed individual-level study app use data from 8 studies that accumulated nearly 3.”
InfluentialProchaska, J. J., Vogel, E. A., Chieng, A., Kendra, M., Baiocchi, M., Pajarito, S., & Robinson, A. (2021) · Journal of Medical Internet Research · doi:10.2196/24850 · Cited by 257
“commensurate with other digital mental health solution trial attrition rates [47,57].”
Metadata
- Authors
- Pratap, A., Neto, E. C., Snyder, P., Stepnowsky, C., Elhadad, N., Grant, D., Mohebbi, M. H., Mooney, S., Suver, C., Wilbanks, J., Mangravite, L., Heagerty, P. J., Areán, P., & Omberg, L.
- Affiliations
- Sage Bionetworks (US)
- University of California San Diego (US)
- Columbia University (US)
- University of Washington (US)
- Corresponding author
- Pratap, Abhishek
- ORCID
- Venue
- npj Digital Medicine
- Published
- 2020-02-17
- Volume
- 3
- Issue
- 1
- Pages
- 21-21
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC7026051
- OpenAlex
- W2978342502
- Cited by
- 410
- References
- 84
- Citations by year
- 2026: 65 · 2025: 71 · 2024: 75 · 2023: 56 · 2022: 81 · 2021: 38 · 2020: 24
- FWCI
- 51.78
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- npj Digital Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Synapse
- Funders
- U.S. Department of Health & Human Services | NIH | National Institute of Mental Health (MH100466)
- Robert Wood Johnson Foundation (73205)
- American Sleep Apnea Foundation, Washington, DC
- U.S. Department of Health & Human Services | NIH | National Center for Advancing Translational Sciences (1UL1TR002319-01)
- Topics
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Ethics in Clinical Research
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Keywords
- participant retention, participant attrition, smartphone app usage, participant recruitment, clinician referral, app usage patterns, demographic representativeness, health equity, real-world data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Indicators of retention in remote digital health studies: a cross-study evaluation of 100,000 participants. https://app.cochat.ai/papers/10.1038/s41746-020-0224-8
- The paper
- Pratap, A., Neto, E. C., Snyder, P., Stepnowsky, C., Elhadad, N., Grant, D., Mohebbi, M. H., Mooney, S., Suver, C., Wilbanks, J., Mangravite, L., Heagerty, P. J., Areán, P., & Omberg, L. (2020). Indicators of retention in remote digital health studies: a cross-study evaluation of 100,000 participants. npj Digital Medicine. https://doi.org/10.1038/s41746-020-0224-8
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.