Development of a decision-making checklist tool to support technology selection in digital health research
Open accessThe DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
Digital technologies offer researchers new approaches to test personalized and adaptive health interventions tailored to an individual. Yet, research leveraging technologies to capture personal health data involve technical and ethical consideration during the study design phase. No guidance exists to facilitate responsible digital technology selection for research purposes. A stakeholder-engaged and iterative approach was used to develop, test, and refine a checklist designed to aid researchers in selecting technologies for their research. First, stakeholders (n = 7) discussed and informed key decision-making domains to guide app/device selection derived from the American Psychiatric Association’s framework that included safety, evidence, usability, and interoperability. We added “ethical principles” to the APA’s hierarchical model and created a checklist that was used by a small group of behavioral scientists (n = 7). Findings revealed the “ethical principles” domains of respect, beneficence, and justice cut across each decision-making domains and the checklist questions/prompts were revised accordingly and can be found at thecore.ucsd.edu. The refined checklist contains four decision-making domains with prompts/questions and ethical principles embedded within the domains of privacy, risk/benefit, data management, and access/evidence. This checklist is the first step in leading the narrative of decision-making when selecting digital health technologies for research. Given the dynamic and rapidly evolving nature of digital health technology use in research, this tool will need to be further evaluated for usefulness in technology selection.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In any case, such checklists are an expressed priority when ML practitioners are asked directly about their needs, and should therefore be co-designed with their close input [2,17,21,192].”
From Operationalising ethics in artificial intelligence for healthcare: a framework for AI developers (2023) · cites this paper as [21]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Many authors state that ndings should not be seen as a replacement for more in-depth assessments, such as those using NHS digital assessment criteria, but rather as an adjunct [5], [87], [88].”
From Security and Privacy of mHealth Applications: A Scoping Review (2020) · cites this paper as [87]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Most of the papers in this category 70% (7/10) presented new frameworks developed by the study authors, and the methods adopted in their creation varied: from reviews of related literature and pre-existing app evaluation frameworks [84], [86], [89] to more empirical methods such as engaging an expert panel of key stakeholders [87] and analysis of mHealth applications [85].”
From Security and Privacy of mHealth Applications: A Scoping Review (2020) · cites this paper as [87]
In the citing paper
“The ethical use of digital health tools remains an active area of investigation, with frameworks emerging but none yet consistently adopted55,56.”
From Selecting and describing control conditions in mobile health randomized controlled trials: a proposed typology (2023) · cites this paper as [55]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Huckvale, K., Prieto, J. T., Tilney, M., Benghozi, P.-J., & Car, J. (2015) · BMC Medicine · doi:10.1186/s12916-015-0444-y · Cited by 376
Torous, J. B., Chan, S. R., Gipson, S. Y.-M. T., Kim, J. W., Nguyen, T.-Q., Luo, J., & Wang, P. (2018) · Psychiatric Services · doi:10.1176/appi.ps.201700423 · Cited by 184 · cited 2×
“Individual medical societies such as the American Psychiatric Association (APA) released app-related guidelines for their clinician members [15] and new initiatives are exploring guidance development.”
BackgroundMethodSelf-citationMethods for Evaluating the Content, Usability, and Efficacy of Commercial Mobile Health Apps
Jake-Schoffman, D. E., Silfee, V. J., Waring, M. E., Boudreaux, E. D., Sadasivam, R. S., Mullen, S. P., Carey, J. L., Hayes, R. B., Ding, E. Y., Bennett, G. G., et al. (2017) · JMIR mHealth and uHealth · doi:10.2196/mhealth.8758 · Cited by 165
Privacy in the Digital World: Medical and Health Data Outside of HIPAA Protections
Glenn, T., & Monteith, S. (2014) · Current Psychiatry Reports · doi:10.1007/s11920-014-0494-4 · Cited by 145
Nebeker, C., Harlow, J., Espinoza Giacinto, R., Orozco-Linares, R., Bloss, C. S., & Weibel, N. (2017) · AJOB Empirical Bioethics · doi:10.1080/23294515.2017.1403980 · Cited by 104
Self-citationData Security and Privacy in Apps for Dementia: An Analysis of Existing Privacy Policies
Rosenfeld, L., Torous, J., & Vahia, I. V. (2017) · The American Journal of Geriatric Psychiatry · doi:10.1016/j.jagp.2017.04.009 · Cited by 100 · cited 1×
“A review of dementia apps found that less than half of those that collected personal health information offered any written description of how that data are used and shared, and in those that did, 42% of privacy policies were not relevant to the app or personal data collected [9].”
BackgroundSelf-citationPrivacy Policies of Android Diabetes Apps and Sharing of Health Information
Blenner, S. R., Köllmer, M., Rouse, A. J., Daneshvar, N., Williams, C., & Andrews, L. B. (2016) · JAMA · doi:10.1001/jama.2015.19426 · Cited by 97
Torous, J. B., & Nebeker, C. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.6793 · Cited by 88
Self-citationNebeker, C., Lagare, T., Takemoto, M., Lewars, B., Crist, K., Bloss, C. S., & Kerr, J. (2016) · Translational Behavioral Medicine · doi:10.1007/s13142-016-0426-4 · Cited by 85
Self-citationNebeker, C., Murray, K., Holub, C., Haughton, J., & Arredondo, E. M. (2017) · JMIR mHealth and uHealth · doi:10.2196/mhealth.6494 · Cited by 59
Self-citation
Research works printed but not linked: 4
- Nebeker, 2016, Exploring mobile imaging, sensing and tracking ethics (MIST-E) in behavioral science research: participant perspectives from the iWatch study, Transl Behav Med, 6, 57
- US Department of Health Human Services, 2016, Examining Oversight of the Privacy & Security of Health Data Collected by Entities not Regulated by HIPAA
- U.S. Department of Health and Human Services, 45 CFR 46, 1991, Subpart A––basic HHS policy for protection of human research subjects, Fed Reg, 56, 28012
- Saldaña, 2015, The Coding Manual for Qualitative Researchers
Cited by
Torous, J., Jän Myrick, K., Rauseo-Ricupero, N., & Firth, J. (2020) · JMIR Mental Health · doi:10.2196/18848 · Cited by 946
“A focus on equity and ethics will ensure digital health truly increases access to care [40].”
BackgroundSelf-citationAn overview of and recommendations for more accessible digital mental health services
Lattie, E. G., Stiles-Shields, C., & Graham, A. K. (2022) · Nature Reviews Psychology · doi:10.1038/s44159-021-00003-1 · Cited by 446
Digital tools against COVID-19: taxonomy, ethical challenges, and navigation aid
Gasser, U., Ienca, M., Scheibner, J., Sleigh, J., & Vayena, E. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30137-0 · Cited by 314
Transdiagnostic clinical staging in youth mental health: a first international consensus statement
Shah, J. L., Scott, J., McGorry, P. D., Cross, S. P. M., Keshavan, M. S., Nelson, B., Wood, S. J., Marwaha, S., Yung, A. R., Scott, E. M., et al. (2020) · World Psychiatry · doi:10.1002/wps.20745 · Cited by 291
Nebeker, C., Torous, J., & Bartlett Ellis, R. J. (2019) · BMC Medicine · doi:10.1186/s12916-019-1377-7 · Cited by 241
“The checklist was informed via developmental research involving a focus group discussion, and a design exercise with behavioral scientists [25].”
BackgroundMethodSelf-citation
Metadata
- Authors
- Nebeker, C., Bartlett Ellis, R. J., & Torous, J.
- Affiliations
- University of California San Diego (US)
- Indiana University School of Nursing, Indianapolis, IN, USA
- Beth Israel Deaconess Medical Center (US)
- Corresponding author
- Nebeker, Camille
- Venue
- Translational Behavioral Medicine
- Publisher
- Oxford University Press (OUP)
- Published
- 2019-05-23
- Volume
- 10
- Issue
- 4
- Pages
- 1004-1015
- ISSN
- 1613-9860 (ISSN-L)
- 1869-6716
- PMC
- PMC7543075
- OpenAlex
- W2945020779
- Cited by
- 99
- References
- 13
- Citations by year
- 2026: 10 · 2025: 10 · 2024: 18 · 2023: 12 · 2022: 16 · 2021: 19 · 2020: 11 · 2019: 3
- FWCI
- 16.69
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model (vor)
- Locations
- Translational Behavioral Medicine (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- IUScholarWorks (Indiana University) (submittedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- Funders
- Robert Wood Johnson Foundation (#72876)
- National Institute of Mental Health (1K23MH116130-01)
- Topics
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Ethics in Clinical Research
- Keywords
- research ethics, ethical principles, privacy, risk-benefit assessment, data management, interoperability, stakeholder engagement
- MeSH terms
- Humans, Research Design, Technology, Biomedical Technology, Checklist
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Development of a decision-making checklist tool to support technology selection in digital health research. https://app.cochat.ai/papers/10.1093/tbm/ibz074
- The paper
- Nebeker, C., Bartlett Ellis, R. J., & Torous, J. (2020). Development of a decision-making checklist tool to support technology selection in digital health research. Translational Behavioral Medicine. https://doi.org/10.1093/tbm/ibz074
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.