A design and evaluation framework for digital health interventions
Open accessThe DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
Abstract
Digital health interventions (DHIs) have the potential to help the growing number of chronic disease patients better manage their everyday lives. However, guidelines for the systematic development of DHIs are still scarce. The current work has, therefore, the objective to propose a framework for the design and evaluation of DHIs (DEDHI). The DEDHI framework is meant to support both researchers and practitioners alike from early conceptual DHI models to large-scale implementations of DHIs in the healthcare market.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Evidence based medicine: what it is and what it isn't
Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., Haynes, R. B., & Richardson, W. S. (1996) · BMJ · doi:10.1136/bmj.312.7023.71 · Cited by 15,769
Michie, S., van Stralen, M. M., & West, R. (2011) · Implementation Science · doi:10.1186/1748-5908-6-42 · Cited by 13,697
Framework for design and evaluation of complex interventions to improve health
Campbell, M. (2000) · BMJ · doi:10.1136/bmj.321.7262.694 · Cited by 3,474 · cited 4×
“of DHIs from MOST [19, 20, 21] for the Phases 1, 2 and 3 and from related work on intervention design and life cycle models [15, 16, 22] for Phase 4 as outlined in Section 2.”
MethodInfluentialMobile app rating scale: a new tool for assessing the quality of health mobile apps
Stoyanov, S. R., Hides, L., Kavanagh, D. J., Zelenko, O., Tjondronegoro, D., & Mani, M. (2015) · JMIR mHealth and uHealth · doi:10.2196/mhealth.3422 · Cited by 2,233 · cited 1×
“For example, first evaluation criteria have been proposed during the last decade [8, 9, 18, 27, 60, 80, 83].”
BackgroundNahum-Shani, I., Smith, S. N., Spring, B. J., Collins, L. M., Witkiewitz, K., Tewari, A., & Murphy, S. A. (2018) · Annals of Behavioral Medicine · doi:10.1007/s12160-016-9830-8 · Cited by 2,171 · cited 1×
“, just-in-time adaptive interventions [61, 62] and corresponding assessment methods such as micro-randomized trials [48] that heavily rely on the use of technology and, finally, (c) because it also focuses on behavioral health interventions at the individual-level which is relevant for chronic health problems [46, 53].”
BackgroundFord, I., & Norrie, J. (2016) · New England Journal of Medicine · doi:10.1056/nejmra1510059 · Cited by 1,405 · cited 1×
“study nurses that are experienced with clinical trials [32].”
BackgroundIntervention Mapping: A Process for Developing Theory and Evidence-Based Health Education Programs
Bartholomew, L. K., Parcel, G. S., & Kok, G. (1998) · Health Education & Behavior · doi:10.1177/109019819802500502 · Cited by 1,102 · cited 1×
“Intervention Mapping [6, 7] (1998) Health education interventions with a focus on theory as justificatory knowledge Increasing condom use among men with HIV [57]”
BackgroundCollins, L. M., Murphy, S. A., & Strecher, V. (2007) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2007.01.022 · Cited by 1,082
Development and validation of the user version of the Mobile application rating scale (uMARS)
Stoyanov, S. R., Hides, L., Kavanagh, D. J., & Wilson, H. (2016) · JMIR mhealth and uhealth · doi:10.2196/mhealth.5849 · Cited by 872 · cited 1×
“[45, 79, 81] 41, 12.4%”
InfluentialEvaluating Digital Health Interventions
Murray, E., Hekler, E. B., Andersson, G., Collins, L. M., Doherty, A., Hollis, C., Rivera, D. E., West, R., & Wyatt, J. C. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.06.008 · Cited by 847
Microrandomized trials: An experimental design for developing just-in-time adaptive interventions.
Klasnja, P., Hekler, E. B., Shiffman, S., Boruvka, A., Almirall, D., Tewari, A., & Murphy, S. A. (2015) · Health Psychology · doi:10.1037/hea0000305 · Cited by 671 · cited 1×
“, just-in-time adaptive interventions [61, 62] and corresponding assessment methods such as micro-randomized trials [48] that heavily rely on the use of technology and, finally, (c) because it also focuses on behavioral health interventions at the individual-level which is relevant for chronic health problems [46, 53].”
BackgroundA user-centered model for designing consumer mobile health (mHealth) applications (apps)
Schnall, R., Rojas, M., Bakken, S., Brown, W., Carballo-Dieguez, A., Carry, M., Gelaude, D., Mosley, J. P., & Travers, J. (2016) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2016.02.002 · Cited by 583
Nahum-Shani, I., Hekler, E. B., & Spruijt-Metz, D. (2015) · Health Psychology · doi:10.1037/hea0000306 · Cited by 568 · cited 1×
“, just-in-time adaptive interventions [61, 62] and corresponding assessment methods such as micro-randomized trials [48] that heavily rely on the use of technology and, finally, (c) because it also focuses on behavioral health interventions at the individual-level which is relevant for chronic health problems [46, 53].”
BackgroundSix steps in quality intervention development (6SQuID)
Wight, D., Wimbush, E., Jepson, R., & Doi, L. (2016) · Journal of Epidemiology and Community Health · doi:10.1136/jech-2015-205952 · Cited by 496 · cited 2×
“tions [87] and policies [24] at the population-level to behavioral health interventions [56] and DHIs [59] at the individual-level.”
BackgroundDeterminants of successful telemedicine implementations: a literature study
Broens, T. H. F., Huis in't Veld, R. M. H. A., Vollenbroek-Hutten, M. M. R., Hermens, H. J., van Halteren, A. T., & Nieuwenhuis, L. J. M. (2007) · Journal of Telemedicine and Telecare · doi:10.1258/135763307781644951 · Cited by 466 · cited 4×
“Due to the fact that MOST does not consider a phase after a DHI has been successfully evaluated in a randomized controlled trial, a corresponding implementation phase is added from both related design and evaluation frameworks from Table 1 [15, 16, 22] and a DHI life cycle model [13].”
MethodInfluentialTherapeutic alliance in videoconferencing psychotherapy: A review
Simpson, S. G., & Reid, C. L. (2014) · Australian Journal of Rural Health · doi:10.1111/ajr.12149 · Cited by 420 · cited 2×
“, special demands for group therapy [76]) 4, 3.”
BackgroundMummah, S. A., Robinson, T. N., King, A. C., Gardner, C. D., & Sutton, S. (2016) · Journal of Medical Internet Research · doi:10.2196/jmir.5927 · Cited by 388
Towards a consensus around standards for smartphone apps and digital mental health
Torous, J., Andersson, G., Bertagnoli, A., Christensen, H., Cuijpers, P., Firth, J., Haim, A., Hsin, H., Hollis, C., Lewis, S., et al. (2019) · World Psychiatry · doi:10.1002/wps.20592 · Cited by 325
Internet interventions: Past, present and future
Andersson, G. (2018) · Internet Interventions · doi:10.1016/j.invent.2018.03.008 · Cited by 315 · cited 1×
“Here, relevant articles were screened back to the year 2000, which can be determined as the start of systematic research on DHIs [3, 4].”
BackgroundA Review of Telehealth Service Implementation Frameworks
Van Dyk, L. (2014) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph110201279 · Cited by 300 · cited 1×
“els describe the phases that systems undergo while they evolve from a prototypical development to an operational product [84].”
BackgroundThe Multiphase Optimization Strategy for Engineering Effective Tobacco Use Interventions
Collins, L. M., Baker, T. B., Mermelstein, R. J., Piper, M. E., Jorenby, D. E., Smith, S. S., Christiansen, B. A., Schlam, T. R., Cook, J. W., & Fiore, M. C. (2011) · Annals of Behavioral Medicine · doi:10.1007/s12160-010-9253-x · Cited by 300
Brown, W., Yen, P.-Y., Rojas, M., & Schnall, R. (2013) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2013.08.001 · Cited by 299 · cited 1×
“[14] 11, 3.3%”
InfluentialCollins, L. M. (2018) · Statistics for Social and Behavioral Sciences · doi:10.1007/978-3-319-72206-1 · Cited by 257 · cited 12×
“Due to the fact that MOST does not consider a phase after a DHI has been successfully evaluated in a randomized controlled trial, a corresponding implementation phase is added from both related design and evaluation frameworks from Table 1 [15, 16, 22] and a DHI life cycle model [13].”
MethodInfluentialCriteria for assessing the quality of mHealth apps: a systematic review
Nouri, R., R Niakan Kalhori, S., Ghazisaeedi, M., Marchand, G., & Yasini, M. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy050 · Cited by 231
Baumel, A., Faber, K., Mathur, N., Kane, J. M., & Muench, F. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7270 · Cited by 206 · cited 1×
“For example, first evaluation criteria have been proposed during the last decade [8, 9, 18, 27, 60, 80, 83].”
BackgroundAssessing the quality and usability of smartphone apps for pain self-management
Reynoldson, C., Stones, C., Allsop, M., Gardner, P., Bennett, M. I., Closs, S. J., Jones, R., & Knapp, P. (2014) · Pain Medicine · doi:10.1111/pme.12327 · Cited by 178 · cited 2×
“Funding: This work was co-funded by Health Promotion Switzerland, and the European Social Fund and the Free State of Saxony (Grant no. 100310385).”
BackgroundTelemedicine across borders: A systematic review of factors that hinder or support implementation
Saliba, V., Legido-Quigley, H., Hallik, R., Aaviksoo, A., Car, J., & McKee, M. (2012) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2012.08.003 · Cited by 173 · cited 7×
“, demographics, health status [37], resistance to change [52, 71]) and the healthcare provider (e.”
BackgroundInfluentialTowards a Framework for Evaluating Mobile Mental Health Apps
Chan, S., Torous, J., Hinton, L., & Yellowlees, P. (2015) · Telemedicine and e-Health · doi:10.1089/tmj.2015.0002 · Cited by 163 · cited 3×
“[17, 49, 74, 88] 41, 12.4%”
InfluentialHage, E., Roo, J. P., van Offenbeek, M. A., & Boonstra, A. (2013) · BMC Health Services Research · doi:10.1186/1472-6963-13-19 · Cited by 137 · cited 9×
“, lack of broad scope [37, 43], technology isolated from care processes [30]) 6, 5.”
BackgroundInfluentialA privacy framework for mobile health and home-care systems
Kotz, D., Avancha, S., & Baxi, A. (2009) · Proceedings of the first ACM workshop on Security and privacy in medical and home-care systems · doi:10.1145/1655084.1655086 · Cited by 127
Kowatsch, T., Nißen, M. K., Shih, C.-H. I., Rüegger, D., Volland, D., Filler, A., Künzler, F., Barata, F., Hung, S., Büchter, D., et al. (2017) · ETH Zurich · doi:10.3929/ethz-b-000218776 · Cited by 74 · cited 1×
“The goal of the search strategywas to update and complement prior findings [63] due to the broader focus of the current work on DHIs which includes not only mobile health interventions but alsoweb-based interventions and hybrid interventions in which also guidance by human health professionals are foreseen [51].”
BackgroundSelf-citationBobrow, K., Farmer, A., Cishe, N., Nwagi, N., Namane, M., Brennan, T. P., Springer, D., Tarassenko, L., & Levitt, N. (2018) · BMC Health Services Research · doi:10.1186/s12913-017-2808-9 · Cited by 46 · cited 1×
“Medical Research Council (MRC) Framework [15, 22] (2000) Complex intervention for public health practice with a focus on evaluation methods Improving blood pressure control via SMS [11]”
MethodTelehealth—An Effective Delivery Method for Diabetes Self-Management Education?
Fitzner, K., & Moss, G. (2013) · Population Health Management · doi:10.1089/pop.2012.0054 · Cited by 45 · cited 5×
“, lack of broad scope [37, 43], technology isolated from care processes [30]) 6, 5.”
BackgroundInfluentialmHealth Quality: A Process to Seal the Qualified Mobile Health Apps
Mobin, Y., Jérôme, B., Pierre, D., Lucas, P., & Guillaume, M. (2016) · Studies in Health Technology and Informatics · doi:10.3233/978-1-61499-678-1-205 · Cited by 27
Christopoulou, S., Kotsilieris, T., & Anagnostopoulos, I. (2018) · Healthcare · doi:10.3390/healthcare6030109 · Cited by 20
Cited by
Perceptions and opinions of patients about mental health chatbots: scoping review
Abd-Alrazaq, A. A., Alajlani, M., Ali, N., Denecke, K., Bewick, B. M., & Househ, M. (2021) · Journal of Medical Internet Research · doi:10.2196/17828 · Cited by 301
“The first attempts toward evaluation frameworks for digital health interventions [82] and health chatbots [83,84] have been recently published.”
Whitehead, L., Talevski, J., Fatehi, F., & Beauchamp, A. (2023) · Journal of Medical Internet Research · doi:10.2196/42719 · Cited by 261
Hauser-Ulrich, S., Künzli, H., Meier-Peterhans, D., & Kowatsch, T. (2020) · JMIR mHealth and uHealth · doi:10.2196/15806 · Cited by 210
“A recent study [93] showed that conversational agent-based interventions target neurological disorders [94-96], addictions [97,98], mental-physical wellness [99,100], nutritional metabolic disorders [101-103], and sexually transmitted disease [104].”
BackgroundSelf-citationTechnical metrics used to evaluate health care chatbots: scoping review
Abd-Alrazaq, A., Safi, Z., Alajlani, M., Warren, J., Househ, M., & Denecke, K. (2020) · Journal of Medical Internet Research · doi:10.2196/18301 · Cited by 150
Jahnel, T., Pan, C.-C., Pedros Barnils, N., Muellmann, S., Freye, M., Dassow, H.-H., Lange, O., Reinschluessel, A. V., Rogowski, W., & Gerhardus, A. (2024) · Journal of Medical Internet Research · doi:10.2196/54269 · Cited by 15
“Other frameworks focus on evaluation and assessment criteria along the life cycle of digital health interventions yet still lacking a public health focus [22].”
Metadata
- Authors
- Kowatsch, T., Otto, L., Harperink, S., Cotti, A., & Schlieter, H.
- Venue
- it - Information Technology
- Publisher
- Walter de Gruyter GmbH
- Published
- 2019-10-01
- Volume
- 61
- Issue
- 5-6
- Pages
- 253-263
- ISSN
- 1611-2776 (ISSN-L)
- 2196-7032
- OpenAlex
- W2990180535
- Cited by
- 116
- References
- 35
- FWCI
- 15.05
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.