The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Developing and Evaluating Digital Interventions to Promote Behavior Change in Health and Health Care: Recommendations Resulting From an International Workshop
Open accessAbstract
Devices and programs using digital technology to foster or support behavior change (digital interventions) are increasingly ubiquitous, being adopted for use in patient diagnosis and treatment, self-management of chronic diseases, and in primary prevention. They have been heralded as potentially revolutionizing the ways in which individuals can monitor and improve their health behaviors and health care by improving outcomes, reducing costs, and improving the patient experience. However, we are still mainly in the age of promise rather than delivery. Developing and evaluating these digital interventions presents new challenges and new versions of old challenges that require use of improved and perhaps entirely new methods for research and evaluation. This article discusses these challenges and provides recommendations aimed at accelerating the rate of progress in digital behavior intervention research and practice. Areas addressed include intervention development in a rapidly changing technological landscape, promoting user engagement, advancing the underpinning science and theory, evaluating effectiveness and cost-effectiveness, and addressing issues of regulatory, ethical, and information governance. This article is the result of a two-day international workshop on how to create, evaluate, and implement effective digital interventions in relation to health behaviors. It was held in London in September 2015 and was supported by the United Kingdom's Medical Research Council (MRC), the National Institute for Health Research (NIHR), the Methodology Research Programme (PI Susan Michie), and the Robert Wood Johnson Foundation of the United States (PI Kevin Patrick). Important recommendations to manage the rapid pace of change include considering using emerging techniques from data science, machine learning, and Bayesian approaches and learning from other disciplines including computer science and engineering. With regard to assessing and promoting engagement, a key conclusion was that sustained engagement is not always required and that for each intervention it is useful to establish what constitutes "effective engagement," that is, sufficient engagement to achieve the intended outcomes. The potential of digital interventions for testing and advancing theories of behavior change by generating ecologically valid, real-time objective data was recognized. Evaluations should include all phases of the development cycle, designed for generalizability, and consider new experimental designs to make the best use of rich data streams. Future health economics analyses need to recognize and model the complex and potentially far-reaching costs and benefits of digital interventions. In terms of governance, developers of digital behavior interventions should comply with existing regulatory frameworks, but with consideration for emerging standards around information governance, ethics, and interoperability.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To date, low uptake and poor engagement are commonly observed with digital interventions, which are often insufficient to sustain behavior change [15,16].”
From Influences on the Uptake of and Engagement With Health and Well-Being Smartphone Apps: Systematic Review (2020) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Within the realm of digital interventions, using suitable strategies, such as user-centered participatory design, can heighten user engagement, thereby rendering intervention measures more appealing and efficacious [26].”
From Assessing the Effectiveness of Digital Health Behavior Strategies on Type 2 Diabetes Management: Systematic Review and Network Meta-Analysis (2025) · cites this paper as [26]
In the citing paper
“Behavioural theory can guide the design of mHealth interventions aimed at changing health behaviours.60”
From mHealth text and voice communication for monitoring people with chronic diseases in low-resource settings: a realist review (2018) · cites this paper as [60]
In the citing paper
“An important reason for the lack of justifications for the intended use of eHealth technologies might be that there is a lack of knowledge regarding the working mechanisms of technology-based applications [8].”
From Clarifying the Concept of Adherence to eHealth Technology: Systematic Review on When Usage Becomes Adherence (2017) · cites this paper as [8]
In the citing paper
“the results of a recent literature review of theoretical perspectives on adherence showed that adherence is a multidimensional concept, influenced by a range of technological, environmental, and individual factors altogether that cannot be evaluated by technology usage alone [8,84].”
From Clarifying the Concept of Adherence to eHealth Technology: Systematic Review on When Usage Becomes Adherence (2017) · cites this paper as [8]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Eysenbach, G., & CONSORT-EHEALTH Group (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1923 · Cited by 1,863 · cited 1×
“The advice provided here is specific to DBCIs, but it should also build on the good practice of reporting general mHealth interventions published elsewhere to ensure adequate description of the technical components of the intervention [44,45].”
BackgroundYardley, L., Morrison, L., Bradbury, K., & Muller, I. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4055 · Cited by 1,442 · cited 2×
“These qualitative approaches are central to participatory user-centered design, which is the key to developing and evaluating DBCIs in order to ensure that they are engaging and effective [16,17].”
BackgroundSelf-citationA Holistic Framework to Improve the Uptake and Impact of eHealth Technologies
van Gemert-Pijnen, J. E., Nijland, N., van Limburg, M., Ossebaard, H. C., Kelders, S. M., Eysenbach, G., & Seydel, E. R. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1672 · Cited by 1,126 · cited 2×
“These qualitative approaches are central to participatory user-centered design, which is the key to developing and evaluating DBCIs in order to ensure that they are engaging and effective [16,17].”
BackgroundUnderstanding and Promoting Effective Engagement With Digital Behavior Change Interventions
Yardley, L., Spring, B. J., Riper, H., Morrison, L. G., Crane, D. H., Curtis, K., Merchant, G. C., Naughton, F., & Blandford, A. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.06.015 · Cited by 1,118 · cited 3×
“The workshop led to a series of publications [3-7].”
BackgroundSelf-citationBuilding Ontologies with Basic Formal Ontology
Arp, R., Smith, B., & Spear, A. D. (2015) · The MIT Press eBooks · doi:10.7551/mitpress/9780262527811.001.0001 · Cited by 1,083 · cited 1×
“Such knowledge-organizing structures are called “ontologies” [22,23].”
The impact of guidance on Internet-based mental health interventions — A systematic review
Baumeister, H., Reichler, L., Munzinger, M., & Lin, J. (2014) · Internet Interventions · doi:10.1016/j.invent.2014.08.003 · Cited by 1,037 · cited 1×
“Adding human support is also known to promote engagement with many interventions [18].”
BackgroundEvaluating Digital Health Interventions
Murray, E., Hekler, E. B., Andersson, G., Collins, L. M., Doherty, A., Hollis, C., Rivera, D. E., West, R., & Wyatt, J. C. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.06.008 · Cited by 847 · cited 4×
“The workshop led to a series of publications [3-7].”
BackgroundInfluentialSelf-citationRamsey, S. D., Willke, R. J., Glick, H., Reed, S. D., Augustovski, F., Jonsson, B., Briggs, A., & Sullivan, S. D. (2015) · Value in Health · doi:10.1016/j.jval.2015.02.001 · Cited by 811 · cited 1×
“However, existing economic approaches (eg, ISPOR guidance [29]) will need to be adapted to take into account the way digital health interventions are delivered, covering the whole life cycle of the intervention.”
BackgroundMichie, S., Wood, C. E., Johnston, M., Abraham, C., Francis, J. J., & Hardeman, W. (2015) · Health Technology Assessment · doi:10.3310/hta19990 · Cited by 680 · cited 1×
“This includes designing DBCIs on a modular basis with a clear understanding of which behavior change techniques [43] constitute that module and how they link to the theoretically postulated mechanisms of action.”
BackgroundSelf-citationAgarwal, S., LeFevre, A. E., Lee, J., L’Engle, K., Mehl, G., Sinha, C., & Labrique, A. (2016) · BMJ · doi:10.1136/bmj.i1174 · Cited by 639
Flagged first, then by citations
Research works printed but not linked: 8
- West, R, 2016, A Guide to the Development and Evaluation of Digital Behaviour Change Intervention
- Michie, S, 2014, ABC of Behaviour Change Theories: An Essential Resource for Researchers, Policy Makers and Practitioners
- Christmas, S, 2015, Thinking about behaviour change: an interdisciplinary dialogue
- 2016, Optimizing the Nation's Investment in Academic Research A New Regulatory Framework for the 21st Century
- 2014, Guidance: Medical device stand-alone software including apps
- 2014, Personalised Health and Care 2020: Using data and technology to transform outcomes for patients and citizens
- 1998, Data Protection Act 1998
- 2016, Regulation (EU) of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (General Data Protection Regulation)
Cited by
Sorted by citations
Taxonomy of approaches to developing interventions to improve health: a systematic methods overview
O’Cathain, A., Croot, L., Sworn, K., Duncan, E., Rousseau, N. S., Turner, K. M., Yardley, L., & Hoddinott, P. (2019) · Pilot and Feasibility Studies · doi:10.1186/s40814-019-0425-6 · Cited by 625
“future rather than current guidance [23].”
Self-citationSkivington, K., Matthews, L., Simpson, S. A., Craig, P., Baird, J., Blazeby, J. M., Boyd, K., Craig, N. J., French, D., McIntosh, E., et al. (2021) · Health Technology Assessment · doi:10.3310/hta25570 · Cited by 585
Liverpool, S., Mota, C. P., Sales, C. M. D., Čuš, A., Carletto, S., Hancheva, C., Sousa, S., Cerón, S. C., Moreno-Peral, P., Pietrabissa, G., et al. (2020) · Journal of Medical Internet Research · doi:10.2196/16317 · Cited by 408
“There is much debate in the literature on appropriate methods for evaluating the impact of DHIs [31].”
Thieme, A., Belgrave, D., & Doherty, G. (2020) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3398069 · Cited by 406
Short, C. E., DeSmet, A., Woods, C., Williams, S. L., Maher, C., Middelweerd, A., Müller, A. M., Wark, P. A., Vandelanotte, C., Poppe, L., et al. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.9397 · Cited by 400
“Although there is evidence that eHealth and mHealth behavior change interventions can be effective, low levels of adherence and high levels of attrition have been commonly reported [1-3].”
BackgroundSzinay, D., Jones, A., Chadborn, T., Brown, J., & Naughton, F. (2020) · Journal of Medical Internet Research · doi:10.2196/17572 · Cited by 289
“To date, low uptake and poor engagement are commonly observed with digital interventions, which are often insufficient to sustain behavior change [15,16].”
BackgroundSieverink, F., Kelders, S. M., & van Gemert-Pijnen, J. E. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.8578 · Cited by 331
“For every technology, a proportion of the users will not use the intervention at all, will stop using the technology after a period, or will not use the available elements of the technology as intended [1,6-8].”
Background(2019) · doi:10.2196/13166 · Cited by 222
“The implementation cost of DHIs is usually substantial, and HTA is, therefore, essential to inform the decision makers on the potential impact of the DHIs on both clinical and health economic outcomes [14].”
Camden, C., & Silva, M. (2020) · Physical & Occupational Therapy In Pediatrics · doi:10.1080/01942638.2020.1825032 · Cited by 237
“It is however important to recognize that when used as stand-alone strategy, most apps fail to sustain long-term engagement (Michie et al., 2017); outcomes improved when online solutions are supported by person-person contact/coaching (Ingersoll et al., 2016).”
BackgroundStockwell, S., Schofield, P., Fisher, A., Firth, J., Jackson, S. E., Stubbs, B., & Smith, L. (2019) · Experimental Gerontology · doi:10.1016/j.exger.2019.02.020 · Cited by 212
“, 2007), reducing costs and improving patient experience and outcomes (Michie et al., 2017).”
Systematic review and meta-analysis of the effectiveness of chatbots on lifestyle behaviours
Singh, B., Olds, T., Brinsley, J., Dumuid, D., Virgara, R., Matricciani, L., Watson, A., Szeto, K., Eglitis, E., Miatke, A., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00856-1 · Cited by 224
Digital Interventions on Healthy Lifestyle Management: Systematic Review
Chatterjee, A., Prinz, A., Gerdes, M. W., & Martínez, S. (2021) · Journal of Medical Internet Research · doi:10.2196/26931 · Cited by 229
“• Included studies with the Scale for the Assessment of Narrative Review Articles scale 2 (107 articles) [1,32-137]”
BackgroundSlater, H., Campbell, J. M., Stinson, J. N., Burley, M. M., & Briggs, A. M. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.8888 · Cited by 155
“Beyond implementation, ongoing evaluation and monitoring of mobile and other digital health interventions is deemed critical to inform health policy and practice [80,81].”
Müller, A. M., Maher, C. A., Vandelanotte, C., Hingle, M., Middelweerd, A., Lopez, M. L., DeSmet, A., Short, C. E., Nathan, N., Hutchesson, M. J., et al. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.8954 · Cited by 180
“In addition, an international workshop addressed how eHealth and mHealth interventions should incorporate psychological theory and behavior change techniques in their design [42].”
Musiat, P., Johnson, C., Atkinson, M., Wilksch, S., & Wade, T. (2022) · Psychological Medicine · doi:10.1017/s0033291721004621 · Cited by 166
“Depending on the intervention and disorder, the completion of homework, monitoring records, or behavioural experiments between online sessions are likely key to the therapeutic efficacy (Michie, Yardley, West, Patrick, & Greaves, 2017).”
BackgroundLanzing, M. (2018) · Philosophy & Technology · doi:10.1007/s13347-018-0316-4 · Cited by 184
“The potential of behavioral change through self-tracking lies in highly personalized online choice architectures enabled by smart algorithms that learn from and adapt to the behavior of the user (Michie et al. 2017).”
BackgroundThe use of eHealth to promote physical activity in cancer survivors: a systematic review
Haberlin, C., O’Dwyer, T., Mockler, D., Moran, J., O’Donnell, D. M., & Broderick, J. M. (2018) · Supportive Care in Cancer · doi:10.1007/s00520-018-4305-z · Cited by 139
Keleko, A. T., Kamsu-Foguem, B., Ngouna, R. H., & Tongne, A. (2022) · AI and Ethics · doi:10.1007/s43681-021-00132-6 · Cited by 171
Sherifali, D., Nerenberg, K. A., Wilson, S., Semeniuk, K., Ali, M. U., Redman, L. M., & Adamo, K. B. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.8006 · Cited by 122
“Applying this principle of patient engagement to the development of new eHealth apps is beneficial and necessary [44,45].”
BackgroundKarekla, M., Kasinopoulos, O., Neto, D. D., Ebert, D. D., Van Daele, T., Nordgreen, T., Höfer, S., Oeverland, S., & Jensen, K. L. (2019) · European Psychologist · doi:10.1027/1016-9040/a000349 · Cited by 117
Pham, Q., Graham, G., Carrion, C., Morita, P. P., Seto, E., Stinson, J. N., & Cafazzo, J. A. (2019) · JMIR mHealth and uHealth · doi:10.2196/11941 · Cited by 131
“Recently, scholars sought to further the conceptualization of engagement by proposing that it may be more valuable to identify the mechanisms that underlie effective engagement, defined as sufficient engagement with an intervention to achieve intended outcomes [14,15].”
BackgroundMethodCarraça, E. V., Encantado, J., Battista, F., Beaulieu, K., Blundell, J. E., Busetto, L., van Baak, M. A., Dicker, D., Ermolao, A., Farpour‐Lambert, N. J., et al. (2021) · Obesity Reviews · doi:10.1111/obr.13258 · Cited by 154
“reach at low cost and can be adapted to individual needs and the information can be delivered in an engaging and interactive way, with greater fidelity to intervention content.(10,15) Although digital interventions are promising for health behavior change, research on their”
BackgroundZarnowiecki, D. M., Mauch, C. E., Middleton, G., Matwiejczyk, L., Watson, W. L., Dibbs, J., Dessaix, A., & Golley, R. K. (2020) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-020-0915-1 · Cited by 128
“Evaluation research is also needed to determine a sufficient level of engagement to achieve behavioural change, as engagement appears to be the factor constraining the effectiveness of digital nutrition promotion interventions [83].”
BackgroundEvaluating the Impact of Physical Activity Apps and Wearables: Interdisciplinary Review
McCallum, C., Rooksby, J., & Gray, C. M. (2018) · JMIR mhealth and uhealth · doi:10.2196/mhealth.9054 · Cited by 126
“It is nevertheless surprising that few studies used single-case designs and new factorial approaches, as it has been suggested that mHealth technologies can support the data collection procedures and experimental setup these research designs require (ie, frequent measurement and several experimental conditions) [25,26…”
BackgroundInfluentialA Ledger of Me: Personalizing Healthcare Using Blockchain Technology
Leeming, G., Cunningham, J. A., & Ainsworth, J. D. (2019) · Frontiers in Medicine · doi:10.3389/fmed.2019.00171 · Cited by 126
Kouri, A., Yamada, J., Cheung, J. L. S., de Velde, S. V., & Gupta, S. (2022) · Implementation Science · doi:10.1186/s13012-022-01199-3 · Cited by 135
“…qualitative CDSS research that has emphasized the importance of implementation context and attaining good “clinician-patient-system integration” when developing CDSSs [39], and highlight the importance of further integrating implementation science principles into CDSS intervention design [40].”
BackgroundSchwartz, S. M., Wildenhaus, K. J., Bucher, A., & Byrd, B. (2020) · Frontiers in Computer Science · doi:10.3389/fcomp.2020.00031 · Cited by 118
“…and their time-series data are paramount in this framework, the experience is best designed around treatment phases (Pertschuk et al., 1978; Hayes et al., 1999; Dallery et al., 2013; Pham et al., 2016; Michie et al., 2017 ) and changes in those phases based on a new (novel) treatment circumstance.”
BackgroundIyamu, I., Gómez-Ramírez, O., Xu, A. X., Chang, H.-J., Watt, S., Mckee, G., & Gilbert, M. (2022) · DIGITAL HEALTH · doi:10.1177/20552076221102255 · Cited by 118
“35,48,77,82,83,86,90,91 Further, interventions were frequently implemented on a small scale and their associated evaluations were described as poorly generalizable.”
BackgroundInfluentialDomin, A., Spruijt‐Metz, D., Theisen, D., Ouzzahra, Y., & Vögele, C. (2021) · JMIR mhealth and uhealth · doi:10.2196/24308 · Cited by 126
“This creates a problem, as software is meant to change, progress, evolve, and adapt to its user in short periods [32,51,99,108,110-114].”
BackgroundMethodInfluentialYeager, C. M., & Benight, C. C. (2018) · mHealth · doi:10.21037/mhealth.2018.08.04 · Cited by 120
“DHIs are defined as products or services that use computer technology to promote behavior change and have also been referred to as digital behavior change interventions (17).”
BackgroundBoulley, G. E., Leroy, T., Bernetière, C., Paquienséguy, F., Desfriches‐Doria, O., & Préau, M. (2018) · Psycho-Oncology · doi:10.1002/pon.4867 · Cited by 112
“Digital health interventions are becoming increasingly ubiquitous in primary prevention and detection, diagnosis, treatment, and in the management of associated side effects.(8) They can be delivered through digital tools (D‐tools) including computers, websites, mobile”
BackgroundGoldberg, S. B., Imhoff-Smith, T., Bolt, D. M., Wilson-Mendenhall, C. D., Dahl, C. J., Davidson, R. J., & Rosenkranz, M. A. (2020) · JMIR Mental Health · doi:10.2196/23825 · Cited by 97
“Similar to attrition, low engagement is a rule rather than an exception for mHealth [120].”
BackgroundChien, I., Enrique, A., Palacios, J., Regan, T., Keegan, D., Carter, D., Tschiatschek, S., Nori, A., Thieme, A., Richards, D., et al. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.10791 · Cited by 111
Digital interventions to improve adherence to maintenance medication in asthma
Chan, A., De Simoni, A., Wileman, V., Holliday, L., Newby, C. J., Chisari, C., Ali, S., Zhu, N., Padakanti, P., Pinprachanan, V., et al. (2022) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013030.pub2 · Cited by 105
Mclaughlin, M., Delaney, T., Hall, A., Byaruhanga, J., Mackie, P., Grady, A., Reilly, K., Campbell, E., Sutherland, R., Wiggers, J., et al. (2021) · Journal of Medical Internet Research · doi:10.2196/23180 · Cited by 105
“This further highlights the importance of defining the types of engagement outcomes as well as using multiple indicators of engagement when trying to understand the relationship between engagement variables and the effectiveness of DHIs [58,59].”
BackgroundVoorheis, P., Zhao, A., Kuluski, K., Pham, Q., Scott, T., Sztur, P., Khanna, N., Ibrahim, M., & Petch, J. (2022) · JMIR mHealth and uHealth · doi:10.2196/35799 · Cited by 94
“This conclusion has been reiterated across the literature, with experts agreeing that methodological guidance is required to design effectively engaging DBCIs [23].”
BackgroundTaufik, D., Verain, M. C. D., Bouwman, E. P., & Reinders, M. J. (2019) · Trends in Food Science & Technology · doi:10.1016/j.tifs.2019.09.019 · Cited by 115
Nurmi, J., Knittle, K. P., Ginchev, T., Khattak, F., Helf, C., Zwickl, P., Castellano‐Tejedor, C., Lusilla, P., Costa‐Requena, J., Ravaja, N., et al. (2019) · JMIR mhealth and uhealth · doi:10.2196/12884 · Cited by 110
“involvement that is necessary for the intervention to achieve intended outcomes [27,31].”
BackgroundGoldberg, S. B., Imhoff-Smith, T., Bolt, D. M., Wilson-Mendenhall, C. D., Dahl, C. J., Davidson, R. J., & Rosenkranz, M. A. (2020) · doi:10.2196/preprints.23825 · Cited by 97
Asbjørnsen, R. A., Wentzel, J., Smedsrød, M. L., Hjelmesæth, J., Clark, M. M., Solberg Nes, L., & Van Gemert-Pijnen, J. E. W. C. (2020) · Journal of Medical Internet Research · doi:10.2196/22598 · Cited by 84
“To create meaningful and effective digital behavior change interventions, it is essential to identify the end user values and needs and the related PSD principles and BCTs during the requirement specification process [23,32,36,39,61].”
BackgroundSystematic review of context-aware digital behavior change interventions to improve health
Craig, K. J. T., Morgan, L. C., Chen, C.-H., Michie, S., Fusco, N., Snowdon, J. L., Scheufele, E. L., Gagliardi, T., & Sill, S. (2020) · Translational Behavioral Medicine · doi:10.1093/tbm/ibaa099 · Cited by 108
“Digital behavior change interventions (DBCIs) employ digital health technologies for behavior modification for the maintenance and improvement of health [9].”
BackgroundSelf-citationAl-Dhahir, I., Reijnders, T., Faber, J. S., van den Berg-Emons, R. J. G., Janssen, V. R., Kraaijenhagen, R. A., Visch, V., Chavannes, N. H., & Evers, A. W. M. (2022) · Journal of Medical Internet Research · doi:10.2196/34229 · Cited by 91
“These results seem to be consistent with other research that found that behavioral theories were seldom applied in interventions [88,97,98].”
InfluentialWhat is an appropriate level of evidence for a digital health intervention?
Greaves, F., Joshi, I., Campbell, M., Roberts, S., Patel, N., & powell, J. A. (2018) · The Lancet · doi:10.1016/s0140-6736(18)33129-5 · Cited by 96
Self-citationBeintner, I., Vollert, B., Zarski, A.-C., Bolinski, F., Musiat, P., Görlich, D., Ebert, D. D., & Jacobi, C. (2019) · Journal of Medical Internet Research · doi:10.2196/14181 · Cited by 89
“Furthermore, it is important to identify differential usage patterns in multicomponent interventions and user characteristics that are associated with these patterns [13-15].”
Korpershoek, Y. J. G., Hermsen, S., Schoonhoven, L., Schuurmans, M. J., & Trappenburg, J. C. A. (2020) · Journal of Medical Internet Research · doi:10.2196/15449 · Cited by 78
“Within these designs, it is important to evaluate self-management skills and behavior change as outcomes, and the way this is assessed should be clearly reported [19,77,78].”
BackgroundShen, H., van der Kleij, R., van der Boog, P. J. M., Chang, X., & Chavannes, N. H. (2019) · Journal of Medical Internet Research · doi:10.2196/12384 · Cited by 93
Wang, T., Wang, W., Liang, J., Nuo, M., Wen, Q., Wei, W., Han, H., & Lei, J. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00692-9 · Cited by 84
Tailored web-based interventions for pain: Systematic review and meta-analysis
Martorella, G., Boitor, M., Berube, M., Fredericks, S., Le May, S., & Gélinas, C. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.8826 · Cited by 88
“50) on pain intensity did not use content matching but rather a hybrid format involving telephone contact with a therapist [53,54], carefully choosing behavioral change techniques that fit the targeted behavior could enhance the effects of these interventions [12,68].”
BackgroundRemote symptom monitoring integrated into electronic health records: A systematic review
Gandrup, J., Ali, S. M., McBeth, J., van der Veer, S. N., & Dixon, W. G. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa177 · Cited by 90
“40”
Lewkowitz, A. K., Whelan, A., Ayala, N. K., Hardi, A., Stoll, C., Battle, C. L., Tuuli, M. G., Ranney, M. L., & Miller, E. S. (2023) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2023.06.028 · Cited by 98
Guo, S. H.-M., Hsing, H.-C., Lin, J., & Lee, C. (2020) · JMIR mhealth and uhealth · doi:10.2196/18404 · Cited by 90
“Thus, some researchers have made efforts to increase the evidence base for mobile eHealth design and development [16,17].”
BackgroundSin, J., Galeazzi, G. M., McGregor, E., Collom, J., Taylor, A. F., Barrett, B. M., Lawrence, V. C., & Henderson, C. (2020) · Journal of Medical Internet Research · doi:10.2196/20581 · Cited by 83
“To examine the (cost-) effectiveness of the identified interventions, we included only empirical studies using a web-based RCT design for optimal external and internal validity [26] and with intervention recipients’ outcomes reported using validated quantitati ve measures.”
BackgroundMethodGuo, Y., Xu, Z.-M., Qiao, J.-Y., Hong, Y. A., Zhang, H.-X., Zeng, C.-B., Cai, W.-P., Li, L.-H., & Liu, C.-C. (2018) · JMIR mHealth and uHealth · doi:10.2196/10274 · Cited by 64
“Finally, the efficacy trial must have a sufficient sample size and multiple follow-ups for the observation of intervention effects [29].”
BackgroundElavsky, S., Knapová, L., Klocek, A., & Šmahel, D. (2018) · Journal of Aging and Physical Activity · doi:10.1123/japa.2017-0410 · Cited by 87
Digital Therapeutics in Parkinson’s Disease: Practical Applications and Future Potential
Ellis, T. D., & Earhart, G. M. (2021) · Journal of Parkinson s Disease · doi:10.3233/jpd-202407 · Cited by 78
“episodic, goal setting; provision of real-time, rather than delayed, feedback to reinforce positive behavioral change; more frequent adjustments of individually tailored programs; and ongoing connections with healthcare professionals through messaging features [12, 13].”
BackgroundEnsuring electronic medical record simulation through better training, modeling, and evaluation
Zhang, Z., Yan, C., Mesa, D. A., Sun, J., & Malin, B. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz161 · Cited by 83
Methods for Human-Centered eHealth Development: Narrative Scoping Review
Kip, H., Keizer, J., da Silva, M. C., Beerlage-de Jong, N., Köhle, N., & Kelders, S. M. (2022) · Journal of Medical Internet Research · doi:10.2196/31858 · Cited by 67
“For that purpose, a user-centered, iterative, and multi-method development process in which all stakeholders are actively involved is recommended [7-10].”
Perski, O., BLANDFORD, A. E., Garnett, C. V., Crane, D., West, R., & Michie, S. (2019) · Translational Behavioral Medicine · doi:10.1093/tbm/ibz039 · Cited by 70
Self-citationArgent, R., Slevin, P., Bevilacqua, A., Neligan, M. C., Daly, A., & Caulfield, B. (2019) · Sensors · doi:10.3390/s19020432 · Cited by 75
“To encourage engagement, it is important that the design of biofeedback systems adopts a user-centred iterative process [19], where developers consult end-users to evaluate the system, identify their usability criteria, and understand the perceived benefits and challenges of its implementation in the real-world [20].”
BackgroundKlonoff, D. C. (2019) · Journal of Diabetes Science and Technology · doi:10.1177/1932296818820303 · Cited by 73
ACSM Call to Action Statement: COVID-19 Considerations for Sports and Physical Activity
Denay, K. L., Breslow, R. G., Turner, M. N., Nieman, D. C., Roberts, W. O., & Best, T. M. (2020) · Current Sports Medicine Reports · doi:10.1249/jsr.0000000000000739 · Cited by 74
“In the related fields of behavior change and mental health, increasing use of digital platforms has improved health care delivery and patient engagement (18,19).”
BackgroundPartridge, S. R., & Redfern, J. (2018) · Healthcare · doi:10.3390/healthcare6030070 · Cited by 70
“Digital behavior change interventions, defined as “a product or service that uses computer technology to promote behavior change” [20], use various technologies for delivery such as websites, social media, text messages, smartphones apps or wearable devices [20–22].”
BackgroundBell, I., Rossell, S. L., Farhall, J., Hayward, M., Lim, M. H., Fielding-Smith, S. F., & Thomas, N. (2019) · Schizophrenia Research · doi:10.1016/j.schres.2019.10.026 · Cited by 70
What Makes Artificial Intelligence Exceptional in Health Technology Assessment?
Bélisle‐Pipon, J., Couture, V., Roy, M., Ganache, I., Goetghebeur, M. M., & Cohen, I. G. (2021) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2021.736697 · Cited by 82
“According to Michie et al. (2017), AIHTs presents “ new challenges and new versions of old challenges ” which require new evaluative methods and legislative motivation to address health AIHT-speci fi c ethical and regulatory issues.”
BackgroundInfluentialBerman, M. A., Guthrie, N. L., Edwards, K. L., Appelbaum, K. J., Njike, V. Y., Eisenberg, D. M., & Katz, D. L. (2018) · JMIR Diabetes · doi:10.2196/diabetes.9591 · Cited by 71
“It was then redeveloped as a native app for Android and iOS using human-centered software design principles [28] and subject to basic usability testing prior to the start of this study.”
MethodMiller, S., Ainsworth, B., Yardley, L., Milton, A., Weal, M. J., Smith, P. W. F., & Morrison, L. G. (2018) · Journal of Medical Internet Research · doi:10.2196/10966 · Cited by 66
“It is also our hope that the framework will be applied as part of a mixed-methods approach, triangulating usage analyses with insights and experiences collected qualitatively [6,8,10].”
Self-citationvan Aubel, E., Bakker, J. M., Batink, T., Michielse, S., Goossens, L., Lange, I., Schruers, K., Lieverse, R., Marcelis, M., van Amelsvoort, T., et al. (2020) · Behaviour Research and Therapy · doi:10.1016/j.brat.2020.103592 · Cited by 57
Optimizing Digital Integrated Care via Micro‐Randomized Trials
Walton, A., Nahum‐Shani, I., Crosby, L., Klasnja, P., & Murphy, S. (2018) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt.1079 · Cited by 66
Tighe, S. A., Ball, K., Kensing, F., Kayser, L., Rawstorn, J. C., & Maddison, R. (2020) · Journal of Medical Internet Research · doi:10.2196/16774 · Cited by 73
“In summary, the adaptive nature of a digital platform accommodates the requirement for flexibility in self-management support, which could facilitate a diverse range of users and life circumstances [66,132].”
Boulton, E., Hawley-Hague, H., French, D., Mellone, S., Zacchi, A., Clemson, L. M., Vereijken, B., & Todd, C. J. (2019) · Progress in Cardiovascular Diseases · doi:10.1016/j.pcad.2019.01.003 · Cited by 64
“The importance of a person-centred and iterative approach,which is refined tomeet end-user requirements, has been recommended by an international expert workshop for developing digital behaviour change interventions.(50) The feedback received from end users, consortium members and the Advisory Board in our pilot studi…”
BackgroundBrewer, L. C., Hayes, S. N., Caron, A. R., Derby, D. A., Breutzman, N. S., Wicks, A., Raman, J., Smith, C. M., Schaepe, K., Sheets, R. E., et al. (2019) · PLoS ONE · doi:10.1371/journal.pone.0218724 · Cited by 75
Methods for Human-Centered eHealth Development: Narrative Scoping Review (Preprint)
Kip, H., Keizer, J., da Silva, M. C., Beerlage-de Jong, N., Köhle, N., & Kelders, S. M. (2021) · doi:10.2196/preprints.31858 · Cited by 67
Distinguishing features in the assessment of mHealth apps
Tarricone, R., Petracca, F., Ciani, O., & Cucciniello, M. (2021) · Expert Review of Pharmacoeconomics & Outcomes Research · doi:10.1080/14737167.2021.1891883 · Cited by 62
Chikersal, P., Belgrave, D. C. M., Doherty, G. J., Enrique, Á., Palacios, J. E., Richards, D., & Thieme, A. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376341 · Cited by 68
Simpson, S. A., Matthews, L., Pugmire, J., McConnachie, A., McIntosh, E., Coulman, E., Hughes, K., Kelson, M. J., Morgan‐Trimmer, S., Murphy, S., et al. (2020) · Public Health Research · doi:10.3310/phr08030 · Cited by 64
mHealth Improved Fruit and Vegetable Accessibility and Intake in Young Children
Bakırcı-Taylor, A. L., Reed, D. B., McCool, B., & Dawson, J. (2019) · Journal of Nutrition Education and Behavior · doi:10.1016/j.jneb.2018.11.008 · Cited by 62
Digital Health Solutions for Indigenous Mental Well-Being
Hensel, J. M., Ellard, K., Koltek, M., Wilson, G., & Sareen, J. (2019) · Current Psychiatry Reports · doi:10.1007/s11920-019-1056-6 · Cited by 61
Hutchesson, M. J., Gough, C., Müller, A. M., Short, C. E., Whatnall, M. C., Ahmed, M., Pearson, N., Yin, Z., Ashton, L. M., Maher, C., et al. (2021) · Obesity Reviews · doi:10.1111/obr.13295 · Cited by 60
Damen, I., Brombacher, H., Lallemand, C., Brankaert, R., Brombacher, A., van Wesemael, P. J. V., & Vos, S. B. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17020499 · Cited by 59
“This is despite the fact that digital technologies have the potential to “revolutionize the way individuals can monitor and improve their health” [20].”
BackgroundSocial media interventions for precision public health: promises and risks
Dunn, A. G., Mandl, K. D., & Coiera, E. W. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0054-0 · Cited by 62
Schwarz, A. F., Huertas‐Delgado, F. J., Cardon, G., & DeSmet, A. (2020) · Games for Health Journal · doi:10.1089/g4h.2019.0058 · Cited by 61
“123 Furthermore, it is important to mention that not every person is equally engaged in the same game, 124 therefore, customization may be further applied to study effective engagement 18 for individual players.”
Chen, A. T., Wu, S., Tomasino, K. N., Lattie, E. G., & Mohr, D. C. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103187 · Cited by 58
“For example, the dimension, “Mechanisms”, does not provide a nuanced differentiation of mechanisms of change, which has been recognized as a need in the study of engagement of digital interventions[59].”
BackgroundWainer, A., Arnold, Z. E., Leonczyk, C., & Soorya, L. V. (2021) · Molecular Autism · doi:10.1186/s13229-021-00443-9 · Cited by 57
“Our iterative process and usercentered framework provides a data-driven approach to better meet diverse user needs [67].”
BackgroundMethodInfluentialAllan, S., Bradstreet, S., Mcleod, H., Farhall, J., Lambrou, M., Gleeson, J., Clark, A., EMPOWER Group, & Gumley, A. (2019) · Journal of Medical Internet Research · doi:10.2196/14366 · Cited by 44
“User experience has been described as a neglected area within digital intervention research [47] and psychosis more specifically [48].”
BackgroundJiménez, P., & Bregenzer, A. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.8769 · Cited by 54
“Therefore, it is suggested to evaluate the cost-effectiveness of the intervention in addition to the effectiveness of the intervention [65].”
Clinicians’ perspectives on inertial measurement units in clinical practice
Routhier, F., Duclos, N. C., Lacroix, É., Lettre, J., Turcotte, É., Hamel, N., Michaud, F., Duclos, C., Archambault, P. S., & Bouyer, L. J. (2020) · PLoS ONE · doi:10.1371/journal.pone.0241922 · Cited by 54
“as early as in the design process [11, 16, 26, 27].”
BackgroundThe Role of Wearable Devices in Multiple Sclerosis
Sparaco, M., Lavorgna, L., Conforti, R., Tedeschi, G., & Bonavita, S. (2018) · Multiple Sclerosis International · doi:10.1155/2018/7627643 · Cited by 54
Sankaran, S., Dendale, P., & Coninx, K. (2019) · JMIR mHealth and uHealth · doi:10.2196/10874 · Cited by 48
“Psychological studies have shown that effectively implementing behavior change interventions supports habit formation and facilitates in sustaining motivation over a longer period [15-17].”
BackgroundWalsh, D. M. J., Moran, K., Cornelissen, V., Buys, R., Claes, J., Zampognaro, P., Melillo, F., Maglaveras, N., Chouvarda, I., Triantafyllidis, A. K., et al. (2018) · Translational Behavioral Medicine · doi:10.1093/tbm/iby017 · Cited by 51
“unique to eHealth interventions due to the pace of development, interoperability concerns and the need for compliance with regulatory, ethical, and security requirements [55].”
BackgroundResultRobinson, A., Oksuz, U., Slight, R. D., Slight, S. P., & Husband, A. (2020) · JMIR mhealth and uhealth · doi:10.2196/19237 · Cited by 55
“Considering the rapidly progressive nature of digital health interventions and digital assistive technology research, cocreation of a person-centered digital support network may help surgical patient cohorts to benefit from pre- and postoperative behavior changes on both a short- and long-term basis [19,99].”
BackgroundVarsi, C., Ledel Solem, I. K., Eide, H., Børøsund, E., Kristjansdottir, O. B., Heldal, K., Waxenberg, L. B., Weiss, K. E., Schreurs, K. M. G., Morrison, E. J., et al. (2021) · BMC Health Services Research · doi:10.1186/s12913-021-06278-7 · Cited by 53
A Digital Intervention for Adolescent Depression (MoodHwb): Mixed Methods Feasibility Evaluation
Jones, R. B., Thapar, A., Rice, F. J., Mars, B., Agha, S. S., Smıth, D. J., Merry, S., Stallard, P. N., Thapar, A. K., Jones, I. R., et al. (2020) · JMIR Mental Health · doi:10.2196/14536 · Cited by 51
“Participants did not use it frequently; however, they found it helpful overall, and its use might vary according to the context and needs of the user (eg, the stage and severity of their difficulties) [37].”
Bray, L., Sharpe, A., Gichuru, P., Fortune, P.-M., Blake, L., & Appleton, V. (2020) · Journal of Medical Internet Research · doi:10.2196/17367 · Cited by 50
“This is against the backdrop of the need for improved innovation and implementation [24] to meet these health care challenges.”
BackgroundNelligan, R. K., Hinman, R. S., Atkins, L., & Bennell, K. L. (2019) · JMIR mhealth and uhealth · doi:10.2196/14619 · Cited by 51
“The BCW also provided structure for the transparent and thorough reporting of intervention design, vital for future evidence synthesis that will enable a greater understanding of how digital behavior change interventions may have their effect [44].”
BackgroundMethodŠilarova, B., Sharp, S. J., Usher‐Smith, J. A., Lucas, J., Payne, R., Shefer, G., Moore, C. S., Girling, C., Lawrence, K., Tolkien, Z., et al. (2019) · Heart · doi:10.1136/heartjnl-2018-314211 · Cited by 48
Martins, M. V., Koert, E., Sylvest, R., Maeda, E., Moura‐Ramos, M., Hammarberg, K., & Harper, J. C. (2023) · Human Reproduction · doi:10.1093/humrep/dead253 · Cited by 55
“Having participants test the resource will show that engagement goes far beyond their ability to use technology, which often does not correlate with behaviour change (Michie et al. , 2017).”
BackgroundInfluentialFlaherty, S. J., McCarthy, M. B., Collins, A., & McAuliffe, F. M. (2019) · Journal of Marketing Management · doi:10.1080/0267257x.2019.1576756 · Cited by 51
“As discussed earlier, one of the challenges faced in the area of health apps is ensuring that the app is used for the time period necessary to affect behaviour change, although this may differ for different behaviours and interventions which add further complexity (Michie et al., 2017).”
BackgroundWillmott, T. J., Pang, B., Rundle-Thiele, S. R., & Badejo, A. (2019) · Health Psychology Review · doi:10.1080/17437199.2019.1625280 · Cited by 49
Ferrand, J. L., Hockensmith, R., Houghton, R. F., & Buhi, E. R. (2020) · Journal of Medical Internet Research · doi:10.2196/19018 · Cited by 46
“While considerable research has been conducted assessing digital health approaches, such as text message–based approaches and mobile apps for a range of health behaviors [7-9], few have included smart assistant responses to health-related queries.”
BackgroundLong-Term Efficacy of Audiologist-Guided Internet-Based Cognitive Behavior Therapy for Tinnitus
Beukes, E. W., Allen, P. M., Baguley, D., Manchaiah, V. K. C., & Andersson, G. (2018) · American Journal of Audiology · doi:10.1044/2018_aja-imia3-18-0004 · Cited by 46
Metadata
- Authors
- Michie, S., Yardley, L., West, R., Patrick, K., & Greaves, F.
- Affiliations
- University College London (GB)
- University of Oxford (GB)
- University of California San Diego (US)
- Public Health England (GB)
- Corresponding author
- Michie, Susan
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2017-06-29
- Volume
- 19
- Issue
- 6
- Pages
- e232
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC5509948
- OpenAlex
- W2724146920
- Cited by
- 865
- References
- 32
- Citations by year
- 2026: 75 · 2025: 94 · 2024: 98 · 2023: 94 · 2022: 122 · 2021: 120 · 2020: 94 · 2019: 98 · 2018: 59 · 2017: 11
- FWCI
- 81.54
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- ePrints Soton (University of Southampton) (acceptedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Explore Bristol Research (submittedVersion, cc-by)
- Bristol Research (University of Bristol) (submittedVersion, cc-by)
- Spiral (Imperial College London) (submittedVersion, other-oa)
- Funders
- Robert Wood Johnson Foundation
- Cancer Research UK (22962)
- National Institute for Health and Care Research
- Medical Research Council (MR/K023195/1)
- Economic and Social Research Council (ES/F029624/1)
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Social Sciences › Psychology › Applied Psychology › Behavioral Health and Interventions
- Keywords
- digital interventions, behavior change, health behavior, health care, intervention development, user engagement, cost-effectiveness, health economics, regulatory framework, information governance, ethics, interoperability, machine learning, data science, Bayesian approach, chronic disease self-management, primary prevention
- MeSH terms
- Delivery of Health Care, Humans, Health Behavior, Telemedicine, Internet
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
