The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Developing and Evaluating Digital Interventions to Promote Behavior Change in Health and Health Care: Recommendations Resulting From an International Workshop
Open accessAbstract
Devices and programs using digital technology to foster or support behavior change (digital interventions) are increasingly ubiquitous, being adopted for use in patient diagnosis and treatment, self-management of chronic diseases, and in primary prevention. They have been heralded as potentially revolutionizing the ways in which individuals can monitor and improve their health behaviors and health care by improving outcomes, reducing costs, and improving the patient experience. However, we are still mainly in the age of promise rather than delivery. Developing and evaluating these digital interventions presents new challenges and new versions of old challenges that require use of improved and perhaps entirely new methods for research and evaluation. This article discusses these challenges and provides recommendations aimed at accelerating the rate of progress in digital behavior intervention research and practice. Areas addressed include intervention development in a rapidly changing technological landscape, promoting user engagement, advancing the underpinning science and theory, evaluating effectiveness and cost-effectiveness, and addressing issues of regulatory, ethical, and information governance. This article is the result of a two-day international workshop on how to create, evaluate, and implement effective digital interventions in relation to health behaviors. It was held in London in September 2015 and was supported by the United Kingdom's Medical Research Council (MRC), the National Institute for Health Research (NIHR), the Methodology Research Programme (PI Susan Michie), and the Robert Wood Johnson Foundation of the United States (PI Kevin Patrick). Important recommendations to manage the rapid pace of change include considering using emerging techniques from data science, machine learning, and Bayesian approaches and learning from other disciplines including computer science and engineering. With regard to assessing and promoting engagement, a key conclusion was that sustained engagement is not always required and that for each intervention it is useful to establish what constitutes "effective engagement," that is, sufficient engagement to achieve the intended outcomes. The potential of digital interventions for testing and advancing theories of behavior change by generating ecologically valid, real-time objective data was recognized. Evaluations should include all phases of the development cycle, designed for generalizability, and consider new experimental designs to make the best use of rich data streams. Future health economics analyses need to recognize and model the complex and potentially far-reaching costs and benefits of digital interventions. In terms of governance, developers of digital behavior interventions should comply with existing regulatory frameworks, but with consideration for emerging standards around information governance, ethics, and interoperability.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To date, low uptake and poor engagement are commonly observed with digital interventions, which are often insufficient to sustain behavior change [15,16].”
From Influences on the Uptake of and Engagement With Health and Well-Being Smartphone Apps: Systematic Review (2020) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Within the realm of digital interventions, using suitable strategies, such as user-centered participatory design, can heighten user engagement, thereby rendering intervention measures more appealing and efficacious [26].”
From Assessing the Effectiveness of Digital Health Behavior Strategies on Type 2 Diabetes Management: Systematic Review and Network Meta-Analysis (2025) · cites this paper as [26]
In the citing paper
“Behavioural theory can guide the design of mHealth interventions aimed at changing health behaviours.60”
From mHealth text and voice communication for monitoring people with chronic diseases in low-resource settings: a realist review (2018) · cites this paper as [60]
In the citing paper
“An important reason for the lack of justifications for the intended use of eHealth technologies might be that there is a lack of knowledge regarding the working mechanisms of technology-based applications [8].”
From Clarifying the Concept of Adherence to eHealth Technology: Systematic Review on When Usage Becomes Adherence (2017) · cites this paper as [8]
In the citing paper
“the results of a recent literature review of theoretical perspectives on adherence showed that adherence is a multidimensional concept, influenced by a range of technological, environmental, and individual factors altogether that cannot be evaluated by technology usage alone [8,84].”
From Clarifying the Concept of Adherence to eHealth Technology: Systematic Review on When Usage Becomes Adherence (2017) · cites this paper as [8]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Eysenbach, G., & CONSORT-EHEALTH Group (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1923 · Cited by 1,863 · cited 1×
“The advice provided here is specific to DBCIs, but it should also build on the good practice of reporting general mHealth interventions published elsewhere to ensure adequate description of the technical components of the intervention [44,45].”
BackgroundYardley, L., Morrison, L., Bradbury, K., & Muller, I. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4055 · Cited by 1,442 · cited 2×
“These qualitative approaches are central to participatory user-centered design, which is the key to developing and evaluating DBCIs in order to ensure that they are engaging and effective [16,17].”
BackgroundSelf-citationA Holistic Framework to Improve the Uptake and Impact of eHealth Technologies
van Gemert-Pijnen, J. E., Nijland, N., van Limburg, M., Ossebaard, H. C., Kelders, S. M., Eysenbach, G., & Seydel, E. R. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1672 · Cited by 1,126 · cited 2×
“These qualitative approaches are central to participatory user-centered design, which is the key to developing and evaluating DBCIs in order to ensure that they are engaging and effective [16,17].”
BackgroundUnderstanding and Promoting Effective Engagement With Digital Behavior Change Interventions
Yardley, L., Spring, B. J., Riper, H., Morrison, L. G., Crane, D. H., Curtis, K., Merchant, G. C., Naughton, F., & Blandford, A. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.06.015 · Cited by 1,118 · cited 3×
“The workshop led to a series of publications [3-7].”
BackgroundSelf-citationBuilding Ontologies with Basic Formal Ontology
Arp, R., Smith, B., & Spear, A. D. (2015) · The MIT Press eBooks · doi:10.7551/mitpress/9780262527811.001.0001 · Cited by 1,083 · cited 1×
“Such knowledge-organizing structures are called “ontologies” [22,23].”
The impact of guidance on Internet-based mental health interventions — A systematic review
Baumeister, H., Reichler, L., Munzinger, M., & Lin, J. (2014) · Internet Interventions · doi:10.1016/j.invent.2014.08.003 · Cited by 1,037 · cited 1×
“Adding human support is also known to promote engagement with many interventions [18].”
BackgroundEvaluating Digital Health Interventions
Murray, E., Hekler, E. B., Andersson, G., Collins, L. M., Doherty, A., Hollis, C., Rivera, D. E., West, R., & Wyatt, J. C. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.06.008 · Cited by 847 · cited 4×
“The workshop led to a series of publications [3-7].”
BackgroundInfluentialSelf-citationRamsey, S. D., Willke, R. J., Glick, H., Reed, S. D., Augustovski, F., Jonsson, B., Briggs, A., & Sullivan, S. D. (2015) · Value in Health · doi:10.1016/j.jval.2015.02.001 · Cited by 811 · cited 1×
“However, existing economic approaches (eg, ISPOR guidance [29]) will need to be adapted to take into account the way digital health interventions are delivered, covering the whole life cycle of the intervention.”
BackgroundMichie, S., Wood, C. E., Johnston, M., Abraham, C., Francis, J. J., & Hardeman, W. (2015) · Health Technology Assessment · doi:10.3310/hta19990 · Cited by 680 · cited 1×
“This includes designing DBCIs on a modular basis with a clear understanding of which behavior change techniques [43] constitute that module and how they link to the theoretically postulated mechanisms of action.”
BackgroundSelf-citationAgarwal, S., LeFevre, A. E., Lee, J., L’Engle, K., Mehl, G., Sinha, C., & Labrique, A. (2016) · BMJ · doi:10.1136/bmj.i1174 · Cited by 639
Complex interventions or complex systems? Implications for health economic evaluation
Shiell, A., Hawe, P., & Gold, L. H. (2008) · BMJ · doi:10.1136/bmj.39569.510521.ad · Cited by 630 · cited 1×
“Economic appraisals will need to recognize this complexity—in terms of interventions, outcomes, and causal pathways [30]—and use appropriate methods [31] to assess them.”
MethodMohr, D. C., Schueller, S. M., Montague, E., Burns, M. N., & Rashidi, P. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.3077 · Cited by 510 · cited 1×
“Analysis of this very different pattern of engagement requires careful consideration of the relationship between the “micro” level of immediate engagement with the digital dimension of the intervention and the “macro” level of engagement with longer-term behavior change (see next section) [13].”
BackgroundA strategy for optimizing and evaluating behavioral interventions
Collins, L. M., Murphy, S. A., Nair, V. N., & Strecher, V. J. (2005) · Annals of Behavioral Medicine · doi:10.1207/s15324796abm3001_8 · Cited by 492 · cited 1×
“Theories, models, and methods to support this approach can be found in systems engineering and related sectors, for example, control theory [8], use of factorial or fractionated evaluation designs [9], and system optimization strategies [10] (please refer to [2] and Multimedia Appendix 2 for more details).”
MethodEllis, K., Kerr, J., Godbole, S. V., Lanckriet, G. R. G., Wing, D., & Marshall, S. J. (2014) · Physiological Measurement · doi:10.1088/0967-3334/35/11/2191 · Cited by 354 · cited 1×
“Machine learning and related approaches are increasingly being used to solve big data challenges, including health behavior assessment and interventions [12].”
BackgroundCollins, L. M., Dziak, J. J., Kugler, K. C., & Trail, J. B. (2014) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2014.06.021 · Cited by 335 · cited 1×
“Theories, models, and methods to support this approach can be found in systems engineering and related sectors, for example, control theory [8], use of factorial or fractionated evaluation designs [9], and system optimization strategies [10] (please refer to [2] and Multimedia Appendix 2 for more details).”
MethodWho Owns the Data? Open Data for Healthcare
Kostkova, P., Brewer, H., de Lusignan, S., Fottrell, E., Goldacre, B., Hart, G., Koczan, P., Knight, P., Marsolier, C., McKendry, R. A., et al. (2016) · Frontiers in Public Health · doi:10.3389/fpubh.2016.00007 · Cited by 230 · cited 1×
“These data have the potential to be a valuable resource for health care research and service planning [40], but the public may not be aware of these potential uses and may have reservations about how the data are used [41].”
BackgroundMohr, D. C., Schueller, S. M., Riley, W. T., Brown, C. H., Cuijpers, P., Duan, N., Kwasny, M. J., Stiles-Shields, C., & Cheung, K. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4391 · Cited by 216 · cited 1×
“While RCTs will continue to have value, they should be complemented with adaptive research designs, A/B testing, N-of-1 studies, and other research methods that yield insights in a shorter time frame or in ways that reflect the granular nature of the intervention effects [28].”
BackgroundComplex interventions and their implications for systematic reviews: a pragmatic approach
Petticrew, M. P., Anderson, L. M., Elder, R. W., Grimshaw, J., Hopkins, D. P., Hahn, R. A., Krause, L. K., Kristjansson, E., Mercer, S. L., Sipe, T., et al. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.06.004 · Cited by 214 · cited 1×
“Economic appraisals will need to recognize this complexity—in terms of interventions, outcomes, and causal pathways [30]—and use appropriate methods [31] to assess them.”
MethodAgile science: creating useful products for behavior change in the real world
Hekler, E. B., Klasnja, P., Riley, W. T., Buman, M. P., Huberty, J., Rivera, D. E., & Martin, C. A. (2016) · Translational Behavioral Medicine · doi:10.1007/s13142-016-0395-7 · Cited by 208 · cited 1×
“However, this requires the application of more “agile” science [42].”
MethodDynamic Models of Behavior for Just-in-Time Adaptive Interventions
Spruijt‐Metz, D., & Nilsen, W. (2014) · IEEE Pervasive Computing · doi:10.1109/mprv.2014.46 · Cited by 186 · cited 1×
“It also enables DBCIs to intervene at opportune moments (sometimes referred to as ecological momentary interventions [20] or “just-in-time” adaptive interventions [14,21].”
BackgroundBehavior change interventions: the potential of ontologies for advancing science and practice
Larsen, K. R. T., Michie, S., Hekler, E. B., Gibson, B., Spruijt‐Metz, D., Ahern, D. K., Cole-Lewis, H. J., Ellis, R. J. B., Hesse, B. W., Moser, R. P., et al. (2016) · Journal of Behavioral Medicine · doi:10.1007/s10865-016-9768-0 · Cited by 179 · cited 3×
“Such knowledge-organizing structures are called “ontologies” [22,23].”
BackgroundSelf-citationAdvancing Models and Theories for Digital Behavior Change Interventions
Hekler, E. B., Michie, S., Pavel, M., Rivera, D. E., Collins, L. M., Jimison, H. B., Garnett, C., Parral, S., & Spruijt-Metz, D. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.06.013 · Cited by 174 · cited 4×
“The workshop led to a series of publications [3-7].”
BackgroundInfluentialSelf-citationThe Pace of Technologic Change
Patrick, K., Hekler, E. B., Estrin, D., Mohr, D. C., Riper, H., Crane, D., Godino, J., & Riley, W. T. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.05.001 · Cited by 169 · cited 4×
“Thus, the methods by which we deploy, evaluate, and improve these DBCIs are moving from relatively data-poor, infrequent, and typically post-hoc assessments to methods that incorporate continuous measurement of the intervention effects in real time [3].”
BackgroundMethodInfluentialSelf-citationRivera, D. E., Pew, M. D., & Collins, L. M. (2007) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2006.10.020 · Cited by 151 · cited 1×
“Theories, models, and methods to support this approach can be found in systems engineering and related sectors, for example, control theory [8], use of factorial or fractionated evaluation designs [9], and system optimization strategies [10] (please refer to [2] and Multimedia Appendix 2 for more details).”
MethodThomas, J. G., & Bond, D. S. (2015) · Health Psychology · doi:10.1037/hea0000304 · Cited by 117 · cited 3×
“For some users and contexts, just one in-depth period of engagement with the DBCI may be sufficient to initiate new habits or teach new skills, whereas for other types of behavior change or other users, brief but timely context-triggered prompts may be needed long-term, whenever the behavior is required [14].”
BackgroundCurrent Issues and Future Directions for Research Into Digital Behavior Change Interventions
Yardley, L., Choudhury, T., Patrick, K., & Michie, S. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.07.019 · Cited by 112 · cited 3×
“The challenges involved in developing, evaluating, and implementing effective digital behavior change interventions (DBCIs), and preventing the use of counterproductive ones, have only just begun to be delineated, let alone met [1].”
BackgroundSelf-citationAn Ecological Framework for Cancer Communication: Implications for Research
Patrick, K., Intille, S. S., & Zabinski, M. F. (2005) · Journal of Medical Internet Research · doi:10.2196/jmir.7.3.e23 · Cited by 98 · cited 1×
“It also enables DBCIs to intervene at opportune moments (sometimes referred to as ecological momentary interventions [20] or “just-in-time” adaptive interventions [14,21].”
BackgroundSelf-citationDiagnostic Verification of Temperature Forecasts
Murphy, A. H., Brown, B. G., & Chen, Y. (1989) · Weather and Forecasting · doi:10.1175/1520-0434(1989)004 · Cited by 97
Designing and undertaking a health economics study of digital health interventions
McNamee, P., Murray, E., Kelly, M. P., Bojke, L., Chilcott, J., Fischer, A., West, R., & Yardley, L. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.05.007 · Cited by 82 · cited 4×
“The workshop led to a series of publications [3-7].”
BackgroundInfluentialSelf-citationMorrison, L. G., Hargood, C., Lin, S. X., Dennison, L., Joseph, J., Hughes, S., Michaelides, D. T., Johnston, D., Johnston, M., Michie, S., et al. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.3579 · Cited by 79
Self-citationEvans, B. J. (2016) · American Journal of Law & Medicine · doi:10.1177/0098858817700245 · Cited by 50 · cited 1×
“These data have the potential to be a valuable resource for health care research and service planning [40], but the public may not be aware of these potential uses and may have reservations about how the data are used [41].”
Garnett, C. V., Crane, D., Michie, S., West, R., & Brown, J. (2016) · BMC Public Health · doi:10.1186/s12889-016-3140-8 · Cited by 44 · cited 1×
“…of this approach is “Drink Less,” a mobile app [25] to reduce excessive alcohol consumption in which five theoretically distinct modules were tested in a full factorial trial (Normative feedback, Feedback and self-monitoring, Identity change, Action planning, and Cognitive bias re-training) [26].”
BackgroundSelf-citation
Flagged first, then by citations
Research works printed but not linked: 8
- West, R, 2016, A Guide to the Development and Evaluation of Digital Behaviour Change Intervention
- Michie, S, 2014, ABC of Behaviour Change Theories: An Essential Resource for Researchers, Policy Makers and Practitioners
- Christmas, S, 2015, Thinking about behaviour change: an interdisciplinary dialogue
- 2016, Optimizing the Nation's Investment in Academic Research A New Regulatory Framework for the 21st Century
- 2014, Guidance: Medical device stand-alone software including apps
- 2014, Personalised Health and Care 2020: Using data and technology to transform outcomes for patients and citizens
- 1998, Data Protection Act 1998
- 2016, Regulation (EU) of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (General Data Protection Regulation)
Cited by
Sorted by citations
Taxonomy of approaches to developing interventions to improve health: a systematic methods overview
O’Cathain, A., Croot, L., Sworn, K., Duncan, E., Rousseau, N. S., Turner, K. M., Yardley, L., & Hoddinott, P. (2019) · Pilot and Feasibility Studies · doi:10.1186/s40814-019-0425-6 · Cited by 625
“future rather than current guidance [23].”
Self-citationSkivington, K., Matthews, L., Simpson, S. A., Craig, P., Baird, J., Blazeby, J. M., Boyd, K., Craig, N. J., French, D., McIntosh, E., et al. (2021) · Health Technology Assessment · doi:10.3310/hta25570 · Cited by 585
Liverpool, S., Mota, C. P., Sales, C. M. D., Čuš, A., Carletto, S., Hancheva, C., Sousa, S., Cerón, S. C., Moreno-Peral, P., Pietrabissa, G., et al. (2020) · Journal of Medical Internet Research · doi:10.2196/16317 · Cited by 408
“There is much debate in the literature on appropriate methods for evaluating the impact of DHIs [31].”
Thieme, A., Belgrave, D., & Doherty, G. (2020) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3398069 · Cited by 406
Short, C. E., DeSmet, A., Woods, C., Williams, S. L., Maher, C., Middelweerd, A., Müller, A. M., Wark, P. A., Vandelanotte, C., Poppe, L., et al. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.9397 · Cited by 400
“Although there is evidence that eHealth and mHealth behavior change interventions can be effective, low levels of adherence and high levels of attrition have been commonly reported [1-3].”
Background
Metadata
- Authors
- Michie, S., Yardley, L., West, R., Patrick, K., & Greaves, F.
- Affiliations
- University College London (GB)
- University of Oxford (GB)
- University of California San Diego (US)
- Public Health England (GB)
- Corresponding author
- Michie, Susan
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2017-06-29
- Volume
- 19
- Issue
- 6
- Pages
- e232
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC5509948
- OpenAlex
- W2724146920
- Cited by
- 865
- References
- 32
- Citations by year
- 2026: 75 · 2025: 94 · 2024: 98 · 2023: 94 · 2022: 122 · 2021: 120 · 2020: 94 · 2019: 98 · 2018: 59 · 2017: 11
- FWCI
- 81.54
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- ePrints Soton (University of Southampton) (acceptedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Explore Bristol Research (submittedVersion, cc-by)
- Bristol Research (University of Bristol) (submittedVersion, cc-by)
- Spiral (Imperial College London) (submittedVersion, other-oa)
- Funders
- Robert Wood Johnson Foundation
- Cancer Research UK (22962)
- National Institute for Health and Care Research
- Medical Research Council (MR/K023195/1)
- Economic and Social Research Council (ES/F029624/1)
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Social Sciences › Psychology › Applied Psychology › Behavioral Health and Interventions
- Keywords
- digital interventions, behavior change, health behavior, health care, intervention development, user engagement, cost-effectiveness, health economics, regulatory framework, information governance, ethics, interoperability, machine learning, data science, Bayesian approach, chronic disease self-management, primary prevention
- MeSH terms
- Delivery of Health Care, Humans, Health Behavior, Telemedicine, Internet
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
