The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Developing and Evaluating Digital Interventions to Promote Behavior Change in Health and Health Care: Recommendations Resulting From an International Workshop
Open accessAbstract
Devices and programs using digital technology to foster or support behavior change (digital interventions) are increasingly ubiquitous, being adopted for use in patient diagnosis and treatment, self-management of chronic diseases, and in primary prevention. They have been heralded as potentially revolutionizing the ways in which individuals can monitor and improve their health behaviors and health care by improving outcomes, reducing costs, and improving the patient experience. However, we are still mainly in the age of promise rather than delivery. Developing and evaluating these digital interventions presents new challenges and new versions of old challenges that require use of improved and perhaps entirely new methods for research and evaluation. This article discusses these challenges and provides recommendations aimed at accelerating the rate of progress in digital behavior intervention research and practice. Areas addressed include intervention development in a rapidly changing technological landscape, promoting user engagement, advancing the underpinning science and theory, evaluating effectiveness and cost-effectiveness, and addressing issues of regulatory, ethical, and information governance. This article is the result of a two-day international workshop on how to create, evaluate, and implement effective digital interventions in relation to health behaviors. It was held in London in September 2015 and was supported by the United Kingdom's Medical Research Council (MRC), the National Institute for Health Research (NIHR), the Methodology Research Programme (PI Susan Michie), and the Robert Wood Johnson Foundation of the United States (PI Kevin Patrick). Important recommendations to manage the rapid pace of change include considering using emerging techniques from data science, machine learning, and Bayesian approaches and learning from other disciplines including computer science and engineering. With regard to assessing and promoting engagement, a key conclusion was that sustained engagement is not always required and that for each intervention it is useful to establish what constitutes "effective engagement," that is, sufficient engagement to achieve the intended outcomes. The potential of digital interventions for testing and advancing theories of behavior change by generating ecologically valid, real-time objective data was recognized. Evaluations should include all phases of the development cycle, designed for generalizability, and consider new experimental designs to make the best use of rich data streams. Future health economics analyses need to recognize and model the complex and potentially far-reaching costs and benefits of digital interventions. In terms of governance, developers of digital behavior interventions should comply with existing regulatory frameworks, but with consideration for emerging standards around information governance, ethics, and interoperability.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To date, low uptake and poor engagement are commonly observed with digital interventions, which are often insufficient to sustain behavior change [15,16].”
From Influences on the Uptake of and Engagement With Health and Well-Being Smartphone Apps: Systematic Review (2020) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Within the realm of digital interventions, using suitable strategies, such as user-centered participatory design, can heighten user engagement, thereby rendering intervention measures more appealing and efficacious [26].”
From Assessing the Effectiveness of Digital Health Behavior Strategies on Type 2 Diabetes Management: Systematic Review and Network Meta-Analysis (2025) · cites this paper as [26]
In the citing paper
“Behavioural theory can guide the design of mHealth interventions aimed at changing health behaviours.60”
From mHealth text and voice communication for monitoring people with chronic diseases in low-resource settings: a realist review (2018) · cites this paper as [60]
In the citing paper
“An important reason for the lack of justifications for the intended use of eHealth technologies might be that there is a lack of knowledge regarding the working mechanisms of technology-based applications [8].”
From Clarifying the Concept of Adherence to eHealth Technology: Systematic Review on When Usage Becomes Adherence (2017) · cites this paper as [8]
In the citing paper
“the results of a recent literature review of theoretical perspectives on adherence showed that adherence is a multidimensional concept, influenced by a range of technological, environmental, and individual factors altogether that cannot be evaluated by technology usage alone [8,84].”
From Clarifying the Concept of Adherence to eHealth Technology: Systematic Review on When Usage Becomes Adherence (2017) · cites this paper as [8]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Eysenbach, G., & CONSORT-EHEALTH Group (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1923 · Cited by 1,863 · cited 1×
“The advice provided here is specific to DBCIs, but it should also build on the good practice of reporting general mHealth interventions published elsewhere to ensure adequate description of the technical components of the intervention [44,45].”
BackgroundYardley, L., Morrison, L., Bradbury, K., & Muller, I. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4055 · Cited by 1,442 · cited 2×
“These qualitative approaches are central to participatory user-centered design, which is the key to developing and evaluating DBCIs in order to ensure that they are engaging and effective [16,17].”
BackgroundSelf-citationA Holistic Framework to Improve the Uptake and Impact of eHealth Technologies
van Gemert-Pijnen, J. E., Nijland, N., van Limburg, M., Ossebaard, H. C., Kelders, S. M., Eysenbach, G., & Seydel, E. R. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1672 · Cited by 1,126 · cited 2×
“These qualitative approaches are central to participatory user-centered design, which is the key to developing and evaluating DBCIs in order to ensure that they are engaging and effective [16,17].”
BackgroundUnderstanding and Promoting Effective Engagement With Digital Behavior Change Interventions
Yardley, L., Spring, B. J., Riper, H., Morrison, L. G., Crane, D. H., Curtis, K., Merchant, G. C., Naughton, F., & Blandford, A. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.06.015 · Cited by 1,118 · cited 3×
“The workshop led to a series of publications [3-7].”
BackgroundSelf-citationBuilding Ontologies with Basic Formal Ontology
Arp, R., Smith, B., & Spear, A. D. (2015) · The MIT Press eBooks · doi:10.7551/mitpress/9780262527811.001.0001 · Cited by 1,083 · cited 1×
“Such knowledge-organizing structures are called “ontologies” [22,23].”
The impact of guidance on Internet-based mental health interventions — A systematic review
Baumeister, H., Reichler, L., Munzinger, M., & Lin, J. (2014) · Internet Interventions · doi:10.1016/j.invent.2014.08.003 · Cited by 1,037 · cited 1×
“Adding human support is also known to promote engagement with many interventions [18].”
BackgroundEvaluating Digital Health Interventions
Murray, E., Hekler, E. B., Andersson, G., Collins, L. M., Doherty, A., Hollis, C., Rivera, D. E., West, R., & Wyatt, J. C. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.06.008 · Cited by 847 · cited 4×
“The workshop led to a series of publications [3-7].”
BackgroundInfluentialSelf-citationRamsey, S. D., Willke, R. J., Glick, H., Reed, S. D., Augustovski, F., Jonsson, B., Briggs, A., & Sullivan, S. D. (2015) · Value in Health · doi:10.1016/j.jval.2015.02.001 · Cited by 811 · cited 1×
“However, existing economic approaches (eg, ISPOR guidance [29]) will need to be adapted to take into account the way digital health interventions are delivered, covering the whole life cycle of the intervention.”
BackgroundMichie, S., Wood, C. E., Johnston, M., Abraham, C., Francis, J. J., & Hardeman, W. (2015) · Health Technology Assessment · doi:10.3310/hta19990 · Cited by 680 · cited 1×
“This includes designing DBCIs on a modular basis with a clear understanding of which behavior change techniques [43] constitute that module and how they link to the theoretically postulated mechanisms of action.”
BackgroundSelf-citationAgarwal, S., LeFevre, A. E., Lee, J., L’Engle, K., Mehl, G., Sinha, C., & Labrique, A. (2016) · BMJ · doi:10.1136/bmj.i1174 · Cited by 639
Flagged first, then by citations
Research works printed but not linked: 8
- West, R, 2016, A Guide to the Development and Evaluation of Digital Behaviour Change Intervention
- Michie, S, 2014, ABC of Behaviour Change Theories: An Essential Resource for Researchers, Policy Makers and Practitioners
- Christmas, S, 2015, Thinking about behaviour change: an interdisciplinary dialogue
- 2016, Optimizing the Nation's Investment in Academic Research A New Regulatory Framework for the 21st Century
- 2014, Guidance: Medical device stand-alone software including apps
- 2014, Personalised Health and Care 2020: Using data and technology to transform outcomes for patients and citizens
- 1998, Data Protection Act 1998
- 2016, Regulation (EU) of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (General Data Protection Regulation)
Cited by
Sorted by citations
Taxonomy of approaches to developing interventions to improve health: a systematic methods overview
O’Cathain, A., Croot, L., Sworn, K., Duncan, E., Rousseau, N. S., Turner, K. M., Yardley, L., & Hoddinott, P. (2019) · Pilot and Feasibility Studies · doi:10.1186/s40814-019-0425-6 · Cited by 625
“future rather than current guidance [23].”
Self-citationSkivington, K., Matthews, L., Simpson, S. A., Craig, P., Baird, J., Blazeby, J. M., Boyd, K., Craig, N. J., French, D., McIntosh, E., et al. (2021) · Health Technology Assessment · doi:10.3310/hta25570 · Cited by 585
Liverpool, S., Mota, C. P., Sales, C. M. D., Čuš, A., Carletto, S., Hancheva, C., Sousa, S., Cerón, S. C., Moreno-Peral, P., Pietrabissa, G., et al. (2020) · Journal of Medical Internet Research · doi:10.2196/16317 · Cited by 408
“There is much debate in the literature on appropriate methods for evaluating the impact of DHIs [31].”
Thieme, A., Belgrave, D., & Doherty, G. (2020) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3398069 · Cited by 406
Short, C. E., DeSmet, A., Woods, C., Williams, S. L., Maher, C., Middelweerd, A., Müller, A. M., Wark, P. A., Vandelanotte, C., Poppe, L., et al. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.9397 · Cited by 400
“Although there is evidence that eHealth and mHealth behavior change interventions can be effective, low levels of adherence and high levels of attrition have been commonly reported [1-3].”
Background
Metadata
- Authors
- Michie, S., Yardley, L., West, R., Patrick, K., & Greaves, F.
- Affiliations
- University College London (GB)
- University of Oxford (GB)
- University of California San Diego (US)
- Public Health England (GB)
- Corresponding author
- Michie, Susan
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2017-06-29
- Volume
- 19
- Issue
- 6
- Pages
- e232
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC5509948
- OpenAlex
- W2724146920
- Cited by
- 865
- References
- 32
- Citations by year
- 2026: 75 · 2025: 94 · 2024: 98 · 2023: 94 · 2022: 122 · 2021: 120 · 2020: 94 · 2019: 98 · 2018: 59 · 2017: 11
- FWCI
- 81.54
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- ePrints Soton (University of Southampton) (acceptedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Explore Bristol Research (submittedVersion, cc-by)
- Bristol Research (University of Bristol) (submittedVersion, cc-by)
- Spiral (Imperial College London) (submittedVersion, other-oa)
- Funders
- Robert Wood Johnson Foundation
- Cancer Research UK (22962)
- National Institute for Health and Care Research
- Medical Research Council (MR/K023195/1)
- Economic and Social Research Council (ES/F029624/1)
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Social Sciences › Psychology › Applied Psychology › Behavioral Health and Interventions
- Keywords
- digital interventions, behavior change, health behavior, health care, intervention development, user engagement, cost-effectiveness, health economics, regulatory framework, information governance, ethics, interoperability, machine learning, data science, Bayesian approach, chronic disease self-management, primary prevention
- MeSH terms
- Delivery of Health Care, Humans, Health Behavior, Telemedicine, Internet
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
