Developing and evaluating digital interventions to promote behavior change in health and health care: recommendations resulting from an international workshop
Open accessAbstract
Devices and programs using digital technology to foster or support behavior change (digital interventions) are increasingly ubiquitous, being adopted for use in patient diagnosis and treatment, self-management of chronic diseases, and in primary prevention. They have been heralded as potentially revolutionizing the ways in which individuals can monitor and improve their health behaviors and health care by improving outcomes, reducing costs, and improving the patient experience. However, we are still mainly in the age of promise rather than delivery. Developing and evaluating these digital interventions presents new challenges and new versions of old challenges that require use of improved and perhaps entirely new methods for research and evaluation. This article discusses these challenges and provides recommendations aimed at accelerating the rate of progress in digital behavior intervention research and practice. Areas addressed include intervention development in a rapidly changing technological landscape, promoting user engagement, advancing the underpinning science and theory, evaluating effectiveness and cost-effectiveness, and addressing issues of regulatory, ethical, and information governance. This article is the result of a two-day international workshop on how to create, evaluate, and implement effective digital interventions in relation to health behaviors. It was held in London in September 2015 and was supported by the United Kingdom's Medical Research Council (MRC), the National Institute for Health Research (NIHR), the Methodology Research Programme (PI Susan Michie), and the Robert Wood Johnson Foundation of the United States (PI Kevin Patrick). Important recommendations to manage the rapid pace of change include considering using emerging techniques from data science, machine learning, and Bayesian approaches and learning from other disciplines including computer science and engineering. With regard to assessing and promoting engagement, a key conclusion was that sustained engagement is not always required and that for each intervention it is useful to establish what constitutes "effective engagement," that is, sufficient engagement to achieve the intended outcomes. The potential of digital interventions for testing and advancing theories of behavior change by generating ecologically valid, real-time objective data was recognized. Evaluations should include all phases of the development cycle, designed for generalizability, and consider new experimental designs to make the best use of rich data streams. Future health economics analyses need to recognize and model the complex and potentially far-reaching costs and benefits of digital interventions. In terms of governance, developers of digital behavior interventions should comply with existing regulatory frameworks, but with consideration for emerging standards around information governance, ethics, and interoperability.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Low uptake and poor engagement are commonly observed with digital interventions, which are often insufficient to sustain behavior change.
“To date, low uptake and poor engagement are commonly observed with digital interventions, which are often insufficient to sustain behavior change [15,16].”
From Influences on the Uptake of and Engagement With Health and Well-Being Smartphone Apps: Systematic Review (2020) · cites this paper as [16]
Table 1 in the Introduction says engagement with digital interventions is often too limited to support behavior change. It does not establish that low uptake is commonly observed. The Results section also cautions that sustained engagement is not always necessary for behavior change, so the claim’s wording about sustaining change is broader than the paper’s point. (LLM, conf=0.85)
Experimental — accuracy not yet measuredUsing suitable strategies, such as user-centered participatory design, can heighten user engagement, thereby rendering intervention measures more appealing and efficacious.
“Within the realm of digital interventions, using suitable strategies, such as user-centered participatory design, can heighten user engagement, thereby rendering intervention measures more appealing and efficacious [26].”
From Assessing the Effectiveness of Digital Health Behavior Strategies on Type 2 Diabetes Management: Systematic Review and Network Meta-Analysis (2025) · cites this paper as [26]
Behavioural theory can guide the design of mHealth interventions aimed at changing health behaviours.
“Behavioural theory can guide the design of mHealth interventions aimed at changing health behaviours.60”
From mHealth text and voice communication for monitoring people with chronic diseases in low-resource settings: a realist review (2018) · cites this paper as [60]
An important reason for the lack of justifications for the intended use of eHealth technologies might be a lack of knowledge regarding the working mechanisms of technology-based applications.
“An important reason for the lack of justifications for the intended use of eHealth technologies might be that there is a lack of knowledge regarding the working mechanisms of technology-based applications [8].”
From Clarifying the Concept of Adherence to eHealth Technology: Systematic Review on When Usage Becomes Adherence (2017) · cites this paper as [8]
Adherence is a multidimensional concept, influenced by a range of technological, environmental, and individual factors altogether that cannot be evaluated by technology usage alone.
“the results of a recent literature review of theoretical perspectives on adherence showed that adherence is a multidimensional concept, influenced by a range of technological, environmental, and individual factors altogether that cannot be evaluated by technology usage alone [8,84].”
From Clarifying the Concept of Adherence to eHealth Technology: Systematic Review on When Usage Becomes Adherence (2017) · cites this paper as [8]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Eysenbach, G., & CONSORT-EHEALTH Group (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1923 · Cited by 1,863 · cited 1× · see citing works
“The advice provided here is specific to DBCIs, but it should also build on the good practice of reporting general mHealth interventions published elsewhere to ensure adequate description of the technical components of the intervention [44,45].”
BackgroundYardley, L., Morrison, L., Bradbury, K., & Muller, I. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4055 · Cited by 1,442 · see citing works
Self-citationA Holistic Framework to Improve the Uptake and Impact of eHealth Technologies
van Gemert-Pijnen, J. E., Nijland, N., van Limburg, M., Ossebaard, H. C., Kelders, S. M., Eysenbach, G., & Seydel, E. R. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1672 · Cited by 1,126 · see citing works
Understanding and Promoting Effective Engagement With Digital Behavior Change Interventions
Yardley, L., Spring, B. J., Riper, H., Morrison, L. G., Crane, D. H., Curtis, K., Merchant, G. C., Naughton, F., & Blandford, A. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.06.015 · Cited by 1,118 · see citing works
Self-citationBuilding Ontologies with Basic Formal Ontology
Arp, R., Smith, B., & Spear, A. D. (2015) · The MIT Press eBooks · doi:10.7551/mitpress/9780262527811.001.0001 · Cited by 1,083 · see citing works
The impact of guidance on Internet-based mental health interventions — A systematic review
Baumeister, H., Reichler, L., Munzinger, M., & Lin, J. (2014) · Internet Interventions · doi:10.1016/j.invent.2014.08.003 · Cited by 1,037 · see citing works
Evaluating Digital Health Interventions
Murray, E., Hekler, E. B., Andersson, G., Collins, L. M., Doherty, A., Hollis, C., Rivera, D. E., West, R., & Wyatt, J. C. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.06.008 · Cited by 847 · see citing works
Self-citationRamsey, S. D., Willke, R. J., Glick, H., Reed, S. D., Augustovski, F., Jonsson, B., Briggs, A., & Sullivan, S. D. (2015) · Value in Health · doi:10.1016/j.jval.2015.02.001 · Cited by 811 · cited 1× · see citing works
“However, existing economic approaches (eg, ISPOR guidance [29]) will need to be adapted to take into account the way digital health interventions are delivered, covering the whole life cycle of the intervention.”
BackgroundMichie, S., Wood, C. E., Johnston, M., Abraham, C., Francis, J. J., & Hardeman, W. (2015) · Health Technology Assessment · doi:10.3310/hta19990 · Cited by 680 · see citing works
Self-citationAgarwal, S., LeFevre, A. E., Lee, J., L’Engle, K., Mehl, G., Sinha, C., & Labrique, A. (2016) · BMJ · doi:10.1136/bmj.i1174 · Cited by 639 · see citing works
Complex interventions or complex systems? Implications for health economic evaluation
Shiell, A., Hawe, P., & Gold, L. H. (2008) · BMJ · doi:10.1136/bmj.39569.510521.ad · Cited by 630 · see citing works
Mohr, D. C., Schueller, S. M., Montague, E., Burns, M. N., & Rashidi, P. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.3077 · Cited by 510 · see citing works
A strategy for optimizing and evaluating behavioral interventions
Collins, L. M., Murphy, S. A., Nair, V. N., & Strecher, V. J. (2005) · Annals of Behavioral Medicine · doi:10.1207/s15324796abm3001_8 · Cited by 492 · see citing works
Ellis, K., Kerr, J., Godbole, S. V., Lanckriet, G. R. G., Wing, D., & Marshall, S. J. (2014) · Physiological Measurement · doi:10.1088/0967-3334/35/11/2191 · Cited by 344 · see citing works
Collins, L. M., Dziak, J. J., Kugler, K. C., & Trail, J. B. (2014) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2014.06.021 · Cited by 335 · see citing works
Who Owns the Data? Open Data for Healthcare
Kostkova, P., Brewer, H., de Lusignan, S., Fottrell, E., Goldacre, B., Hart, G., Koczan, P., Knight, P., Marsolier, C., McKendry, R. A., et al. (2016) · Frontiers in Public Health · doi:10.3389/fpubh.2016.00007 · Cited by 230 · see citing works
Mohr, D. C., Schueller, S. M., Riley, W. T., Brown, C. H., Cuijpers, P., Duan, N., Kwasny, M. J., Stiles-Shields, C., & Cheung, K. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4391 · Cited by 216 · see citing works
Agile science: creating useful products for behavior change in the real world
Hekler, E. B., Klasnja, P., Riley, W. T., Buman, M. P., Huberty, J., Rivera, D. E., & Martin, C. A. (2016) · Translational Behavioral Medicine · doi:10.1007/s13142-016-0395-7 · Cited by 208 · see citing works
Complex interventions and their implications for systematic reviews: a pragmatic approach
Petticrew, M. P., Anderson, L. M., Elder, R. W., Grimshaw, J., Hopkins, D. P., Hahn, R. A., Krause, L. K., Kristjansson, E., Mercer, S. L., Sipe, T., et al. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.06.004 · Cited by 204 · see citing works
Dynamic Models of Behavior for Just-in-Time Adaptive Interventions
Spruijt‐Metz, D., & Nilsen, W. (2014) · IEEE Pervasive Computing · doi:10.1109/mprv.2014.46 · Cited by 188 · see citing works
Behavior change interventions: the potential of ontologies for advancing science and practice
Larsen, K. R. T., Michie, S., Hekler, E. B., Gibson, B., Spruijt‐Metz, D., Ahern, D. K., Cole-Lewis, H. J., Ellis, R. J. B., Hesse, B. W., Moser, R. P., et al. (2016) · Journal of Behavioral Medicine · doi:10.1007/s10865-016-9768-0 · Cited by 174 · see citing works
Self-citationAdvancing Models and Theories for Digital Behavior Change Interventions
Hekler, E. B., Michie, S., Pavel, M., Rivera, D. E., Collins, L. M., Jimison, H. B., Garnett, C., Parral, S., & Spruijt-Metz, D. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.06.013 · Cited by 174 · see citing works
Self-citationThe Pace of Technologic Change
Patrick, K., Hekler, E. B., Estrin, D., Mohr, D. C., Riper, H., Crane, D., Godino, J., & Riley, W. T. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.05.001 · Cited by 169 · see citing works
Self-citationRivera, D. E., Pew, M. D., & Collins, L. M. (2007) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2006.10.020 · Cited by 151 · see citing works
Thomas, J. G., & Bond, D. S. (2015) · Health Psychology · doi:10.1037/hea0000304 · Cited by 117 · see citing works
Current Issues and Future Directions for Research Into Digital Behavior Change Interventions
Yardley, L., Choudhury, T., Patrick, K., & Michie, S. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.07.019 · Cited by 112 · see citing works
Self-citationAn Ecological Framework for Cancer Communication: Implications for Research
Patrick, K., Intille, S. S., & Zabinski, M. F. (2005) · Journal of Medical Internet Research · doi:10.2196/jmir.7.3.e23 · Cited by 98 · see citing works
Self-citationDiagnostic Verification of Temperature Forecasts
Murphy, A. H., Brown, B. G., & Chen, Y. (1989) · Weather and Forecasting · doi:10.1175/1520-0434(1989)004 · Cited by 97 · see citing works
Designing and undertaking a health economics study of digital health interventions
McNamee, P., Murray, E., Kelly, M. P., Bojke, L., Chilcott, J., Fischer, A., West, R., & Yardley, L. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.05.007 · Cited by 82 · see citing works
Self-citationMorrison, L. G., Hargood, C., Lin, S. X., Dennison, L., Joseph, J., Hughes, S., Michaelides, D. T., Johnston, D., Johnston, M., Michie, S., et al. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.3579 · Cited by 79 · see citing works
Self-citationEvans, B. J. (2016) · American Journal of Law & Medicine · doi:10.1177/0098858817700245 · Cited by 50 · see citing works
Garnett, C. V., Crane, D., Michie, S., West, R., & Brown, J. (2016) · BMC Public Health · doi:10.1186/s12889-016-3140-8 · Cited by 42 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 3
- [32]Committee on Federal Research Regulations and Reporting Requirements Optimizing the Nation's Investment in Academic Research A New Regulatory Framework for the 21st Century 2016 7 27 Washington, DC The National Academies Press 27386613
- [37]US Government Publishing Office Conference Report Health Insurance Portability and Accountability Act of 1996 1996 7 31 104th Congress, 2d session July 31, 1996 House of Representatives, Washington DC
- [39]European Union Regulation (EU) of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (General Data Protection Regulation) 2016 Brussels Official Journal of the European Union Commission
Cited by
Sorted by citations
Taxonomy of approaches to developing interventions to improve health: a systematic methods overview
O’Cathain, A., Croot, L., Sworn, K., Duncan, E., Rousseau, N. S., Turner, K. M., Yardley, L., & Hoddinott, P. (2019) · Pilot and Feasibility Studies · doi:10.1186/s40814-019-0425-6 · Cited by 581 · see citing works
Self-citationSkivington, K., Matthews, L., Simpson, S. A., Craig, P., Baird, J., Blazeby, J. M., Boyd, K., Craig, N. J., French, D., McIntosh, E., et al. (2021) · Health Technology Assessment · doi:10.3310/hta25570 · Cited by 535 · see citing works
Short, C. E., DeSmet, A., Woods, C., Williams, S. L., Maher, C., Middelweerd, A., Müller, A. M., Wark, P. A., Vandelanotte, C., Poppe, L., et al. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.9397 · Cited by 400 · see citing works
Liverpool, S., Mota, C. P., Sales, C. M. D., Čuš, A., Carletto, S., Hancheva, C., Sousa, S., Cerón, S. C., Moreno-Peral, P., Pietrabissa, G., et al. (2020) · Journal of Medical Internet Research · doi:10.2196/16317 · Cited by 408 · see citing works
Thieme, A., Belgrave, D., & Doherty, G. (2020) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3398069 · Cited by 406 · see citing works
Szinay, D., Jones, A., Chadborn, T., Brown, J., & Naughton, F. (2020) · Journal of Medical Internet Research · doi:10.2196/17572 · Cited by 289 · see citing works
“To date, low uptake and poor engagement are commonly observed with digital interventions, which are often insufficient to sustain behavior change [15,16].”
BackgroundSieverink, F., Kelders, S. M., & van Gemert-Pijnen, J. E. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.8578 · Cited by 331 · see citing works
Camden, C., & Silva, M. (2020) · Physical & Occupational Therapy In Pediatrics · doi:10.1080/01942638.2020.1825032 · Cited by 237 · see citing works
Stockwell, S., Schofield, P., Fisher, A., Firth, J., Jackson, S. E., Stubbs, B., & Smith, L. (2019) · Experimental Gerontology · doi:10.1016/j.exger.2019.02.020 · Cited by 212 · see citing works
Systematic review and meta-analysis of the effectiveness of chatbots on lifestyle behaviours
Singh, B., Olds, T., Brinsley, J., Dumuid, D., Virgara, R., Matricciani, L., Watson, A., Szeto, K., Eglitis, E., Miatke, A., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00856-1 · Cited by 215 · see citing works
Digital Interventions on Healthy Lifestyle Management: Systematic Review
Chatterjee, A., Prinz, A., Gerdes, M. W., & Martínez, S. (2021) · Journal of Medical Internet Research · doi:10.2196/26931 · Cited by 214 · see citing works
Slater, H., Campbell, J. M., Stinson, J. N., Burley, M. M., & Briggs, A. M. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.8888 · Cited by 155 · see citing works
Müller, A. M., Maher, C. A., Vandelanotte, C., Hingle, M., Middelweerd, A., Lopez, M. L., DeSmet, A., Short, C. E., Nathan, N., Hutchesson, M. J., et al. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.8954 · Cited by 180 · see citing works
Musiat, P., Johnson, C., Atkinson, M., Wilksch, S., & Wade, T. (2022) · Psychological Medicine · doi:10.1017/s0033291721004621 · Cited by 166 · see citing works
Lanzing, M. (2018) · Philosophy & Technology · doi:10.1007/s13347-018-0316-4 · Cited by 160 · see citing works
The use of eHealth to promote physical activity in cancer survivors: a systematic review
Haberlin, C., O’Dwyer, T., Mockler, D., Moran, J., O’Donnell, D. M., & Broderick, J. M. (2018) · Supportive Care in Cancer · doi:10.1007/s00520-018-4305-z · Cited by 139 · see citing works
Keleko, A. T., Kamsu-Foguem, B., Ngouna, R. H., & Tongne, A. (2022) · AI and Ethics · doi:10.1007/s43681-021-00132-6 · Cited by 155 · see citing works
Sherifali, D., Nerenberg, K. A., Wilson, S., Semeniuk, K., Ali, M. U., Redman, L. M., & Adamo, K. B. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.8006 · Cited by 122 · see citing works
Karekla, M., Kasinopoulos, O., Neto, D. D., Ebert, D. D., Van Daele, T., Nordgreen, T., Höfer, S., Oeverland, S., & Jensen, K. L. (2019) · European Psychologist · doi:10.1027/1016-9040/a000349 · Cited by 117 · see citing works
Pham, Q., Graham, G., Carrion, C., Morita, P. P., Seto, E., Stinson, J. N., & Cafazzo, J. A. (2019) · JMIR mHealth and uHealth · doi:10.2196/11941 · Cited by 131 · see citing works
Carraça, E. V., Encantado, J., Battista, F., Beaulieu, K., Blundell, J. E., Busetto, L., van Baak, M. A., Dicker, D., Ermolao, A., Farpour‐Lambert, N. J., et al. (2021) · Obesity Reviews · doi:10.1111/obr.13258 · Cited by 133 · see citing works
Evaluating the Impact of Physical Activity Apps and Wearables: Interdisciplinary Review
McCallum, C., Rooksby, J., & Gray, C. M. (2018) · JMIR mhealth and uhealth · doi:10.2196/mhealth.9054 · Cited by 126 · see citing works
Zarnowiecki, D. M., Mauch, C. E., Middleton, G., Matwiejczyk, L., Watson, W. L., Dibbs, J., Dessaix, A., & Golley, R. K. (2020) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-020-0915-1 · Cited by 128 · see citing works
A Ledger of Me: Personalizing Healthcare Using Blockchain Technology
Leeming, G., Cunningham, J. A., & Ainsworth, J. D. (2019) · Frontiers in Medicine · doi:10.3389/fmed.2019.00171 · Cited by 126 · see citing works
Kouri, A., Yamada, J., Cheung, J. L. S., de Velde, S. V., & Gupta, S. (2022) · Implementation Science · doi:10.1186/s13012-022-01199-3 · Cited by 125 · see citing works
Schwartz, S. M., Wildenhaus, K. J., Bucher, A., & Byrd, B. (2020) · Frontiers in Computer Science · doi:10.3389/fcomp.2020.00031 · Cited by 118 · see citing works
Iyamu, I., Gómez-Ramírez, O., Xu, A. X., Chang, H.-J., Watt, S., Mckee, G., & Gilbert, M. (2022) · DIGITAL HEALTH · doi:10.1177/20552076221102255 · Cited by 118 · see citing works
“35,48,77,82,83,86,90,91 Further, interventions were frequently implemented on a small scale and their associated evaluations were described as poorly generalizable.”
BackgroundInfluentialDomin, A., Spruijt‐Metz, D., Theisen, D., Ouzzahra, Y., & Vögele, C. (2021) · JMIR mhealth and uhealth · doi:10.2196/24308 · Cited by 118 · see citing works
Yeager, C. M., & Benight, C. C. (2018) · mHealth · doi:10.21037/mhealth.2018.08.04 · Cited by 115 · see citing works
Boulley, G. E., Leroy, T., Bernetière, C., Paquienséguy, F., Desfriches‐Doria, O., & Préau, M. (2018) · Psycho-Oncology · doi:10.1002/pon.4867 · Cited by 112 · see citing works
Goldberg, S. B., Imhoff-Smith, T., Bolt, D. M., Wilson-Mendenhall, C. D., Dahl, C. J., Davidson, R. J., & Rosenkranz, M. A. (2020) · JMIR Mental Health · doi:10.2196/23825 · Cited by 97 · see citing works
“Similar to attrition, low engagement is a rule rather than an exception for mHealth [120].”
BackgroundChien, I., Enrique, A., Palacios, J., Regan, T., Keegan, D., Carter, D., Tschiatschek, S., Nori, A., Thieme, A., Richards, D., et al. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.10791 · Cited by 111 · see citing works
Digital interventions to improve adherence to maintenance medication in asthma
Chan, A., De Simoni, A., Wileman, V., Holliday, L., Newby, C. J., Chisari, C., Ali, S., Zhu, N., Padakanti, P., Pinprachanan, V., et al. (2022) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013030.pub2 · Cited by 105 · see citing works
Mclaughlin, M., Delaney, T., Hall, A., Byaruhanga, J., Mackie, P., Grady, A., Reilly, K., Campbell, E., Sutherland, R., Wiggers, J., et al. (2021) · Journal of Medical Internet Research · doi:10.2196/23180 · Cited by 105 · see citing works
Voorheis, P., Zhao, A., Kuluski, K., Pham, Q., Scott, T., Sztur, P., Khanna, N., Ibrahim, M., & Petch, J. (2022) · JMIR mHealth and uHealth · doi:10.2196/35799 · Cited by 94 · see citing works
Taufik, D., Verain, M. C. D., Bouwman, E. P., & Reinders, M. J. (2019) · Trends in Food Science & Technology · doi:10.1016/j.tifs.2019.09.019 · Cited by 102 · see citing works
Nurmi, J., Knittle, K. P., Ginchev, T., Khattak, F., Helf, C., Zwickl, P., Castellano‐Tejedor, C., Lusilla, P., Costa‐Requena, J., Ravaja, N., et al. (2019) · JMIR mhealth and uhealth · doi:10.2196/12884 · Cited by 93 · see citing works
Asbjørnsen, R. A., Wentzel, J., Smedsrød, M. L., Hjelmesæth, J., Clark, M. M., Solberg Nes, L., & Van Gemert-Pijnen, J. E. W. C. (2020) · Journal of Medical Internet Research · doi:10.2196/22598 · Cited by 84 · see citing works
Systematic review of context-aware digital behavior change interventions to improve health
Craig, K. J. T., Morgan, L., Chen, C.-H., Michie, S., Fusco, N., Snowdon, J., Scheufele, E. L., Gagliardi, T. A., & Sill, S. (2020) · Translational Behavioral Medicine · doi:10.1093/tbm/ibaa099 · Cited by 87 · see citing works
Self-citationAl-Dhahir, I., Reijnders, T., Faber, J. S., van den Berg-Emons, R. J. G., Janssen, V. R., Kraaijenhagen, R. A., Visch, V., Chavannes, N. H., & Evers, A. W. M. (2022) · Journal of Medical Internet Research · doi:10.2196/34229 · Cited by 91 · see citing works
What is an appropriate level of evidence for a digital health intervention?
Greaves, F., Joshi, I., Campbell, M., Roberts, S., Patel, N., & powell, J. A. (2018) · The Lancet · doi:10.1016/s0140-6736(18)33129-5 · Cited by 92 · see citing works
Self-citationBeintner, I., Vollert, B., Zarski, A.-C., Bolinski, F., Musiat, P., Görlich, D., Ebert, D. D., & Jacobi, C. (2019) · Journal of Medical Internet Research · doi:10.2196/14181 · Cited by 89 · see citing works
Korpershoek, Y. J. G., Hermsen, S., Schoonhoven, L., Schuurmans, M. J., & Trappenburg, J. C. A. (2020) · Journal of Medical Internet Research · doi:10.2196/15449 · Cited by 78 · see citing works
Shen, H., van der Kleij, R., van der Boog, P. J. M., Chang, X., & Chavannes, N. H. (2019) · Journal of Medical Internet Research · doi:10.2196/12384 · Cited by 93 · see citing works
Wang, T., Wang, W., Liang, J., Nuo, M., Wen, Q., Wei, W., Han, H., & Lei, J. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00692-9 · Cited by 84 · see citing works
Tailored web-based interventions for pain: Systematic review and meta-analysis
Martorella, G., Boitor, M., Berube, M., Fredericks, S., Le May, S., & Gélinas, C. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.8826 · Cited by 88 · see citing works
Remote symptom monitoring integrated into electronic health records: A systematic review
Gandrup, J., Ali, S. M., McBeth, J., van der Veer, S. N., & Dixon, W. G. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa177 · Cited by 90 · see citing works
Lewkowitz, A. K., Whelan, A., Ayala, N. K., Hardi, A., Stoll, C., Battle, C. L., Tuuli, M. G., Ranney, M. L., & Miller, E. S. (2023) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2023.06.028 · Cited by 86 · see citing works
Guo, S. H.-M., Hsing, H.-C., Lin, J., & Lee, C. (2020) · JMIR mhealth and uhealth · doi:10.2196/18404 · Cited by 83 · see citing works
Sin, J., Galeazzi, G. M., McGregor, E., Collom, J., Taylor, A. F., Barrett, B. M., Lawrence, V. C., & Henderson, C. (2020) · Journal of Medical Internet Research · doi:10.2196/20581 · Cited by 79 · see citing works
Guo, Y., Xu, Z.-M., Qiao, J.-Y., Hong, Y. A., Zhang, H.-X., Zeng, C.-B., Cai, W.-P., Li, L.-H., & Liu, C.-C. (2018) · JMIR mHealth and uHealth · doi:10.2196/10274 · Cited by 64 · see citing works
Elavsky, S., Knapová, L., Klocek, A., & Šmahel, D. (2018) · Journal of Aging and Physical Activity · doi:10.1123/japa.2017-0410 · Cited by 80 · see citing works
Digital Therapeutics in Parkinson’s Disease: Practical Applications and Future Potential
Ellis, T. D., & Earhart, G. M. (2021) · Journal of Parkinson s Disease · doi:10.3233/jpd-202407 · Cited by 78 · see citing works
Ensuring electronic medical record simulation through better training, modeling, and evaluation
Zhang, Z., Yan, C., Mesa, D. A., Sun, J., & Malin, B. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz161 · Cited by 76 · see citing works
Methods for Human-Centered eHealth Development: Narrative Scoping Review
Kip, H., Keizer, J., da Silva, M. C., Beerlage-de Jong, N., Köhle, N., & Kelders, S. M. (2022) · Journal of Medical Internet Research · doi:10.2196/31858 · Cited by 67 · see citing works
Perski, O., BLANDFORD, A. E., Garnett, C. V., Crane, D., West, R., & Michie, S. (2019) · Translational Behavioral Medicine · doi:10.1093/tbm/ibz039 · Cited by 70 · see citing works
Self-citationArgent, R., Slevin, P., Bevilacqua, A., Neligan, M. C., Daly, A., & Caulfield, B. (2019) · Sensors · doi:10.3390/s19020432 · Cited by 75 · see citing works
Klonoff, D. C. (2019) · Journal of Diabetes Science and Technology · doi:10.1177/1932296818820303 · Cited by 73 · see citing works
ACSM Call to Action Statement: COVID-19 Considerations for Sports and Physical Activity
Denay, K. L., Breslow, R. G., Turner, M. N., Nieman, D. C., Roberts, W. O., & Best, T. M. (2020) · Current Sports Medicine Reports · doi:10.1249/jsr.0000000000000739 · Cited by 72 · see citing works
Partridge, S. R., & Redfern, J. (2018) · Healthcare · doi:10.3390/healthcare6030070 · Cited by 70 · see citing works
Bell, I. H., Rossell, S. L., Farhall, J., Hayward, M., Lim, M. H., Fielding-Smith, S. F., & Thomas, N. (2019) · Schizophrenia Research · doi:10.1016/j.schres.2019.10.026 · Cited by 70 · see citing works
What Makes Artificial Intelligence Exceptional in Health Technology Assessment?
Bélisle‐Pipon, J., Couture, V., Roy, M., Ganache, I., Goetghebeur, M. M., & Cohen, I. G. (2021) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2021.736697 · Cited by 72 · see citing works
Berman, M. A., Guthrie, N. L., Edwards, K. L., Appelbaum, K. J., Njike, V. Y., Eisenberg, D. M., & Katz, D. L. (2018) · JMIR Diabetes · doi:10.2196/diabetes.9591 · Cited by 70 · see citing works
Miller, S., Ainsworth, B., Yardley, L., Milton, A., Weal, M. J., Smith, P. W. F., & Morrison, L. G. (2018) · Journal of Medical Internet Research · doi:10.2196/10966 · Cited by 66 · see citing works
Self-citationvan Aubel, E., Bakker, J. M., Batink, T., Michielse, S., Goossens, L., Lange, I., Schruers, K., Lieverse, R., Marcelis, M., van Amelsvoort, T., et al. (2020) · Behaviour Research and Therapy · doi:10.1016/j.brat.2020.103592 · Cited by 57 · see citing works
Optimizing Digital Integrated Care via Micro‐Randomized Trials
Walton, A., Nahum‐Shani, I., Crosby, L., Klasnja, P., & Murphy, S. (2018) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt.1079 · Cited by 66 · see citing works
Tighe, S. A., Ball, K., Kensing, F., Kayser, L., Rawstorn, J. C., & Maddison, R. (2020) · Journal of Medical Internet Research · doi:10.2196/16774 · Cited by 68 · see citing works
Boulton, E., Hawley-Hague, H., French, D., Mellone, S., Zacchi, A., Clemson, L. M., Vereijken, B., & Todd, C. J. (2019) · Progress in Cardiovascular Diseases · doi:10.1016/j.pcad.2019.01.003 · Cited by 64 · see citing works
Brewer, L. C., Hayes, S. N., Caron, A. R., Derby, D. A., Breutzman, N. S., Wicks, A., Raman, J., Smith, C. M., Schaepe, K., Sheets, R. E., et al. (2019) · PLoS ONE · doi:10.1371/journal.pone.0218724 · Cited by 67 · see citing works
Methods for Human-Centered eHealth Development: Narrative Scoping Review (Preprint)
Kip, H., Keizer, J., da Silva, M. C., Beerlage-de Jong, N., Köhle, N., & Kelders, S. M. (2021) · doi:10.2196/preprints.31858 · Cited by 67 · see citing works
Distinguishing features in the assessment of mHealth apps
Tarricone, R., Petracca, F., Ciani, O., & Cucciniello, M. (2021) · Expert Review of Pharmacoeconomics & Outcomes Research · doi:10.1080/14737167.2021.1891883 · Cited by 62 · see citing works
Chikersal, P., Belgrave, D. C. M., Doherty, G. J., Enrique, Á., Palacios, J. E., Richards, D., & Thieme, A. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376341 · Cited by 63 · see citing works
Simpson, S. A., Matthews, L., Pugmire, J., McConnachie, A., McIntosh, E., Coulman, E., Hughes, K., Kelson, M. J., Morgan‐Trimmer, S., Murphy, S., et al. (2020) · Public Health Research · doi:10.3310/phr08030 · Cited by 63 · see citing works
mHealth Improved Fruit and Vegetable Accessibility and Intake in Young Children
Bakırcı-Taylor, A. L., Reed, D. B., McCool, B., & Dawson, J. (2019) · Journal of Nutrition Education and Behavior · doi:10.1016/j.jneb.2018.11.008 · Cited by 62 · see citing works
Digital Health Solutions for Indigenous Mental Well-Being
Hensel, J. M., Ellard, K., Koltek, M., Wilson, G., & Sareen, J. (2019) · Current Psychiatry Reports · doi:10.1007/s11920-019-1056-6 · Cited by 61 · see citing works
Hutchesson, M. J., Gough, C., Müller, A. M., Short, C. E., Whatnall, M. C., Ahmed, M., Pearson, N., Yin, Z., Ashton, L. M., Maher, C., et al. (2021) · Obesity Reviews · doi:10.1111/obr.13295 · Cited by 60 · see citing works
Damen, I., Brombacher, H., Lallemand, C., Brankaert, R., Brombacher, A., van Wesemael, P. J. V., & Vos, S. B. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17020499 · Cited by 59 · see citing works
Social media interventions for precision public health: promises and risks
Dunn, A. G., Mandl, K. D., & Coiera, E. W. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0054-0 · Cited by 59 · see citing works
Schwarz, A. F., Huertas‐Delgado, F. J., Cardon, G., & DeSmet, A. (2020) · Games for Health Journal · doi:10.1089/g4h.2019.0058 · Cited by 61 · see citing works
Chen, A. T., Wu, S., Tomasino, K. N., Lattie, E. G., & Mohr, D. C. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103187 · Cited by 58 · see citing works
“For example, the dimension, “Mechanisms”, does not provide a nuanced differentiation of mechanisms of change, which has been recognized as a need in the study of engagement of digital interventions[59].”
BackgroundWainer, A., Arnold, Z. E., Leonczyk, C., & Soorya, L. V. (2021) · Molecular Autism · doi:10.1186/s13229-021-00443-9 · Cited by 57 · see citing works
Allan, S., Bradstreet, S., Mcleod, H., Farhall, J., Lambrou, M., Gleeson, J., Clark, A., EMPOWER Group, & Gumley, A. (2019) · Journal of Medical Internet Research · doi:10.2196/14366 · Cited by 44 · see citing works
Jiménez, P., & Bregenzer, A. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.8769 · Cited by 54 · see citing works
Clinicians’ perspectives on inertial measurement units in clinical practice
Routhier, F., Duclos, N. C., Lacroix, É., Lettre, J., Turcotte, É., Hamel, N., Michaud, F., Duclos, C., Archambault, P. S., & Bouyer, L. J. (2020) · PLoS ONE · doi:10.1371/journal.pone.0241922 · Cited by 54 · see citing works
The Role of Wearable Devices in Multiple Sclerosis
Sparaco, M., Lavorgna, L., Conforti, R., Tedeschi, G., & Bonavita, S. (2018) · Multiple Sclerosis International · doi:10.1155/2018/7627643 · Cited by 54 · see citing works
Sankaran, S., Dendale, P., & Coninx, K. (2019) · JMIR mhealth and uhealth · doi:10.2196/10874 · Cited by 57 · see citing works
Walsh, D. M. J., Moran, K., Cornelissen, V., Buys, R., Claes, J., Zampognaro, P., Melillo, F., Maglaveras, N., Chouvarda, I., Triantafyllidis, A. K., et al. (2018) · Translational Behavioral Medicine · doi:10.1093/tbm/iby017 · Cited by 51 · see citing works
Robinson, A., Oksuz, U., Slight, R. D., Slight, S. P., & Husband, A. (2020) · JMIR mhealth and uhealth · doi:10.2196/19237 · Cited by 55 · see citing works
Varsi, C., Ledel Solem, I. K., Eide, H., Børøsund, E., Kristjansdottir, O. B., Heldal, K., Waxenberg, L. B., Weiss, K. E., Schreurs, K. M. G., Morrison, E. J., et al. (2021) · BMC Health Services Research · doi:10.1186/s12913-021-06278-7 · Cited by 53 · see citing works
A Digital Intervention for Adolescent Depression (MoodHwb): Mixed Methods Feasibility Evaluation
Jones, R. B., Thapar, A., Rice, F. J., Mars, B., Agha, S. S., Smıth, D. J., Merry, S., Stallard, P. N., Thapar, A. K., Jones, I. R., et al. (2020) · JMIR Mental Health · doi:10.2196/14536 · Cited by 51 · see citing works
Bray, L., Sharpe, A., Gichuru, P., Fortune, P.-M., Blake, L., & Appleton, V. (2020) · Journal of Medical Internet Research · doi:10.2196/17367 · Cited by 50 · see citing works
Nelligan, R. K., Hinman, R. S., Atkins, L., & Bennell, K. L. (2019) · JMIR mhealth and uhealth · doi:10.2196/14619 · Cited by 51 · see citing works
Šilarova, B., Sharp, S. J., Usher‐Smith, J. A., Lucas, J., Payne, R., Shefer, G., Moore, C. S., Girling, C., Lawrence, K., Tolkien, Z., et al. (2019) · Heart · doi:10.1136/heartjnl-2018-314211 · Cited by 49 · see citing works
Martins, M. V., Koert, E., Sylvest, R., Maeda, E., Moura‐Ramos, M., Hammarberg, K., & Harper, J. C. (2023) · Human Reproduction · doi:10.1093/humrep/dead253 · Cited by 49 · see citing works
Flaherty, S. J., McCarthy, M. B., Collins, A., & McAuliffe, F. M. (2019) · Journal of Marketing Management · doi:10.1080/0267257x.2019.1576756 · Cited by 51 · see citing works
Willmott, T. J., Pang, B., Rundle-Thiele, S. R., & Badejo, A. (2019) · Health Psychology Review · doi:10.1080/17437199.2019.1625280 · Cited by 47 · see citing works
Ferrand, J. L., Hockensmith, R., Houghton, R. F., & Buhi, E. R. (2020) · Journal of Medical Internet Research · doi:10.2196/19018 · Cited by 46 · see citing works
Long-Term Efficacy of Audiologist-Guided Internet-Based Cognitive Behavior Therapy for Tinnitus
Beukes, E. W., Allen, P. M., Baguley, D., Manchaiah, V. K. C., & Andersson, G. (2018) · American Journal of Audiology · doi:10.1044/2018_aja-imia3-18-0004 · Cited by 46 · see citing works
Taylor, C. B., Ruzek, J. I., Fitzsimmons‐Craft, E. E., Sadeh‐Sharvit, S., Topooco, N., Weissman, R. S., Eisenberg, D., Mohr, D. C., Graham, A. K., Jacobi, C., et al. (2020) · Journal of Medical Internet Research · doi:10.2196/17493 · Cited by 46 · see citing works
- Authors
- Michie, S., Yardley, L., West, R., Patrick, K., & Greaves, F.
- Affiliations
- University College London (GB)
- University of Oxford (GB)
- University of California San Diego (US)
- Public Health England (GB)
- Corresponding author
- Michie, Susan
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2017-06-29
- Volume
- 19
- Issue
- 6
- Pages
- e232
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC5509948
- OpenAlex
- W2724146920
- Cited by
- 865
- References
- 32
- Citations by year
- 2026: 75 · 2025: 94 · 2024: 98 · 2023: 94 · 2022: 122 · 2021: 120 · 2020: 94 · 2019: 98 · 2018: 59 · 2017: 11
- FWCI
- 81.54
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- ePrints Soton (University of Southampton) (acceptedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Explore Bristol Research (submittedVersion, cc-by)
- Bristol Research (University of Bristol) (submittedVersion, cc-by)
- Spiral (Imperial College London) (submittedVersion, other-oa)
- Funders
- Robert Wood Johnson Foundation
- Cancer Research UK (22962)
- National Institute for Health and Care Research
- Medical Research Council (MR/K023195/1)
- Economic and Social Research Council (ES/F029624/1)
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Social Sciences › Psychology › Applied Psychology › Behavioral Health and Interventions
- Keywords
- digital interventions, behavior change, health behavior, health care, intervention development, user engagement, cost-effectiveness, health economics, regulatory framework, information governance, ethics, interoperability, machine learning, data science, Bayesian approach, chronic disease self-management, primary prevention
- MeSH terms
- Delivery of Health Care, Humans, Health Behavior, Telemedicine, Internet
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
