Designing and undertaking a health economics study of digital health interventions
Open accessAbstract
This paper introduces and discusses key issues in the economic evaluation of digital health interventions. The purpose is to stimulate debate so that existing economic techniques may be refined or new methods developed. The paper does not seek to provide definitive guidance on appropriate methods of economic analysis for digital health interventions. This paper describes existing guides and analytic frameworks that have been suggested for the economic evaluation of healthcare interventions. Using selected examples of digital health interventions, it assesses how well existing guides and frameworks align to digital health interventions. It shows that digital health interventions may be best characterized as complex interventions in complex systems. Key features of complexity relate to intervention complexity, outcome complexity, and causal pathway complexity, with much of this driven by iterative intervention development over time and uncertainty regarding likely reach of the interventions among the relevant population. These characteristics imply that more-complex methods of economic evaluation are likely to be better able to capture fully the impact of the intervention on costs and benefits over the appropriate time horizon. This complexity includes wider measurement of costs and benefits, and a modeling framework that is able to capture dynamic interactions among the intervention, the population of interest, and the environment. The authors recommend that future research should develop and apply more-flexible modeling techniques to allow better prediction of the interdependency between interventions and important environmental influences.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Reach, uptake and context are particularly salient for digital health interventions because impact and cost-effectiveness depend on the total number of users, and effectiveness may depend on context.
“The concepts of reach, uptake and context are particularly salient for DHI, as impact and cost-effectiveness are highly dependent on the total number of users (McNamee 2016), and effectiveness may be highly dependent on context.”
From Evaluating Digital Health Interventions (2016) · cites this paper as McNamee 2016
Establishing and optimising reach and uptake requires human-centred design methods from engineering and computer science.
“Establishing and optimising potential reach and uptake requires methods used in engineering and computer science, collectively referred to as human-centred design 25–27.”
From Evaluating Digital Health Interventions (2016) · cites this paper as [25]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Developing and evaluating complex interventions: the new Medical Research Council guidance
Craig, P., Dieppe, P., Macintyre, S., Michie, S., Nazareth, I., & Petticrew, M. (2008) · BMJ · doi:10.1136/bmj.a1655 · Cited by 10,995 · see citing works
The Spread of Obesity in a Large Social Network over 32 Years
Christakis, N. A., & Fowler, J. H. (2007) · New England Journal of Medicine · doi:10.1056/nejmsa066082 · Cited by 5,007 · see citing works
The collective dynamics of smoking in a large social network
Christakis, N. A., & Fowler, J. H. (2008) · New England Journal of Medicine · doi:10.1056/nejmsa0706154 · Cited by 2,390 · see citing works
Developing and Evaluating Complex Interventions: New Guidance
Craig, P., Dieppe, P. A., MacIntyre, S., Michie, S., Nazareth, I., & Petticrew, M. P. (2006) · UCL Discovery (University College London) · Cited by 1,877 · see citing works
Yardley, L., Morrison, L., Bradbury, K., & Muller, I. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4055 · Cited by 1,442 · see citing works
Self-citationEffectiveness of telemedicine: A systematic review of reviews
Ekeland, A. G., Bowes, A., & Flottorp, S. (2010) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2010.08.006 · Cited by 1,162 · see citing works
Theorising interventions as events in systems
Hawe, P., Shiell, A., & Riley, T. (2009) · American Journal of Community Psychology · doi:10.1007/s10464-009-9229-9 · Cited by 1,143 · see citing works
Understanding and Promoting Effective Engagement With Digital Behavior Change Interventions
Yardley, L., Spring, B. J., Riper, H., Morrison, L. G., Crane, D. H., Curtis, K., Merchant, G. C., Naughton, F., & Blandford, A. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.06.015 · Cited by 1,118 · see citing works
Self-citationEvidence-Based Public Health: Moving Beyond Randomized Trials
Victora, C. G., Habicht, J.-P., & Bryce, J. (2004) · American Journal of Public Health · doi:10.2105/ajph.94.3.400 · Cited by 1,056 · see citing works
Criteria for evaluating evidence on public health interventions
Rychetnik, L., Frommer, M., Hawe, P., & Shiell, A. (2002) · Journal of Epidemiology and Community Health · doi:10.1136/jech.56.2.119 · Cited by 1,042 · see citing works
Evaluating Digital Health Interventions
Murray, E., Hekler, E. B., Andersson, G., Collins, L. M., Doherty, A., Hollis, C., Rivera, D. E., West, R., & Wyatt, J. C. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.06.008 · Cited by 847 · see citing works
Self-citationRamsey, S. D., Willke, R. J., Glick, H., Reed, S. D., Augustovski, F., Jonsson, B., Briggs, A., & Sullivan, S. D. (2015) · Value in Health · doi:10.1016/j.jval.2015.02.001 · Cited by 811 · cited 1× · see citing works
“To enhance the conduct and reporting of trial-based economic evaluation studies applied to new medicines, medical devices, and procedures, the International Society for Pharmacoeconomics and Outcomes Research has published an updated good research practice guide.(1) This re-emphasizes the need to base economic evidenc…”
MethodAn Experimental Study of Homophily in the Adoption of Health Behavior
Centola, D. (2011) · Science · doi:10.1126/science.1207055 · Cited by 681 · see citing works
Development of a self-report measure of capability wellbeing for adults: the ICECAP-A
Al-Janabi, H., N Flynn, T., & Coast, J. (2012) · Quality of Life Research · doi:10.1007/s11136-011-9927-2 · Cited by 666 · see citing works
Complex interventions or complex systems? Implications for health economic evaluation
Shiell, A., Hawe, P., & Gold, L. H. (2008) · BMJ · doi:10.1136/bmj.39569.510521.ad · Cited by 630 · see citing works
Little, P., Stuart, B., Francis, N., Douglas, E., Tonkin-Crine, S., Anthierens, S., Cals, J. W., Melbye, H., Santer, M., Moore, M., et al. (2013) · The Lancet · doi:10.1016/s0140-6736(13)60994-0 · Cited by 430 · see citing works
Self-citationHesketh, K. R., McMinn, A. M., Ekelund, U., Sharp, S. J., Collings, P. J., Harvey, N. C., Godfrey, K. M., Inskip, H. M., Cooper, C. C., & van Sluijs, E. M. F. (2014) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/1479-5868-11-1 · Cited by 405 · see citing works
Discrete-event simulation is dead, long live agent-based simulation!
Siebers, P., Macal, C. M., Garnett, J., Buxton, D., & Pidd, M. (2010) · Journal of Simulation · doi:10.1057/jos.2010.14 · Cited by 402 · see citing works
Zucchini, S., Scaramuzza, A. E., Bonfanti, R., Buono, P., Cardella, F., Cauvin, V., Cherubini, V., Chiari, G., d’Annunzio, G., Frongia, A. P., et al. (2015) · Journal of Diabetes Research · doi:10.1155/2016/5719470 · Cited by 326 · see citing works
Hawe, P., Shiell, A., Riley, T., & Gold, L. H. (2004) · Journal of Epidemiology & Community Health · doi:10.1136/jech.2003.014415 · Cited by 287 · see citing works
Social network analysis and agent-based modeling in social epidemiology
El‐Sayed, A. M., Scarborough, P., Seemann, L., & Galea, S. (2012) · Epidemiologic Perspectives & Innovations · doi:10.1186/1742-5573-9-1 · Cited by 224 · see citing works
Mistry, H. (2012) · Journal of Telemedicine and Telecare · doi:10.1258/jtt.2011.110505 · Cited by 215 · see citing works
Complex interventions and their implications for systematic reviews: a pragmatic approach
Petticrew, M. P., Anderson, L. M., Elder, R. W., Grimshaw, J., Hopkins, D. P., Hahn, R. A., Krause, L. K., Kristjansson, E., Mercer, S. L., Sipe, T., et al. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.06.004 · Cited by 204 · see citing works
Kelly, M., Morgan, A., Ellis, S., Younger, T., Huntley, J., & Swann, C. (2010) · Social Science & Medicine · doi:10.1016/j.socscimed.2010.06.032 · Cited by 182 · see citing works
Self-citationTime to rethink the systematic review catechism? Moving from ‘what works’ to ‘what happens’
Petticrew, M. P. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0027-1 · Cited by 160 · see citing works
Evaluation of Systems-Oriented Public Health Interventions: Alternative Research Designs
Sanson‐Fisher, R., D’Este, C. A., Carey, M. L., Noble, N., & Paul, C. L. (2014) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-032013-182445 · Cited by 141 · see citing works
Online interventions for depression and anxiety – a systematic review
Saddichha, S., Al-Desouki, M., Lamia, A., Linden, I. A., & Krausz, M. (2014) · Health Psychology and Behavioral Medicine · doi:10.1080/21642850.2014.945934 · Cited by 133 · cited 1× · see citing works
“,(36) there is good evidence that DHIs alone are often not as effective as DHI plus human support or facilitation, where the human input focuses on getting the patient (user) to use the DHI as intended.(37,38) Unlike all other costs associated with DHIs, which are fixed, these facilitation costs are variable as they i…”
BackgroundPowell, K., Wilcox, J., Clonan, A., Bissell, P., Preston, L., Peacock, M., & Holdsworth, M. (2015) · BMC Public Health · doi:10.1186/s12889-015-2314-0 · Cited by 123 · see citing works
Little, P. S., Stuart, B. L., Hobbs, R. F., Moore, M. V., Barnett, J., Popoola, D., Middleton, K., Kelly, J., Mullee, M., Raftery, J. P., et al. (2015) · The Lancet · doi:10.1016/s0140-6736(15)60127-1 · Cited by 120 · see citing works
Self-citationA Framework for Developing the Structure of Public Health Economic Models
Squires, H., Chilcott, J. B., Akehurst, R. L., Burr, J. A., & Kelly, M. P. (2016) · Value in Health · doi:10.1016/j.jval.2016.02.011 · Cited by 114 · see citing works
Self-citationEvaluating complex and unfolding interventions in real time
Ling, T. (2012) · Evaluation · doi:10.1177/1356389011429629 · Cited by 110 · see citing works
Weaver, E. R., Horyniak, D., Jenkinson, R., Dietze, P., & Lim, M. S. C. (2013) · JMIR mhealth and uhealth · doi:10.2196/mhealth.2709 · Cited by 107 · see citing works
Dennison, L., Morrison, L., Lloyd, S., Phillips, D., Stuart, B., Williams, S., Bradbury, K., Roderick, P., Murray, E., Michie, S., et al. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.3199 · Cited by 98 · see citing works
Self-citationZhang, D. “., Giabbanelli, P. J., Arah, O. A., & Zimmerman, F. J. (2014) · American Journal of Public Health · doi:10.2105/ajph.2014.301934 · Cited by 90 · see citing works
Yardley, L., Ware, L. J., Smith, E. R., Williams, S., Bradbury, K. J., Arden-Close, E. J., Mullee, M. A., Moore, M. V., Peacock, J. L., Lean, M. E., et al. (2014) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/1479-5868-11-67 · Cited by 84 · see citing works
Self-citationAgent-Based Models and Systems Science Approaches to Public Health
Maglio, P. P., & Mabry, P. L. (2011) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2010.11.010 · Cited by 72 · see citing works
Mistry, H., Garnvwa, H., & Oppong, R. (2014) · Telemedicine Journal and e-Health · doi:10.1089/tmj.2013.0259 · Cited by 65 · see citing works
Frerichs, L., Araz, Ö. M., & Huang, T. T. ‐. (2013) · PLoS ONE · doi:10.1371/journal.pone.0082887 · Cited by 58 · see citing works
Leroux, J. S., Moore, S., & Dubé, L. (2013) · Journal of Obesity · doi:10.1155/2013/348249 · Cited by 47 · see citing works
Habit, custom, and power: A multi-level theory of population health
Zimmerman, F. J. (2013) · Social Science & Medicine · doi:10.1016/j.socscimed.2012.12.029 · Cited by 43 · see citing works
Hofmann, M., Dack, C., Barker, C., & Murray, E. (2015) · Journal of Diabetes Research · doi:10.1155/2016/1476384 · Cited by 33 · see citing works
Self-citationThe appraisal of public health interventions: the use of theory
Threlfall, A., Meah, S., Fischer, A. J., et al. (2014) · Journal of Public Health · doi:10.1093/pubmed/fdu044 · Cited by 33 · see citing works
Self-citationThe appraisal of public health interventions: an overview
Fischer, A. J., Threlfall, A., Meah, S., Cookson, R., Rutter, H., & Kelly, M. P. (2013) · Journal of Public Health · doi:10.1093/pubmed/fdt076 · Cited by 32 · see citing works
Self-citationChalabi, Z., & Lorenc, T. (2013) · Preventive Medicine · doi:10.1016/j.ypmed.2013.07.013 · Cited by 20 · see citing works
McCambridge, J., O'Donnell, O., Godfrey, C., Khadjesari, Z., Linke, S., Murray, E., & Wallace, P. (2010) · BMC Research Notes · doi:10.1186/1756-0500-3-172 · Cited by 9 · see citing works
Self-citationStaff, T. P. O. (2014) · PLoS ONE · doi:10.1371/journal.pone.0097204 · Cited by 6 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [44]44. Kahneman D. Thinking, fast and slow . First ed. London: Penguin; 2012.
Cited by
Sorted by citations
Michie, S., Yardley, L., West, R., Patrick, K., & Greaves, F. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7126 · Cited by 865 · see citing works
Self-citationEvaluating Digital Health Interventions
Murray, E., Hekler, E. B., Andersson, G., Collins, L. M., Doherty, A., Hollis, C., Rivera, D. E., West, R., & Wyatt, J. C. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.06.008 · Cited by 847 · see citing works
Self-citationThe Pace of Technologic Change
Patrick, K., Hekler, E. B., Estrin, D., Mohr, D. C., Riper, H., Crane, D., Godino, J., & Riley, W. T. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.05.001 · Cited by 169 · see citing works
Aref-Adib, G., McCloud, T., Ross, J., O'Hanlon, P., Appleton, V., Rowe, S., Murray, E., Johnson, S., & Lobban, F. (2019) · The Lancet Psychiatry · doi:10.1016/s2215-0366(18)30302-x · Cited by 118 · see citing works
Self-citationCurrent Issues and Future Directions for Research Into Digital Behavior Change Interventions
Yardley, L., Choudhury, T., Patrick, K., & Michie, S. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.07.019 · Cited by 112 · see citing works
Self-citationMitchell, L. M., Joshi, U., Patel, V., Lu, C., & Naslund, J. A. (2021) · Journal of Affective Disorders · doi:10.1016/j.jad.2021.01.092 · Cited by 80 · see citing works
Distinguishing features in the assessment of mHealth apps
Tarricone, R., Petracca, F., Ciani, O., & Cucciniello, M. (2021) · Expert Review of Pharmacoeconomics & Outcomes Research · doi:10.1080/14737167.2021.1891883 · Cited by 62 · see citing works
Digital interventions in mental health: Evidence syntheses and economic modelling
Gega, L., Jankovic, D., Saramago, P., Marshall, D., Dawson, S., Brabyn, S., Nikolaidis, G. F., Melton, H., Churchill, R., & Bojke, L. (2022) · Health Technology Assessment · doi:10.3310/rcti6942 · Cited by 48 · see citing works
Joshi, U., Naslund, J. A., Anand, A., Tugnawat, D., Vishwakarma, R., Bhan, A., Patel, V., & Lu, C. (2022) · Psychiatry Research · doi:10.1016/j.psychres.2021.114299 · Cited by 24 · see citing works
Assessing the costs and cost-effectiveness of ICare internet-based interventions (protocol)
Beecham, J., Bonin, E.-M., Görlich, D., Baños, R., Beintner, I., Buntrock, C., Bolinski, F., Botella, C., Ebert, D. D., Herrero, R., et al. (2019) · Internet Interventions · doi:10.1016/j.invent.2018.02.009 · Cited by 13 · see citing works
Integrating remote blood pressure monitoring into NHS primary care: a human factors perspective
Micocci, M., Butt, O., Zhou, S., El-Osta, A., Buckle, P., & Hanna, G. B. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1697787 · Cited by 1 · see citing works
- Authors
- McNamee, P., Murray, E., Kelly, M. P., Bojke, L., Chilcott, J., Fischer, A., West, R., & Yardley, L.
- Affiliations
- University of Aberdeen (GB)
- University College London (GB)
- University of Cambridge (GB)
- University of York (GB)
- National Institute for Health and Care Excellence (GB)
- University of Southampton (GB)
- Corresponding author
- McNamee, Paul
- Venue
- American Journal of Preventive Medicine
- Publisher
- Elsevier BV
- Published
- 2016-11
- Volume
- 51
- Issue
- 5
- Pages
- 852-860
- ISSN
- 0749-3797 (ISSN-L)
- OpenAlex
- W2344797598
- Cited by
- 82
- References
- 46
- Citations by year
- 2026: 4 · 2025: 9 · 2024: 7 · 2023: 9 · 2022: 14 · 2021: 7 · 2020: 9 · 2019: 6 · 2018: 9 · 2017: 3 · 2016: 3 · 2015: 1
- FWCI
- 13.62
- Type
- journal article
- Language
- en
- Open access
- Licence
- cc-by-nc-nd
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- Locations
- American Journal of Preventive Medicine (publishedVersion)
- PubMed (publishedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- ePrints Soton (University of Southampton) (acceptedVersion)
- Apollo (University of Cambridge) (acceptedVersion, cc-by-nc-nd)
- Bristol Research (University of Bristol) (submittedVersion, cc-by-nc-nd)
- Aberdeen University Research Archive (Aberdeen University) (submittedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- National Science Foundation (1539846)
- Scottish Government
- Cancer Research UK (14135)
- National Institute for Health and Care Research (RP-PG-0609-10135)
- Chief Scientist Office, Scottish Government Health and Social Care Directorate
- Medical Research Council (MR/K023195/1)
- Economic and Social Research Council (ES/G007489/1)
- Topics
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Health Systems, Economic Evaluations, Quality of Life
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Keywords
- digital health interventions, economic evaluation, complex interventions, complex systems, intervention complexity, dynamic modeling, cost-benefit analysis
- MeSH terms
- Cost-Benefit Analysis, Health Promotion, Humans, Research Design, Telecommunications, Program Evaluation, Health Behavior
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
