CoChat
Sign Up
Cited by 847Mobile Health and mHealth ApplicationsDigital Mental Health InterventionsHealth Systems, Economic Evaluations, Quality of Life

Evaluating Digital Health Interventions

Open access
Murray, Elizabeth, Hekler, Eric B., Andersson, Gerhard, Collins, Linda M., Doherty, Aiden, Hollis, Chris, Rivera, Daniel E., West, Robert +1 more·2016·American Journal of Preventive Medicine
DOI 10.1016/j.amepre.2016.06.008PMC PMC5324832
Open this paper in CoChat — free accountOpen PDF

Abstract

Digital health interventions have enormous potential as scalable tools to improve health and healthcare delivery by improving effectiveness, efficiency, accessibility, safety, and personalization. Achieving these improvements requires a cumulative knowledge base to inform development and deployment of digital health interventions. However, evaluations of digital health interventions present special challenges. This paper aims to examine these challenges and outline an evaluation strategy in terms of the research questions needed to appraise such interventions. As they are at the intersection of biomedical, behavioral, computing, and engineering research, methods drawn from all of these disciplines are required. Relevant research questions include defining the problem and the likely benefit of the digital health intervention, which in turn requires establishing the likely reach and uptake of the intervention, the causal model describing how the intervention will achieve its intended benefit, key components, and how they interact with one another, and estimating overall benefit in terms of effectiveness, cost effectiveness, and harms. Although RCTs are important for evaluation of effectiveness and cost effectiveness, they are best undertaken only when: (1) the intervention and its delivery package are stable; (2) these can be implemented with high fidelity; and (3) there is a reasonable likelihood that the overall benefits will be clinically meaningful (improved outcomes or equivalent outcomes at lower cost). Broadening the portfolio of research questions and evaluation methods will help with developing the necessary knowledge base to inform decisions on policy, practice, and research.

Integrity signals

Registry

Crossref · 10.1016/j.amepre.2016.06.008

Cites

93 of 93 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 35 cited sources checked · 0 supported · 0 flagged

  • Interpretationcitation

    Establishing and optimising reach and uptake requires human-centred design methods from engineering and computer science.

    “Establishing and optimising potential reach and uptake requires methods used in engineering and computer science, collectively referred to as human-centred design 25–27.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [25] McNamee, 2016, Designing and undertaking a health economics study of digital health interventions, Am J Prev Med source

      Not checked in the free run.

    • cites [26] Agarwal, 2011

      Not checked in the free run.

    • cites [27] Maguire, 2001, Methods to support human-centred design, Int J Hum Comput Stud source

      Not checked in the free run.

  • Premisecitation

    Rapid change in the wider technology landscape requires digital health interventions to constantly evolve and be updated just to remain useful.

    “Difficulties include the rapid change of the wider technology landscape (Patrick 2016), which requires DHI’s to constantly evolve and be updated just to remain useful, let alone improve.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Patrick 2016

      Not checked in the free run.

  • Interpretationreasoning

    Effects seen when a digital health intervention is used in a controlled environment may not be replicated in real-world use.

    “For example, effects seen when a DHI is used in a controlled environment (laboratory or clinical office) may not be replicated if used in the “wild”, with many competing demands on user’s attention.”
    Quote in the paperSubstantial claim
  • Premisecitation

    Digital health interventions can improve outcomes in people with long-term conditions such as cardiovascular disease, diabetes, and mental health conditions.

    “improve outcomes in people with long term conditions 7 such as cardiovascular disease (McClean in press), diabetes 8 and mental health conditions 9”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [7] McLean, 2016, Digital interventions to promote self-management in adults with hypertension systematic review and meta-analysis, J Hypertens source

      Not checked in the free run.

    • cites [8] Pal, 2013, Computer-based diabetes self-management interventions for adults with type 2 diabetes mellitus, Cochrane Database Syst Rev

      Not checked in the free run.

    • cites [9] Hollis, 2015, Technological innovations in mental healthcare: harnessing the digital revolution, Br J Psychiatry source

      Not checked in the free run.

  • Premisecitation

    Randomised controlled trials have repeatedly overturned assumptions drawn from observational or non-randomised studies.

    “The importance of undertaking a RCT, and not relying solely on formative studies is evidenced by the fact that RCTs have repeatedly overturned assumptions drawn from observational or non-randomised studies (e.g.50, 51).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [50] Martin CA, Deshpande S, Hekler EB, Rivera DE. A system identification approach for improving behavioral interventions based on Social Cognitive Theory. Paper presented at: American Control Conference (ACC), July 1-3, 2015; Chicago, IL. source

      Not checked in the free run.

    • cites [51] Mohr, 2015, Trials of intervention principles: evaluation methods for evolving behavioral intervention technologies, J Med Internet Res source

      Not checked in the free run.

  • Classified as probably not a substantial claim. 15
  • May combine two claims. 18
  • The quote only partly states this claim. 1

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    System identification approaches obtain dynamic systems models that are the basis for designing control systems achieving optimisation.

    “System identification approaches are used in engineering to obtain dynamic systems models; these in turn are the basis for the design of control systems which achieve optimization. 45”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [45] Collins, 2014, Factorial experiments: efficient tools for evaluation of intervention components, Am J Prev Med source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Evaluations of digital health interventions present special challenges.

    “However, evaluations of DHI present special challenges.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The potential of digital health interventions has scarcely been realised, partly because of difficulties in generating an accumulating knowledge base to guide decisions about them.

    “To date, the potential of DHIs has scarcely been realized, partly because of difficulties generating an accumulating knowledge-base for guiding decisions about DHI’s.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The public, patients, clinicians, policy-makers and healthcare commissioners all have to make decisions about digital health interventions now.

  • Classified as probably not a substantial claim.

    Interpretationnone given

    Randomised controlled trials are important for evaluating the effectiveness and cost-effectiveness of digital health interventions.

    “While Randomised Controlled Trials (RCTs) are important for evaluation of effectiveness and cost-effectiveness”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Because digital health interventions lie at the intersection of biomedical, behavioural, computing and engineering research, methods from all these disciplines are needed.

    “As DHI are at the intersection of biomedical, behavioural, computing and engineering research, methods drawn from all these disciplines are required.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation

    Experimental approaches for optimisation include full or fractional factorial experiments, sequential multiple-assignment randomised trials, and system identification techniques.

    “The experimental approaches used for optimization include full or fractional factorial experiments 38, 39, the sequential multiple-assignment randomized trial (SMART), 40 and system identification techniques 41, 42.”
    Quote in the paperNot a substantial claim0 of 5 sources checked
    • cites [38] Caldwell, 2012, Translational research in South Africa: evaluating implementation quality using a factorial design, Child Youth Care Forum source

      Not checked in the free run.

    • cites [39] Strecher, 2008, Web-based smoking-cessation programs: results of a randomized trial, Am J Prev Med source

      Not checked in the free run.

    • cites [40] Collins, 2009, Design of experiments with multiple independent variables: a resource management perspective on complete and reduced factorial designs, Psychol Methods source

      Not checked in the free run.

    • cites [41] Dziak, 2012, Multilevel factorial experiments for developing behavioral interventions: power, sample size, and resource considerations, Psychol Methods source

      Not checked in the free run.

    • cites [42] Almirall, 2014, Introduction to SMART designs for the development of adaptive interventions: with application to weight loss research, Transl Behav Med source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    Achieving improvements in health and healthcare from digital health interventions requires a cumulative knowledge base to inform their development and deployment.

    “Achieving these improvements requires a cumulative knowledge base to inform development and deployment of DHI.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Digital health interventions have enormous potential as scalable tools to improve health and healthcare delivery by improving effectiveness, efficiency, accessibility, safety and personalisation.

  • Classified as probably not a substantial claim.

    Interpretationnone given

    For many digital health interventions, users include healthcare professionals who prescribe the intervention and monitor outcomes.

    “For many DHI, ‘users’ will include healthcare professionals (HCPs) who ‘prescribe’ the DHI and monitor outcomes.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationcitation

    Factorial experimental design is useful and economical for examining effects of individual intervention components and is the only experimental design enabling full examination of all interactions.

    “The factorial experimental design can be a useful and economical approach for examining the effects of individual intervention components, and is the only experimental design that enables full examination of all interactions. 38”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [38] Caldwell, 2012, Translational research in South Africa: evaluating implementation quality using a factorial design, Child Youth Care Forum source

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    Reach, uptake and context are particularly salient for digital health interventions because impact and cost-effectiveness depend on the total number of users, and effectiveness may depend on context.

    “The concepts of reach, uptake and context are particularly salient for DHI, as impact and cost-effectiveness are highly dependent on the total number of users (McNamee 2016), and effectiveness may be highly dependent on context.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites McNamee 2016 source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Most digital health interventions are highly complex with multiple components, so their development needs a period of optimisation.

    “Most DHI are highly complex interventions, containing multiple components so the development process needs to include a period of optimisation.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    The emphasis in trials of pharmaceutical products is on internal validity and reducing bias, and extensive work has confirmed the importance of this.

    “The emphasis in trials of pharmaceutical products is on internal validity and reducing bias, and extensive work has confirmed the importance of this. 54”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [54] Kennedy-Martin, 2015, A literature review on the representativeness of randomized controlled trial samples and implications for the external validity of trial results, Trials source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given

    Digital health interventions are typically complex interventions with multiple components and multiple aims.

    “They are typically complex interventions with multiple components, and many have multiple aims”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationnone given

    System identification experiments are idiographic and work best when planned changes are introduced to adjustable components of an intervention.

    “System identification experiments are inherently idiographic in nature, and work best when planned changes (preferably random or pseudo-random in nature) are introduced to adjustable components of an intervention (e.g. dosages).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationnone given

    Idiosyncratic wants and needs of users and the influence of context on effectiveness are difficulties for evaluating digital health interventions.

    “Other difficulties include the idiosyncratic wants and needs of users and the influence of context on effectiveness.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given

    Once research questions 3-6 have been addressed, the research team are likely to be able to estimate the direction and magnitude of the effect of the digital health intervention.

    “Once RQ 3–6 have been addressed, the research team are likely to be able to estimate the direction and magnitude of the effect of the DHI.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationcitation

    Deciding how to balance external and internal validity is a challenge for many trials and is particularly salient for trials of digital health interventions.

    “Deciding how to balance external and internal validity is a challenge for many trials 52, but is particularly salient for trials of DHI.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [52] Baron, 2015, A trial of calcium and vitamin D for the prevention of colorectal adenomas, N Engl J Med source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Digital health interventions are not harm free, although data on actual harms are relatively sparse.

    “Digital health interventions are not harm free, although to date, the data on actual harms are relatively sparse.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationcitation

    Specification of digital health interventions is required for replication of trial results, comparison between interventions, synthesising data across trials in systematic reviews and meta-analyses, and may help determine criteria for substantial equivalence.

    “Such specification is required for replication of trial results, comparison between DHI, synthesising data across trials in systematic reviews and meta-analyses 55 and may help with determining the criteria for ‘substantial equivalence’ of digital interventions.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [55] Rothwell, 2005, External validity of randomised controlled trials: “to whom do the results of this trial apply?”, Lancet source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationnone given

    Identification and quantification of expected harms can be undertaken as part of a randomised controlled trial, but unexpected harms require alternative strategies for identification and quantification.

    “Identification and quantification of expected harms (such as increased anxiety) can be undertaken as part of an RCT, but unexpected harms will require alternative strategies for identification and quantification.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationnone given

    Detailed specification of a digital health intervention is important but may be hard to achieve, particularly where there is a high degree of tailoring, adaptive learning and user choice.

    “Detailed specification of the DHI is important, but may be hard to achieve, particularly where there is a high degree of tailoring, adaptive learning and user choice.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    In pragmatic trials aiming to determine the effectiveness of a new treatment compared to current best practice, the comparator is typically treatment as usual.

    “In pragmatic trials which aim to determine the effectiveness of a new treatment compared to current best practice, the comparator is typically ‘treatment as usual’ (TAU).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    How best to combine rigor with efficiency in evaluating digital health interventions requires a great deal of methodological research.

    “How best to combine rigor with efficiency in evaluating DHI requires a great deal of methodological research.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationnone given

    Embedding data collection into the digital health intervention in randomised controlled trials may introduce systematic bias or confound the intervention with the measurement method.

    “There is a temptation in RCTs of DHI to embed data collection into the intervention, but this may introduce systematic bias or confound the intervention with the measurement method.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    Answers to the previous ten questions should enable an assessment of the overall utility of the digital health intervention and confidence in this assessment.

    “Answers to the previous ten questions should enable an assessment of the overall utility of the DHI (e.g., balancing its effects, usage, scalability, costs and safety), along with an estimate of how confidence in this assessment.”
    Quote in the paperNot a substantial claim

  • May combine two claims. The quote only partly states this claim.

    Interpretationcitation

    Digital health interventions can result in harm through various mechanisms, including being designed for harmful outcomes, making fraudulent claims, containing inaccurate information, misinterpretation of accurate information, opportunity costs, disillusionment, and data leaks.

    “There are various mechanisms by which DHI could result in harm.”
    • Also stated in: § Defining the likely benefit of the DHI · hedged
      “Finally, DHIs may ‘leak’ personal data because of inadequate security and encryption functions. 59”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [58] 2014

      Not checked in the free run.

    • cites [59] Khadjesari, 2015, Negotiating the ‘grey area between normal social drinking and being a smelly tramp’: a qualitative study of people searching for help online to reduce their drinking, Health Expect source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationnone given

    Selection of a suitable comparator is determined by the research question addressed, which varies with the stage of the research.

    “The selection of a suitable comparator is determined by the research question addressed, which will vary with the stage of the research.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationcitation

    In trials of digital health interventions, participants in the treatment-as-usual group may access many other digital interventions and may search online for better resources, which risks undermining the trial.

    “However, in trials of DHIs, the participants in the TAU group may have access to a myriad of other digital interventions.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [57] Hoffmann, 2014, Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide, BMJ source

      Not checked in the free run.

28 of 28

  • The paper relies on this without support.

    Digital health interventions have enormous potential as scalable tools to improve health and healthcare delivery by improving effectiveness, efficiency, accessibility, safety and personalisation.

    “Digital health interventions (DHI) have enormous potential as scalable tools to improve health and healthcare delivery by improving effectiveness, efficiency, accessibility, safety and personalisation.”

  • The paper relies on this without support.

    The public, patients, clinicians, policy-makers and healthcare commissioners all have to make decisions about digital health interventions now.

    “However, the public, patients, clinicians, policy-makers and healthcare commissioners all have to make decisions on DHI now”

  • The paper relies on this without support.

    The wider technological landscape is likely to continue to evolve.

    “The wider technological landscape is likely to continue to evolve, and investigators must judge what impact this will have on the generalisability of their findings.”
3 of 3

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Digital interventions, such as smartphone apps, offer an alternative approach with potential for broader reach and greater acceptance.

    “Digital interventions, such as smartphone applications (“apps”), offer an alternative approach, with the potential for broader reach and greater acceptance [40, 41].”

    From Can an Acceptance and Commitment Therapy-Based Smartphone App Help Individuals with Mental Health Disorders Quit Smoking? (2024) · cites this paper as [40]

    The Background & Aims section describes smartphone-based interventions as potentially scalable and able to provide remote access to treatment, supporting the broader-reach part of the claim. The Research Questions section treats acceptability as something to assess and improve; it does not say digital interventions have greater acceptance than alternatives. Page numbers were not available in our copy. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Digital health interventions (DHIs) have already been used in a variety of contexts for promoting well-being.

    “DHIs have already been used in a variety of contexts for promoting well-being, from delivering healthcare and education to personalised diet and fitness plans.”

    From Mixed-methods feasibility outcomes for a novel ACT-based video game ‘ACTing Minds’ to support mental health (2024) · cites this paper as [10]

  • The benefits of DMH interventions, including increased engagement and motivation, fidelity to intervention protocols, and opportunities for personalization, remain largely unrealized.

    “Specifically, the benefits of DMH interventions, including increased engagement and motivation, fidelity to intervention protocols, and opportunities for personalization [12,48,50,53], remain largely unrealized.”

    From Use of the Principles of Design Thinking to Address Limitations of Digital Mental Health Interventions for Youth: Viewpoint (2019) · cites this paper as [48]

  • DMH interventions are content focused and not user focused, resulting in a large disconnect between the world in which youth live and the content and style of DMH interventions.

    “In addition, DMH interventions are content focused (ie, CBT techniques) and not user focused (ie, they are not designed around how and when young people prefer to engage with digital experiences) [48], resulting in a large disconnect between the world in which youth live and the content and style of DMH interventions.”

    From Use of the Principles of Design Thinking to Address Limitations of Digital Mental Health Interventions for Youth: Viewpoint (2019) · cites this paper as [48]

Cites

Flagged first, then by citations

Research works printed but not linked: 3

  1. Patrick, 2016, Rapid rate of technological development and its implications for research on digital behavior interventions, Am J Prev Med
  2. Michie S, West R. Guide to Development and Evaluation of Digital Behaviour Change Interventions in Healthcare. London: UCL Centre for Behaviour Change; In press
  3. Hekler EB, Michie S, Rivera DE, et al. Developing and refining models and theories suitable for digital health interventions. Am J Prev Med. In press.

Cited by

Sorted by citations

Related

Authors
Murray, E., Hekler, E. B., Andersson, G., Collins, L. M., Doherty, A., Hollis, C., Rivera, D. E., West, R., & Wyatt, J. C.
Affiliations
  • University College London (GB)
  • Arizona State University (US)
  • Linköping University (SE)
  • Pennsylvania State University (US)
  • University of Oxford (GB)
  • University of Nottingham (GB)
  • University of Southampton (GB)
Corresponding author
Murray, Elizabeth
ORCID
Venue
American Journal of Preventive Medicine
Publisher
Elsevier BV
Published
2016-11
Volume
51
Issue
5
Pages
843-851
ISSN
0749-3797 (ISSN-L)
DOI
10.1016/j.amepre.2016.06.008
PMC
PMC5324832
OpenAlex
W2530268835
Cited by
847
References
93
Citations by year
2026: 115 · 2025: 134 · 2024: 113 · 2023: 97 · 2022: 117 · 2021: 117 · 2020: 64 · 2019: 44 · 2018: 26 · 2017: 14 · 2016: 5
FWCI
55.18
Type
journal article
Language
en
Open access
Licence
www.elsevier.com/tdm/userlicense/1.0/ (tdm)
Locations
  • American Journal of Preventive Medicine (publishedVersion)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
  • ePrints Soton (University of Southampton) (acceptedVersion)
  • UCL Discovery (University College London) (submittedVersion)
Funders
  • National Science Foundation (1539846)
  • Robert Wood Johnson Foundation
  • Cancer Research UK (14135)
  • National Institute for Health and Care Research (RP-PG-0609-10135, NF-SI-0514-10119)
  • British Heart Foundation (RE/13/1/30181)
  • National Institutes of Health
  • Medical Research Council (MR/K023195/1, MR/L004933/1, MR/L004933/2)
  • Economic and Social Research Council (ES/G007489/1)
  • National Institute on Drug Abuse (P50DA039838)
  • Office of Behavioral and Social Sciences Research
Topics
  • Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
  • Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Health Systems, Economic Evaluations, Quality of Life
Keywords
digital health interventions, assessment methods, research questions, randomized controlled trials, cost-effectiveness, intervention effectiveness, harm, implementation fidelity, causal models, scalability, health policy decision-making
MeSH terms
Health Promotion, Humans, Research Design, Telecommunications, Program Evaluation, Health Behavior

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read under definition version 3 (extract-v3) on 2026-09-28.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-09-28.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review