Cited by 149Mobile Health and mHealth ApplicationsHealth Policy Implementation ScienceDigital Mental Health Interventions

Measuring the implementation of behavioral intervention technologies: Recharacterization of established outcomes

Open access
Hermes, Eric DA, Lyon, Aaron R, Schueller, Stephen M, Glass, Joseph E·2019·Journal of Medical Internet Research

Published version of a preprint

Abstract

Behavioral intervention technologies (BITs) are websites, software, mobile apps, and sensors designed to help users address or change behaviors, cognitions, and emotional states. BITs have the potential to transform health care delivery, and early research has produced promising findings of efficacy. BITs also favor new models of health care delivery and provide novel data sources for measurement. However, there are few examples of successful BIT implementation and a lack of consensus on as well as inadequate descriptions of BIT implementation measurement. The aim of this viewpoint paper is to provide an overview and characterization of implementation outcomes for the study of BIT use in routine practice settings. Eight outcomes for the evaluation of implementation have been previously described: acceptability, adoption, appropriateness, feasibility, fidelity, implementation cost, penetration, and sustainability. In a proposed recharacterization of these outcomes with respect to BIT implementation, definitions are clarified, expansions to the level of analysis are identified, and unique measurement characteristics are discussed. Differences between BIT development and implementation, an increased focus on consumer-level outcomes, the expansion of providers who support BIT use, and the blending of BITs with traditional health care services are specifically discussed. BITs have the potential to transform health care delivery. Realizing this potential, however, will hinge on high-quality research that consistently and accurately measures how well such technologies have been integrated into health services. This overview and characterization of implementation outcomes support BIT research by identifying and proposing solutions for key theoretical and practical measurement challenges.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.2196/11752

Cites

72 of 72 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The study draws from recent recommendations in the implementation science literature on characterizing implementation outcomes for behavioral intervention technology.

    “The study draws from recent recommendations in the implementation science literature on characterizing implementation outcomes for behavioral intervention technology (Hermes et al., 2019).”

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

1 of the 73 references couldn’t be matched to a known paper.

Research works printed but not linked: 3

  1. [3]England MJ Butler AS Gonzalez ML Psychosocial interventions for mental and substance use disorders: A framework for establishing evidence-based standards 2015 Washington, DC National Academy Press 209 26203478
  2. [44]Grudin J Pruitt J Personas, participatory design and product development: An Infrastructure for Engagement Proceedings of Participation and Design Conference (PDC2002) 2002 Participation and Design Conference (PDC2002) June 23-25, 2002 Sweden 144 61
  3. [48]Rubin J Chisnell D Spool J Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests 2008 New York, NY Wiley, John & Sons 384

Cited by

Sorted by citations

Authors
Hermes, E. D., Lyon, A. R., Schueller, S. M., & Glass, J. E.
Affiliations
  • Yale University (US)
  • University of Washington (US)
  • University of California, Irvine (US)
  • Kaiser Permanente Washington Health Research Institute (US)
Corresponding author
Hermes, Eric DA
Venue
Journal of Medical Internet Research
Published
2019-01-25
Volume
21
Issue
1
Pages
e11752
ISSN
1438-8871 (ISSN-L)
OpenAlex
W2897648388
Cited by
149
References
73
Citations by year
2026: 12 · 2025: 25 · 2024: 19 · 2023: 23 · 2022: 24 · 2021: 23 · 2020: 18 · 2019: 5
FWCI
17.56
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Journal of Medical Internet Research (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
  • National Institute on Alcohol Abuse and Alcoholism (K01AA023859)
  • National Institute on Minority Health and Health Disparities (L60 MD009373)
Topics
  • Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
  • Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
Keywords
implementation outcomes, implementation research, acceptability, adoption, appropriateness, feasibility, fidelity, implementation costs, penetration, sustainability, health care delivery, digital health, measurement challenges
MeSH terms
Behavior Therapy, Humans, Technology, Translational Research, Biomedical, Mobile Applications

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review