Measures of fidelity of delivery of, and engagement with, complex, face‐to‐face health behaviour change interventions: A systematic review of measure quality
Open accessThe DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Abstract
Purpose: Understanding the effectiveness of complex, face‐to‐face health behaviour change interventions requires high‐quality measures to assess fidelity of delivery and engagement. This systematic review aimed to (1) identify the types of measures used to monitor fidelity of delivery of, and engagement with, complex, face‐to‐face health behaviour change interventions and (2) describe the reporting of psychometric and implementation qualities. Methods: Electronic databases were searched, systematic reviews and reference lists were hand‐searched, and 21 experts were contacted to identify articles. Studies that quantitatively measured fidelity of delivery of, and/or engagement with, a complex, face‐to‐face health behaviour change intervention for adults were included. Data on interventions, measures, and psychometric and implementation qualities were extracted and synthesized using narrative analysis. Results: Sixty‐six studies were included: 24 measured both fidelity of delivery and engagement, 20 measured fidelity of delivery, and 22 measured engagement. Measures of fidelity of delivery included observation ( n = 17; 38.6%), self‐report ( n = 15; 34%), quantitatively rated qualitative interviews ( n = 1; 2.3%), or multiple measures ( n = 11; 25%). Measures of engagement included self‐report ( n = 18; 39.1%), intervention records ( n = 11; 24%), or multiple measures ( n = 17; 37%). Fifty‐one studies (77%) reported at least one psychometric or implementation quality; 49 studies (74.2%) reported at least one psychometric quality, and 17 studies (25.8%) reported at least one implementation quality. Conclusion: Fewer than half of the reviewed studies measured both fidelity of delivery of, and engagement with complex, face‐to‐face health behaviour change interventions. More studies reported psychometric qualities than implementation qualities. Interpretation of intervention outcomes from fidelity of delivery and engagement measurements may be limited due to a lack of reporting of psychometric and implementation qualities. Statement of contribution: What is already known on this subject? Evidence of fidelity and engagement is needed to understand effectiveness of complex interventions Evidence of fidelity and engagement are rarely reported High‐quality measures are needed to measure fidelity and engagement What does this study add? Evidence that indicators of quality of measures are reported in some studies Evidence that psychometric qualities are reported more frequently than implementation qualities A recommendation for intervention evaluations to report indicators of quality of fidelity and engagement measures
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Durlak, J. A., & DuPre, E. P. (2008) · American Journal of Community Psychology · doi:10.1007/s10464-008-9165-0 · Cited by 5,304
The Delphi Technique: Making Sense of Consensus
Hsu, C.-C., & Sandford, B. A. (2007) · University of Massachusetts Amherst · doi:10.7275/pdz9-th90 · Cited by 4,500 · cited 1×
“This systematic method could be used for building a consensus (Hsu & Sandford, 2007) regarding which psychometric and implementation qualities are most important, and which qualities should be given the most weighting when developing and evaluating fidelity of delivery and engagement measures.”
MethodHow we design feasibility studies
Bowen, D. J., Kreuter, M., Spring, B., Cofta-Woerpel, L., Linnan, L., Weiner, D., Bakken, S., Kaplan, C. P., Squiers, L., Fabrizio, C., et al. (2009) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2009.02.002 · Cited by 4,021
Framework for design and evaluation of complex interventions to improve health
Campbell, M. (2000) · BMJ · doi:10.1136/bmj.321.7262.694 · Cited by 3,474 · cited 5×
“5%)(31) Baranowski & Stables (2000): (n = 2; 3.3%)(42,50) Saunders et al. (2005) (n = 1; 1.”
InfluentialContent Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability
Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00826.x · Cited by 3,068
Bellg, A. J., Treatment Fidelity Workgroup of the NIH Behavior Change Consortium, Borrelli, B., Resnick, B., Hecht, J., Minicucci, D. S., Ory, M., Ogedegbe, G., Orwig, D., Ernst, D., et al. (2004) · Health Psychology · doi:10.1037/0278-6133.23.5.443 · Cited by 2,620
A conceptual framework for implementation fidelity
Carroll, C., Patterson, M., Wood, S., Booth, A., Rick, J., & Balain, S. (2007) · Implementation Science · doi:10.1186/1748-5908-2-40 · Cited by 2,559
Popay, J., Roberts, H., Sowden, A., Petticrew, M., Arai, L., Rodgers, M., Britten, N., Roen, K., & Duffy, S. (2006) · Lancaster University · doi:10.13140/2.1.1018.4643 · Cited by 2,070 · cited 1×
“Due to the heterogeneity of studies, a descriptive rather than quantitative synthesis of data was conducted (Deeks, Higgins, & Altman, 2008; Popay et al., 2006).”
Analysing Data and Undertaking Meta‐Analyses
Deeks, J. J., Higgins, J. P., & Altman, D. G. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch9 · Cited by 1,800
Process evaluation in randomised controlled trials of complex interventions
Oakley, A., Strange, V., Bonell, C., Allen, E., & Stephenson, J. (2006) · BMJ · doi:10.1136/bmj.332.7538.413 · Cited by 1,247 · cited 1×
“…than implementation qualities A recommendation for intervention evaluations to report indicators of quality of fidelity and engagement measures Most interventions aimed at changing health behaviours are complex in that they contain multiple components (Campbell et al., 2000; Oakley et al., 2006).”
BackgroundSaunders, R. P., Evans, M. H., & Joshi, P. (2005) · Health Promotion Practice · doi:10.1177/1524839904273387 · Cited by 1,031
Treatment fidelity in outcome studies
Moncher, F. J., & Prinz, R. J. (1991) · Clinical Psychology Review · doi:10.1016/0272-7358(91)90103-2 · Cited by 797 · cited 2×
“Furthermore, researchers have previously outlined some strengths and weaknesses of different measures of fidelity of delivery and engagement (e.g., Borrelli, 2011; Breitenstein et al., 2010; Moncher & Prinz, 1991).”
BackgroundDealing with missing data in a multi-question depression scale: a comparison of imputation methods
Shrive, F. M., Stuart, H., Quan, H., & Ghali, W. A. (2006) · BMC Medical Research Methodology · doi:10.1186/1471-2288-6-57 · Cited by 650
Roy-Byrne, P., Craske, M. G., Sullivan, G., Rose, R. D., Edlund, M. J., Lang, A. J., Bystritsky, A., Welch, S. S., Chavira, D. A., Golinelli, D., et al. (2010) · JAMA · doi:10.1001/jama.2010.608 · Cited by 452
Farmer, A., Wade, A., Goyder, E., Yudkin, P., French, D., Craven, A., Holman, R., Kinmonth, A.-L., & Neil, A. (2007) · BMJ · doi:10.1136/bmj.39247.447431.be · Cited by 407
Smith, D. E., Heckemeyer, C. M., Kratt, P. P., & Mason, D. A. (1997) · Diabetes Care · doi:10.2337/diacare.20.1.52 · Cited by 359
TREATMENT INTEGRITY IN APPLIED BEHAVIOR ANALYSIS WITH CHILDREN
Gresham, F. M., Gansle, K. A., & Noell, G. H. (1993) · Journal of Applied Behavior Analysis · doi:10.1901/jaba.1993.26-257 · Cited by 299
Miller, W. R., & Rollnick, S. (2014) · Contemporary Clinical Trials · doi:10.1016/j.cct.2014.01.005 · Cited by 262 · cited 1×
“However, some qualities which are recommended by research were not frequently reported; an example of this is routine audio-recording (Gresham, Gansle, & Noell, 1993; Miller & Rollnick, 2014).”
BackgroundCampbell, N. C., Ritchie, L. D., Thain, J., Deans, H. G., Rawles, J. M., & Squair, J. L. (1998) · Heart · doi:10.1136/hrt.80.5.447 · Cited by 258
Taking healthcare interventions from trial to practice
Glasziou, P., Chalmers, I., Altman, D. G., Bastian, H., Boutron, I., Brice, A., Jamtvedt, G., Farmer, A., Ghersi, D., Groves, T., et al. (2010) · BMJ · doi:10.1136/bmj.c3852 · Cited by 209
Self-citationOckene, I. S., Tellez, T. L., Rosal, M. C., Reed, G. W., Mordes, J., Merriam, P. A., Olendzki, B. C., Handelman, G., Nicolosi, R., & Ma, Y. (2012) · American Journal of Public Health · doi:10.2105/ajph.2011.300357 · Cited by 201
Treatment Fidelity in Behavior Change Research: A Case Example
Resnick, B., Inguito, P., Orwig, D., Yahiro, J. Y., Hawkes, W., Werner, M., Zimmerman, S., & Magaziner, J. (2005) · Nursing Research · doi:10.1097/00006199-200503000-00010 · Cited by 136 · cited 1×
“…fidelity model/NIH Behaviour change Consortium framework (Bellg et al., 2004) (n = 6; 9.1%)6,10,20,22,26,39 RE-AIM framework (n = 1; 1.5%)29 Resnick et al. (2005) (n = 1; 1.5%)31 Baranowski & Stables (2000): (n = 2; 3.3%)42,50 Saunders et al. (2005) (n = 1; 1.5%)42 Hasson (2010) based on…”
InfluentialBrug, J., Spikmans, F., Aartsen, C., Breedveld, B., Bes, R., & Fereira, I. (2007) · Journal of Nutrition Education and Behavior · doi:10.1016/j.jneb.2006.08.010 · Cited by 120
Rixon, L., Baron, J., McGale, N., Lorencatto, F., Francis, J., & Davies, A. (2016) · BMC Health Services Research · doi:10.1186/s12913-016-1904-6 · Cited by 117 · cited 6×
“Systematic reviews of measures used to monitor fidelity of delivery demonstrate that these measures have consistently been used in intervention research, in both educational (Maynard, Peters, Vaughn,& Sarteschi, 2013) and health settings (Rixon et al., 2016).”
BackgroundMethodInfluentialTraining and fidelity monitoring of behavioral interventions in multi-site addictions research
Baer, J. S., Ball, S. A., Campbell, B. K., Miele, G. M., Schoener, E. P., & Tracy, K. (2007) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2006.08.028 · Cited by 108 · cited 1×
“…that the reporting of psychometric qualities in the complex, face-to-face health behaviour change interventions included in this review,may not be as infrequent as previously suggested in different populations (Baer et al., 2007; Breitenstein et al., 2010; Maynard et al., 2013; Rixon et al., 2016).”
BackgroundA Brief Culturally Tailored Intervention for Puerto Ricans With Type 2 Diabetes
Osborn, C. Y., Amico, K. R., Cruz, N., O'Connell, A. A., Perez-Escamilla, R., Kalichman, S. C., Wolf, S. A., & Fisher, J. D. (2010) · Health Education & Behavior · doi:10.1177/1090198110366004 · Cited by 96
Montgomery, P., Underhill, K., Gardner, F., Operario, D., & Mayo-Wilson, E. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.03.006 · Cited by 92
Gabbay, R. A., Añel‐Tiangco, R. M., Dellasega, C., Mauger, D. T., Adelman, A., & Van Horn, D. H. A. (2013) · Journal of Diabetes · doi:10.1111/1753-0407.12030 · Cited by 85
A method for assessing fidelity of delivery of telephone behavioral support for smoking cessation
Lorencatto, F., West, R., Bruguera, C., & Michie, S. (2014) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0035149 · Cited by 52 · cited 3×
“Fidelity of delivery has been assessed by self-report measures (Bellg et al., 2004), and by audio-recording, which is considered to be the gold standard (Bellg et al., 2004; Borrelli, 2011; Lorencatto et al., 2014).”
MethodSelf-citationLorencatto, F., West, R., Seymour, N., & Michie, S. (2013) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0032106 · Cited by 45
Self-citationFidelity in After-School Program Intervention Research: A Systematic Review
Maynard, B. R., Peters, K. E., Vaughn, M. G., & Sarteschi, C. M. (2013) · Research on Social Work Practice · doi:10.1177/1049731513491150 · Cited by 30 · cited 3×
“A systematic reviewof fidelity of delivery in after-school programmes found that no studies reported reliability (Maynard et al., 2013), and a systematic review of intervention receipt in health research found that 26% of studies reported on reliability and validity (Rixon et al., 2016).”
Background
Cited by
The Handbook of Behavior Change
(2020) · Cambridge University Press eBooks · doi:10.1017/9781108677318 · Cited by 407
Short, C. E., DeSmet, A., Woods, C., Williams, S. L., Maher, C., Middelweerd, A., Müller, A. M., Wark, P. A., Vandelanotte, C., Poppe, L., et al. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.9397 · Cited by 400
“Some researchers have defined engagement solely as a psychological process relating to user perceptions and experience, whereas others consider engagement a purely behavioral construct, synonymous with intervention usage [4,7].”
BackgroundChanging Behavior Using the Theory of Planned Behavior
Ajzen, I., & Schmidt, P. (2020) · The Handbook of Behavior Change · doi:10.1017/9781108677318.002 · Cited by 266
Changing Behavior Using the Model of Action Phases
Keller, L., Gollwitzer, P. M., & Sheeran, P. (2020) · The Handbook of Behavior Change · doi:10.1017/9781108677318.006 · Cited by 147
Saban, M., Dagan, E., & Drach-Zahavy, A. (2021) · Journal of Emergency Nursing · doi:10.1016/j.jen.2020.09.009 · Cited by 24
Metadata
- Authors
- Walton, H., Spector, A., Tombor, I., & Michie, S.
- Venue
- British Journal of Health Psychology
- Publisher
- Wiley
- Published
- 2017-08
- Volume
- 22
- Issue
- 4
- Pages
- 872-903
- ISSN
- 1359-107X (ISSN-L)
- 2044-8287
- OpenAlex
- W2740999512
- Cited by
- 238
- References
- 32
- FWCI
- 5.53
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.