CoChat

A systematic review of measures of implementation players and processes: Summarizing the dearth of psychometric evidence

Open access

The DOI matches its Crossref record; 91 of the 91 references it cites were found in a registry or an index.

Abstract

Background: Measurement is a critical component for any field. Systematic reviews are a way to locate measures and uncover gaps in current measurement practices. The present study identified measures used in behavioral health settings that assessed all constructs within the Process domain and two constructs from the Inner setting domain as defined by the Consolidated Framework for Implementation Research (CFIR). While previous conceptual work has established the importance social networks and key stakeholders play throughout the implementation process, measurement studies have not focused on investigating the quality of how these activities are being carried out. Methods: The review occurred in three phases: Phase I, data collection included (1) search string generation, (2) title and abstract screening, (3) full text review, (4) mapping to CFIR-constructs, and (5) “cited-by” searches. Phase II, data extraction, consisted of coding information relevant to the nine psychometric properties included in the Psychometric And Pragmatic Rating Scale (PAPERS). In Phase III, data analysis was completed. Results: Measures were identified in only seven constructs: Structural characteristics ( n = 13), Networks and communication ( n = 29), Engaging ( n = 1), Opinion leaders ( n = 5), Champions ( n = 5), Planning ( n = 5), and Reflecting and evaluating ( n = 5). No quantitative assessment measures of Formally appointed implementation leaders, External change agents, or Executing were identified. Internal consistency and norms were reported on most often, whereas no studies reported on discriminant validity or responsiveness. Not one measure in the sample reported all nine psychometric properties evaluated by the PAPERS. Scores in the identified sample of measures ranged from “-2” to “10” out of a total of “36.” Conclusions: Overall measures demonstrated minimal to adequate evidence and available psychometric information was limited. The majority were study specific, limiting their generalizability. Future work should focus on more rigorous measure development and testing of currently existing measures, while moving away from creating new, single use measures. Plain Language Summary: How we measure the processes and players involved for implementing evidence-based interventions is crucial to understanding what factors are helping or hurting the intervention’s use in practice and how to take the intervention to scale. Unfortunately, measures of these factors—stakeholders, their networks and communication, and their implementation activities—have received little attention. This study sought to identify and evaluate the quality of these types of measures. Our review focused on collecting measures used for identifying influential staff members, known as opinion leaders and champions, and investigating how they plan, execute, engage, and evaluate the hard work of implementation. Upon identifying these measures, we collected all published information about their uses to evaluate the quality of their evidence with respect to their ability to produce consistent results across items within each use (i.e., reliable) and if they assess what they are intending to measure (i.e., valid). Our searches located over 40 measures deployed in behavioral health settings for evaluation. We observed a dearth of evidence for reliability and validity and when evidence existed the quality was low. These findings tell us that more measurement work is needed to better understand how to optimize players and processes for the purposes of successful implementation.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1177/26334895211002474

Cites

91 of 91 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1. Structural characteristics: head-to-head comparison of measures or scales.
Figure 1. Structural characteristics: head-to-head comparison of measures or scales.CC BY-NC · PubMed Central
Figure 2. Networks and communications: head-to-head comparison of measures or scales.
Figure 2. Networks and communications: head-to-head comparison of measures or scales.CC BY-NC · PubMed Central
Figure 3. General process: head-to-head comparison of measures or scales.
Figure 3. General process: head-to-head comparison of measures or scales.CC BY-NC · PubMed Central
Figure 4. Planning: head-to-head comparison of measures or scales.
Figure 4. Planning: head-to-head comparison of measures or scales.CC BY-NC · PubMed Central
Figure 5. Engaging, opinion leaders, and champions: head-to-head comparison of measures or scales.
Figure 5. Engaging, opinion leaders, and champions: head-to-head comparison of measures or scales.CC BY-NC · PubMed Central
Figure 6. Reflecting and evaluating: head-to-head comparison of measures or scales.
Figure 6. Reflecting and evaluating: head-to-head comparison of measures or scales.CC BY-NC · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 91 references

Cited by

See all 19

Metadata

Authors
Dorsey, C. N., Mettert, K. D., Puspitasari, A. J., Damschroder, L. J., & Lewis, C. C.
Affiliations
  • Kaiser Permanente Washington Health Research Institute (US)
  • Mayo Clinic (US)
  • VA Ann Arbor Healthcare System (US)
Corresponding author
Dorsey, Caitlin N
Venue
Implementation Research and Practice
Published
2021-01
Volume
2
Pages
26334895211002474-26334895211002474
ISSN
2633-4895 (ISSN-L)
OpenAlex
W3144193208
Cited by
15
References
91
Citations by year
2026: 1 · 2025: 6 · 2024: 1 · 2023: 1 · 2022: 5 · 2021: 1
FWCI
0.41
Type
journal article
Language
en
Open access
Locations
  • Implementation Research and Practice (publishedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
National Institute of Mental Health (R01MH106510)
Topics
  • Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
  • Health Sciences › Nursing › Research and Theory › Health Education and Validation
  • Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
Keywords
implementation research, Consolidated Framework for Implementation Research, psychometric properties, opinion leaders, implementation champions, stakeholder networks, social networks, implementation leadership, implementation planning, implementation evaluation, behavioral health

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A systematic review of measures of implementation players and processes: Summarizing the dearth of psychometric evidence. https://app.cochat.ai/papers/10.1177/26334895211002474
The paper
Dorsey, C. N., Mettert, K. D., Puspitasari, A. J., Damschroder, L. J., & Lewis, C. C. (2021). A systematic review of measures of implementation players and processes: Summarizing the dearth of psychometric evidence. Implementation Research and Practice. https://doi.org/10.1177/26334895211002474

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.