Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index; 7 of the 26 sources its claims cite were checked for support.

Psychometric assessment of three newly developed implementation outcome measures

Open access

Abstract

BACKGROUND: Implementation outcome measures are essential for monitoring and evaluating the success of implementation efforts. Yet, currently available measures lack conceptual clarity and have largely unknown reliability and validity. This study developed and psychometrically assessed three new measures: the Acceptability of Intervention Measure (AIM), Intervention Appropriateness Measure (IAM), and Feasibility of Intervention Measure (FIM). METHODS: Thirty-six implementation scientists and 27 mental health professionals assigned 31 items to the constructs and rated their confidence in their assignments. The Wilcoxon one-sample signed rank test was used to assess substantive and discriminant content validity. Exploratory and confirmatory factor analysis (EFA and CFA) and Cronbach alphas were used to assess the validity of the conceptual model. Three hundred twenty-six mental health counselors read one of six randomly assigned vignettes depicting a therapist contemplating adopting an evidence-based practice (EBP). Participants used 15 items to rate the therapist's perceptions of the acceptability, appropriateness, and feasibility of adopting the EBP. CFA and Cronbach alphas were used to refine the scales, assess structural validity, and assess reliability. Analysis of variance (ANOVA) was used to assess known-groups validity. Finally, half of the counselors were randomly assigned to receive the same vignette and the other half the opposite vignette; and all were asked to re-rate acceptability, appropriateness, and feasibility. Pearson correlation coefficients were used to assess test-retest reliability and linear regression to assess sensitivity to change. RESULTS: All but five items exhibited substantive and discriminant content validity. A trimmed CFA with five items per construct exhibited acceptable model fit (CFI = 0.98, RMSEA = 0.08) and high factor loadings (0.79 to 0.94). The alphas for 5-item scales were between 0.87 and 0.89. Scale refinement based on measure-specific CFAs and Cronbach alphas using vignette data produced 4-item scales (α's from 0.85 to 0.91). A three-factor CFA exhibited acceptable fit (CFI = 0.96, RMSEA = 0.08) and high factor loadings (0.75 to 0.89), indicating structural validity. ANOVA showed significant main effects, indicating known-groups validity. Test-retest reliability coefficients ranged from 0.73 to 0.88. Regression analysis indicated each measure was sensitive to change in both directions. CONCLUSIONS: The AIM, IAM, and FIM demonstrate promising psychometric properties. Predictive validity assessment is planned.

Overview

CoChat Summary

## TL;DR

This paper developed and psychometrically tested three new implementation outcome measures: the Acceptability of Intervention Measure (AIM), Intervention Appropriateness Measure (IAM), and Feasibility of Intervention Measure (FIM). Across three studies, the measures showed promising content validity, factor structure, internal consistency, test-retest reliability, and sensitivity to change, though discriminant validity between acceptability and appropriateness remained uncertain.

## Key findings

  • In study 1, judges showed high agreement on item assignments: ICC = 0.89 across all 31 items, with construct-specific ICCs of 0.82 for acceptability, 0.94 for appropriateness, and 0.87 for feasibility.
  • For content validity, all but five items showed substantive and discriminant content validity; the paper specifically flags items such as “This EBP is good enough,” “This EBP will do,” “This EBP is pretty good,” “This EBP seems right,” and “This EBP seems right on the button” as lacking sufficient discriminant content validity.
  • A 3-factor CFA supported the conceptual model: in study 1, fit was CFI = 0.98 and RMSEA = 0.08 with factor loadings from 0.79 to 0.94; in study 2, fit was CFI = 0.96 and RMSEA = 0.08 with factor loadings from 0.75 to 0.89.
  • The final short scales were 4-item measures with good reliability: Cronbach alphas were 0.85, 0.91, and 0.89 in study 2 for acceptability, appropriateness, and feasibility, respectively; test-retest coefficients in study 3 were 0.80, 0.73, and 0.88.
  • Known-groups validity was supported by ANOVA: the manipulated construct explained 42% of variance for acceptability, 37% for appropriateness, and 42% for feasibility; regression models explained 41%, 42%, and 46% of the variance in change scores, respectively.

## Methods

  • The paper reports three studies: study 1 assessed substantive and discriminant content validity with implementation scientists and mental health professionals; study 2 tested structural validity, reliability, and known-groups validity in a vignette experiment; study 3 re-administered the vignette experiment weeks later to assess test-retest reliability and sensitivity to change.
  • Study 1 included 36 implementation scientists and 27 mental health professionals who assigned 31 items to constructs and rated confidence; study 2 included 326 mental health counselors who rated 15 items across six randomly assigned vignettes; study 3 followed up with 192 respondents from study 2, with 95 used for test-retest and 97 for sensitivity analyses.
  • Analyses included Wilcoxon one-sample signed rank tests, ICCs, EFA, CFA, Cronbach alphas, ANOVA, Pearson correlations corrected for attenuation, and linear regression.

## Limitations

  • The paper states that correlations among acceptability, appropriateness, and feasibility were sometimes fairly high, and discriminant validity was not fully tested because of a survey design error.
  • The study also states that replication is needed in samples with different backgrounds and characteristics, and with different methods or materials.
  • Evident from the text: the vignette design was incomplete because two of the eight possible vignettes were not fielded, which limited the exploration of discriminant validity and interaction effects, especially between acceptability and appropriateness.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s13012-017-0635-3

Cites

35 of 35 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Confirmatory factor analysis of substantive validity data N = 63. χ2(87) = 120.6, p =.01; CFI = 0.98 ; RMSEA = 0.08, CI [0.04-0.11]. All path loadings are significant at p < .05
Fig. 1 Confirmatory factor analysis of substantive validity data N = 63. χ2(87) = 120.6, p =.01; CFI = 0.98 ; RMSEA = 0.08, CI [0.04-0.11]. All path loadings are significant at p < .05CC BY · PubMed Central
Fig. 2 Confirmatory factor analysis of structural validity data N = 326. χ2(49) = 147.9, p = <.001; CFI = 0.96; RMSEA = 0.079, CI [0.06-0.09]. All path loadings are significant at p < .05
Fig. 2 Confirmatory factor analysis of structural validity data N = 326. χ2(49) = 147.9, p = <.001; CFI = 0.96; RMSEA = 0.079, CI [0.06-0.09]. All path loadings are significant at p < .05CC BY · PubMed Central

Verification

Claims the paper makes

5 Valid · 0 Not valid · 13 Not checked · 88 Own results and interpretations

48 more claims

Create a free account to see all 106 claims and to verify the 19 references still open.

See all claims — free account
  • It may state two separate claims as one. 35
  • The quote only partly states this claim. 1
  • Probably not a substantial claim. 24

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 35 references

Research works printed but not linked: 5

  1. Muthén LK, Muthén BO. Mplus user’s guide. seventh edition ed. Muthén & Muthén: Los Angeles; 2012.
  2. Cohen J. Statistical power analysis for the behavioral sciences. Second edition ed. Lawrence Earlbaum Associates: Hillsdale; 1988.
  3. WMK T, JP D. The research methods knowledge base. 3rd ed. Cincinnati: Atomic Dog Publishing; 2006.
  4. Proctor EK, Powell BJ, Feely M. Measurement in dissemination and implementation science. In: Kendall RSBPC, editor. Dissemination and implementation of evidence-based practices in child and adolescent mental health. New York: Oxford University Press; 2014. p. 22–43.
  5. Powell BJ, Weiner BJ, Stanick CF, Halko H, Dorsey C, Lewis CC. Stakeholders’ perceptions of criteria for pragmatic measurement in implementation: a concept mapping approach (oral presentation). 9th Annual Conference on the Science of Dissemination & Implementation. Washington, D. C: Academy Health and the National Institutes of Health; 2016.

Cited by

See all 2,819

Metadata

Authors
Weiner, B. J., Lewis, C. C., Stanick, C., Powell, B. J., Dorsey, C. N., Clary, A. S., Boynton, M. H., & Halko, H.
Affiliations
  • University of Washington (US)
  • Kaiser Permanente Washington Health Research Institute (US)
  • University of North Carolina at Chapel Hill (US)
  • University of Montana (US)
Corresponding author
Weiner, Bryan J.
Venue
Implementation Science
Published
2017-08-29
Volume
12
Issue
1
Pages
108-108
ISSN
1748-5908 (ISSN-L)
OpenAlex
W2750615195
Cited by
2,819
References
35
Citations by year
2026: 549 · 2025: 655 · 2024: 548 · 2023: 376 · 2022: 298 · 2021: 229 · 2020: 98 · 2019: 45 · 2018: 14 · 2017: 2
FWCI
69.79
Type
journal article
Language
en
Open access
Licence
Locations
  • Implementation Science (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Open Scholarship Institutional Repository (Washington University in St. Louis) (submittedVersion, other-oa)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
National Institute of Mental Health (R01MH106510)
Topics
  • Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
  • Health Sciences › Nursing › Research and Theory › Health Education and Validation
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
Keywords
acceptability, appropriateness, feasibility, evidence-based practice adoption, mental health counselors, scale development, content validity, structural validity, known-groups validity, test-retest reliability, sensitivity to change, vignettes, psychometric properties, reliability and validity, mental health services
MeSH terms
Factor Analysis, Statistical, Feasibility Studies, Female, Health Plan Implementation, Humans, Male, Psychometrics, Surveys and Questionnaires, Reproducibility of Results, Outcome Assessment, Health Care

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Psychometric assessment of three newly developed implementation outcome measures. https://app.cochat.ai/papers/10.1186/s13012-017-0635-3
The paper
Weiner, B. J., Lewis, C. C., Stanick, C., Powell, B. J., Dorsey, C. N., Clary, A. S., Boynton, M. H., & Halko, H. (2017). Psychometric assessment of three newly developed implementation outcome measures. Implementation Science. https://doi.org/10.1186/s13012-017-0635-3

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.