Measures of outer setting constructs for implementation research: A systematic review and analysis of psychometric quality
Open accessThe DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
Background: Despite their influence, outer setting barriers (e.g., policies, financing) are an infrequent focus of implementation research. The objective of this systematic review was to identify and assess the psychometric properties of measures of outer setting used in behavioral and mental health research. Methods: Data collection involved (a) search string generation, (b) title and abstract screening, (c) full-text review, (d) construct mapping, and (e) measure forward searches. Outer setting constructs were defined using the Consolidated Framework for Implementation Research (CFIR). The search strategy included four relevant constructs separately: (a) cosmopolitanism, (b) external policy and incentives, (c) patient needs and resources, and (d) peer pressure. Information was coded using nine psychometric criteria: (a) internal consistency, (b) convergent validity, (c) discriminant validity, (d) known-groups validity, (e) predictive validity, (f) concurrent validity, (g) structural validity, (h) responsiveness, and (i) norms. Frequencies were calculated to summarize the availability of psychometric information. Information quality was rated using a 5-point scale and a final median score was calculated for each measure. Results: Systematic searches yielded 20 measures: four measures of the general outer setting domain, seven of cosmopolitanism, four of external policy and incentives, four of patient needs and resources, and one measure of peer pressure. Most were subscales within full scales assessing implementation context. Typically, scales or subscales did not have any psychometric information available. Where information was available, the quality was most often rated as “1-minimal” or “2-adequate.” Conclusion: To our knowledge, this is the first systematic review to focus exclusively on measures of outer setting factors used in behavioral and mental health research and comprehensively assess a range of psychometric criteria. The results highlight the limited quantity and quality of measures at this level. Researchers should not assume “one size fits all” when measuring outer setting constructs. Some outer setting constructs may be more appropriately and efficiently assessed using objective indices or administrative data reflective of the system rather than the individual.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures








Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Systematic reviews of implementation context measures have been conducted, all of which have found significant gaps and signal the continued need for measure validation and development [89–95].”
From The updated consolidated framework for implementation research based on user feedback (2022) · cites this paper as [92]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009) · Implementation Science · doi:10.1186/1748-5908-4-50 · Cited by 14,496 · cited 6×
“This systematic review is part of a larger project to identify measures of all implementation constructs associated with the CFIR (Damschroder et al., 2009), which are included in this special section.”
BackgroundMethodInfluentialAdvancing a Conceptual Model of Evidence-Based Practice Implementation in Public Service Sectors
Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0327-7 · Cited by 3,240 · cited 2×
“Most frameworks distinguish between contextual factors related to the organizational setting in which implementation is taking place and factors related to the wider environment in which the organization operates (Aarons et al., 2011; Damschroder et al., 2009; Feldstein & Glasgow, 2008).”
BackgroundMethodBridging Research and Practice
Tabak, R. G., Khoong, E. C., Chambers, D. A., & Brownson, R. C. (2012) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2012.05.024 · Cited by 1,386 · cited 1×
“There are a plethora of implementation models and frameworks that categorize context into groups of factors that are hypothesized to affect implementation outcomes (Tabak et al., 2012).”
BackgroundDissemination and Implementation Research in HealthTranslating Science to Practice
Brownson, R. C., Colditz, G. A., & Proctor, E. K. (2012) · doi:10.1093/acprof:oso/9780199751877.001.0001 · Cited by 1,355
Feldstein, A. C., & Glasgow, R. E. (2008) · The Joint Commission Journal on Quality and Patient Safety · doi:10.1016/s1553-7250(08)34030-6 · Cited by 987 · cited 1×
“Most frameworks distinguish between contextual factors related to the organizational setting in which implementation is taking place and factors related to the wider environment in which the organization operates (Aarons et al., 2011; Damschroder et al., 2009; Feldstein & Glasgow, 2008).”
BackgroundAssessing organizational readiness for change
Lehman, W. E. K., Greener, J. M., & Simpson, D. D. (2002) · Journal of Substance Abuse Treatment · doi:10.1016/s0740-5472(02)00233-7 · Cited by 863 · cited 2×
“Note the median ratings of “3-good” for known-groups validity and “2-adequate” for convergent validity were based on two subscales within the Texas Christian University Organizational Readiness for Change (TCU-ORC) (Lehman et al., 2002).”
MethodChaudoir, S. R., Dugan, A. G., & Barr, C. H. (2013) · Implementation Science · doi:10.1186/1748-5908-8-22 · Cited by 800 · cited 10×
“Our results align with the findings from other reviews which examined measures of implementation determinants at multiple levels in general health (Chaudoir et al., 2013), public and community health (Clinton-McHarg et al., 2016), and state mental health systems (Chor et al., 2015) where outer…”
BackgroundResultInfluentialWeiner, B. J., Amick, H., & Lee, S.-Y. D. (2008) · Medical Care Research and Review · doi:10.1177/1077558708317802 · Cited by 786 · cited 2×
“A number of reviews have concentrated solely on the measurement of inner setting constructs (Emmons et al., 2012; French et al., 2009; Weiner et al., 2008).”
BackgroundKAPLAN, H. C., BRADY, P. W., DRITZ, M. C., HOOPER, D. K., LINAM, W. M., FROEHLE, C. M., & MARGOLIS, P. (2010) · Milbank Quarterly · doi:10.1111/j.1468-0009.2010.00611.x · Cited by 714 · cited 7×
“” There was inconsistent evidence of an association between these factors and QI success (Kaplan et al., 2010).”
BackgroundInfluentialPurtle, J., Peters, R., & Brownson, R. C. (2015) · Implementation Science · doi:10.1186/s13012-015-0367-1 · Cited by 446 · cited 1×
“This is perhaps not surprising as policy is understudied in implementation research, at least in the United States where less than 10% of dissemination and implementation research funded by the National Institutes of Health (NIH) between 2007 and 2014 focussed on policy (Purtle et al., 2015).”
BackgroundSelf-citationLewis, C. C., Fischer, S., Weiner, B. J., Stanick, C., Kim, M., & Martinez, R. G. (2015) · Implementation Science · doi:10.1186/s13012-015-0342-x · Cited by 430 · cited 1×
“Similar to other systematic reviews of implementation measures (Lewis et al., 2015), traditional systematic review methods were not useful for identifying articles with measures of outer setting constructs.”
ResultSelf-citationBruns, E. J., Parker, E. M. L., Hensley, S., Pullmann, M. D., Benjamin, P. H., Lyon, A. R., & Hoagwood, K. (2019) · Implementation Science · doi:10.1186/s13012-019-0944-9 · Cited by 125 · cited 2×
“While outer setting factors may be difficult to influence, research has also found conditions in the outer setting that are both influential and mutable (Bruns et al., 2019; Palinkas et al., 2014).”
BackgroundSelf-citationMeasuring competition in health care markets.
Baker, L. C. (2001) · PubMed Central · Cited by 123
An updated protocol for a systematic review of implementation-related measures
Lewis, C. C., Mettert, K. D., Dorsey, C. N., Martinez, R. G., Weiner, B. J., Nolen, E., Stanick, C., Halko, H., & Powell, B. J. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0728-3 · Cited by 118
Self-citationPalinkas, L. A., Fuentes, D., Finno, M., Garcia, A. R., Holloway, I. W., & Chamberlain, P. (2014) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-012-0437-5 · Cited by 116 · cited 2×
“While outer setting factors may be difficult to influence, research has also found conditions in the outer setting that are both influential and mutable (Bruns et al., 2019; Palinkas et al., 2014).”
BackgroundClinton-McHarg, T., Yoong, S. L., Tzelepis, F., Regan, T., Fielding, A., Skelton, E., Kingsland, M., Ooi, J. Y., & Wolfenden, L. (2016) · Implementation Science · doi:10.1186/s13012-016-0512-5 · Cited by 116 · cited 6×
“Similar to the findings of another review of implementation measures in public health contexts (Clinton-McHarg et al., 2016), few measures reported criterion-predictive validity or known-groups validity.”
BackgroundMethodResultInfluentialDavidoff, F., Batalden, P. B., Stevens, D., Ogrinc, G. S., Mooney, S. E., & for the SQUIRE development group (2008) · Journal of General Internal Medicine · doi:10.1007/s11606-008-0797-4 · Cited by 109
How the study of networks informs knowledge translation and implementation: a scoping review
Glegg, S., Jenkins, E., & Kothari, A. R. (2019) · Implementation Science · doi:10.1186/s13012-019-0879-1 · Cited by 106 · cited 1×
“In the main, these analyses are based on self-report surveys (Glegg et al., 2019).”
BackgroundThe National Criminal Justice Treatment Practices survey: Multilevel survey methods and procedures
Taxman, F. S., Young, D. W., Wiersema, B., Rhodes, A. G., & Mitchell, S. (2007) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2007.01.002 · Cited by 105 · cited 3×
“, 2011) which was developed for child mental health services, and scales from the NCJTP Survey which was developed for use in correctional settings (Taxman et al., 2007).”
BackgroundInfluentialFrench, B., Thomas, L. H., Baker, P., Burton, C. R., Pennington, L., & Roddam, H. (2009) · Implementation Science · doi:10.1186/1748-5908-4-28 · Cited by 87
Measures for Predictors of Innovation Adoption
Chor, K. H. B., Wisdom, J. P., Olin, S.-C. S., Hoagwood, K. E., & Horwitz, S. M. (2015) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-014-0551-7 · Cited by 85 · cited 9×
“Similarly, in a systematic review of predictors of EBP adoption, half of “external system” measures identified were derived from computations of frequency data, based on reviews of state documents, or open-ended surveys or interviews with key informants (Chor et al., 2015).”
BackgroundInfluentialSystems Antecedents for Dissemination and Implementation: A Review and Analysis of Measures
Emmons, K. M., Weiner, B., Fernandez, M. E., & Tu, S.-P. (2012) · Health Education & Behavior · doi:10.1177/1090198111409748 · Cited by 72 · cited 2×
“A number of reviews have concentrated solely on the measurement of inner setting constructs (Emmons et al., 2012; French et al., 2009; Weiner et al., 2008).”
BackgroundBrookman-Frazee, L., Stahmer, A., Stadnick, N., Chlebowski, C., Herschell, A., & Garland, A. F. (2016) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-014-0622-9 · Cited by 64
Measurement of a model of implementation for health care: toward a testable theory
Cook, J. M., O’Donnell, C., Dinnen, S., Coyne, J. C., Ruzek, J. I., & Schnurr, P. P. (2012) · Implementation Science · doi:10.1186/1748-5908-7-59 · Cited by 59 · cited 4×
“Third, similar to other potentially influential determinants of implementation, there are no explicit recommendations for operational definitions or items to measure outer setting constructs (Cook et al., 2012).”
BackgroundMethodInfluentialInnovative culture in long-term care settings: The influence of organizational characteristics
Nieboer, A. P., & Strating, M. M. H. (2012) · Health Care Management Review · doi:10.1097/hmr.0b013e318222416b · Cited by 41 · cited 5×
“One measure of peer pressure was identified: Nieboer and Stratling’s measure of environmental competitiveness (Nieboer & Strating, 2012).”
BackgroundMethodInfluentialPurtle, J., Brownson, R. C., & Proctor, E. K. (2016) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-016-0752-3 · Cited by 41 · cited 3×
“Responsibility for behavioral and mental health care is diffuse (Purtle et al., 2017).”
BackgroundSelf-citationSmith, D., Damschroder, L. J., Kim, S. Y. H., & Ubel, P. A. (2012) · Psychiatric Services · doi:10.1176/appi.ps.201000036 · Cited by 33 · cited 1×
“This complexity is coupled with public and policymaker stigma toward people with mental illness (Purtle et al., 2017; Smith et al., 2012).”
BackgroundBoothroyd, R. A., Greenbaum, P. E., Wang, W., Kutash, K., & Friedman, R. M. (2011) · The Journal of Behavioral Health Services & Research · doi:10.1007/s11414-011-9239-x · Cited by 16 · cited 3×
“, 2016), which also used CFIR as the organizing framework, identified only two of the same measures of outer setting constructs as this review: the Systems of Care Implementation Survey (SOCIS; Boothroyd et al., 2011) which was developed for child mental health services, and scales from the NCJTP Survey which was deve…”
MethodInfluentialDavidoff, F., Batalden, P. B., Ogrinc, G. S., & Mooney, S. E. (2008) · Canadian Journal of Diabetes · doi:10.1016/s1499-2671(08)24008-6 · Cited by 14
A Longitudinal Study of Medical Practices’ Treatment of Patients Who Use Tobacco
Ramsay, P. P., Shortell, S. M., Casalino, L. P., Rodríguez, H. P., & Rittenhouse, D. R. (2015) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2015.07.005 · Cited by 12 · cited 2×
“consistency and norms from only one measure, the National Survey of Physician Organizations (NSPO) survey (Ramsay et al., 2016).”
Background
Cited by
The updated consolidated framework for implementation research based on user feedback
Damschroder, L. J., Reardon, C. M., Widerquist, M. A. O., & Lowery, J. (2022) · Implementation Science · doi:10.1186/s13012-022-01245-0 · Cited by 4,452
Lengnick-Hall, R., Stadnick, N. A., Dickson, K. S., Moullin, J. C., & Aarons, G. A. (2021) · Implementation Science · doi:10.1186/s13012-021-01099-y · Cited by 119
“However, a recent review of measures of outer context constructs used in behavioral health research highlighted the limited quantitative investigation of system-level and policy influences both in terms of quantity and quality [64].”
BackgroundFour very basic ways to think about policy in implementation science
Purtle, J., Moucheraud, C., Yang, L. H., & Shelley, D. (2023) · Implementation Science Communications · doi:10.1186/s43058-023-00497-1 · Cited by 68
“Recent reviews have summarized measures of health policy implementation [8, 15, 16], synthesized evidence on policymaker-focused dissemination strategies [9], and identified priorities for methodological innovation in policy implementation research [10].”
BackgroundSelf-citationMielke, J., Leppla, L., Valenta, S., Zullig, L. L., Zúñiga, F., Staudacher, S., Teynor, A., & Geest, S. D. (2022) · Implementation Science Communications · doi:10.1186/s43058-022-00354-7 · Cited by 55
Mielke, J., Brunkert, T., Zúñiga, F., Simon, M., Zullig, L. L., & Geest, S. D. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01772-w · Cited by 51
“, Standards for Reporting implementation Studies (StaRI) [22, 23]) ambiguous regarding how contextual analysis to report [18, 24].”
Background
Metadata
- Authors
- McHugh, S., Dorsey, C. N., Mettert, K., Purtle, J., Bruns, E., & Lewis, C. C.
- Affiliations
- University College Cork (IE)
- Kaiser Permanente Washington Health Research Institute (US)
- Drexel University (US)
- University of Washington (US)
- Corresponding author
- McHugh, Sheena
- Venue
- Implementation Research and Practice
- Publisher
- SAGE Publications
- Published
- 2020-01
- Volume
- 1
- Pages
- 2633489520940022-2633489520940022
- ISSN
- 2633-4895 (ISSN-L)
- PMC
- PMC9924255
- OpenAlex
- W3045176209
- Cited by
- 51
- References
- 30
- Citations by year
- 2026: 5 · 2025: 3 · 2024: 10 · 2023: 11 · 2022: 11 · 2021: 11
- FWCI
- 1.57
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Implementation Research and Practice (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Cork Open Research Archive (University College Cork, Ireland) (submittedVersion, cc-by-nc)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Lenus, The Irish Health Repository (Dr Steevens Hospital Library) (submittedVersion)
- Funders
- health research board (2017)
- National Institute of Mental Health (1R01MH106510)
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Health Sciences › Nursing › Research and Theory › Health Education and Validation
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- implementation context, psychometric properties, measurement quality, behavioral health, mental health, cosmopolitanism, peer pressure, Consolidated Framework for Implementation Research, scale validation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.