The DOI matches its Crossref record; 31 of the 32 references it cites were found in a registry or an index.
Advancing implementation science through measure development and evaluation: a study protocol
Open accessAbstract
BACKGROUND: Significant gaps related to measurement issues are among the most critical barriers to advancing implementation science. Three issues motivated the study aims: (a) the lack of stakeholder involvement in defining pragmatic measure qualities; (b) the dearth of measures, particularly for implementation outcomes; and (c) unknown psychometric and pragmatic strength of existing measures. Aim 1: Establish a stakeholder-driven operationalization of pragmatic measures and develop reliable, valid rating criteria for assessing the construct. Aim 2: Develop reliable, valid, and pragmatic measures of three critical implementation outcomes, acceptability, appropriateness, and feasibility. Aim 3: Identify Consolidated Framework for Implementation Research and Implementation Outcome Framework-linked measures that demonstrate both psychometric and pragmatic strength. METHODS/DESIGN: For Aim 1, we will conduct (a) interviews with stakeholder panelists (N = 7) and complete a literature review to populate pragmatic measure construct criteria, (b) Q-sort activities (N = 20) to clarify the internal structure of the definition, (c) Delphi activities (N = 20) to achieve consensus on the dimension priorities, (d) test-retest and inter-rater reliability assessments of the emergent rating system, and (e) known-groups validity testing of the top three prioritized pragmatic criteria. For Aim 2, our systematic development process involves domain delineation, item generation, substantive validity assessment, structural validity assessment, reliability assessment, and predictive validity assessment. We will also assess discriminant validity, known-groups validity, structural invariance, sensitivity to change, and other pragmatic features. For Aim 3, we will refine our established evidence-based assessment (EBA) criteria, extract the relevant data from the literature, rate each measure using the EBA criteria, and summarize the data. DISCUSSION: The study outputs of each aim are expected to have a positive impact as they will establish and guide a comprehensive measurement-focused research agenda for implementation science and provide empirically supported measures, tools, and methods for accomplishing this work.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Furthermore, there is an increasing number of measures of implementation determinants [74, 75] (e.g., implementation leadership [76], implementation climate [77, 78], or implementation intentions [79]).”
From Ten recommendations for using implementation frameworks in research and practice (2020) · cites this paper as [74]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To date, there is no consensus on measures, and psychometric properties have not been reported for most measures, although efforts are underway to increase measurement rigor and identify psychometrically sound measures (75).”
From The Sustainability of Evidence-Based Interventions and Practices in Public Health and Health Care (2018) · cites this paper as [75]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To date, there is no consensus on measures, and psychometric properties have not been reported for most measures, although efforts are underway to increase measurement rigor and identify psychometrically sound measures (75).”
From The Sustainability of Evidence-Based Interventions and Practices in Public Health and Health Care (2018) · cites this paper as [75]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Standardised tools to assess these outcomes are often unavailable due to the unique nature of interventions being implemented and the variable (and changing) implementation context in which the research is undertaken [64].”
From Guidance for conducting feasibility and pilot studies for implementation trials (2020) · cites this paper as [64]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“few if any have been assessed for other measurement properties that are of key importance to implementation researchers and other implementation stakeholders (e.g., intermediaries, policymakers, practice leaders), such as stability, responsiveness, and predictive validity [2–4].”
From Psychometric assessment of three newly developed implementation outcome measures (2017) · cites this paper as [4]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rogers, E. M., Singhal, A., & Quinlan, M. M. (2019) · An Integrated Approach to Communication Theory and Research · doi:10.4324/9780203710753-35 · Cited by 37,322
(1991) · Cited by 15,554
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009) · Implementation Science · doi:10.1186/1748-5908-4-50 · Cited by 14,496 · cited 7×
“Aim 3: identify CFIR- and IOF-linked measures that demonstrate both psychometric and pragmatic strength”
MethodInfluentialPower analysis and determination of sample size for covariance structure modeling.
MacCallum, R. C., Browne, M. W., & Sugawara, H. M. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.2.130 · Cited by 10,418 · cited 3×
“The estimated power [25] for the model is 100 %.”
BackgroundMethodInfluentialProctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0319-7 · Cited by 8,721
Foundations of Behavioral Research
Stockton, R., & Kerlinger, F. N. (1974) · American Educational Research Journal · doi:10.2307/1162206 · Cited by 7,347 · cited 1×
“Reliability, the precision of a measure, is a necessary condition for validity [27].”
BackgroundResearch guidelines for the Delphi survey technique
Hasson, F. (2000) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.2000.01567.x · Cited by 5,951 · cited 1×
“In the first round, the stakeholders will again be asked to distribute 100 points among the dimensions, taking into account the measures of central tendency and levels of dispersion from the first round [13].”
BackgroundLost in knowledge translation: Time for a map?
Graham, I. D., Logan, J., Harrison, M. B., Straus, S. E., Tetroe, J., Caswell, W., & Robinson, N. (2006) · Journal of Continuing Education in the Health Professions · doi:10.1002/chp.47 · Cited by 4,705 · cited 1×
“In 2006, it was noted that “implementation science” had 29 related terms in use around the globe [9], which threatens the cumulative knowledge that can be gained.”
BackgroundMessick, S. (1995) · American Psychologist · doi:10.1037/0003-066x.50.9.741 · Cited by 4,198
A Brief Tutorial on the Development of Measures for Use in Survey Questionnaires
Hinkin, T. R. (1998) · Organizational Research Methods · doi:10.1177/109442819800100106 · Cited by 4,095 · cited 3×
“We predict that our measures will demonstrate sufficient inter-item [21] and test-retest reliability.”
MethodInfluential
Research works printed but not linked: 4
- Grimshaw J, Eccles M, Thomas R, MacLennan G, Ramsay C, Fraser C, et al. Toward evidence-based quality improvement. J Gen Intern Med. 2006;21:S14–20.
- Nunnally JC. Bernstein: psychometric theory. New York: McGraw-Hill; 1994.
- Donnelly J, Trochim W. The research methods knowledge base. Ohio: Atomic Dog Publishing; 2007.
- Muthén LK, Muthén BO. Mplus. Compr Model Program Appl Res Users Guide. 2012;5:E22.
Cited by
Psychometric assessment of three newly developed implementation outcome measures
Weiner, B. J., Lewis, C. C., Stanick, C., Powell, B. J., Dorsey, C. N., Clary, A. S., Boynton, M. H., & Halko, H. (2017) · Implementation Science · doi:10.1186/s13012-017-0635-3 · Cited by 2,819
“To address this limitation, we are working with stakeholders to define and operationalize pragmatic features of measures, with the goal of developing rating criteria that can be used to assess the pragmatic properties of measures, much like the psychometric properties of measures are assessed [4].”
Self-citationThe Sustainability of Evidence-Based Interventions and Practices in Public Health and Health Care
Shelton, R. C., Cooper, B. R., & Stirman, S. W. (2018) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-040617-014731 · Cited by 806
Ten recommendations for using implementation frameworks in research and practice
Moullin, J. C., Dickson, K. S., Stadnick, N. A., Albers, B., Nilsen, P., Broder-Fingert, S., Mukasa, B., & Aarons, G. A. (2020) · Implementation Science Communications · doi:10.1186/s43058-020-00023-7 · Cited by 488
Guidance for conducting feasibility and pilot studies for implementation trials
Pearson, N., Naylor, P.-J., Ashe, M. C., Fernandez, M., Yoong, S. L., & Wolfenden, L. (2020) · Pilot and Feasibility Studies · doi:10.1186/s40814-020-00634-w · Cited by 475
A systematic review of empirical studies examining mechanisms of implementation in health
Lewis, C. C., Boyd, M. R., Walsh-Bailey, C., Lyon, A. R., Beidas, R., Mittman, B., Aarons, G. A., Weiner, B. J., & Chambers, D. A. (2020) · Implementation Science · doi:10.1186/s13012-020-00983-3 · Cited by 363
“However, conceptual, theoretical, and empirical work is needed to (a) articulate the theorized mechanisms for the 70+ strategies and proximal outcomes [128], (b) identify measures of implementation mechanisms and evaluate their psychometric evidence base [131] and pragmatic qualities [132], and (c) attempt to identify…”
BackgroundSelf-citation
Metadata
- Authors
- Lewis, C. C., Weiner, B. J., Stanick, C., & Fischer, S. M.
- Affiliations
- University of Washington (US)
- University of North Carolina at Chapel Hill (US)
- University of Montana (US)
- Indiana University Bloomington (US)
- Corresponding author
- Lewis, Cara C.
- Venue
- Implementation Science
- Published
- 2015-07-22
- Volume
- 10
- Issue
- 1
- Pages
- 102-102
- ISSN
- 1748-5908 (ISSN-L)
- PMC
- PMC4511441
- OpenAlex
- W1522008408
- Cited by
- 115
- References
- 32
- Citations by year
- 2026: 2 · 2025: 7 · 2024: 3 · 2023: 21 · 2022: 9 · 2021: 18 · 2020: 16 · 2019: 13 · 2018: 8 · 2017: 8 · 2016: 7 · 2015: 2
- FWCI
- 15.20
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Implementation Science (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Carolina Digital Repository (University of North Carolina at Chapel Hill) (submittedVersion)
- UNC Libraries
- Funders
- Georgia Clinical and Translational Science Alliance
- National Institutes of Health (2010-2015, UL1TR000083)
- National Institute of Mental Health (R01MH106510)
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Health Sciences › Nursing › Research and Theory › Health Education and Validation
- Keywords
- implementation outcomes, measurement development, stakeholder engagement, acceptability, appropriateness, feasibility, psychometric properties, grading criteria, Consolidated Framework for Implementation Research, structural invariance, sensitivity to change, known-groups validity
- MeSH terms
- Humans, Research Design, Program Evaluation, Program Development
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Advancing implementation science through measure development and evaluation: a study protocol. https://app.cochat.ai/papers/10.1186/s13012-015-0287-0
- The paper
- Lewis, C. C., Weiner, B. J., Stanick, C., & Fischer, S. M. (2015). Advancing implementation science through measure development and evaluation: a study protocol. Implementation Science. https://doi.org/10.1186/s13012-015-0287-0
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.