Toward criteria for pragmatic measurement in implementation research and practice: a stakeholder-driven approach using concept mapping
Open accessAbstract
BACKGROUND: Advancing implementation research and practice requires valid and reliable measures of implementation determinants, mechanisms, processes, strategies, and outcomes. However, researchers and implementation stakeholders are unlikely to use measures if they are not also pragmatic. The purpose of this study was to establish a stakeholder-driven conceptualization of the domains that comprise the pragmatic measure construct. It built upon a systematic review of the literature and semi-structured stakeholder interviews that generated 47 criteria for pragmatic measures, and aimed to further refine that set of criteria by identifying conceptually distinct categories of the pragmatic measure construct and providing quantitative ratings of the criteria's clarity and importance. METHODS: Twenty-four stakeholders with expertise in implementation practice completed a concept mapping activity wherein they organized the initial list of 47 criteria into conceptually distinct categories and rated their clarity and importance. Multidimensional scaling, hierarchical cluster analysis, and descriptive statistics were used to analyze the data. FINDINGS: The 47 criteria were meaningfully grouped into four distinct categories: (1) acceptable, (2) compatible, (3) easy, and (4) useful. Average ratings of clarity and importance at the category and individual criteria level will be presented. CONCLUSIONS: This study advances the field of implementation science and practice by providing clear and conceptually distinct domains of the pragmatic measure construct. Next steps will include a Delphi process to develop consensus on the most important criteria and the development of quantifiable pragmatic rating criteria that can be used to assess measures.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The short versions facilitate measurement of sustainability capacity in research, evaluation, and practice settings.
“reduce the burden of assessment and facilitate measurement of sustainability capacity in research, evaluation, and practice setting [5, 17, 18, 22].”
From Lowering the burden: Shorter versions of the Program Sustainability Assessment Tool (PSAT) and Clinical Sustainability Assessment Tool (CSAT) (2024) · cites this paper as [18]
The Background and Methods > Data analysis passages identify low burden and brevity as criteria for pragmatic measures, which supports the general rationale for reducing assessment burden. They do not discuss short versions of a sustainability-capacity measure or show that such versions facilitate its use in research, evaluation, and practice settings. No page number is available in our copy. (LLM, conf=0.82)
Experimental — accuracy not yet measuredPragmatic assessment tools are important in implementation science because they ensure consistent measurement of constructs and advance implementation practice efforts.
“Pragmatic assessment tools, often described as tools that are usable and brief, are important in implementation science because they ensure consistent measurement of constructs as well as advance implementation practice efforts [17, 18].”
From Lowering the burden: Shorter versions of the Program Sustainability Assessment Tool (PSAT) and Clinical Sustainability Assessment Tool (CSAT) (2024) · cites this paper as [18]
The Background says implementation science needs valid, reliable measures that are also pragmatic. The Discussion says such measures could benefit research and practice, but it does not say that being pragmatic ensures consistent measurement of constructs; it also frames the practice benefit as prospective. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009) · Implementation Science · doi:10.1186/1748-5908-4-50 · Cited by 14,496 · see citing works
Self-citationPalinkas, L. A., Horwitz, S. M., Green, C. A., Wisdom, J. P., Duan, N., & Hoagwood, K. (2015) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-013-0528-y · Cited by 12,038 · see citing works
Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0319-7 · Cited by 8,721 · see citing works
The Delphi Technique: Making Sense of Consensus
Hsu, C.-C., & Sandford, B. A. (2007) · University of Massachusetts Amherst · doi:10.7275/pdz9-th90 · Cited by 4,500 · cited 1× · see citing works
“Next steps will include developing consensus on the relative priorities for these categories and criteria through a Delphi [18] study; developing rating criteria with concrete, measurable anchors; and assessing inter-rater reliability and known-groups validity of the criteria [3].”
Psychometric assessment of three newly developed implementation outcome measures
Weiner, B. J., Lewis, C. C., Stanick, C., Powell, B. J., Dorsey, C. N., Clary, A. S., Boynton, M. H., & Halko, H. (2017) · Implementation Science · doi:10.1186/s13012-017-0635-3 · Cited by 2,819 · see citing works
Self-citationConcept Mapping for Planning and Evaluation
Kane, M., & Trochim, W. M. K. (2006) · Sage eBooks · doi:10.4135/9781412983730 · Cited by 1,141 · see citing works
Waltz, T. J., Powell, B. J., Matthieu, M. M., Damschroder, L. J., Chinman, M. J., Smith, J. L., Proctor, E. K., & Kirchner, J. E. (2015) · Implementation Science · doi:10.1186/s13012-015-0295-0 · Cited by 1,108 · see citing works
Self-citationKontos, P. C., & Poland, B. D. (2009) · Implementation Science · doi:10.1186/1748-5908-4-1 · Cited by 760 · see citing works
Quality and rigor of the concept mapping methodology: A pooled study analysis
Rosas, S. R., & Kane, M. (2011) · Evaluation and Program Planning · doi:10.1016/j.evalprogplan.2011.10.003 · Cited by 598 · see citing works
Concept mapping: an introduction to structured conceptualization in health care
Trochim, W., & Kane, M. (2005) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzi038 · Cited by 477 · see citing works
Pai, M., Lloyd, N. S., Cheng, J., Thabane, L., Spencer, F. A., Cook, D. J., Haynes, R. B., Schünemann, H. J., & Douketis, J. D. (2013) · Implementation Science · doi:10.1186/1748-5908-8-1 · Cited by 376 · see citing works
Glasgow, R. E., & Riley, W. T. (2013) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2013.03.010 · Cited by 318 · see citing works
Aarons, G. A., Wells, R. S., Zagursky, K., Fettes, D. L., & Palinkas, L. A. (2009) · American Journal of Public Health · doi:10.2105/ajph.2009.161711 · Cited by 299 · see citing works
Gama, E. M., Madan, J. J., Banda, H., Squire, S. B., Thomson, R., & Namakhoma, I. (2015) · Implementation Science · doi:10.1186/s13012-014-0195-8 · Cited by 155 · see citing works
Lewis, C. C., Stanick, C. F., Martinez, R. G., Weiner, B. J., Kim, M., Barwick, M., & Comtois, K. A. (2016) · Implementation Science · doi:10.1186/s13012-014-0193-x · Cited by 155 · see citing works
Self-citationDissemination and Implementation Science Training Needs: Insights From Practitioners and Researchers
Tabak, R. G., Padek, M. M., Kerner, J. F., Stange, K. C., Proctor, E. K., Dobbins, M., Colditz, G. A., Chambers, D., & Brownson, R. C. (2017) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.10.005 · Cited by 131 · see citing works
Advancing implementation science through measure development and evaluation: a study protocol
Lewis, C. C., Weiner, B. J., Stanick, C., & Fischer, S. M. (2015) · Implementation Science · doi:10.1186/s13012-015-0287-0 · Cited by 115 · see citing works
Self-citationDissemination and Implementation of Evidence-Based Practices in Child and Adolescent Mental Health
Beidas, R. S., Kendall, P. C., & McClelland, H. (2017) · Clinical Psychology Forum · doi:10.53841/bpscpf.2017.1.300.46 · Cited by 109 · see citing works
Stanick, C. F., Halko, H. M., Dorsey, C. N., Weiner, B. J., Powell, B. J., Palinkas, L. A., & Lewis, C. C. (2018) · BMC Health Services Research · doi:10.1186/s12913-018-3709-2 · Cited by 54 · see citing works
Self-citationLobb, R., Pinto, A. D., & Lofters, A. (2013) · Implementation Science · doi:10.1186/1748-5908-8-37 · Cited by 47 · see citing works
Smith, L. R., Damschroder, L. J., Lewis, C. C., & Weiner, B. J. (2015) · Implementation Science · doi:10.1186/1748-5908-10-s1-a11 · Cited by 45 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
A systematic review of empirical studies examining mechanisms of implementation in health
Lewis, C. C., Boyd, M. R., Walsh-Bailey, C., Lyon, A. R., Beidas, R., Mittman, B., Aarons, G. A., Weiner, B. J., & Chambers, D. A. (2020) · Implementation Science · doi:10.1186/s13012-020-00983-3 · Cited by 363 · see citing works
Self-citationQuantitative measures of health policy implementation determinants and outcomes: a systematic review
Allen, P., Pilar, M., Walsh-Bailey, C., Hooley, C., Mazzucca, S., Lewis, C. C., Mettert, K. D., Dorsey, C. N., Purtle, J., Kepper, M. M., et al. (2020) · Implementation Science · doi:10.1186/s13012-020-01007-w · Cited by 164 · see citing works
Self-citationBoyd, M. R., Powell, B. J., Endicott, D., & Lewis, C. C. (2018) · Behavior Therapy · doi:10.1016/j.beth.2017.11.012 · Cited by 137 · see citing works
Self-citationStanick, C. F., Halko, H. M., Nolen, E. A., Powell, B. J., Dorsey, C. N., Mettert, K. D., Weiner, B. J., Barwick, M., Wolfenden, L., Damschroder, L. J., et al. (2021) · Translational Behavioral Medicine · doi:10.1093/tbm/ibz164 · Cited by 117 · see citing works
Self-citationAn updated protocol for a systematic review of implementation-related measures
Lewis, C. C., Mettert, K. D., Dorsey, C. N., Martinez, R. G., Weiner, B. J., Nolen, E., Stanick, C., Halko, H., & Powell, B. J. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0728-3 · Cited by 118 · see citing works
Self-citationReilly, K. L., Kennedy, S., Porter, G., & Estabrooks, P. (2020) · Frontiers in Public Health · doi:10.3389/fpubh.2020.00430 · Cited by 102 · see citing works
Meza, R. D., Triplett, N. S., Woodard, G. S., Martin, P., Khairuzzaman, A. N., Jamora, G., & Dorsey, S. (2021) · Implementation Science · doi:10.1186/s13012-021-01104-4 · Cited by 76 · see citing works
Powell, B. J., Mettert, K. D., Dorsey, C. N., Weiner, B. J., Stanick, C. F., Lengnick-Hall, R., Ehrhart, M. G., Aarons, G. A., Barwick, M. A., Damschroder, L. J., et al. (2021) · Implementation Research and Practice · doi:10.1177/26334895211018862 · Cited by 79 · see citing works
Self-citationPuspitasari, A. J., Heredia, D., Gentry, M., Sawchuk, C., Theobald, B., Moore, W., Tiede, M., Galardy, C., & Schak, K. (2021) · Cognitive and Behavioral Practice · doi:10.1016/j.cbpra.2021.05.002 · Cited by 45 · see citing works
Advancing implementation frameworks with a mixed methods case study in child behavioral health
Barwick, M., Barac, R., Kimber, M., Akrong, L., Johnson, S. N., Cunningham, C. E., Bennett, K., Ashbourne, G., & Godden, T. (2020) · Translational Behavioral Medicine · doi:10.1093/tbm/ibz005 · Cited by 33 · see citing works
Self-citationA Taxonomy for External Support for Practice Transformation
Solberg, L. I., Kuzel, A., Parchman, M. L., Shelley, D. R., Dickinson, W. P., Walunas, T. L., Nguyen, A. M., Fagnan, L. J., Cykert, S., Cohen, D. J., et al. (2021) · The Journal of the American Board of Family Medicine · doi:10.3122/jabfm.2021.01.200225 · Cited by 18 · see citing works
Triplett, N. S., Munson, S., Mbwayo, A., Mutavi, T., Weiner, B. J., Collins, P., Amanya, C., & Dorsey, S. (2021) · Implementation Science Communications · doi:10.1186/s43058-020-00102-9 · Cited by 30 · see citing works
Self-citationMalone, S., Prewitt, K., McKay, V., Zabotka, L., Bacon, C., & Luke, D. A. (2024) · Implementation Science Communications · doi:10.1186/s43058-024-00656-y · Cited by 12 · see citing works
“We were able to reduce the number of items of both measures while maintaining strong psychometric properties, and, in doing so, reduce the burden of assessment and facilitate measurement of sustainability capacity in research, evaluation, and practice setting [5, 17, 18, 22].”
BackgroundKuo, C. C., Sibeko, G., Akande, M., Allie, S., Tisaker, N., Stein, D. J., & Becker, S. J. (2021) · Addiction Science & Clinical Practice · doi:10.1186/s13722-021-00236-8 · Cited by 8 · see citing works
- Authors
- Powell, B. J., Stanick, C. F., Halko, H. M., Dorsey, C. N., Weiner, B. J., Barwick, M. A., Damschroder, L. J., Wensing, M., Wolfenden, L., & Lewis, C. C.
- Affiliations
- University of North Carolina at Chapel Hill (US)
- University of Montana (US)
- Kaiser Permanente Washington Health Research Institute (US)
- University of Washington (US)
- University of Toronto (CA)
- VA Ann Arbor Healthcare System (US)
- University Hospital Heidelberg (DE)
- University of Newcastle Australia (AU)
- Corresponding author
- Powell, Byron J.
- ORCID
- Venue
- Implementation Science
- Published
- 2017-10-03
- Volume
- 12
- Issue
- 1
- Pages
- 118-118
- ISSN
- 1748-5908 (ISSN-L)
- PMC
- PMC5627503
- OpenAlex
- W2760901594
- Cited by
- 144
- References
- 21
- Citations by year
- 2026: 5 · 2025: 6 · 2024: 8 · 2023: 22 · 2022: 16 · 2021: 41 · 2020: 23 · 2019: 13 · 2018: 8 · 2017: 2
- FWCI
- 20.39
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Implementation Science (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- heiDOK (Heidelberg University) (acceptedVersion, other-oa)
- TSpace (University of Toronto) (submittedVersion, cc-by)
- Carolina Digital Repository (University of North Carolina at Chapel Hill) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Radboud Repository (Radboud University) (submittedVersion)
- Radboud Repository (Radboud University) (submittedVersion, other-oa)
- NOVA (University of Newcastle Australia) (submittedVersion, cc-by)
- UNC Libraries
- Funders
- National Institute of Mental Health (R01MH106510, R25MH080916, L30MH108060, K01MH113806)
- National Center for Advancing Translational Sciences (UL1TR001111)
- National Institute of Allergy and Infectious Diseases (P30AI050410)
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Education › Health Education and Validation
- Keywords
- implementation research, implementation practices, stakeholder perspectives, concept mapping, measurement criteria, implementation factors, implementation outcomes, acceptability, compatibility, ease of use, usefulness
- MeSH terms
- Stakeholder Participation, Health Plan Implementation, Health Services Research, Humans, Interviews as Topic, Cluster Analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
