Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: results from the Expert Recommendations for Implementing Change (ERIC) study
Open accessAbstract
BACKGROUND: Poor terminological consistency for core concepts in implementation science has been widely noted as an obstacle to effective meta-analyses. This inconsistency is also a barrier for those seeking guidance from the research literature when developing and planning implementation initiatives. The Expert Recommendations for Implementing Change (ERIC) study aims to address one area of terminological inconsistency: discrete implementation strategies involving one process or action used to support a practice change. The present report is on the second stage of the ERIC project that focuses on providing initial validation of the compilation of 73 implementation strategies that were identified in the first phase. FINDINGS: Purposive sampling was used to recruit a panel of experts in implementation science and clinical practice (N = 35). These key stakeholders used concept mapping sorting and rating activities to place the 73 implementation strategies into similar groups and to rate each strategy's relative importance and feasibility. Multidimensional scaling analysis provided a quantitative representation of the relationships among the strategies, all but one of which were found to be conceptually distinct from the others. Hierarchical cluster analysis supported organizing the 73 strategies into 9 categories. The ratings data reflect those strategies identified as the most important and feasible. CONCLUSIONS: This study provides initial validation of the implementation strategies within the ERIC compilation as being conceptually distinct. The categorization and strategy ratings of importance and feasibility may facilitate the search for, and selection of, strategies that are best suited for implementation efforts in a particular setting.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Understanding the existing implementation landscape may support targeted selection and application of implementation strategies and may guide implementation strategy refinement.
“Understanding the existing implementation landscape may support targeted selection and application of implementation strategies (Waltz et al., 2015) and may guide implementation strategy refinement.”
From Implementation-as-Usual in Community-Based Organizations Providing Specialized Services to Individuals with Autism Spectrum Disorder: A Mixed Methods Study (2021) · cites this paper as Waltz et al., 2015
The ERIC taxonomy comprises 73 implementation strategies organized into nine clusters.
“In 2015, the Expert Recommendations for Implementing Change (ERIC) study organized a panel of implementation scientists to compile a standardized set of implementation strategy terms and definitions [2–4].”
From A systematic review of experimentally tested implementation strategies across health and human service settings: evidence from 2010-2022 (2024) · cites this paper as [5]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Powell, B. J., Waltz, T. J., Chinman, M. J., Damschroder, L. J., Smith, J. L., Matthieu, M. M., Proctor, E. K., & Kirchner, J. E. (2015) · Implementation Science · doi:10.1186/s13012-015-0209-1 · Cited by 4,963 · see citing works
Self-citationImplementation strategies: recommendations for specifying and reporting
Proctor, E. K., Powell, B. J., & McMillen, J. C. (2013) · Implementation Science · doi:10.1186/1748-5908-8-139 · Cited by 2,585 · see citing works
Self-citationConcept Mapping for Planning and Evaluation
Kane, M., & Trochim, W. M. K. (2006) · Sage eBooks · doi:10.4135/9781412983730 · Cited by 1,141 · see citing works
A compilation of strategies for implementing clinical innovations in health and mental health
Powell, B. J., McMillen, J. C., Proctor, E. K., Carpenter, C. R., Griffey, R. T., Bunger, A. C., Glass, J. E., & York, J. L. (2012) · Medical Care Research and Review · doi:10.1177/1077558711430690 · Cited by 1,069 · see citing works
Self-citationSpecifying and reporting complex behaviour change interventions: the need for a scientific method
Michie, S., Fixsen, D., Grimshaw, J. M., & Eccles, M. P. (2009) · Implementation Science · doi:10.1186/1748-5908-4-40 · Cited by 976 · see citing works
Individual determinants of research utilization by nurses: a systematic review update
Squires, J. E., Estabrooks, C. A., Gustavsson, P., & Wallin, L. (2011) · Implementation Science · doi:10.1186/1748-5908-6-1 · Cited by 798 · see citing works
Kontos, P. C., & Poland, B. D. (2009) · Implementation Science · doi:10.1186/1748-5908-4-1 · Cited by 760 · see citing works
Evidence-based de-implementation for contradicted, unproven, and aspiring healthcare practices
Prasad, V., & Ioannidis, J. P. (2014) · Implementation Science · doi:10.1186/1748-5908-9-1 · Cited by 723 · see citing works
Vollmar, H. C., Mayer, H., Ostermann, T., Butzlaff, M. E., Sandars, J. E., Wilm, S., & Rieger, M. A. (2010) · Implementation Science · doi:10.1186/1748-5908-5-1 · Cited by 693 · see citing works
Quality and rigor of the concept mapping methodology: A pooled study analysis
Rosas, S. R., & Kane, M. (2011) · Evaluation and Program Planning · doi:10.1016/j.evalprogplan.2011.10.003 · Cited by 598 · see citing works
Concept mapping: an introduction to structured conceptualization in health care
Trochim, W., & Kane, M. (2005) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzi038 · Cited by 477 · see citing works
Pai, M., Lloyd, N. S., Cheng, J., Thabane, L., Spencer, F. A., Cook, D. J., Haynes, R. B., Schünemann, H. J., & Douketis, J. D. (2013) · Implementation Science · doi:10.1186/1748-5908-8-1 · Cited by 376 · see citing works
McKibbon, K. A., Lokker, C., Wilczynski, N. L., Ciliska, D., Dobbins, M., Davis, D. A., Haynes, R. B., & Straus, S. E. (2010) · Implementation Science · doi:10.1186/1748-5908-5-16 · Cited by 345 · see citing works
Expert recommendations for implementing change (ERIC): protocol for a mixed methods study
Waltz, T. J., Powell, B. J., Chinman, M. J., Smith, J. L., Matthieu, M. M., Proctor, E. K., Damschroder, L. J., & Kirchner, J. E. (2014) · Implementation Science · doi:10.1186/1748-5908-9-39 · Cited by 202 · see citing works
Self-citationColquhoun, H., Leeman, J., Michie, S., Lokker, C., Bragge, P., Hempel, S., McKibbon, K. A., Peters, G.-J. Y., Stevens, K. R., Wilson, M. G., et al. (2014) · Implementation Science · doi:10.1186/1748-5908-9-51 · Cited by 180 · see citing works
Refining a taxonomy for guideline implementation: results of an exercise in abstract classification
Mazza, D., Bairstow, P., Buchan, H., Chakraborty, S. P., Van Hecke, O., Grech, C., & Kunnamo, I. (2013) · Implementation Science · doi:10.1186/1748-5908-8-32 · Cited by 173 · see citing works
Hempel, S., Rubenstein, L. V., Shanman, R., Foy, R., Golder, S., Danz, M., & Shekelle, P. (2011) · Implementation Science · doi:10.1186/1748-5908-6-85 · Cited by 40 · see citing works
McKibbon, K. A., Lokker, C., Wilczynski, N. L., Haynes, R. B., Ciliska, D., Dobbins, M., Davis, D. A., & Straus, S. E. (2012) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.10.014 · Cited by 37 · see citing works
Cohen, A. N., Hamilton, A. B., Ritchie, M. J., Mittman, B. S., Kirchner, J. E., Wyatt, G. E., Fortney, J. C., Hellemann, G., Liu, H., Curran, G. M., et al. (2015) · Implementation Science · doi:10.1186/1748-5908-10-s1-a29 · Cited by 34 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Waltz, T. J., Powell, B. J., Fernández, M. E., Abadie, B., & Damschroder, L. J. (2019) · Implementation Science · doi:10.1186/s13012-019-0892-4 · Cited by 1,136 · see citing works
Self-citationLewis, C. C., Klasnja, P., Powell, B. J., Lyon, A. R., Tuzzio, L., Jones, S., Walsh-Bailey, C., & Weiner, B. (2018) · Frontiers in Public Health · doi:10.3389/fpubh.2018.00136 · Cited by 588 · see citing works
Self-citationShelton, R. C., Chambers, D. A., & Glasgow, R. E. (2020) · Frontiers in Public Health · doi:10.3389/fpubh.2020.00134 · Cited by 448 · see citing works
Perry, C. K., Damschroder, L. J., Hemler, J. R., Woodson, T. T., Ono, S. S., & Cohen, D. J. (2019) · Implementation Science · doi:10.1186/s13012-019-0876-4 · Cited by 288 · see citing works
Self-citationTracking implementation strategies: a description of a practical approach and early findings
Bunger, A. C., Powell, B. J., Robertson, H. A., MacDowell, H., Birken, S. A., & Shea, C. (2017) · Health Research Policy and Systems · doi:10.1186/s12961-017-0175-y · Cited by 208 · see citing works
Self-citationLyon, A. R., Cook, C. R., Brown, E. C., Locke, J., Davis, C., Ehrhart, M., & Aarons, G. A. (2018) · Implementation Science · doi:10.1186/s13012-017-0705-6 · Cited by 199 · see citing works
Lyon, A. R., Brewer, S. K., & Areán, P. A. (2020) · American Psychologist · doi:10.1037/amp0000652 · Cited by 152 · see citing works
Rudd, B. N., Davis, M., & Beidas, R. S. (2020) · Implementation Science · doi:10.1186/s13012-020-01060-5 · Cited by 157 · see citing works
Varsi, C., Solberg Nes, L., Kristjansdottir, O. B., Kelders, S. M., Stenberg, U., Zangi, H. A., Børøsund, E., Weiss, K. E., Stubhaug, A., Asbjørnsen, R. A., et al. (2019) · Journal of Medical Internet Research · doi:10.2196/14255 · Cited by 148 · see citing works
Powell, B. J., Stanick, C. F., Halko, H. M., Dorsey, C. N., Weiner, B. J., Barwick, M. A., Damschroder, L. J., Wensing, M., Wolfenden, L., & Lewis, C. C. (2017) · Implementation Science · doi:10.1186/s13012-017-0649-x · Cited by 144 · see citing works
Self-citationHagermoser Sanetti, L. M., & Collier-Meek, M. A. (2019) · Journal of School Psychology · doi:10.1016/j.jsp.2019.07.008 · Cited by 129 · see citing works
Rogal, S. S., Yakovchenko, V., Waltz, T. J., Powell, B. J., Kirchner, J. E., Proctor, E. K., Gonzalez, R., Park, A., Ross, D., Morgan, T. R., et al. (2017) · Implementation Science · doi:10.1186/s13012-017-0588-6 · Cited by 98 · see citing works
Self-citationLourida, I., Abbott, R. A., Rogers, M., Lang, I. A., Stein, K., Kent, B., & Thompson Coon, J. (2017) · BMC Geriatrics · doi:10.1186/s12877-017-0528-y · Cited by 100 · see citing works
Nathan, N., Powell, B. J., Shelton, R. C., Laur, C. V., Wolfenden, L., Hailemariam, M., Yoong, S. L., Sutherland, R., Kingsland, M., Waltz, T. J., et al. (2022) · Frontiers in Health Services · doi:10.3389/frhs.2022.905909 · Cited by 105 · see citing works
Self-citationAshcraft, L. E., Goodrich, D. E., Hero, J., Phares, A., Bachrach, R. L., Quinn, D. A., Qureshi, N., Ernecoff, N. C., Lederer, L. G., Scheunemann, L. P., et al. (2024) · Implementation Science · doi:10.1186/s13012-024-01369-5 · Cited by 89 · see citing works
Self-citationDopp, A. R., Narcisse, M.-R., Mundey, P., Silovsky, J. F., Smith, A. B., Mandell, D., Funderburk, B. W., Powell, B. J., Schmidt, S., Edwards, D., et al. (2020) · Implementation Research and Practice · doi:10.1177/2633489520939980 · Cited by 79 · see citing works
Self-citationBalis, L. E., Houghtaling, B., & Harden, S. M. (2022) · Translational Behavioral Medicine · doi:10.1093/tbm/ibac061 · Cited by 63 · see citing works
Rogal, S. S., Yakovchenko, V., Waltz, T. J., Powell, B. J., Gonzalez, R., Park, A., Chartier, M., Ross, D., Morgan, T. R., Kirchner, J. E., et al. (2019) · Implementation Science · doi:10.1186/s13012-019-0881-7 · Cited by 52 · see citing works
Self-citationMurrell, J. E., Pisegna, J. L., & Juckett, L. A. (2021) · Implementation Science · doi:10.1186/s13012-021-01178-0 · Cited by 60 · see citing works
Lovero, K. L., Kemp, C. G., Wagenaar, B. H., Giusto, A., Greene, M. C., Powell, B. J., & Proctor, E. K. (2023) · Implementation Science · doi:10.1186/s13012-023-01310-2 · Cited by 52 · see citing works
Self-citationEngell, T., Stadnick, N. A., Aarons, G. A., & Barnett, M. L. (2023) · Global Implementation Research and Applications · doi:10.1007/s43477-023-00077-4 · Cited by 37 · see citing works
Yakovchenko, V., Morgan, T. R., Chinman, M. J., Powell, B. J., Gonzalez, R., Park, A., Malone, P. S., Chartier, M., Ross, D., & Rogal, S. S. (2021) · BMC Health Services Research · doi:10.1186/s12913-021-07312-4 · Cited by 27 · see citing works
Self-citationA Taxonomy for External Support for Practice Transformation
Solberg, L. I., Kuzel, A., Parchman, M. L., Shelley, D. R., Dickinson, W. P., Walunas, T. L., Nguyen, A. M., Fagnan, L. J., Cykert, S., Cohen, D. J., et al. (2021) · The Journal of the American Board of Family Medicine · doi:10.3122/jabfm.2021.01.200225 · Cited by 18 · see citing works
Drahota, A., Meza, R. D., Bustos, T. E., Sridhar, A., Martinez, J. I., Brikho, B., Stahmer, A. C., & Aarons, G. A. (2021) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-020-01084-5 · Cited by 23 · see citing works
Godbee, K., Gunn, J., Lautenschlager, N. T., & Palmer, V. J. (2020) · Australian Journal of Primary Health · doi:10.1071/py19249 · Cited by 20 · see citing works
Motamedi, M., Lau, A. S., Lind, T., Lui, J. H., Rodriguez, A., Smith, A., & Brookman-Frazee, L. (2021) · Implementation Research and Practice · doi:10.1177/2633489520982903 · Cited by 17 · see citing works
Randall, C. L. (2023) · Community Dentistry and Oral Epidemiology · doi:10.1111/cdoe.12841 · Cited by 17 · see citing works
Liaw, S. Y., Rusli, K. D. B., Tan, J. Z., Wee, Y. H. C., Neo, N. W. S., & Chua, W. L. (2025) · Nurse Education in Practice · doi:10.1016/j.nepr.2025.104462 · Cited by 17 · see citing works
Berner, M., Vazquez, A., & Mcdougall, M. E. (2019) · Public Administration Quarterly · doi:10.1177/073491491904300402 · Cited by 3 · see citing works
- Authors
- Waltz, T. J., Powell, B. J., Matthieu, M. M., Damschroder, L. J., Chinman, M. J., Smith, J. L., Proctor, E. K., & Kirchner, J. E.
- Affiliations
- Eastern Michigan University (US)
- University of North Carolina at Chapel Hill (US)
- United States Department of Veterans Affairs (US)
- VA Ann Arbor Healthcare System (US)
- RAND Corporation (US)
- Washington University in St. Louis (US)
- Corresponding author
- Waltz, Thomas J.
- ORCID
- Venue
- Implementation Science
- Published
- 2015-08-07
- Volume
- 10
- Issue
- 1
- Pages
- 109-109
- ISSN
- 1748-5908 (ISSN-L)
- PMC
- PMC4527340
- OpenAlex
- W2121795537
- Cited by
- 1,108
- References
- 19
- Citations by year
- 2026: 156 · 2025: 164 · 2024: 217 · 2023: 164 · 2022: 121 · 2021: 109 · 2020: 67 · 2019: 46 · 2018: 26 · 2017: 24 · 2016: 10 · 2015: 3
- FWCI
- 38.00
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Implementation Science (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Carolina Digital Repository (University of North Carolina at Chapel Hill) (submittedVersion)
- Digital Commons@Becker (Washington University School of Medicine) (submittedVersion)
- UNC Libraries
- Funders
- U.S. Department of Veterans Affairs
- Doris Duke Charitable Foundation
- Purdue University
- University of Pennsylvania
- Health Services Research and Development (QLP 55-025)
- Harvard University
- University of Minnesota (55-025)
- Washington University in St. Louis
- University of Washington
- Johns Hopkins University
- Indiana University-Purdue University Indianapolis
- University of Arkansas for Medical Sciences
- Fahs-Beck Fund for Research and Experimentation
- University of Massachusetts Boston
- University of Toronto
- Quality Enhancement Research Initiative
- University of California, Los Angeles
- University of California, San Diego
- VISN 4 Mental Illness Research, Education, and Clinical Center
- VA Pittsburgh Healthcare System
- National Institute of Mental Health (F31MH098478)
- National Institute of Diabetes and Digestive and Kidney Diseases (P30DK092926)
- NIH Clinical Center
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- implementation strategies, concept mapping, Expert Recommendations for Implementing Change, terminology consistency, feasibility, importance ratings, expert validation, practice change
- MeSH terms
- Health Priorities, Humans, Terminology as Topic, Organizational Innovation, Program Evaluation, Program Development
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
