Methods and metrics challenges of delivery-system research
Open accessAbstract
BACKGROUND: Many delivery-system interventions are fundamentally about change in social systems (both planned and unplanned). This systems perspective raises a number of methodological challenges for studying the effects of delivery-system change--particularly for answering questions related to whether the change will work under different conditions and how the change is integrated (or not) into the operating context of the delivery system. METHODS: The purpose of this paper is to describe the methodological and measurement challenges posed by five key issues in delivery-system research: (1) modeling intervention context; (2) measuring readiness for change; (3) assessing intervention fidelity and sustainability; (4) assessing complex, multicomponent interventions; and (5) incorporating time in delivery-system models to discuss recommendations for addressing these issues. For each issue, we provide recommendations for how research may be designed and implemented to overcome these challenges. RESULTS AND CONCLUSIONS: We suggest that a more refined understanding of the mechanisms underlying delivery-system interventions (treatment theory) and the ways in which outcomes for different classes of individuals change over time are fundamental starting points for capturing the heterogeneity in samples of individuals exposed to delivery-system interventions. To support the research recommendations outlined in this paper and to advance understanding of the "why" and "how" questions of delivery-system change and their effects, funding agencies should consider supporting studies with larger organizational sample sizes; longer duration; and nontraditional, mixed-methods designs.A version of this paper was prepared under contract with the Agency for Healthcare Research and Quality (AHRQ), US Department of Health and Human Services for presentation and discussion at a meeting on "The Challenge and Promise of Delivery System Research," held in Sterling, VA, on February 16-17, 2011. The opinions in the paper are those of the author and do not represent the views or recommendations of AHRQ or the US Department of Health and Human Services.1.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Implementation strategies are part of 'packaged' approaches whose specific elements are poorly understood.
“are part of ‘packaged’ approaches whose specific elements are poorly understood [12]”
From Implementation strategies: recommendations for specifying and reporting (2013) · cites this paper as [12]
Implementation strategies are inherently complex social interventions, as they address multifaceted and complicated processes within interpersonal, organizational, and community contexts.
“Implementation strategies are inherently complex social interventions, as they address multifaceted and complicated processes within interpersonal, organizational, and community contexts [12,56-58].”
From Implementation strategies: recommendations for specifying and reporting (2013) · cites this paper as [12]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Research Design: Qualitative, Quantitative, and Mixed Methods Approaches
Creswell, J. W. (1994) · Cited by 35,227 · see citing works
Hierarchical Linear Models: Applications and Data Analysis Methods.
Longford, N. T., Bryk, A. S., & Raudenbush, S. W. (1993) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2075823 · Cited by 18,768 · see citing works
Research design: qualitative, quantitative, and mixed methods approaches
Caldas, M. (2003) · Revista de Administração Contemporânea · doi:10.1590/s1415-65552003000100015 · Cited by 18,670 · see citing works
Institute of Medicine (2000) · National Academies Press eBooks · doi:10.17226/9728 · Cited by 14,174 · see citing works
Hierarchical Linear Models: Applications and Data Analysis Methods
Davidian, M. (2003) · Journal of the American Statistical Association · doi:10.1198/jasa.2003.s288 · Cited by 13,820 · see citing works
Crossing the Quality Chasm: A New Health System for the 21st Century
Baker, A. (2001) · BMJ · doi:10.1136/bmj.323.7322.1192 · Cited by 10,754 · see citing works
Crossing the Quality Chasm: A New Health System for the 21st Century
Committee on Quality of Health Care in America (2002) · Journal for Healthcare Quality · doi:10.1111/j.1945-1474.2002.tb00463.x · Cited by 8,508 · see citing works
Wall, A. (2000) · British Journal of Healthcare Management · doi:10.12968/bjhc.2000.6.9.19311 · Cited by 5,909 · see citing works
Goldstein, H. (2006) · Technometrics · doi:10.1198/tech.2006.s417 · Cited by 4,960 · see citing works
The Essential Impact of Context on Organizational Behavior
Johns, G. (2006) · Academy of Management Review · doi:10.5465/amr.2006.20208687 · Cited by 3,545 · see citing works
Evaluation: A Systematic Approach
Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (1979) · Cited by 2,943 · see citing works
Courgeau, D., & Goldstein, H. (1997) · Population · doi:10.2307/1534624 · Cited by 2,590 · see citing works
Higher Education: Handbook of Theory and Research
Smart, J. C. (2007) · Springer eBooks · doi:10.1007/978-1-4020-5666-6 · Cited by 2,375 · see citing works
Evaluation: A Systematic Approach
Wallace, D. P. (2005) · Library & Information Science Research · doi:10.1016/j.lisr.2004.09.009 · Cited by 1,938 · see citing works
Bringing context back into epidemiology: variables and fallacies in multilevel analysis
Diez-Roux, A. V. (1998) · American Journal of Public Health · doi:10.2105/ajph.88.2.216 · Cited by 1,317 · see citing works
Kitson, A. L., Rycroft-Malone, J., Harvey, G., McCormack, B., Seers, K., & Titchen, A. (2008) · Implementation Science · doi:10.1186/1748-5908-3-1 · Cited by 1,314 · see citing works
Chen, H.-T. (1990) · Medical Entomology and Zoology · Cited by 1,227 · see citing works
Location, location, location: contextualizing organizational research*
Rousseau, D. M., & Fried, Y. (2001) · Journal of Organizational Behavior · doi:10.1002/job.78 · Cited by 1,094 · see citing works
Fidelity Criteria: Development, Measurement, and Validation
Mowbray, C. T., Holter, M. C., Teague, G. B., & Bybee, D. I. (2003) · American Journal of Evaluation · doi:10.1177/109821400302400303 · Cited by 884 · see citing works
Individual determinants of research utilization by nurses: a systematic review update
Squires, J. E., Estabrooks, C. A., Gustavsson, P., & Wallin, L. (2011) · Implementation Science · doi:10.1186/1748-5908-6-1 · Cited by 798 · see citing works
Weiner, B. J., Amick, H., & Lee, S.-Y. D. (2008) · Medical Care Research and Review · doi:10.1177/1077558708317802 · Cited by 786 · see citing works
Advances in mixed-method evaluation : the challenges and benefits of integrating diverse paradigms
Greene, J. C., & Caracelli, V. J. (1997) · Bibliothèque et Archives nationales du Québec (Québec government) · Cited by 748 · see citing works
Lipsey, M. W. (1991) · Evaluation and Program Planning · doi:10.1016/0149-7189(91)90027-e · Cited by 598 · see citing works
Nutting, P. A., Miller, W. L., Crabtree, B. F., Jaén, C. R., Stewart, E. E., & Stange, K. C. (2009) · The Annals of Family Medicine · doi:10.1370/afm.1002 · Cited by 472 · see citing works
Comparing Personal Trajectories and Drawing Causal Inferences from Longitudinal Data
Raudenbush, S. W. (2001) · Annual Review of Psychology · doi:10.1146/annurev.psych.52.1.501 · Cited by 458 · see citing works
Muthén, B., & Muthén, L. K. (2000) · Journal of Studies on Alcohol · doi:10.15288/jsa.2000.61.290 · Cited by 359 · see citing works
Higher Education: Handbook of Theory and Research
Smart, J. C. (2005) · Springer eBooks · doi:10.1007/1-4020-3279-x · Cited by 344 · see citing works
Learning more by crossing levels: evidence from airplanes, hospitals, and orchestras
Hackman, J. R. (2003) · Journal of Organizational Behavior · doi:10.1002/job.226 · Cited by 342 · see citing works
Promoting intervention fidelity
Dumas, J. E., Lynch, A. M., Laughlin, J. E., Smith, E. P., & Prinz, R. J. (2001) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(00)00272-5 · Cited by 277 · see citing works
Using program theory in evaluation
Bickman, L. (1987) · Jossey-Bass eBooks · Cited by 265 · see citing works
The meaning and measurement of implementation climate
Weiner, B. J., Belden, C. M., Bergmire, D. M., & Johnston, M. (2011) · Implementation Science · doi:10.1186/1748-5908-6-78 · Cited by 260 · see citing works
Hedeker, D., Gibbons, R. D., & Flay, B. R. (1994) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.62.4.757 · Cited by 254 · see citing works
Hedeker, D., Gibbons, R. D., & Flay, B. R. (1994) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.62.4.757 · Cited by 237 · see citing works
Nutting, P. A., Crabtree, B. F., Miller, W. L., Stewart, E. E., Stange, K. C., & Jaén, C. R. (2010) · The Annals of Family Medicine · doi:10.1370/afm.1075 · Cited by 193 · see citing works
Mihalic, S. W., Fagan, A. A., & Argamaso, S. (2008) · Implementation Science · doi:10.1186/1748-5908-3-5 · Cited by 173 · see citing works
Collins, L. M., & Graham, J. A. (2002) · Drug and Alcohol Dependence · doi:10.1016/s0376-8716(02)00217-x · Cited by 169 · see citing works
Advances in Patient Safety: New Directions and Alternative Approaches (Vol. 2: Culture and Redesign)
Henriksen, K., Battles, J. B., Keyes, M. A., & Grady, M. L. (2008) · Cited by 145 · see citing works
Organizational Readiness for Change and opinions toward treatment innovations
Fuller, B. E., Rieckmann, T. R., Nunes, E. V., Miller, M. J., Arfken, C. L., Edmundson, E., & McCarty, D. W. (2007) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2006.12.026 · Cited by 123 · see citing works
Griffin, M. A. (2007) · Journal of Organizational Behavior · doi:10.1002/job.489 · Cited by 114 · see citing works
Goldstein (H.) — Multilevel Statistical Models
Courgeau, D. (1997) · Population · doi:10.3917/popu.p1997.52n4.1046 · Cited by 111 · see citing works
Measuring persistence of implementation: QUERI Series
Initiative, T. H. Q. E. R., Bowman, C. C., Sobo, E. J., Asch, S. M., & Gifford, A. L. (2008) · Implementation Science · doi:10.1186/1748-5908-3-21 · Cited by 100 · see citing works
Benn, J. J., Burnett, S. J., Parand, A., Pinto, A. C. S. N., Iskander, S., & Vincent, C. A. (2009) · Social Science & Medicine · doi:10.1016/j.socscimed.2009.09.051 · Cited by 98 · see citing works
Brekke, J. S., Long, J. D., Nesbitt, N., & Sobel, E. (1997) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.65.3.464 · Cited by 92 · see citing works
English, M., Nzinga, J., Mbindyo, P., Ayieko, P., Irimu, G., & Mbaabu, L. (2011) · Implementation science : IS · doi:10.1186/1748-5908-6-124 · Cited by 79 · see citing works
Innovation in mental health services: what are the key components of success?
Brooks, H. L., Pilgrim, D. M., & Rogers, A. E. (2011) · Implementation Science · doi:10.1186/1748-5908-6-120 · Cited by 72 · see citing works
Brekke, J. S., Long, J. D., Nesbitt, N., et al. (1997) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.65.3.464 · Cited by 71 · see citing works
New directions in program evaluation
(1986) · Evaluation Practice · doi:10.1016/s0886-1633(86)80033-2 · Cited by 69 · see citing works
Johnson, J. K., Miller, S. H., Miller, H., Sheldon, M., & Horowitz, D. (2010) · Cited by 65 · see citing works
Assessing program fidelity in substance abuse health services research
Orwin, R. G. (2000) · Addiction · doi:10.1080/09652140020004250 · Cited by 53 · see citing works
Sullivan, G., Blevins, D., & Kauth, M. R. (2008) · Implementation Science · doi:10.1186/1748-5908-3-33 · Cited by 51 · see citing works
Increasing the relevance of research to health care managers
Alexander, J. A., Hearld, L. R., Jiang, H., & Fraser, I. (2007) · Health Care Management Review · doi:10.1097/01.hmr.0000267792.09686.e3 · Cited by 50 · see citing works
Self-citationRiley, B., MacDonald, J.-A., Mansi, O., Kothari, A. R., Kurtz, D., von Tettenborn, L., & Edwards, N. C. (2008) · Implementation Science · doi:10.1186/1748-5908-3-27 · Cited by 45 · see citing works
Berwick, D. M. (2007) · Medical Care · doi:10.1097/mlr.0b013e31815d18e6 · Cited by 15 · see citing works
Improving the Health Care Work Environment: Implications for Research, Practice, and Policy
Harrison, M. I., Henriksen, K., & Hughes, R. G. (2007) · The Joint Commission Journal on Quality and Patient Safety · doi:10.1016/s1553-7250(07)33115-2 · Cited by 9 · see citing works
Henriksen, K., Jb, B., Ma, K., & Grady, M. L. (2008) · Cited by 6 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- Pawson R Tilley N Realistic Evaluation 1997 London: Sage Publications, Ltd
- Dumas JE Lynch AM Laughlin JE Smith EP Prinz RJ Promoting implementation fidelity: Conceptual issues, methods, and preliminary results from the Early Alliance prevention trial Am J Prev Med 2001 20 Supplement 1 38 47 11146259
Cited by
Sorted by citations
Implementation strategies: recommendations for specifying and reporting
Proctor, E. K., Powell, B. J., & McMillen, J. C. (2013) · Implementation Science · doi:10.1186/1748-5908-8-139 · Cited by 2,585 · see citing works
Enhancing the Impact of Implementation Strategies in Healthcare: A Research Agenda
Powell, B. J., Fernandez, M. E., Williams, N. J., Aarons, G. A., Beidas, R. S., Lewis, C. C., McHugh, S. M., & Weiner, B. J. (2019) · Frontiers in Public Health · doi:10.3389/fpubh.2019.00003 · Cited by 775 · see citing works
Organizational readiness for implementing change: a psychometric assessment of a new measure
Shea, C. M., Jacobs, S., Esserman, D. A., Bruce, K., & Weiner, B. J. (2014) · Implementation Science · doi:10.1186/1748-5908-9-7 · Cited by 863 · see citing works
Keith, R. E., Crosson, J. C., O’Malley, A. S., Cromp, D., & Taylor, E. F. (2017) · Implementation Science · doi:10.1186/s13012-017-0550-7 · Cited by 710 · see citing works
Elwyn, G. J., Scholl, I., Tietbohl, C. K., Mann, M. K., Edwards, A., Clay, C. F., Légaré, F., van der Weijden, T., Lewis, C. L., Wexler, R., et al. (2013) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-13-s2-s14 · Cited by 515 · see citing works
A Systematic Review of Strategies for Implementing Empirically Supported Mental Health Interventions
Powell, B. J., Proctor, E. K., & Glass, J. E. (2014) · Research on Social Work Practice · doi:10.1177/1049731513505778 · Cited by 268 · see citing works
Boyd, M. R., Powell, B. J., Endicott, D., & Lewis, C. C. (2018) · Behavior Therapy · doi:10.1016/j.beth.2017.11.012 · Cited by 137 · see citing works
Importance of Mixed Methods in Pragmatic Trials and Dissemination and Implementation Research
Albright, K., Gechter, K., & Kempe, A. (2013) · Academic Pediatrics · doi:10.1016/j.acap.2013.06.010 · Cited by 146 · see citing works
Rixon, L., Baron, J., McGale, N., Lorencatto, F., Francis, J., & Davies, A. (2016) · BMC Health Services Research · doi:10.1186/s12913-016-1904-6 · Cited by 117 · see citing works
Burnout and Leadership Style in Behavioral Health Care: a Literature Review
Kelly, R. J., & Hearld, L. R. (2020) · The Journal of Behavioral Health Services & Research · doi:10.1007/s11414-019-09679-z · Cited by 118 · see citing works
Self-citationWare, N., Pisarski, E. E., Tam, M., Wyatt, M. A., Atukunda, E., Musiimenta, A., Bangsberg, D., & Haberer, J. (2016) · AIDS (London) · doi:10.1097/qad.0000000000001035 · Cited by 93 · see citing works
Ludwick, T., Turyakira, E., Kyomuhangi, T., Manalili, K., Robinson, S., & Brenner, J. L. (2018) · Human Resources for Health · doi:10.1186/s12960-018-0272-1 · Cited by 73 · see citing works
Creating and Supporting a Mixed Methods Health Services Research Team
Bowers, B. J., Cohen, L. W., Elliot, A., Grabowski, D. C., Fishman, N. W., Sharkey, S. S., Zimmerman, S. I., Horn, S. D., & Kemper, P. F. (2013) · Health Services Research · doi:10.1111/1475-6773.12118 · Cited by 74 · see citing works
Lam, H., Quinn, M. T., Cipriano-Steffens, T., Jayaprakash, M., Koebnick, E., Randal, F. T., Liebovitz, D., Polite, B., & Kim, K. C. (2021) · Implementation Science Communications · doi:10.1186/s43058-021-00150-9 · Cited by 64 · see citing works
Guise, J., Chang, C., Viswanathan, M., Glick, S., Treadwell, J. R., Umscheid, C. A., Whitlock, E. P., Fu, R. F., Berliner, E., Paynter, R. A., et al. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.06.010 · Cited by 56 · see citing works
Financial incentives and maternal health: where do we go from here?
Morgan, L. L., Stanton, M. E., Higgs, E. S., Balster, R. L., Bellows, B., Brandes, N., Comfort, A. B., Eichler, R., Glassman, A. L., Hatt, L. E., et al. (2013) · PubMed Central · Cited by 51 · see citing works
Using qualitative comparative analysis in a systematic review of a complex intervention
Kahwati, L. C., Jacobs, S., Kane, H., Lewis, M. A., Viswanathan, M., & Golin, C. E. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0256-y · Cited by 50 · see citing works
Jalali, M. S., Rahmandad, H., Bullock, S. L., Lee, S. H., Gittelsohn, J., & Ammerman, A. S. (2018) · Social Science & Medicine · doi:10.1016/j.socscimed.2018.12.021 · Cited by 49 · see citing works
With the PMC study team, Templeton, A. R., Young, L. C., Bish, A., Gnich, W., Cassie, H., Treweek, S. P., Bonetti, D. L., Stirling, D., MacPherson, L. M. D., et al. (2015) · Implementation Science · doi:10.1186/s13012-015-0366-2 · Cited by 45 · see citing works
Integrating Mixed Methods in Health Services and Delivery System Research
Miller, W. L., Crabtree, B. F., Harrison, M. I., & Fennell, M. L. (2013) · Health Services Research · doi:10.1111/1475-6773.12123 · Cited by 46 · see citing works
Lindig, A., Hahlweg, P., Christalle, E., & Scholl, I. (2020) · BMJ Open · doi:10.1136/bmjopen-2019-034380 · Cited by 44 · see citing works
Zapka, J., Simpson, K., Hiott, L., Langston, L., Fakhry, S. M., & Ford, D. W. (2013) · BMC Health Services Research · doi:10.1186/1472-6963-13-33 · Cited by 43 · see citing works
“A Natural Progression”: Australian Women's Attitudes About an Individualized Breast Screening Model
Lippey, J., Keogh, L. A., Mann, G. B., Campbell, I., & Forrest, L. E. (2019) · Cancer Prevention Research · doi:10.1158/1940-6207.capr-18-0443 · Cited by 42 · see citing works
Lee, S.-Y. D., Weiner, B. J., Harrison, M. I., & Belden, C. (2012) · Medical Care Research and Review · doi:10.1177/1077558712458539 · Cited by 41 · see citing works
Wu, H. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.106124 · Cited by 34 · see citing works
Gilmer, T., Katz, M. L., Stefančić, A., & Palinkas, L. A. (2013) · Health Services Research · doi:10.1111/1475-6773.12119 · Cited by 31 · see citing works
Herman, P. M., Dodds, S. E., Logue, M. D., Abraham, I. L., Rehfeld, R. A., Grizzle, A. J., Urbine, T. F., Horwitz, R., Crocker, R. L., & Maizes, V. H. (2014) · BMC Complementary and Alternative Medicine · doi:10.1186/1472-6882-14-132 · Cited by 29 · see citing works
Replication or Innovation? Structuration in Policy Implementation
Sandfort, J. R., & Moulton, S. (2019) · Perspectives on Public Management and Governance · doi:10.1093/ppmgov/gvz029 · Cited by 26 · see citing works
McKay, V. R., Hoffer, L., Combs, T., & Dolcini, M. M. (2018) · Implementation Science · doi:10.1186/s13012-018-0767-0 · Cited by 24 · see citing works
Cancelliere, C., Sutton, D. A., Côté, P., French, S. D., Taylor-Vaisey, A. L., & Mior, S. A. (2019) · Implementation Science · doi:10.1186/s13012-019-0931-1 · Cited by 22 · see citing works
Precision Implementation: An Approach to Mechanism Testing in Implementation Research
Frank, H. E., Kemp, J. J., Benito, K., & Freeman, J. B. (2022) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-022-01218-x · Cited by 20 · see citing works
Laker, C., Cella, M., Callard, F. J., & Wykes, T. (2019) · Journal of Psychiatric and Mental Health Nursing · doi:10.1111/jpm.12577 · Cited by 18 · see citing works
Communities of practice in Alberta Health Services: advancing a learning organisation
Auer, A., Hanson, P. A., Brady-Fryer, B., Alati-it, J., & Johnson, A. L. (2020) · Health Research Policy and Systems · doi:10.1186/s12961-020-00603-y · Cited by 18 · see citing works
Making, A. H. I. P. C. F. D. U. F. D. (2022) · Global Health: Science and Practice Journal · doi:10.9745/ghsp-d-21-00666 · Cited by 15 · see citing works
Treatment manuals, training and successful provision of stop smoking behavioural support
Brose, L. S., McEwen, A., Michie, S., West, R., Chew, X. Y., & Lorencatto, F. (2015) · Behaviour Research and Therapy · doi:10.1016/j.brat.2015.05.013 · Cited by 17 · see citing works
Cooperation and conflict in intra‐hospital transfers: A qualitative analysis
Germack, H. D., Fekieta, R., Britton, M. C., Feder, S. L., Rosenberg, A. P., & Chaudhry, S. I. (2019) · Nursing Open · doi:10.1002/nop2.434 · Cited by 16 · see citing works
Dickinson, L. M., Dickinson, W. P., Nutting, P. A., Fisher, L., Harbrecht, M. G., Crabtree, B. F., Glasgow, R. E., & West, D. R. (2014) · Journal of General Internal Medicine · doi:10.1007/s11606-014-3131-3 · Cited by 16 · see citing works
Costa, L., Bobay, K. L., Hughes, R. G., Bahr, S. J., Siclovan, D. M., Nuccio, S. A., & Weiss, M. (2020) · Nursing Outlook · doi:10.1016/j.outlook.2020.07.005 · Cited by 15 · see citing works
Utilising a collective case study system theory mixed methods approach: a rural health example
Adams, R. B., Jones, A., Lefmann, S. A., & Sheppard, L. A. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-94 · Cited by 15 · see citing works
Care integration within and outside health system boundaries
Singer, S. J., Sinaiko, A. D., Tietschert, M. V., Kerrissey, M. J., Phillips, R. S., Martin, V., Joseph, G., Bahadurzada, H., & Agniel, D. M. (2020) · Health Services Research · doi:10.1111/1475-6773.13578 · Cited by 15 · see citing works
Evidence-Based Programs in “Real World” Settings: Finding the Best Fit
Powell, B. J., Bosk, E. A., Wilen, J. S., Danko, C. M., Scoyoc, A. V., & Banman, A. (2015) · Child maltreatment · doi:10.1007/978-3-319-16327-7_7 · Cited by 14 · see citing works
Barie, P. S., & Ho, V. P. (2012) · Surgical Clinics of North America · doi:10.1016/j.suc.2012.09.001 · Cited by 12 · see citing works
Technology-enabled performance monitoring in intensive care: An integrative literature review
DeMellow, J. M., & Kim, T. Y. (2018) · Intensive and Critical Care Nursing · doi:10.1016/j.iccn.2018.07.003 · Cited by 12 · see citing works
Watson, D. P., Snow‐Hill, N. L., Saldana, L., Walden, A. L., Staton, M. D., Kong, A., & Donenberg, G. R. (2020) · Implementation Research and Practice · doi:10.1177/2633489520974974 · Cited by 13 · see citing works
Variations in Patient-Centered Medical Home Capacity
Hearld, L. R., Weech-Maldonado, R. J., & Asagbra, O. E. (2013) · Medical Care Research and Review · doi:10.1177/1077558713498117 · Cited by 11 · see citing works
Self-citationHoben, M., Berendonk, C., Buscher, I., Quasdorf, T., Riesner, C., Wilborn, D., & Behrens, J. (2014) · International Journal of Health Professions · doi:10.2478/ijhp-2014-0002 · Cited by 10 · see citing works
Technical Efficiency and Organ Transplant Performance: A Mixed-Method Approach
De‐Pablos‐Heredero, C., Fernández‐Renedo, C., & Medina, J. (2015) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph120504869 · Cited by 11 · see citing works
Tietbohl, C. K. (2012) · The 34th Annual Meeting of the Society for Medical Decision Making · Cited by 11 · see citing works
Dir, A. L., Watson, D. P., Zhiss, M., Taylor, L., Bray, B. C., & McGuire, A. B. (2021) · Trials · doi:10.1186/s13063-021-05065-3 · Cited by 11 · see citing works
Burn, A., Vainre, M., Humphrey, A., & Howarth, E. (2020) · Implementation Science Communications · doi:10.1186/s43058-020-00078-6 · Cited by 13 · see citing works
Implementing Lean in Academic Primary Care
Daaleman, T. P., Brock, D., Gwynne, M. D., Weir, S., Dickinson, I., Willis, B., & Reid, A. (2018) · Quality Management in Health Care · doi:10.1097/qmh.0000000000000173 · Cited by 10 · see citing works
Leppin, A. L., Boehmer, K. R., Branda, M. E., Shah, N. D., Hargraves, I. G., Dick, S. R., Elwyn, G. J., Ting, H. H., Ye, S., Gilles, R., et al. (2019) · BMC Health Services Research · doi:10.1186/s12913-019-4055-8 · Cited by 10 · see citing works
Ha, D. A., Oanh, T. T. M., Nguyen, H. L., Chiriboga, G., Goldberg, R. J., Phan, V. H., Nguyen, C. T., Giang, N. H., Pham, H. V., Nguyen, T., et al. (2020) · Trials · doi:10.1186/s13063-020-04917-8 · Cited by 11 · see citing works
Kahwati, L. C., Kelly, B., Johnson, M., Clark, R., & Viswanathan, M. (2022) · Research Synthesis Methods · doi:10.1002/jrsm.1602 · Cited by 9 · see citing works
A method to determine the impact of patient-centered care interventions in primary care
Daaleman, T. P., Shea, C. M., Halladay, J. R., & Reed, D. (2014) · Patient Education and Counseling · doi:10.1016/j.pec.2014.09.009 · Cited by 9 · see citing works
Evaluating Infrastructure Development in Complex Home Visiting Systems
Hargreaves, M. K., Cole, R. P., Coffee‐Borden, B., Paulsell, D., & Boller, K. (2013) · American Journal of Evaluation · doi:10.1177/1098214012469271 · Cited by 9 · see citing works
Moore, J. E., Uka, S., Vogel, J. P., Timmings, C., Rashid, S., Gülmezoglu, A. M., & Straus, S. E. (2016) · BMC Public Health · doi:10.1186/s12889-016-3641-5 · Cited by 7 · see citing works
Laker, C., Cella, M., Agbediro, D., Callard, F. J., & Wykes, T. (2019) · BMC Psychiatry · doi:10.1186/s12888-019-2370-6 · Cited by 7 · see citing works
Austin, J. D., Rutten, L. J. F., Fischer, K. J., Ridgeway, J. L., Minteer, S. A., Griffin, J. M., Pachman, D. R., Ruddy, K. J., & Cheville, A. L. (2024) · JCO Oncology Practice · doi:10.1200/op.24.00280 · Cited by 6 · see citing works
Hughes, A. M., Lin, E. H., Hussain, R. A., Gibson, G., Jurasic, M. M., Sharp, L. K., Hubbard, C. C., Poggensee, L., Evans, C. T., McGregor, J. C., et al. (2022) · Journal of the American Pharmacists Association · doi:10.1016/j.japh.2022.08.001 · Cited by 6 · see citing works
An examination of the factor structure of TeamSTEPPS measures in school mental health teams
Wolk, C. B., Locke, J. J., Eduardo Salas, A., Eiraldi, R. B., Cronholm, P. F., & Mandell, D. S. (2019) · Journal of Psychologists and Counsellors in Schools · doi:10.1017/jgc.2019.18 · Cited by 6 · see citing works
Predictors of Temporary Profile Days Among U.S. Army Active Duty Soldiers
Taylor-Clark, T. M., Loan, L. A., Swiger, P. A., Hearld, L. R., Li, P., & Patrician, P. A. (2022) · Military Medicine · doi:10.1093/milmed/usab558 · Cited by 4 · see citing works
Self-citationGilmer, T., Henwood, B. F., McGovern, N., Hurst, S., Burgdorf, J. R., & Innes-Gomberg, D. (2017) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-016-0786-6 · Cited by 3 · see citing works
Greiner, M. G., Shulman, M., Opara, O., Potter, K. M., Voronca, D., Tafessu, H. M., Hefner, K. R., Hamilton, A., Scheele, C., Ho, R., et al. (2023) · Contemporary Clinical Trials · doi:10.1016/j.cct.2023.107148 · Cited by 3 · see citing works
Evaluating the effect of a national collaborative: a cautionary tale
Sales, A. E., & Saint, S. K. (2012) · BMJ Quality & Safety · doi:10.1136/bmjqs-2012-001065 · Cited by 3 · see citing works
Meng, M., Chen, S., Yang, D., Zhang, X., Yang, Y., Wang, Z., Zhāng, J., Li, X., & Hao, Y. (2025) · BMC Nursing · doi:10.1186/s12912-025-03070-4 · Cited by 2 · see citing works
Innovation and Service Delivery: Conceptual and Contextual Framework
Momen, M. N., & Ferdous, J. (2023) · New frontiers in regional science: Asian perspectives · doi:10.1007/978-981-99-0424-2_1 · Cited by 2 · see citing works
Allen, M. L., Somasundaram, K., Leslie, K., & Manski-Nankervis, J.-A. E. (2024) · Global Implementation Research and Applications · doi:10.1007/s43477-024-00124-8 · Cited by 2 · see citing works
Maiorana, A., Bolton, A. T., Koester, K. A., Bourdeau, B., DeLorenzo, L., Rebchook, G., Steward, W. T., Coffey, S., Bacon, O., & Myers, J. J. (2026) · Implementation Science Communications · doi:10.1186/s43058-026-00863-9 · Cited by 1 · see citing works
Taylor-Clark, T. M., Hearld, L. R., Loan, L. A., Swiger, P. A., Li, P., & Patrician, P. A. (2021) · Military Medicine · doi:10.1093/milmed/usab515 · Cited by 1 · see citing works
Self-citationSowon, K., & Chigona, W. (2022) · The Electronic Journal of Information Systems in Developing Countries · doi:10.1002/isd2.12212 · Cited by 1 · see citing works
Conceptualising Healthcare-Seeking as an Activity to Explain Technology Use: A Case of M-healthPreprint
Sowon, K., & Chigona, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.10082 · Cited by 1 · see citing works
Implementation of coordinated healthy lifestyle promotion in primary care : Process and outcomes
Thomas, K. (2015) · Linköping University medical dissertations · doi:10.3384/diss.diva-121492
Keith, R. E., Heitkamp, S., Little, J., Peebles, V., Sarwar, R., Petersen, D. M., & O’Malley, A. S. (2023) · American Journal of Evaluation · doi:10.1177/10982140221116096
McPhee, J., Huett, B., Brogan, L. A., McCurdy, E., NeMoyer, A., Kreimer, R., DeYoung, L., & Goldstein, N. E. S. (2023) · Journal of Applied Juvenile Justice Services · doi:10.52935/23.13316.05
Burn, A.-M., Vainre, M., Humphrey, A., & Howarth, E. (2020) · Figshare · doi:10.6084/m9.figshare.13093970
Burn, A.-M., Vainre, M., Humphrey, A., & Howarth, E. (2020) · Figshare · doi:10.6084/m9.figshare.13093967
Arshad, H., Gillani, A. H., Rehman, J. U., Mallah, M. A., Arshed, M., & Fang, Y. (2026) · Substance Abuse Treatment Prevention and Policy · doi:10.1186/s13011-025-00689-z
- Authors
- Alexander, J. A., & Hearld, L. R.
- Affiliations
- University of Michigan (US)
- University of Alabama at Birmingham (US)
- Corresponding author
- Alexander, Jeffrey A
- ORCID
- Hearld, Larry R
- Venue
- Implementation Science
- Published
- 2012-03-12
- Volume
- 7
- Issue
- 1
- Pages
- 15-15
- ISSN
- 1748-5908 (ISSN-L)
- PMC
- PMC3317852
- OpenAlex
- W2157452301
- Cited by
- 78
- References
- 53
- Citations by year
- 2026: 2 · 2025: 1 · 2024: 3 · 2023: 4 · 2022: 6 · 2021: 4 · 2020: 10 · 2019: 9 · 2018: 5 · 2017: 3 · 2016: 4 · 2015: 5 · 2014: 7 · 2013: 11 · 2012: 4
- FWCI
- 15.36
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Implementation Science (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Deep Blue (University of Michigan) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- U.S. Department of Health and Human Services
- Agency for Healthcare Research and Quality
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Social Sciences › Decision Sciences › Management Science and Operations Research › Evaluation and Performance Assessment
- Health Sciences › Health Professions › Health Information Management › Healthcare Quality and Management
- Keywords
- readiness to change, intervention fidelity, sustainability, treatment theory, mixed methods research design
- MeSH terms
- Delivery of Health Care, Humans, Organizational Innovation, Research Design, Therapeutics, Time Factors, Treatment Outcome, Evidence-Based Practice
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
