Interaction of Theory and Practice to Assess External Validity

Leviton, Laura C.; Trujillo, Mathew D.

doi:10.1177/0193841x15625289

Cited by 36Health Policy Implementation ScienceEvaluation and Performance AssessmentAdvanced Causal Inference Techniques

Interaction of Theory and Practice to Assess External Validity

Leviton, Laura C., Trujillo, Mathew D.·2017·Evaluation Review

DOI 10.1177/0193841x15625289

Open this paper in CoChat — free account

Abstract

Background: Variations in local context bedevil the assessment of external validity: the ability to generalize about effects of treatments. For evaluation, the challenges of assessing external validity are intimately tied to the translation and spread of evidence-based interventions. This makes external validity a question for decision makers, who need to determine whether to endorse, fund, or adopt interventions that were found to be effective and how to ensure high quality once they spread. Objective: To present the rationale for using theory to assess external validity and the value of more systematic interaction of theory and practice. Methods: We review advances in external validity, program theory, practitioner expertise, and local adaptation. Examples are provided for program theory, its adaptation to diverse contexts, and generalizing to contexts that have not yet been studied. The often critical role of practitioner experience is illustrated in these examples. Work is described that the Robert Wood Johnson Foundation is supporting to study treatment variation and context more systematically. Results: Researchers and developers generally see a limited range of contexts in which the intervention is implemented. Individual practitioners see a different and often a wider range of contexts, albeit not a systematic sample. Organized and taken together, however, practitioner experiences can inform external validity by challenging the developers and researchers to consider a wider range of contexts. Researchers have developed a variety of ways to adapt interventions in light of such challenges. Conclusions: In systematic programs of inquiry, as opposed to individual studies, the problems of context can be better addressed. Evaluators have advocated an interaction of theory and practice for many years, but the process can be made more systematic and useful. Systematic interaction can set priorities for assessment of external validity by examining the prevalence and importance of context features and treatment variations. Practitioner interaction with researchers and developers can assist in sharpening program theory, reducing uncertainty about treatment variations that are consistent or inconsistent with the theory, inductively ruling out the ones that are harmful or irrelevant, and helping set priorities for more rigorous study of context and treatment variation.

Integrity signals

Registry

Crossref · 10.1177/0193841x15625289

Cites

83 of 83 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

The classification system's focus on targets of action (levels and determinants) has potential to aid in selecting and strategically combining multiple strategies.
“Consistent with the intent of Proctor et al.’s guidelines, this classification system’s focus on targets of action (levels and determinants) has potential to aid in selecting and strategically combining multiple strategies [45, 46].”
From Beyond “implementation strategies”: classifying the full range of strategies used in implementation science and practice (2017) · cites this paper as [45]

The need to belong: Desire for interpersonal attachments as a fundamental human motivation.
Baumeister, R. F., & Leary, M. R. (1995) · Psychological Bulletin · doi:10.1037//0033-2909.117.3.497 · Cited by 23,245 · see citing works
Governing the Commons: The Evolution of Institutions for Collective Action
Ostrom, E. (1990) · doi:10.1017/cbo9780511807763 · Cited by 20,653 · see citing works
Diffusion of innovations
Valente, T. W. (2003) · Genetics in Medicine · doi:10.1097/01.gim.0000061743.67794.c4 · Cited by 20,555 · see citing works
The reflective practitioner: How professionals think in action
Stewart, C. J. (1984) · Patient Education and Counseling · doi:10.1016/0738-3991(84)90022-3 · Cited by 19,751 · see citing works
Governing the Commons: The Evolution of Institutions for Collective Action
Field, B. C., & Elinor Ostrom, B. (1992) · Land Economics · doi:10.2307/3146384 · Cited by 19,082 · see citing works
Untitled work
(1963) · Medical Entomology and Zoology · Cited by 15,085 · see citing works
Untitled work
(2002) · Cited by 13,211 · see citing works
Construct validity in psychological tests.
Cronbach, L. J., & Meehl, P. E. (1955) · Psychological Bulletin · doi:10.1037/h0040957 · Cited by 11,195 · see citing works
Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide
Hoffmann, T. C., Glasziou, P. P., Boutron, I., Milne, R., Perera, R., Moher, D., Altman, D. G., Barbour, V., Macdonald, H., Johnston, M., et al. (2014) · BMJ · doi:10.1136/bmj.g1687 · Cited by 10,548 · see citing works
Evaluating the public health impact of health promotion interventions: the RE-AIM framework.
Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999) · American Journal of Public Health · doi:10.2105/ajph.89.9.1322 · Cited by 7,217 · see citing works
Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation
Durlak, J. A., & DuPre, E. P. (2008) · American Journal of Community Psychology · doi:10.1007/s10464-008-9165-0 · Cited by 5,304 · see citing works
Cultivating Communities of Practice: A Guide to Managing Knowledge
Wenger, É., & Snyder, W. M. (2002) · CERN Document Server (European Organization for Nuclear Research) · Cited by 4,769 · see citing works
Cultivating Communities of Practice: A Guide to Managing Knowledge
Cuddy, C. (2002) · The Bottom Line Managing Library Finances · doi:10.1108/bl.2002.17015bae.001 · Cited by 3,935 · see citing works
Mind over Machine: The Power of Human Intuition and Expertise in the Era of the Computer
Dreyfus, H. L., Drey-fus, S. E., & Zadeh, L. A. (1987) · IEEE Expert · doi:10.1109/mex.1987.4307079 · Cited by 3,172 · see citing works
Predicting and Changing Behavior
Fishbein, M., & Ajzen, I. (2011) · doi:10.4324/9780203838020 · Cited by 2,807 · see citing works
A primer on decision making : how decisions happen
Heath, C., & March, J. G. (1994) · Medical Entomology and Zoology · Cited by 2,411 · see citing works
Welcome to Implementation Science
Eccles, M. P., & Mittman, B. S. (2006) · Implementation Science · doi:10.1186/1748-5908-1-1 · Cited by 2,254 · see citing works
Communication And Persuasion
(2017) · doi:10.5040/9798400682018.ch-003 · Cited by 2,247 · see citing works
Communication and persuasion
Lumsdaine, A. A. (1954) · Educational Technology Research and Development · doi:10.1007/bf02713272 · Cited by 2,237 · see citing works
Contending with group image: The psychology of stereotype and social identity threat
Steele, C. M., Spencer, S. J., & Aronson, J. (2002) · Advances in Experimental Social Psychology · doi:10.1016/s0065-2601(02)80009-0 · Cited by 2,201 · see citing works
A question of belonging: Race, social fit, and achievement.
Walton, G. M., & Cohen, G. L. (2007) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.92.1.82 · Cited by 2,163 · see citing works
A Brief Social-Belonging Intervention Improves Academic and Health Outcomes of Minority Students
Walton, G. M., & Cohen, G. L. (2011) · Science · doi:10.1126/science.1198364 · Cited by 1,992 · cited 1× · see citing works
“The same intervention at a second elite private institution caused a positive impact on GPA 3 years postintervention (Walton & Cohen, 2011).”
Background
Evaluation: A Systematic Approach
Wallace, D. P. (2005) · Library & Information Science Research · doi:10.1016/j.lisr.2004.09.009 · Cited by 1,938 · see citing works
Disseminating Innovations in Health Care
Berwick, D. M. (2003) · JAMA · doi:10.1001/jama.289.15.1969 · Cited by 1,878 · see citing works
Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use
Patton, M. Q. (2010) · Cited by 1,740 · see citing works
Systems of psychotherapy a transtheoretical analysis
Prochaska, J. O., & Norcross, J. C. (2010) · Cited by 1,497 · see citing works
Complex interventions: how “out of control” can a randomised controlled trial be?
Hawe, P., Shiell, A., & Riley, T. (2004) · BMJ · doi:10.1136/bmj.328.7455.1561 · Cited by 1,244 · see citing works
Culturally adapted mental health intervention: a meta-analytic review
Griner, D., & Smith, T. B. (2006) · Psychotherapy: Theory, Research, Practice, Training · doi:10.1037/0033-3204.43.4.531 · Cited by 1,236 · see citing works
Designing Evaluations of Educational and Social Programs.
Hendrickson, L., & Cronbach, L. J. (1983) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2068975 · Cited by 1,035 · see citing works
Re-revisiting Andersen’s Behavioral Model of Health Services Use: A systematic review of studies from 1998–2011
Babitsch, B., Gohl, D., & Von Lengerke, T. (2012) · German Medical Science GMS Publishing House · doi:10.3205/psm000089 · Cited by 1,011 · see citing works
The primary factors that characterize effective interventions with juvenile offenders: A meta-analytic overview
Lipsey, M. W. (2009) · Victims & Offenders · doi:10.1080/15564880802612573 · Cited by 962 · see citing works
Sense of Belonging as a Predictor of Intentions to Persist Among African American and White First-Year College Students
Hausmann, L. R. M., Schofield, J. W., & Woods, R. L. (2007) · Research in Higher Education · doi:10.1007/s11162-007-9052-9 · Cited by 954 · see citing works
Evaluation : methods for studying programs and policies
Weiss, C. H. (1997) · Cited by 931 · see citing works
Population-Based Prevention of Child Maltreatment: The U.S. Triple P System Population Trial
Prinz, R. J., Sanders, M. R., Shapiro, C. J., Whitaker, D. J., & Lutzker, J. R. (2009) · Prevention Science · doi:10.1007/s11121-009-0123-3 · Cited by 833 · see citing works
Issues and challenges in the design of culturally adapted evidence-based interventions
Castro, F. G., Barrera, M., & Holleran Steiker, L. K. (2010) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-033109-132032 · Cited by 796 · see citing works
Issues in Disseminating and Replicating Effective Prevention Programs
Elliott, D. S., & Mihalic, S. (2004) · Prevention Science · doi:10.1023/b:prev.0000013981.28071.52 · Cited by 794 · see citing works
Demystifying theory and its use in improvement
Davidoff, F., Dixon-Woods, M., Leviton, L., & Michie, S. (2015) · BMJ Quality & Safety · doi:10.1136/bmjqs-2014-003627 · Cited by 714 · cited 2× · see citing works
“Practitioner knowledge is based on both training and lived experience (Davidoff et al., 2015; Stake & Schwandt, 2006).”
BackgroundSelf-citation
Evidence for the impact of quality improvement collaboratives: systematic review
Schouten, L. M. T., Hulscher, M., van Everdingen, J. J. E., Huijsman, R., & Grol, R. P. T. M. (2008) · BMJ · doi:10.1136/bmj.39570.749884.be · Cited by 683 · see citing works
A Comprehensive Meta-Analysis of Triple P-Positive Parenting Program Using Hierarchical Linear Modeling: Effectiveness and Moderating Variables
Nowak, C., & Heinrichs, N. (2008) · Clinical Child and Family Psychology Review · doi:10.1007/s10567-008-0033-0 · Cited by 621 · see citing works
Two brief interventions to mitigate a “chilly climate” transform women’s experience, relationships, and achievement in engineering.
Walton, G. M., Logel, C. R., Peach, J. M., Spencer, S. J., & Zanna, M. P. (2014) · Journal of Educational Psychology · doi:10.1037/a0037461 · Cited by 572 · see citing works
Assessing the long-term effects of the Safe dates program and a booster in preventing and reducing adolescent dating violence victimization and perpetration
Foshee, V. A., Bauman, K. E., Ennett, S. T., Linder, G. F., Benefield, T., & Suchindran, C. (2004) · American Journal of Public Health · doi:10.2105/ajph.94.4.619 · Cited by 522 · see citing works
Critical dimensions in the choice and maintenance of successful treatments: Strength, integrity, and effectiveness.
Yeaton, W. H., & SECHREST, L. B. (1981) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.49.2.156 · Cited by 501 · see citing works
Critical dimensions in the choice and maintenance of successful treatments: Strength, integrity, and effectiveness.
Yeaton, W. H., & SECHREST, L. B. (1981) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.49.2.156 · Cited by 407 · see citing works
Evidence-based Kernels: Fundamental Units of Behavioral Influence
Embry, D. D., & Biglan, A. (2008) · Clinical Child and Family Psychology Review · doi:10.1007/s10567-008-0036-x · Cited by 396 · see citing works
Stepping stones triple P: an RCT of a parenting program with parents of a child diagnosed with an autism Spectrum disorder
Whittingham, K., Sofronoff, K., Sheffield, J., & Sanders, M. R. (2009) · Journal of Abnormal Child Psychology · doi:10.1007/s10802-008-9285-x · Cited by 357 · see citing works
Theory as method: small theories of treatments
Lipsey, M. W. (1993) · New Directions for Program Evaluation · doi:10.1002/ev.1637 · Cited by 322 · see citing works
Untitled work
Rohrbach (2006) · doi:10.1177/0163278706290408 · Cited by 300 · see citing works
The fidelity‐adaptation debate: Implications for the implementation of public sector social programs
Blakely, C. H., Mayer, J. P., Gottschalk, R. G., Schmitt, N., Davidson, W. S., Roitman, D. B., & Emshoff, J. G. (1987) · American Journal of Community Psychology · doi:10.1007/bf00922697 · Cited by 297 · see citing works
An Outcome Evaluation of the Implementation of the Triple P‐Positive Parenting Program in Hong Kong
Leung, C., Sanders, M. R., Leung, S., MAK, R., & Lau, J. T. F. (2003) · Family Process · doi:10.1111/j.1545-5300.2003.00531.x · Cited by 297 · see citing works
Understanding the Common Elements of Evidence-Based Practice
Chorpita, B. F., Becker, K. D., & Daleiden, E. L. (2007) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1097/chi.0b013e318033ff71 · Cited by 292 · see citing works
Moving From the Lab to the Field: The Role of Fidelity and Achieved Relative Intervention Strength
Hulleman, C. S., & Cordray, D. S. (2009) · Journal of Research on Educational Effectiveness · doi:10.1080/19345740802539325 · Cited by 291 · see citing works
Quality of implementation: developing measures crucial to understanding the diffusion of preventive interventions
Dusenbury, L., Brannigan, R., Hansen, W. B., Walsh, J., & Falco, M. (2005) · Health Education Research · doi:10.1093/her/cyg134 · Cited by 278 · see citing works
Factors Associated with Fidelity to Substance Use Prevention Curriculum Guides in the Nation's Middle Schools
Ringwalt, C. L., Ennett, S. T., Johnson, R. E., Rohrbach, L. A., Simons‐Rudolph, A., Vincus, A. A., & Thorne, J. (2003) · Health Education & Behavior · doi:10.1177/1090198103030003010 · Cited by 273 · see citing works
The role of method in treatment effectiveness research: Evidence from meta-analysis.
Wilson, D. B., & Lipsey, M. W. (2001) · Psychological Methods · doi:10.1037/1082-989x.6.4.413 · Cited by 261 · see citing works
Disruptive innovations for designing and diffusing evidence-based interventions
Rotheram-Borus, M. J., Swendeman, D., & Chorpita, B. F. (2012) · American Psychologist · doi:10.1037/a0028180 · Cited by 210 · see citing works
Outreach-based HIV prevention for injecting drug users: a review of published outcome data.
Coyle, S. L., Needle, R. H., & Normand, J. (1998) · PubMed Central · Cited by 192 · see citing works
A Randomized Controlled Trial of Culturally Tailored Dance and Reducing Screen Time to Prevent Weight Gain in Low-Income African American Girls
Robinson, T. N., Matheson, D., KRAEMER, H. C., Wilson, D. M., Obarzanek, E., Thompson, N. S., Alhassan, S., Spencer, T. R., Haydel, K. F., Fujimoto, M., et al. (2010) · Archives of Pediatrics and Adolescent Medicine · doi:10.1001/archpediatrics.2010.197 · Cited by 166 · see citing works
Assessing the Effects of Families for Safe Dates, a Family-Based Teen Dating Abuse Prevention Program
Foshee, V. A., Reyes, H. L. M., Ennett, S. T., Cance, J. D., Bauman, K. E., & Bowling, J. M. (2012) · Journal of Adolescent Health · doi:10.1016/j.jadohealth.2011.12.029 · Cited by 161 · see citing works
Active for Life
Wilcox, S., Dowda, M. C., Leviton, L. C., Bartlett-Prescott, J., Bazzarre, T. L., Campbell‐Voytal, K., Carpenter, R. A., Castro, C. M., Dowdy, D. M., Dunn, A. L., et al. (2008) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2008.07.001 · Cited by 152 · see citing works
Self-citation
The bottom-up approach to integrative validity: A new perspective for program evaluation
Chen, H. T. (2009) · Evaluation and Program Planning · doi:10.1016/j.evalprogplan.2009.10.002 · Cited by 141 · see citing works
Translating research into practice: transitional care for older adults
Naylor, M. D., Feldman, P. H., Keating, S. A., Koren, M. J., Kurtzman, E. T., Maccoy, M. C., & Krakauer, R. S. (2009) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2009.01308.x · Cited by 133 · see citing works
A Focused Approach to Assessing Program Fidelity
Hill, L. G., Maucione, K., & Hood, B. (2006) · Prevention Science · doi:10.1007/s11121-006-0051-4 · Cited by 120 · see citing works
Using Theory to Improve Program and Policy Evaluations.
Sabatier, P., Chen, H.-T., & Rossi, P. H. (1994) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2076456 · Cited by 93 · see citing works
On Discerning Quality in Evaluation
Stake, R. E., & Schwandt, T. A. (2006) · SAGE Publications Ltd eBooks · doi:10.4135/9781848608078.n18 · Cited by 93 · see citing works
Using Theory to Improve Program and Policy Evaluations.
Freeman, D. M., Chen, H.-T., & Rossi, P. H. (1993) · Social Forces · doi:10.2307/2580149 · Cited by 76 · see citing works
Emerging Methods in Family Research
McHale, S. M., Amato, P. R., & BOOTH, A. R. (2013) · National symposium on family issues · doi:10.1007/978-3-319-01562-0 · Cited by 66 · see citing works
A framework to assess the development and capacity of non-profit agencies
Schuh, R. G., & Leviton, L. C. (2006) · Evaluation and Program Planning · doi:10.1016/j.evalprogplan.2005.12.001 · Cited by 65 · see citing works
Self-citation
Increasing the generalisability of improvement research with an improvement replication programme: Figure 1
Øvretveit, J., Leviton, L. C., & Parry, G. J. (2011) · BMJ Quality & Safety · doi:10.1136/bmjqs.2010.046342 · Cited by 56 · see citing works
Self-citation
When is a Program Ready for Rigorous Impact Evaluation? The Role of a Falsifiable Logic Model
Epstein, D., & Klerman, J. A. (2012) · Evaluation Review · doi:10.1177/0193841x12474275 · Cited by 50 · see citing works
Protective Factors Associated with Preadolescent Violence: Preliminary Work on a Cultural Model
Jagers, R. J., Sydnor, K. D., Mouttapa, M., & Flay, B. R. (2007) · American Journal of Community Psychology · doi:10.1007/s10464-007-9121-4 · Cited by 46 · see citing works
Innovation in Education and Criminal Justice: Measuring Fidelity of Implementation and Program Effectiveness
Emshoff, J. G., Blakely, C. H., Gottschalk, R. G., Mayer, J., Davidson, W. S., & Erickson, S. K. (1987) · Educational Evaluation and Policy Analysis · doi:10.3102/01623737009004300 · Cited by 46 · see citing works
Adapting Evidence-Based Interventions Using a Common Theory, Practices, and Principles
Rotheram‐Borus, M. J., Swendeman, D. T., & Becker, K. D. (2013) · Journal of Clinical Child & Adolescent Psychology · doi:10.1080/15374416.2013.836453 · Cited by 41 · see citing works
State-based Home Visiting: Strengthening Programs Through State Leadership
Johnson, K. (2009) · Columbia Academic Commons (Columbia University) · doi:10.7916/d8h70qhh · Cited by 38 · see citing works
Research in Social Problems and Public Policy
(2012) · Research in social problems and public policy · doi:10.1108/s0196-1152(2012)0000020046 · Cited by 38 · see citing works
Mind over machine: The power of human intuition and expertise in the era of the computer
Bouwman, M. J. (1988) · European Journal of Operational Research · doi:10.1016/0377-2217(88)90183-x · Cited by 36 · see citing works
Measuring Fidelity and Adaptation
Bishop, D. C., Pankratz, M. M., Hansen, W. B., Albritton, J., Albritton, L. P., & Strack, J. (2013) · Evaluation & the Health Professions · doi:10.1177/0163278713476882 · Cited by 35 · see citing works
The Process of Adapting a Universal Dating Abuse Prevention Program to Adolescents Exposed to Domestic Violence
Foshee, V. A., Dixon, K. S., Ennett, S. T., Moracco, K. E., Bowling, J. M., Chang, L., & Moss, J. L. (2014) · Journal of Interpersonal Violence · doi:10.1177/0886260514552278 · Cited by 27 · see citing works
Understanding the Challenges Encountered and Adaptations Made by Community Organizations in Translation of Evidence-Based Behavior Change Physical Activity Interventions: A Qualitative Study
Lattimore, D., Griffin, S. F., Wilcox, S., Rheaume, C. E., Dowdy, D. M., Leviton, L. C., & Ory, M. G. (2010) · American Journal of Health Promotion · doi:10.4278/ajhp.081024-qual-252 · Cited by 24 · see citing works
Self-citation
Three Cs of Translating Evidence-Based Programs for Youth and Families to Practice Settings
Freire, K. E., Perkinson, L., Morrel‐Samuels, S., & Zimmerman, M. A. (2015) · New Directions for Child and Adolescent Development · doi:10.1002/cad.20111 · Cited by 17 · see citing works
Assessing the Evidence of Effectiveness of Home Visiting Program Models Implemented in Tribal CommunitiesPreprint
Grosso, P. D., Kleinman, R., Esposito, A. M., Martin, E., & Paulsell, D. (2014) · RePEc: Research Papers in Economics · Cited by 16 · see citing works
Dawning Answers: How the HIV/AIDS Epidemic Has Helped to Strengthen Public Health.
Katz, M. H. (2003) · JAIDS Journal of Acquired Immune Deficiency Syndromes · doi:10.1097/00126334-200306010-00025 · Cited by 8 · see citing works
Evaluation Practice and Theory
Leviton, L. C. (2015) · American Journal of Evaluation · doi:10.1177/1098214015573070 · Cited by 8 · see citing works
Self-citation
Conduct and behavior problems: Intervention and resources for school aged youth
Adelman, H. S., & Taylor, L. K. (2008) · PsycEXTRA Dataset · doi:10.1037/e576072011-001 · Cited by 3 · see citing works

Flagged first, then by citations

1 of the 84 references couldn’t be matched to a known paper.

Sorted by citations

Beyond “implementation strategies”: classifying the full range of strategies used in implementation science and practice
Leeman, J., Birken, S. A., Powell, B. J., Rohweder, C., & Shea, C. M. (2017) · Implementation Science · doi:10.1186/s13012-017-0657-x · Cited by 544 · see citing works
Evidence suggests a need to rethink social capital and social capital interventions
Shiell, A., Hawe, P., & Kavanagh, S. (2018) · Social Science & Medicine · doi:10.1016/j.socscimed.2018.09.006 · Cited by 145 · see citing works
Adaptations of the evidence-based Transitional Care Model in the U.S.
Naylor, M. D., Hirschman, K. B., Toles, M., Jarrin, O. F., Shaid, E., & Pauly, M. V. (2018) · Social Science & Medicine · doi:10.1016/j.socscimed.2018.07.023 · Cited by 108 · see citing works
External Validity and Policy Adaptation: From Impact Evaluation to Policy Design
Williams, M. J. (2018) · The World Bank Research Observer · doi:10.1093/wbro/lky010 · Cited by 66 · see citing works
Generalizing about Public Health Interventions: A Mixed-Methods Approach to External Validity
Leviton, L. C. (2017) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-031816-044509 · Cited by 59 · see citing works
Self-citation
Evidence-Based Interventions Are Necessary but Not Sufficient for Achieving Outcomes in Each Setting in a Complex World
Wandersman, A., Alia, K. A., Cook, B. S., Hsu, L. L.-Y., & Ramaswamy, R. (2016) · American Journal of Evaluation · doi:10.1177/1098214016660613 · Cited by 46 · see citing works
Diffusion of Innovations
Dearing, J. W. (2021) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780198845973.013.23 · Cited by 40 · see citing works
Quality improvement studies in nursing homes: a scoping review
Toles, M., Colon-Emeric, C. S., Moreton, E., Frey, L., & Leeman, J. (2021) · BMC Health Services Research · doi:10.1186/s12913-021-06803-8 · Cited by 36 · see citing works
Qualitative Comparative Analysis: A Mixed-Method Tool for Complex Implementation Questions
Hill, L. G., Cooper, B. R., & Parker, L. A. (2019) · The Journal of Primary Prevention · doi:10.1007/s10935-019-00536-5 · Cited by 33 · see citing works
Delivering evidence based services for all vulnerable families
Acquah, D. K., & Thévenon, O. (2020) · OECD social employment and migration working papers · doi:10.1787/1bb808f2-en · Cited by 20 · see citing works
“Putting meat on the bones”: Understanding the implementation of a community-based early intervention and prevention programme—contextual, person, and programme influences
Hickey, G., McGilloway, S., Leckey, Y., Stokes, A., Bywater, T., & Donnelly, M. (2021) · Prevention Science · doi:10.1007/s11121-020-01170-y · Cited by 15 · see citing works
Implementation strategies to promote community-engaged efforts to counter tobacco marketing at the point of sale
Leeman, J., Myers, A., Grant, J. C., Wangen, M., & Queen, T. L. (2017) · Translational Behavioral Medicine · doi:10.1007/s13142-017-0489-x · Cited by 14 · see citing works
A typology of useful evidence: approaches to increase the practical value of intervention research
Hasson, H., Leviton, L. C., & Schwarz, U. V. T. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-00992-2 · Cited by 10 · see citing works
Self-citation
Reciprocal coproduction as a basis for the diffusion of global health innovations
Dearing, J. W., Masquillier, C., van Olmen, J., Zieff, S. G., Liu, A., & Rollins, A. L. (2023) · BMJ Global Health · doi:10.1136/bmjgh-2023-013134 · Cited by 10 · see citing works
A Critical Reflection of Generalization in Mixed Methods Research
Hong, Q. N., & Fàbregues, S. (2025) · Evaluation Review · doi:10.1177/0193841x251331723 · Cited by 8 · see citing works
Visualizing Expertise: Collaborative Pathway Modeling as a Methodology for Conveying Community-driven Strategies for Change
Hargraves, M., & Denning, C. (2018) · Journal of Agriculture Food Systems and Community Development · doi:10.5304/jafscd.2018.08a.005 · Cited by 7 · see citing works
Using Heatmap Visualization to assess the performance of the DJ30 and NASDAQ100 Indices under diverse VMA trading rules
Chen, Y., Huang, P., Day, M.-Y., Ni, Y., & Liang, M.-C. (2023) · PLoS ONE · doi:10.1371/journal.pone.0284918 · Cited by 7 · see citing works
Understanding the theoretical underpinnings of preventative early parenting interventions: A theory-based evaluation
Hickey, G., McGilloway, S., Leckey, Y., Bywater, T. J., & Donnelly, M. (2023) · Evaluation · doi:10.1177/13563890231165814 · Cited by 4 · see citing works
Bayesian adaptive trials for social policy
Cripps, S., Lopatnikova, A., Afshar, H. M., Gales, B., Marchant, R., Francis, G., Moreira, C., & Fischer, A. (2025) · Data & Policy · doi:10.1017/dap.2024.97 · Cited by 4 · see citing works
Fidelity and Adaptation of Programs: Does Adaptation Thwart Effectiveness?
Nolt, K. L., & Leviton, L. C. (2022) · American Journal of Evaluation · doi:10.1177/10982140221138604 · Cited by 3 · see citing works
Self-citation
Enhancing Students’ Skills in Applying Theory Through Collaborative and Active Learning: A Pre-Post Evaluation of an Innovative Theory Course for Graduate Public Health Students
McGee, R. E., Walker, E. R., McBride, C. M., Nehl, E. J., Rice, W. S., Wise, N. E., Jones-Harrell, C., & Lang, D. L. (2023) · Pedagogy in Health Promotion · doi:10.1177/23733799231186637 · Cited by 3 · see citing works
Editor in Chief’s Comment: External Validity in Systematic Reviews
Klerman, J. A. (2017) · Evaluation Review · doi:10.1177/0193841x17746740 · Cited by 3 · see citing works
Celebrating the Most Valuable Publications from 2022: The 6th Annual Psyche Awards for Excellence in Scholarship
Overholser, J. (2023) · Journal of Contemporary Psychotherapy · doi:10.1007/s10879-023-09581-y · Cited by 3 · see citing works
Learning about improvement to address global health and healthcare challenges—lessons and the future
Øvretveit, J. (2018) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzy015 · Cited by 2 · see citing works
Framework to Support the Transfer of Innovative Interventions in the Disability Field: Lessons from the Transferability of Complex Interventions in Public Health: A Review
Ségard, E., Chervin, P., & Cambon, L. (2024) · Disabilities · doi:10.3390/disabilities4030044 · Cited by 1 · see citing works
The power of the group – Group-based parenting programmes for disadvantaged parents and their infants: a realist review
Verweij, R., Howe, J., Mathijssen, J., & van Bakel, H. J. A. (2026) · International Journal of Nursing Studies Advances · doi:10.1016/j.ijnsa.2026.100591 · Cited by 1 · see citing works
Students’ Difficulties in Learning about AI from Teachers' Perspectives: Insights from an Action Research Approach in Compulsory CS Education
Jetzinger, F., & Michaeli, T. (2026) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3765964.3811641 · Cited by 1 · see citing works
The effect and implementation of the COVID Box, a remote patient monitoring system for patients with a COVID-19 infection in primary care: a matched cohort study
van Hattem, N. E., Mijnsbergen, N. J., van Os, H. J. A., Mertens, B., Eekhof, J. A., Chavannes, N. H., Atsma, D. E., & Bonten, T. N. (2024) · Family Practice · doi:10.1093/fampra/cmae045 · Cited by 1 · see citing works
Codifying Practice Into Theory
Leviton, L. C. (2016) · American Journal of Evaluation · doi:10.1177/1098214016661413 · Cited by 1 · see citing works
Self-citation
Introduction to Special Issue: External Validity and Policy
Westbrook, T. R. (2017) · Evaluation Review · doi:10.1177/0193841x17707908 · Cited by 1 · see citing works
Radicalisation and Counter-Terrorism—Reviewing the Problems in Strategic Communications Role in Counter-Terrorism
Zamir, M., & Kaunert, C. (2025) · Advanced sciences and technologies for security applications · doi:10.1007/978-3-031-89110-6_2 · Cited by 1 · see citing works
External Validity for Complex Interventions: Towards Pragmatic Strategies for Evaluators
Delahais, T. (2025) · Evaluation Review · doi:10.1177/0193841x251343130 · Cited by 1 · see citing works
Additional file 1 of Quality improvement studies in nursing homes: a scoping review
Toles, M., Colon-Emeric, C. S., Moreton, E., Frey, L., & Leeman, J. (2021) · Figshare · doi:10.6084/m9.figshare.15161894
Additional file 2 of Quality improvement studies in nursing homes: a scoping review
Toles, M., Colon-Emeric, C. S., Moreton, E., Frey, L., & Leeman, J. (2021) · Figshare · doi:10.6084/m9.figshare.15161897
Charge de travail et blessure dans le football de haut niveau
Delecroix, B. (2019) · doi:10.70675/f7b5af5dz55a0z46ffz98b8z0964d718e897
Conducting Evaluations in the Context of Tertiary Prevention of Youth Crime: Reflections from the Youth Endowment Fund
Acquah, D. K., Kung, C. S. J., & Sherlock, R. M. (2026) · Behavioral Sciences · doi:10.3390/bs16050626

Authors: Leviton, L. C., & Trujillo, M. D.
Affiliations: Robert Wood Johnson Foundation (US)
Corresponding author: Leviton, Laura C.
ORCID: Leviton, Laura C.
Venue: Evaluation Review
Publisher: SAGE Publications
Published: 2016-01-18
Volume: 41
Issue: 5
Pages: 436-471
ISSN: 0145-4692 (ISSN-L)
0193-841X
1552-3926
DOI: 10.1177/0193841x15625289
OpenAlex: W2343651394
Cited by: 35
References: 84
Citations by year: 2026: 2 · 2025: 4 · 2024: 2 · 2023: 5 · 2022: 1 · 2021: 4 · 2020: 3 · 2019: 2 · 2018: 5 · 2017: 5 · 2016: 2
FWCI: 8.68
Type: journal article
Language: en
Open access: closed
Licence: journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations: Evaluation Review (publishedVersion)
PubMed (publishedVersion)
RePEc: Research Papers in Economics (submittedVersion)
Topics: Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
Social Sciences › Decision Sciences › Management Science and Operations Research › Evaluation and Performance Assessment
Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
Keywords: generalizability, program theory, practitioner expertise, local adaptation, context effects, treatment variation, evidence-based interventions, theory-practice interaction, intervention development, systematic inquiry
MeSH terms: Decision Making, Evaluation Studies as Topic, Models, Theoretical, Policy Making, Reproducibility of Results, Evidence-Based Practice

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Interaction of Theory and Practice to Assess External Validity

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Related

Record

About these checks