Critical Measurement Issues in Translational Research
Abstract
This article summarizes critical evaluation needs, challenges, and lessons learned in translational research. Evaluation can play a key role in enhancing successful application of research-based programs and tools as well as informing program refinement and future research. Discussion centers on what is unique about evaluating programs and policies for implementation impact (or potential for dissemination). Central issues reviewed include the importance of context and local issues, robustness and external validity issues, multiple levels of evaluation, implementation fidelity versus customization, choosing evaluation designs to fit questions, and who participates and characteristics of success at each stage of program recruitment, delivery, and outcome. The use of mixed quantitative and qualitative methods is especially important, and the primary redirection that is needed is to focus on questions of decision makers and potential adoptees rather than the research colleagues.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Implementation science to date suffers a dearth of economic evaluations from which to draw.
“Although budget impact is a key concern to administrators and some funding agencies require budget impact analysis, implementation science to date suffers a dearth of economic evaluations from which to draw [96,97].”
From Writing implementation research grant proposals: ten key ingredients (2012) · cites this paper as [97]
Implementation research often involves some customization of an intervention to fit local context.
“Unlike efficacy and effectiveness studies, implementation research often involves some customization of an intervention to fit local context; accordingly, measurement plans need to address the intervention’s degree of customization versus fidelity [97].”
From Writing implementation research grant proposals: ten key ingredients (2012) · cites this paper as [97]
The field of implementation science has been deemed a Tower of Babel given the lack of conceptual clarity and consistency in its terminology, and concerns about the state of measurement in the field have been well documented.
“The field of implementation science has been deemed a Tower of Babel given the lack of conceptual clarity and consistency in its terminology [26], and concerns about the state of measurement in the field have been well documented [3, 27, 28].”
From Psychometric assessment of three newly developed implementation outcome measures (2017) · cites this paper as [27]
Cites
Crossing the Quality Chasm: A New Health System for the 21st Century
Committee on Quality of Health Care in America (2002) · Journal for Healthcare Quality · doi:10.1111/j.1945-1474.2002.tb00463.x · Cited by 8,508 · see citing works
The quality of health care delivered to adults in the United States
McGlynn, E. A., Asch, S. M., Adams, J., Keesey, J., Hicks, J., DeCristofaro, A., & Kerr, E. A. (2003) · New England Journal of Medicine · doi:10.1056/nejmsa022615 · Cited by 5,079 · see citing works
The American Journal of Managed Care
(2020) · The American Journal of Managed Care · doi:10.37765/ajmc · Cited by 3,238 · see citing works
Realist review - a new method of systematic review designed for complex policy interventions
Pawson, R., Greenhalgh, T., Harvey, G., & Walshe, K. (2005) · Journal of Health Services Research & Policy · doi:10.1258/1355819054308530 · Cited by 3,190 · see citing works
Bellg, A. J., Treatment Fidelity Workgroup of the NIH Behavior Change Consortium, Borrelli, B., Resnick, B., Hecht, J., Minicucci, D. S., Ory, M., Ogedegbe, G., Orwig, D., Ernst, D., et al. (2004) · Health Psychology · doi:10.1037/0278-6133.23.5.443 · Cited by 2,620 · see citing works
(2005) · JAMA · doi:10.1001/jama.294.19.2423-c · Cited by 2,325 · see citing works
Glasgow, R. E., Lichtenstein, E., & Marcus, A. C. (2003) · American Journal of Public Health · doi:10.2105/ajph.93.8.1261 · Cited by 1,836 · see citing works
Self-citationTunis (2003) · doi:10.1001/jama.290.12.1624 · Cited by 1,724 · see citing works
(1999) · Cited by 1,648 · see citing works
The Cultural Adaptation of Prevention Interventions: Resolving Tensions Between Fidelity and Fit
Castro, F. G., Barrera, M., & Martinez, C. R. (2004) · Prevention Science · doi:10.1023/b:prev.0000013980.12412.cd · Cited by 1,228 · see citing works
How Can We Increase Translation of Research into Practice? Types of Evidence Needed
Glasgow, R. E., & Emmons, K. M. (2007) · Annual Review of Public Health · doi:10.1146/annurev.publhealth.28.021406.144145 · Cited by 1,050 · see citing works
Self-citationA primer on leading the improvement of systems: Fig 1
Berwick, D. M. (1996) · BMJ · doi:10.1136/bmj.312.7031.619 · Cited by 793 · see citing works
Glasgow, R., Mckay, H., Piette, J. D., & Reynolds, K. (2001) · Patient Education and Counseling · doi:10.1016/s0738-3991(00)00186-5 · Cited by 643 · see citing works
Self-citationGlasgow, R. E., Klesges, L. M., Dzewaltowski, D. A., Bull, S. S., & Estabrooks, P. (2004) · Annals of Behavioral Medicine · doi:10.1207/s15324796abm2701_2 · Cited by 635 · see citing works
Self-citationKlesges, L. M., Estabrooks, P. A., Dzewaltowski, D. A., Bull, S. S., & Glasgow, R. E. (2005) · Annals of Behavioral Medicine · doi:10.1207/s15324796abm2902s_10 · Cited by 325 · see citing works
Self-citationGlasgow, R. E., Magid, D. J., Beck, A., Ritzwoller, D., & Estabrooks, P. A. (2005) · Medical Care · doi:10.1097/01.mlr.0000163645.41407.09 · Cited by 308 · see citing works
Self-citationTheory as mediating variables: Why aren't community interventions working as desired?
Baranowski, T., Lin, L. S., Wetter, D. W., Resnicow, K., & Hearn, M. D. (1997) · Annals of Epidemiology · doi:10.1016/s1047-2797(97)80011-7 · Cited by 283 · see citing works
Avoiding Type III Errors in Health Education Program Evaluations: A Case Study
Basch, C. E., Sliepcevich, E. M., Gold, R. S., Duncan, D. F., & Kolbe, L. J. (1985) · Health Education Quarterly · doi:10.1177/109019818501200311 · Cited by 244 · see citing works
(2006) · Europe PMC (PubMed Central) · Cited by 216 · see citing works
Development of Level of Institutionalization Scales for Health Promotion Programs
Goodman, R. M., McLeroy, K. R., Steckler, A., & Hoyle, R. H. (1993) · Health Education Quarterly · doi:10.1177/109019819302000208 · Cited by 181 · see citing works
Orleans, C. T. (2000) · Health Psychology · doi:10.1037/0278-6133.19.suppl1.76 · Cited by 174 · see citing works
What Types of Evidence are Most Needed to Advance Behavioral Medicine?
Glasgow, R. E. (2008) · Annals of Behavioral Medicine · doi:10.1007/s12160-007-9008-5 · Cited by 146 · see citing works
Self-citationThe significance of quality of life in health care
Kaplan, R. M. (2003) · Quality of Life Research · doi:10.1023/a:1023547632545 · Cited by 140 · see citing works
Enhancing Theoretical Fidelity: An E-mail—Based Walking Program Demonstration
Rovniak, L. S., Hovell, M. F., Wojcik, J. R., Winett, R. A., & Martinez-Donate, A. P. (2005) · American Journal of Health Promotion · doi:10.4278/0890-1171-20.2.85 · Cited by 90 · see citing works
Practical behavioral trials to advance evidence-based behavioral medicine
Glasgow, R. E., Davidson, K. W., Dobkin, P. L., Ockene, J. K., & Spring, B. J. (2006) · Annals of Behavioral Medicine · doi:10.1207/s15324796abm3101_3 · Cited by 76 · see citing works
Self-citationTHE JOURNAL OF THE AMERICAN MEDICAL ASSOCIATION
(1912) · JAMA · doi:10.1001/jama.1912.04260060107014 · Cited by 68 · see citing works
Tasseff, T. L., & Nies, M. A. (2017) · AJN American Journal of Nursing · doi:10.1097/01.naj.0000526728.75657.9e · Cited by 50 · see citing works
Glasgow, R. E., Strycker, L. A., King, D. K., Toobert, D. J., Rahm, A. K., Jex, M., & Nutting, P. A. (2006) · PubMed · Cited by 35 · see citing works
Self-citationMediation of a middle school skin cancer prevention program.
Reynolds, K. D., Buller, D. B., Yaroch, A. L., Maloy, J. A., & Cutter, G. (2006) · Health Psychology · doi:10.1037/0278-6133.25.5.616 · Cited by 32 · see citing works
Interventions That are CURRES: Cost-Effective, Useful, Realistic, Robust, Evolving, and Sustainable
Rotheram‐Borus, M. J., Flannery, D., & Duan, N. (2004) · Springer eBooks · doi:10.1007/978-3-642-18611-0_17 · Cited by 24 · see citing works
Modest Implementation Efforts, Modest Fidelity, and Modest Outcomes
Bond, G. R. (2007) · Psychiatric Services · doi:10.1176/ps.2007.58.3.334 · Cited by 24 · see citing works
Economic Analysis of the Mediterranean Lifestyle Program for Postmenopausal Women With Diabetes
Ritzwoller, D. P., Toobert, D. J., Sukhanova, A., & Glasgow, R. E. (2006) · The Science of Diabetes Self-Management and Care · doi:10.1177/0145721706291757 · Cited by 24 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Psychometric assessment of three newly developed implementation outcome measures
Weiner, B. J., Lewis, C. C., Stanick, C., Powell, B. J., Dorsey, C. N., Clary, A. S., Boynton, M. H., & Halko, H. (2017) · Implementation Science · doi:10.1186/s13012-017-0635-3 · Cited by 2,819 · see citing works
An Agenda for Research on the Sustainability of Public Health Programs
Scheirer, M. A., & Dearing, J. W. (2011) · American Journal of Public Health · doi:10.2105/ajph.2011.300193 · Cited by 775 · see citing works
DRAHOTA, A., MEZA, R. D., BRIKHO, B., NAAF, M., ESTABILLO, J. A., GOMEZ, E. D., VEJNOSKA, S. F., DUFEK, S., STAHMER, A. C., & AARONS, G. A. (2016) · The Milbank Quarterly · doi:10.1111/1468-0009.12184 · Cited by 439 · see citing works
Preliminary Findings of a Telehealth Approach to Parent Training in Autism
Vismara, L. A., McCormick, C. E. B., Young, G. S., Nadhan, A., & Monlux, K. D. (2013) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-013-1841-8 · Cited by 407 · see citing works
Cabassa, L. J., & Baumann, A. A. (2013) · Implementation Science · doi:10.1186/1748-5908-8-90 · Cited by 290 · see citing works
Writing implementation research grant proposals: ten key ingredients
Proctor, E. K., Powell, B. J., Baumann, A. A., Hamilton, A. M., & Santens, R. L. (2012) · Implementation Science · doi:10.1186/1748-5908-7-96 · Cited by 169 · see citing works
The CIPHER team, Haynes, A. S., Brennan, S., Redman, S., Williamson, A. B., Gallego, G., & Butow, P. N. (2015) · Implementation Science · doi:10.1186/s13012-016-0378-6 · Cited by 68 · see citing works
New Horizons of Translational Research and Research Translation in Social Work
Palinkas, L. A., & Soydan, H. (2011) · Research on Social Work Practice · doi:10.1177/1049731511408738 · Cited by 66 · see citing works
Painter, P., Clark, L., & Olausson, J. M. (2014) · American Journal of Kidney Diseases · doi:10.1053/j.ajkd.2014.01.433 · Cited by 50 · see citing works
The Translational Research Impact Scale
Dembe, A. E., Lynch, M. S., Gugiu, P. C., & Jackson, R. D. (2013) · Evaluation & the Health Professions · doi:10.1177/0163278713506112 · Cited by 49 · see citing works
Operationalizing Social Work Science Through Research–Practice Partnerships
Palinkas, L. A., He, A. S., Choy-Brown, M., & Hertel, A. L. (2016) · Research on Social Work Practice · doi:10.1177/1049731516666329 · Cited by 49 · see citing works
Robinson, C., & Damschroder, L. J. (2023) · Implementation Science Communications · doi:10.1186/s43058-022-00380-5 · Cited by 47 · see citing works
Social Science Implications for Control of Helminth Infections in Southeast Asia
Vandemark, L. M., Jia, T.-W., & Zhou, X. (2010) · Advances in Parasitology/Advances in parasitology · doi:10.1016/s0065-308x(10)73006-2 · Cited by 46 · see citing works
Nathan, S., Bunde‐Birouste, A., Evers, C., Kemp, L., MacKenzie, J., & Henley, R. R. (2010) · BMC Public Health · doi:10.1186/1471-2458-10-587 · Cited by 43 · see citing works
Implementation and effectiveness of the Indian Child Welfare Act: A systematic review
Francis, A. M., Hall, W. J., Ansong, D., Lanier, P., Albritton, T., & McMillan, A. (2023) · Children and Youth Services Review · doi:10.1016/j.childyouth.2022.106799 · Cited by 34 · see citing works
The Assimilation of Evidence-Based Healthcare Innovations: A Management-Based Perspective
Panzano, P. C., Sweeney, H. A., Seffrin, B., Massatti, R. R., & Knudsen, K. J. (2012) · The Journal of Behavioral Health Services & Research · doi:10.1007/s11414-012-9294-y · Cited by 20 · see citing works
Bliss, S., & McCardle, M. (2013) · Clinical social work journal · doi:10.1007/s10615-013-0456-z
Intervention Research in Social Work
Soydan, H. (2015) · Australian Social Work · doi:10.1080/0312407x.2014.993670 · Cited by 19 · see citing works
Manfredi, C., Cho, Y. I., Warnecke, R. B., Saunders, S., & Sullivan, M. (2011) · Health Education Research · doi:10.1093/her/cyr010 · Cited by 16 · see citing works
Social Work’s Role in Collaborative Community–Academic Partnerships: How Our Past Informs Our Future
Adams, D. R. (2018) · Social Work · doi:10.1093/sw/swy046 · Cited by 16 · see citing works
Dissemination and Implementation Research
Proctor, E. K. (2014) · Oxford University Press eBooks · doi:10.1093/acrefore/9780199975839.013.900 · Cited by 16 · see citing works
Pearson, C. R., Smartlowit-Briggs, L., Belcourt, A., Bedard‐Gilligan, M. A., & Kaysen, D. L. (2018) · Health Promotion Practice · doi:10.1177/1524839918762122 · Cited by 15 · see citing works
Preventing Problem Behavior in School through School-Wide Staff Empowerment: Intervention Outcomes
Sørlie, M., Ogden, T., & Olseth, A. R. (2015) · World Journal of Educational Research · doi:10.22158/wjer.v2n2p117 · Cited by 15 · see citing works
Patel, Z. S., Becker, S. J., Olson, M. B., Rinella, H., & Scott, K. (2024) · Implementation Science Communications · doi:10.1186/s43058-024-00549-0 · Cited by 14 · see citing works
George, S. M. S., Huang, S., Vidot, D. C., Smith, J. D., Brown, C. H., & Prado, G. J. (2015) · Translational Behavioral Medicine · doi:10.1007/s13142-015-0344-x · Cited by 12 · see citing works
McMillin, S. E. (2014) · Journal of Evidence-Based Social Work · doi:10.1080/15433714.2013.845029 · Cited by 7 · see citing works
Adherence to three different strategies to prevent early onset GBS infection in newborns
Kolkman, D. G. E., Rijnders, M. E. B., Wouters, M. G. A. J., van Dommelen, P., de Groot, C. J. M., & Fleuren, M. A. H. (2019) · Women and Birth · doi:10.1016/j.wombi.2019.12.004 · Cited by 7 · see citing works
Intellectual Property and Innovation in Translational Medicine
Moreno, P. G., & Joly, Y. (2015) · Elsevier eBooks · doi:10.1016/b978-0-12-800687-0.00030-x · Cited by 5 · see citing works
Exploring the Evidence-Based Practice Process with Social Work Practitioners
Scoresby, K. J., Huslage, M., & Wallis, D. (2024) · Journal of Evidence-Based Social Work · doi:10.1080/26408066.2024.2316038 · Cited by 4 · see citing works
Translating Evidence-Based Interventions Into Practice
Malveaux, F. J., & Butterfoss, F. D. (2011) · Health Promotion Practice · doi:10.1177/1524839911414414 · Cited by 4 · see citing works
Implementation Science and Practice
Proctor, E. K., Powell, B. J., & McGinnis, H. (2011) · Oxford University Press eBooks · doi:10.1093/obo/9780195389678-0012 · Cited by 1 · see citing works
Translational Science and Social Work
Soydan, H., & Palinkas, L. A. (2013) · doi:10.1093/obo/9780195389678-0070 · Cited by 1 · see citing works
Douglas, N. (2013) · Digital Commons - University of South Florida (University of South Florida) · Cited by 1 · see citing works
Smoking cessation support for COPD patients in general practice
Hilberink, S. R. (2013) · Radboud Repository (Radboud University)
Intellectual property and innovation in translational medicine
Moreno, P. G., & Joly, Y. (2021) · Elsevier eBooks · doi:10.1016/b978-0-12-820493-1.00005-2
Using the Practice Skills Inventory in Real Time
O’Hare, T., & Geertsma, J. H. (2013) · Best practices in mental health · doi:10.70256/603143byvhum
「エビデンス」の利用に関する検討:技術移転と追試過程を中心に
津富, 宏. (2010) · 日本評価研究 · doi:10.11278/jjoes.10.1_43
Patel-Syed, Z., Becker, S., Olson, M. B., Rinella, H., & Scott, K. (2024) · Figshare · doi:10.6084/m9.figshare.26680738
Robinson, C., & Damschroder, L. J. (2023) · Figshare · doi:10.6084/m9.figshare.22613759
Robinson, C., & Damschroder, L. J. (2023) · Figshare · doi:10.6084/m9.figshare.22613762
Investigating fidelity of health behaviour change interventions in general practice
Taylor, C. (2012) · The Open Collections (Coventry University)
- Authors
- Glasgow, R. E.
- Affiliations
- Kaiser Permanente (US)
- Corresponding author
- Glasgow, Russell E.
- ORCID
- Glasgow, Russell E.
- Venue
- Research on Social Work Practice
- Publisher
- SAGE Publications
- Published
- 2009-05-27
- Volume
- 19
- Issue
- 5
- Pages
- 560-568
- ISSN
- 1049-7315 (ISSN-L)
- 1552-7581
- OpenAlex
- W2008953761
- Cited by
- 42
- References
- 32
- Citations by year
- 2024: 3 · 2023: 4 · 2021: 1 · 2019: 1 · 2018: 2 · 2017: 1 · 2016: 2 · 2015: 5 · 2014: 4 · 2013: 8 · 2012: 3
- FWCI
- 6.68
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Research on Social Work Practice (publishedVersion)
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Social Sciences › Decision Sciences › Management Science and Operations Research › Evaluation and Performance Assessment
- Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
- Keywords
- translational research, program evaluation, implementation effects, dissemination, generalizability, implementation fidelity, customization, evaluation design, mixed methods, context, decision-makers, program recruitment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review