Pragmatic measures for implementation research: development of the Psychometric and Pragmatic Evidence Rating Scale (PAPERS)
Open accessAbstract
The use of reliable, valid measures in implementation practice will remain limited without pragmatic measures. Previous research identified the need for pragmatic measures, though the characteristic identification used only expert opinion and literature review. Our team completed four studies to develop a stakeholder-driven pragmatic rating criteria for implementation measures. We published Studies 1 (identifying dimensions of the pragmatic construct) and 2 (clarifying the internal structure) that engaged stakeholders—participants in mental health provider and implementation settings—to identify 17 terms/phrases across four categories: Useful, Compatible, Acceptable, and Easy. This paper presents Studies 3 and 4: a Delphi to ascertain stakeholder-prioritized dimensions within a mental health context, and a pilot study applying the rating criteria. Stakeholders (N = 26) participated in a Delphi and rated the relevance of 17 terms/phrases to the pragmatic construct. The investigator team further defined and shortened the list, which were piloted with 60 implementation measures. The Delphi confirmed the importance of all pragmatic criteria, but provided little guidance on relative importance. The investigators removed or combined terms/phrases to obtain 11 criteria. The 6-point rating system assigned to each criterion demonstrated sufficient variability across items. The grey literature did not add critical information. This work produced the first stakeholder-driven rating criteria to assess whether measures are pragmatic. The Psychometric and Pragmatic Evidence Rating Scale (PAPERS) combines the pragmatic criteria with psychometric rating criteria, from previous work. Use of PAPERS can inform development of implementation measures and to assess the quality of existing measures.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The PHQ-9: Validity of a brief depression severity measure
Kroenke, K., Spitzer, R. L., & Williams, J. B. W. (2001) · Journal of General Internal Medicine · doi:10.1046/j.1525-1497.2001.016009606.x · Cited by 44,910 · see citing works
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009) · Implementation Science · doi:10.1186/1748-5908-4-50 · Cited by 14,496 · see citing works
Self-citationProctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0319-7 · Cited by 8,721 · see citing works
Research guidelines for the Delphi survey technique
Hasson, F. (2000) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.2000.01567.x · Cited by 5,951 · see citing works
The Delphi technique: myths and realities
Powell, C. (2003) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.2003.02537.x · Cited by 2,611 · see citing works
Consensus development methods, and their use in clinical guideline development.
Murphy, M., Black, Lamping, McKee, M., Sanderson, Askham, & Marteau (1998) · Health Technology Assessment · doi:10.3310/hta2030 · Cited by 2,042 · see citing works
Consensus development methods, and their use in clinical guideline development.
Murphy, M. P., Black, N. A., Lamping, D. L., et al. (1998) · PubMed · Cited by 965 · see citing works
Lewis, C. C., Fischer, S., Weiner, B. J., Stanick, C., Kim, M., & Martinez, R. G. (2015) · Implementation Science · doi:10.1186/s13012-015-0342-x · Cited by 430 · see citing works
Self-citationGlasgow, R. E., & Riley, W. T. (2013) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2013.03.010 · Cited by 318 · see citing works
Michie, S., Abraham, C., Eccles, M. P., Francis, J. J., Hardeman, W., & Johnston, M. (2011) · Implementation Science · doi:10.1186/1748-5908-6-10 · Cited by 301 · see citing works
Lewis, C. C., Stanick, C. F., Martinez, R. G., Weiner, B. J., Kim, M., Barwick, M., & Comtois, K. A. (2016) · Implementation Science · doi:10.1186/s13012-014-0193-x · Cited by 155 · see citing works
Self-citationPowell, B. J., Stanick, C. F., Halko, H. M., Dorsey, C. N., Weiner, B. J., Barwick, M. A., Damschroder, L. J., Wensing, M., Wolfenden, L., & Lewis, C. C. (2017) · Implementation Science · doi:10.1186/s13012-017-0649-x · Cited by 144 · see citing works
Self-citationAn updated protocol for a systematic review of implementation-related measures
Lewis, C. C., Mettert, K. D., Dorsey, C. N., Martinez, R. G., Weiner, B. J., Nolen, E., Stanick, C., Halko, H., & Powell, B. J. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0728-3 · Cited by 118 · see citing works
Self-citationStanick, C. F., Halko, H. M., Dorsey, C. N., Weiner, B. J., Powell, B. J., Palinkas, L. A., & Lewis, C. C. (2018) · BMC Health Services Research · doi:10.1186/s12913-018-3709-2 · Cited by 54 · see citing works
Self-citationOrdinary search engine users carrying out complex search tasks
Singer, G., Norbisrath, U., & Lewandowski, D. (2012) · Journal of Information Science · doi:10.1177/0165551512466974 · Cited by 45 · see citing works
Web search engines and complex information needs
Singer, G. (2012) · DSpace repository (University of Tartu) · Cited by 5 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [17]17. Maynes R, Everdell I. The Evolution of Google Search Results Pages and their Effects on User Behaviour. USA: Mediative; 2014. [Google Scholar] Articles from Translational Behavioral Medicine are provided here courtesy of Oxford University Press
Cited by
Sorted by citations
Quantitative measures of health policy implementation determinants and outcomes: a systematic review
Allen, P., Pilar, M., Walsh-Bailey, C., Hooley, C., Mazzucca, S., Lewis, C. C., Mettert, K. D., Dorsey, C. N., Purtle, J., Kepper, M. M., et al. (2020) · Implementation Science · doi:10.1186/s13012-020-01007-w · Cited by 164 · see citing works
Self-citationMeasuring implementation outcomes: An updated systematic review of measures’ psychometric properties
Mettert, K., Lewis, C., Dorsey, C., Halko, H., & Weiner, B. (2020) · Implementation Research and Practice · doi:10.1177/2633489520936644 · Cited by 162 · see citing works
Self-citationMalone, S., Prewitt, K., Hackett, R., Lin, J. C., McKay, V., Walsh-Bailey, C., & Luke, D. A. (2021) · Implementation Science Communications · doi:10.1186/s43058-021-00181-2 · Cited by 148 · see citing works
Hall, A., Shoesmith, A., Doherty, E., McEvoy, B., Mettert, K., Lewis, C. C., Wolfenden, L., Yoong, S., Kingsland, M., Shelton, R. C., et al. (2022) · Implementation Science · doi:10.1186/s13012-022-01252-1 · Cited by 81 · see citing works
Self-citationPowell, B. J., Mettert, K. D., Dorsey, C. N., Weiner, B. J., Stanick, C. F., Lengnick-Hall, R., Ehrhart, M. G., Aarons, G. A., Barwick, M. A., Damschroder, L. J., et al. (2021) · Implementation Research and Practice · doi:10.1177/26334895211018862 · Cited by 79 · see citing works
Self-citationLewis, C. C., Mettert, K., & Lyon, A. R. (2021) · Implementation Research and Practice · doi:10.1177/2633489521994197 · Cited by 38 · see citing works
Self-citationDorsey, C. N., Mettert, K. D., Puspitasari, A. J., Damschroder, L. J., & Lewis, C. C. (2021) · Implementation Research and Practice · doi:10.1177/26334895211002474 · Cited by 15 · see citing works
Self-citationAdvancing the science of integrity measurement in school mental health research
Sutherland, K. S., & McLeod, B. D. (2022) · School Mental Health · doi:10.1007/s12310-021-09468-0 · Cited by 14 · see citing works
Malone, S., Prewitt, K., McKay, V., Zabotka, L., Bacon, C., & Luke, D. A. (2024) · Implementation Science Communications · doi:10.1186/s43058-024-00656-y · Cited by 12 · see citing works
“While the PAPERS scale has a cutoff of 10 items for a short tool [22], reducing the PSAT and CSAT still reduces the burden of assessment.”
BackgroundMethodStanick, C., Halko, H., Mettert, K., Dorsey, C., Moullin, J., Weiner, B., Powell, B., & Lewis, C. C. (2021) · Implementation Research and Practice · doi:10.1177/26334895211000458 · Cited by 11 · see citing works
Self-citation
- Authors
- Stanick, C. F., Halko, H. M., Nolen, E. A., Powell, B. J., Dorsey, C. N., Mettert, K. D., Weiner, B. J., Barwick, M., Wolfenden, L., Damschroder, L. J., & Lewis, C. C.
- Affiliations
- Hathaway-Sycamores Child and Family Services, Pasadena, CA, USA
- University of Montana (US)
- University of Washington (US)
- Washington University in St. Louis (US)
- Kaiser Permanente Washington Health Research Institute (US)
- University of Toronto (CA)
- University of Newcastle Australia (AU)
- VA Ann Arbor Healthcare System (US)
- Corresponding author
- Stanick, Cameo F
- ORCID
- Venue
- Translational Behavioral Medicine
- Publisher
- Oxford University Press (OUP)
- Published
- 2019-11-20
- Volume
- 11
- Issue
- 1
- Pages
- 11-20
- ISSN
- 1613-9860 (ISSN-L)
- 1869-6716
- PMC
- PMC7877299
- OpenAlex
- W2991615795
- Cited by
- 117
- References
- 16
- Citations by year
- 2026: 5 · 2025: 10 · 2024: 28 · 2023: 35 · 2022: 12 · 2021: 19 · 2020: 8
- FWCI
- 13.29
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model (vor)
- Locations
- Translational Behavioral Medicine (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- NOVA (University of Newcastle Australia) (submittedVersion)
- Funders
- National Institute of Mental Health (R01 MH106510)
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Education › Health Education and Validation
- Keywords
- implementation research, paper, grading criteria, stakeholder engagement, mental health professionals, implementation measures, Delphi method
- MeSH terms
- Humans, Pilot Projects, Psychometrics, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
