Critical reflections on realist review: insights from customizing the methodology to the needs of participatory research assessment: Realist Review Methodology Reflection
Abstract
Realist review has increased in popularity as a methodology for complex intervention assessment. Our experience suggests that the process of designing a realist review requires its customization to areas under investigation. To elaborate on this idea, we first describe the logic underpinning realist review and then present critical reflections on our application experience, organized in seven areas. These are the following: (1) the challenge of identifying middle range theory; (2) addressing heterogeneity and lack of conceptual clarity; (3) the challenge of appraising the quality of complex evidence; (4) the relevance of capturing unintended outcomes; (5) understanding the process of context, mechanism, and outcome (CMO) configuring; (6) incorporating middle-range theory in the CMO configuration process; and (7) using middle range theory to advance the conceptualization of outcomes - both visible and seemingly 'hidden'. One conclusion from our experience is that the degree of heterogeneity of the evidence base will determine whether theory can drive the development of review protocols from the outset, or will follow only after an intense period of data immersion. We hope that presenting a critical reflection on customizing realist review will convey how the methodology can be tailored to the often complex and idiosyncratic features of health research, leading to innovative evidence syntheses.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Despite the growing body of literature, there is relatively little evidence of PPI's impact and outcomes.
“Despite the growing body of literature outlined in the previous section, there is relatively little evidence of PPI's impact and outcomes.99”
From ReseArch with Patient and Public invOlvement: a RealisT evaluation – the RAPPORT study (2015) · cites this paper as [99]
Cites
Realist review - a new method of systematic review designed for complex policy interventions
Pawson, R., Greenhalgh, T., Harvey, G., & Walshe, K. (2005) · Journal of Health Services Research & Policy · doi:10.1258/1355819054308530 · Cited by 3,190 · see citing works
Self-citationMoral Consciousness and Communicative Action.
Weberman, D., Habermas, J., Lenhardt, C. K., Nicholsen, S. W., & McCarthy, T. (1992) · The Philosophical Review · doi:10.2307/2185962 · Cited by 3,053 · see citing works
Sayer, A. (2000) · doi:10.4135/9781446218730 · Cited by 2,259 · see citing works
Lawson, T. (1997) · Routledge eBooks · doi:10.4324/9780203195390 · Cited by 1,910 · see citing works
Moral consciousness and communicative action
(1991) · Choice Reviews Online · doi:10.5860/choice.28-3821 · Cited by 1,877 · see citing works
Pawson, R. (2006) · doi:10.4135/9781849209120 · Cited by 1,791 · see citing works
Sayer, A. (2000) · Cited by 1,753 · see citing works
Critical realism: essential readings
(1999) · Choice Reviews Online · doi:10.5860/choice.36-5017 · Cited by 1,693 · see citing works
The Value and Challenges of Participatory Research: Strengthening Its Practice
Cargo, M., & Mercer, S. L. (2008) · Annual Review of Public Health · doi:10.1146/annurev.publhealth.29.091307.083824 · Cited by 1,505 · see citing works
Self-citationRAMESES publication standards: realist syntheses
Wong, G., Greenhalgh, T., Westhorp, G., Buckingham, J., & Pawson, R. (2013) · BMC Medicine · doi:10.1186/1741-7015-11-21 · Cited by 1,374 · see citing works
Self-citationJAGOSH, J., MACAULAY, A. C., PLUYE, P., SALSBERG, J., BUSH, P. L., HENDERSON, J., SIRETT, E., WONG, G., CARGO, M., HERBERT, C. P., et al. (2012) · The Milbank Quarterly · doi:10.1111/j.1468-0009.2012.00665.x · Cited by 1,268 · see citing works
Self-citationLasker, R. D., Weiss, E. S., & Miller, R. (2001) · The Milbank Quarterly · doi:10.1111/1468-0009.00203 · Cited by 1,053 · see citing works
Identifying and Defining the Dimensions of Community Capacity to Provide a Basis for Measurement
Goodman, R. M., Speers, M. A., McLeroy, K. R., Fawcett, S. B., Kegler, M. C., Parker, E. A., Smith, S. R., Sterling, T. D., & Wallerstein, N. B. (1998) · Health Education & Behavior · doi:10.1177/109019819802500303 · Cited by 1,007 · see citing works
Unpacking black boxes: mechanisms and theory building in evaluation
Astbury, B., & Leeuw, F. L. (2010) · American Journal of Evaluation · doi:10.1177/1098214010371972 · Cited by 850 · see citing works
Realist synthesis: illustrating the method for implementation research
Rycroft-Malone, J., McCormack, B., Hutchinson, A. M., DeCorby, K., Bucknall, T. K., Kent, B., Schultz, A., Snelgrove-Clarke, E., Stetler, C. B., Titler, M., et al. (2012) · Implementation Science · doi:10.1186/1748-5908-7-33 · Cited by 742 · see citing works
Participatory research maximises community and lay involvement
Macaulay, A. C., Commanda, L., Freeman, W. L., Gibson, N., McCabe, M. L., Robbins, C. M., & Twohig, P. L. (1999) · BMJ · doi:10.1136/bmj.319.7212.774 · Cited by 629 · see citing works
Self-citationInternet-based medical education: a realist review of what works, for whom and in what circumstances
Wong, G., Greenhalgh, T., & Pawson, R. (2010) · BMC Medical Education · doi:10.1186/1472-6920-10-12 · Cited by 443 · see citing works
Self-citationRAMESES publication standards: realist syntheses
Wong, G., Greenhalgh, T., Westhorp, G., Buckingham, J., & Pawson, R. (2013) · Journal of Advanced Nursing · doi:10.1111/jan.12095 · Cited by 385 · see citing works
Self-citationOn Theoretical Sociology: Five Essays, Old and New
Merton, R. Κ. (1967) · Cited by 348 · see citing works
School feeding for improving the physical and psychosocial health of disadvantaged students
Kristjansson, B., Petticrew, M. P., Macdonald, B., Krasevec, J., Janzen, L. A., Greenhalgh, T., Wells, G. A., MacGowan, J., Farmer, A. P., Shea, B., et al. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004676.pub2 · Cited by 337 · see citing works
Self-citationDigging for Nuggets: How ‘Bad’ Research Can Yield ‘Good’ Evidence
Pawson, R. (2006) · International Journal of Social Research Methodology · doi:10.1080/13645570600595314 · Cited by 314 · see citing works
Protocol - realist and meta-narrative evidence synthesis: Evolving Standards (RAMESES)
Greenhalgh, T., Wong, G., Westhorp, G., & Pawson, R. (2011) · BMC Medical Research Methodology · doi:10.1186/1471-2288-11-115 · Cited by 279 · see citing works
Self-citationJagosh, J., Pluye, P., Macaulay, A. C., Salsberg, J., Henderson, J., Sirett, E., Bush, P. L., Seller, R., Wong, G., Greenhalgh, T., et al. (2011) · Implementation Science · doi:10.1186/1748-5908-6-24 · Cited by 131 · see citing works
Self-citationPhilosophy and scientific realism
Archer, M. S., Bhaskar, R., Collier, A., Lawson, T., & Norrie, A. (2013) · doi:10.4324/9781315008592-6 · Cited by 83 · see citing works
Assessing the benefits of participatory research: a rationale for a realist review
Macaulay, A. C., Jagosh, J., Seller, R., Henderson, J., Cargo, M., Greenhalgh, T., Wong, G., Salsberg, J., Green, L. W., Herbert, C. P., et al. (2011) · Global Health Promotion · doi:10.1177/1757975910383936 · Cited by 79 · see citing works
Self-citationCritical Realism: Essential Readings
Joseph, J. (2001) · Historical Materialism · doi:10.1163/156920601794750701 · Cited by 77 · see citing works
Managing Collaborative Inter‐organizational Relations
Hibbert, P., Huxham, C., & Ring, P. S. (2009) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199282944.003.0015 · Cited by 59 · see citing works
Quantitative Methods in Participatory Research
Macaulay, A. C., Jagosh, J., Pluye, P., Bush, P. L., & Salsberg, J. (2013) · Nouvelles pratiques sociales · doi:10.7202/1020827ar · Cited by 16 · see citing works
Self-citationPawson, R., Wong, G., & Owen, L. (2011) · Canadian Medical Association Journal · doi:10.1503/cmaj.100903 · Cited by 10 · see citing works
Self-citationThe logic of scientific discovery
Archer, M., Bhaskar, R., Collier, A., et al. (2013) · doi:10.4324/9781315008592-7 · Cited by 10 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Pluye, P., & Hong, Q. N. (2014) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-032013-182440 · Cited by 1,363 · see citing works
Self-citationJagosh, J., Bush, P. L., Salsberg, J., Macaulay, A. C., Greenhalgh, T., Wong, G., Cargo, M., Green, L. W., Herbert, C. P., & Pluye, P. (2015) · BMC Public Health · doi:10.1186/s12889-015-1949-1 · Cited by 758 · see citing works
Self-citationJagosh, J. (2019) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-031816-044451 · Cited by 346 · see citing works
Self-citationReseArch with Patient and Public invOlvement: a RealisT evaluation – the RAPPORT study
Wilson, P., Mathie, E., Keenan, J., McNeilly, E., Goodman, C., Howe, A., Poland, F., Staniszewska, S., Kendall, S., Munday, D., et al. (2015) · Health Services and Delivery Research · doi:10.3310/hsdr03380 · Cited by 297 · see citing works
“BOX 3 Definition of CMO configurations (adapted from Pawson and Tilley(115) and Jagosh et al.(99))”
InfluentialRivas, C., Vigurs, C., Cameron, J., & Yeo, L. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013135.pub2 · Cited by 65 · see citing works
- Authors
- Jagosh, J., Pluye, P., Wong, G., Cargo, M., Salsberg, J., Bush, P. L., Herbert, C. P., Green, L. W., Greenhalgh, T., & Macaulay, A. C.
- Affiliations
- McGill University (CA)
- University of London (GB)
- University of South Australia (AU)
- Western University (CA)
- University of California, San Francisco (US)
- Centre for Primary Care and Public Health; Queen Mary, University of London; London UK
- Corresponding author
- Jagosh, Justin
- ORCID
- Venue
- Research Synthesis Methods
- Publisher
- Wiley
- Published
- 2013-10-22
- Volume
- 5
- Issue
- 2
- Pages
- 131-141
- ISSN
- 1759-2879 (ISSN-L)
- OpenAlex
- W1506611220
- Cited by
- 165
- References
- 30
- Citations by year
- 2026: 11 · 2025: 13 · 2024: 27 · 2023: 13 · 2022: 9 · 2021: 14 · 2020: 22 · 2019: 10 · 2018: 8 · 2017: 13 · 2016: 7 · 2015: 10 · 2014: 6 · 2013: 2
- FWCI
- 9.18
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- Research Synthesis Methods (publishedVersion)
- PubMed (publishedVersion)
- eScholarship@McGill (McGill) (submittedVersion)
- Funders
- National Institute for Health and Care Research (10/1008/07)
- Canadian Institutes of Health Research (91805_1)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Management Science and Operations Research › Evaluation and Performance Assessment
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Keywords
- realist review, complex interventions, middle-range theory, context-mechanism-outcome configurations, conceptual clarity, unintended consequences
- MeSH terms
- Clinical Trials as Topic, Humans, Research Design, Outcome Assessment, Health Care, Evidence-Based Medicine, Community-Based Participatory Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
