The DOI matches its Crossref record; 93 of the 93 references it cites were found in a registry or an index.
Is realist evaluation keeping its promise? A review of published empirical studies in the field of health systems research
Abstract
This overview aims to stimulate conceptual and practical discussions to help unlock the full potential of realist evaluation in health systems research. Based on a structured literature search, this review maps how the concepts of realist evaluation are applied in health systems research and which methodological problems are encountered. We found a great diversity in the depth of application of the philosophical concepts, use of terminology and scope of application in the research process. Terms of theory-driven evaluation, theories of change and realist evaluation are often used interchangeably. Diverging views exist regarding the nature of ‘mechanism’ and the difference between mechanism and essential context condition. A lack of methodological guidance was highlighted by many authors. Realist evaluation is slowly gaining traction in health systems research, but more clarity is needed concerning the definitions of mechanisms and context and how the configuration of context, mechanism and intervention can be described and assessed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Marchal et al. [18], in a systematic review on realist evaluation in health systems research, also noted that ToC and realist evaluation were often used together or interchangeably.”
From Using theory of change to design and evaluate public health interventions: a systematic review (2015) · cites this paper as [18]
In the citing paper
“Both a published literature review 112 and critical discussions in the applied literature cite several methodological challenges associated with a RE approach, including deciding what constitutes a mechanism, 127 and indeed differentiating between what is a mechanism and what is context.”
From ReseArch with Patient and Public invOlvement: a RealisT evaluation – the RAPPORT study (2015) · cites this paper as [112]
Citations
Cites
Social Theory and Social Structure
Ross, E. J., & Merton, R. Κ. (1958) · The American Catholic Sociological Review · doi:10.2307/3709394 · Cited by 10,318
Social Theory and Social Structure.
Bock, K. E., & Merton, R. Κ. (1969) · American Sociological Review · doi:10.2307/2091967 · Cited by 6,687
Systems Thinking, Systems Practice
Checkland, P. B. (1981) · Cumulative Index of Computer Aided Architectural Design · Cited by 6,256
Systems thinking, systems practice
(1982) · European Journal of Operational Research · doi:10.1016/0377-2217(82)90206-5 · Cited by 3,579
Realist review - a new method of systematic review designed for complex policy interventions
Pawson, R., Greenhalgh, T., Harvey, G., & Walshe, K. (2005) · Journal of Health Services Research & Policy · doi:10.1258/1355819054308530 · Cited by 3,190
A set-theoretic approach to organizational configurations
Fiss, P. C. (2007) · Academy of Management Review · doi:10.5465/amr.2007.26586092 · Cited by 2,225
Pawson, R. (2006) · doi:10.4135/9781849209120 · Cited by 1,791
Critical realism: essential readings
(1999) · Choice Reviews Online · doi:10.5860/choice.36-5017 · Cited by 1,683
Designing and evaluating complex interventions to improve health care
Campbell, N. C., Murray, E., Darbyshire, J., Emery, J., Farmer, A., Griffiths, F., Guthrie, B., Lester, H., Wilson, P., & Kinmonth, A. L. (2007) · BMJ · doi:10.1136/bmj.39108.379965.be · Cited by 1,331
Chen, H.-T. (1990) · Medical Entomology and Zoology · Cited by 1,227
Evidence-Based Public Health: Moving Beyond Randomized Trials
Victora, C. G., Habicht, J.-P., & Bryce, J. (2004) · American Journal of Public Health · doi:10.2105/ajph.94.3.400 · Cited by 1,056
Mintzberg on management: inside our strange world of organizations
(1990) · Choice Reviews Online · doi:10.5860/choice.27-3961 · Cited by 1,042
Unpacking black boxes: mechanisms and theory building in evaluation
Astbury, B., & Leeuw, F. L. (2010) · American Journal of Evaluation · doi:10.1177/1098214010371972 · Cited by 850
Berwick, D. M. (2008) · JAMA · doi:10.1001/jama.299.10.1182 · Cited by 809
Evidence-based policy: the promise of ‘realist synthesis’
Pawson, R. (2002) · Evaluation · doi:10.1177/135638902401462448 · Cited by 729
Theory‐based evaluation: Past, present, and future
Weiss, C. H. (1997) · New Directions for Evaluation · doi:10.1002/ev.1086 · Cited by 634
Bamberger, P. A. (2008) · Academy of Management Journal · doi:10.5465/amj.2008.34789630 · Cited by 616
Lipsey, M. W. (1991) · Evaluation and Program Planning · doi:10.1016/0149-7189(91)90027-e · Cited by 592
Research designs for studies evaluating the effectiveness of change and improvement strategies
Eccles, M. (2003) · Quality and Safety in Health Care · doi:10.1136/qhc.12.1.47 · Cited by 586
A Systematic Review of Theory-Driven Evaluation Practice From 1990 to 2009
Coryn, C. L. S., Noakes, L. A., Westine, C. D., & Schröter, D. C. (2011) · American Journal of Evaluation · doi:10.1177/1098214010389321 · Cited by 531
Mintzberg on management: Inside our strange world of organizations
Petersen, P. B. (1990) · Organizational Dynamics · doi:10.1016/0090-2616(90)90050-y · Cited by 527
Outcome pattern matching and program theory
Trochim, W. M. K. (1989) · Evaluation and Program Planning · doi:10.1016/0149-7189(89)90052-9 · Cited by 481
Chen, H.-T., & Rossi, P. H. (1983) · Evaluation Review · doi:10.1177/0193841x8300700301 · Cited by 420
Studying the Organisation and Delivery of Health Services: Research Methods
Goodwin, N. (2002) · Health & Social Care in the Community · doi:10.1046/j.1365-2524.2002.03832.x · Cited by 385
Theory-Based Evaluation and Types of Complexity
Stame, N. (2004) · Evaluation · doi:10.1177/1356389004043135 · Cited by 383
Theories of Change and Realistic Evaluation: Peas in a Pod or Apples and Oranges?
Blamey, A., & Mackenzie, M. (2007) · Evaluation · doi:10.1177/1356389007082129 · Cited by 383
Chen, H.-T. (2005) · doi:10.4135/9781412985444 · Cited by 365
How Do You Modernize a Health Service? A Realist Evaluation of Whole‐Scale Transformation in London
Greenhalgh, T., Humphrey, C., HUGHES, J., Macfarlane, F., Butler, C., & Pawson, R. (2009) · Milbank Quarterly · doi:10.1111/j.1468-0009.2009.00562.x · Cited by 364
The Importance of External Validity
Steckler, A., & McLeroy, K. R. (2007) · American Journal of Public Health · doi:10.2105/ajph.2007.126847 · Cited by 295
Reconstructing Program Theories: Methods Available and Problems to be Solved
Leeuw, F. L. (2003) · American Journal of Evaluation · doi:10.1016/s1098-2140(02)00271-0 · Cited by 254
Understanding what works—and why—in quality improvement: the need for theory-driven evaluation
Walshe, K. (2007) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzm004 · Cited by 237
The Contribution of Critical Realism as an Underpinning Philosophy for OR/MS and Systems
Mingers, J. (2000) · Journal of the Operational Research Society · doi:10.1057/palgrave.jors.2601033 · Cited by 235
Chen, H.-T., & Rossi, P. H. (1980) · Social Forces · doi:10.1093/sf/59.1.106 · Cited by 219
Evidence, Understanding and Complexity: Evaluation in Non-Linear Systems
Barnes, M., Matka, E., & Sullivan, H. (2003) · Evaluation · doi:10.1177/13563890030093003 · Cited by 208
Critical realism: a way forward for evaluation research in nursing?
McEvoy, P., & Richards, D. A. (2003) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.2003.02730.x · Cited by 203
The theory-driven approach to validity
Chen, H.-T., & Rossi, P. H. (1987) · Evaluation and Program Planning · doi:10.1016/0149-7189(87)90025-5 · Cited by 198
Studying the Organisation and Delivery of Health Services
(2004) · Routledge eBooks · doi:10.4324/9780203481981 · Cited by 198
Byng, R., Norman, I. J., & Redfern, S. (2005) · Evaluation · doi:10.1177/1356389005053198 · Cited by 187
A realistic evaluation: the case of protocol-based care
Rycroft-Malone, J., Fontenla, M., Bick, D., & Seers, K. (2010) · Implementation Science · doi:10.1186/1748-5908-5-38 · Cited by 173
Reconstructing Program Theories: Methods Available and Problems to be Solved
Leeuw, F. L. (2003) · American Journal of Evaluation · doi:10.1177/109821400302400102 · Cited by 170
A realist evaluation of the management of a well- performing regional hospital in Ghana
Marchal, B., Dedzo, M., & Kegels, G. (2010) · BMC Health Services Research · doi:10.1186/1472-6963-10-24 · Cited by 156
Self-citationRealist Evaluation in Practice
Kazi, M. A. F. (2003) · SAGE Publications Ltd eBooks · doi:10.4135/9781849209762 · Cited by 156
Rogers, P., & Weiss, C. H. (2007) · New Directions for Evaluation · doi:10.1002/ev.225 · Cited by 150
New approaches to evaluating community initiatives: Theory, measurement, and analysis
Connell, K. F.-A. A. A. C. K. A. J. P. (1998) · Cited by 136
Strong theory, flexible methods: Evaluating complex community-based initiatives
Judge, K., & Bauld, L. (2001) · Critical Public Health · doi:10.1080/09581590010028237 · Cited by 135
Issues in the theory-driven perspective
Chen, H.-T., & Rossi, P. H. (1989) · Evaluation and Program Planning · doi:10.1016/0149-7189(89)90046-3 · Cited by 132
Realist Evaluation for Practice
Kazi, M. (2003) · The British Journal of Social Work · doi:10.1093/bjsw/33.6.803 · Cited by 130
Constructing Theories of Change: Methods and Sources
Mason, P., & Barnes, M. (2007) · Evaluation · doi:10.1177/1356389007075221 · Cited by 123
Evaluation of health promotion in schools: a realistic evaluation approach using mixed methods
Pommier, J., Guevel, M., & Jourdan, D. (2010) · BMC Public Health · doi:10.1186/1471-2458-10-43 · Cited by 119
Byng, R., Norman, I. J., Redfern, S., & Jones, R. H. (2008) · BMC Health Services Research · doi:10.1186/1472-6963-8-274 · Cited by 114
A realist study of the mechanisms of cardiac rehabilitation
Clark, A. M., Whelan, H. K., Barbour, R. S., & MACINTYRE, P. D. (2005) · Journal of Advanced Nursing · doi:10.1111/j.1365-2648.2005.03601.x · Cited by 108
Belle, S. V., Marchal, B., Dubourg, D., & Kegels, G. (2010) · BMC Public Health · doi:10.1186/1471-2458-10-741 · Cited by 95
Self-citationComplexity and health – yesterday's traditions, tomorrow's future
Sturmberg, J. P., & Martin, C. M. (2009) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2009.01163.x · Cited by 94
Wanted--new methodologies for health service research. Is complexity theory the answer?
Kernick, D. P. (2006) · Family Practice · doi:10.1093/fampra/cml011 · Cited by 89
Building Collaborative Capacity through `Theories of Change'
Sullivan, H. T., Barnes, M., & Matka, E. (2002) · Evaluation · doi:10.1177/1358902002008002514 · Cited by 85
Mackenzie, M., & Blamey, A. (2005) · Evaluation · doi:10.1177/1356389005055538 · Cited by 85
Studying the Organisation and Delivery of Health Services
Blackman, T. (2002) · Dialnet (Universidad de la Rioja) · Cited by 84
Philosophy and scientific realism
Archer, M. S., Bhaskar, R., Collier, A., Lawson, T., & Norrie, A. (2013) · doi:10.4324/9781315008592-6 · Cited by 83
Scriven, A. (2011) · doi:10.1093/obo/9780199756797-0007 · Cited by 82
Realist Evaluation in Practice: Health and Social Work
Weir, E., & Kazi, M. A. F. (2007) · Evaluation Journal of Australasia · doi:10.1177/1035719x0700700108 · Cited by 80
Dickinson, H. (2006) · Health & Social Care in the Community · doi:10.1111/j.1365-2524.2006.00650.x · Cited by 78
What do we mean by rigorous health-systems research?
MILLS, A. J., Gilson, L., Hanson, K., Palmer, N., & Lagarde, M. (2008) · The Lancet · doi:10.1016/s0140-6736(08)61633-5 · Cited by 77
A Theory‐Driven Approach to Evaluating Quality of Nursing Care
Sidani, S., Doran, D. M., & Mitchell, P. H. (2004) · Journal of Nursing Scholarship · doi:10.1111/j.1547-5069.2004.04014.x · Cited by 75
Maluka, S. O., Kamuzora, P., Sebastiån, M. S., Byskov, J., Ndawi, B., Olsen, Ø. E., & Hurtig, A. (2011) · Implementation Science · doi:10.1186/1748-5908-6-11 · Cited by 74
Blaise, P., & Kegels, G. (2004) · The International Journal of Health Planning and Management · doi:10.1002/hpm.769 · Cited by 68
Self-citationDeveloping a managed clinical network in palliative care: a realistic evaluation
Tolson, D., McIntosh, J., Loftus, L., & Cormie, P. (2006) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2005.11.027 · Cited by 68
Tackling health inequalities through partnership working: Learning from a realistic evaluation
Evans, D., & Killoran, A. (2000) · Critical Public Health · doi:10.1080/09581590050075899 · Cited by 66
Evaluating complex public health interventions: theory, methods and scope of realist enquiry
Connelly, J. B. (2007) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2006.00790.x · Cited by 63
The conceptual framework of the theory-driven perspective
Chen, H.-T. (1989) · Evaluation and Program Planning · doi:10.1016/0149-7189(89)90057-8 · Cited by 58
Realist Evaluation as a Framework for the Assessment of Teaching About the Improvement of Care
Ogrinc, G. S., & Batalden, P. B. (2009) · Journal of Nursing Education · doi:10.3928/01484834-20091113-08 · Cited by 55
Wand, T., White, K., & Patching, J. (2010) · Nursing Inquiry · doi:10.1111/j.1440-1800.2010.00488.x · Cited by 53
Marchal, B., Dedzo, M., & Kegels, G. (2010) · BMC Public Health · doi:10.1186/1471-2458-10-787 · Cited by 49
Self-citationReview: Promoting Realism in Evaluation
Julnes, G., Mark, M. M., & Henry, G. T. (1998) · Evaluation · doi:10.1177/13563899822208743 · Cited by 42
Evaluating change in health care practice: lessons from three studies
Redfern, S., Christian, S., & Norman, I. J. (2003) · Journal of Evaluation in Clinical Practice · doi:10.1046/j.1365-2753.2003.00387.x · Cited by 40
Manzano, A. (2011) · Evaluation · doi:10.1177/1356389010389913 · Cited by 36
Quality improvement reports: a new kind of article
Smith, R. J. (2000) · BMJ · doi:10.1136/bmj.321.7274.1428 · Cited by 34
An Application of Theory-Driven Evaluation to a Drop-In Youth Center
Mercier, C., Piat, M., Péladeau, N., & Dagenais, C. (2000) · Evaluation Review · doi:10.1177/0193841x0002400103 · Cited by 34
Analysing process in healthcare: the methodological and theoretical challenges
Calnan, M., & Ferlie, E. B. (2003) · Policy & Politics · doi:10.1332/030557303765371672 · Cited by 33
Is Realist Evaluation a Realistic Approach for Complex Reforms?
Pedersen, L. H., & Rieper, O. (2008) · Evaluation · doi:10.1177/1356389008090856 · Cited by 30
Cole, G. E. (1999) · American Journal of Evaluation · doi:10.1016/s1098-2140(99)00033-8 · Cited by 29
Costner, H. L. (1989) · Evaluation and Program Planning · doi:10.1016/0149-7189(89)90051-7 · Cited by 28
Cole, G. E. (1999) · American Journal of Evaluation · doi:10.1177/109821409902000305 · Cited by 27
Evaluating Urban Regeneration Programmes in Britain
Ho, S. Y. (1999) · Evaluation · doi:10.1177/135638999400830084 · Cited by 24
Realistic Evaluation of an Illicit Drug Deterrence Programme
Leone, L. (2008) · Evaluation · doi:10.1177/1356389007084673 · Cited by 24
A realistic theory of health sector management
Connelly, J. (2000) · Journal of Management in Medicine · doi:10.1108/02689230010362882 · Cited by 23
Douglas, F., Gray, D., & van Teijlingen, E. (2010) · BMC Health Services Research · doi:10.1186/1472-6963-10-49 · Cited by 22
Mackenzie, M., Koshy, P., Leslie, W. S., Lean, M. E. J., & Hankey, C. R. (2009) · European Journal of Clinical Nutrition · doi:10.1038/ejcn.2009.38 · Cited by 18
A methodological inquiry into the evaluation of smoking cessation programmes
Kaneko, M. (1999) · Health Education Research · doi:10.1093/her/14.3.433 · Cited by 14
Developing an evidence base for intermediate care delivered by GPs with a special interest.
Kernick, D. P., & Mannion, R. (2005) · PubMed · Cited by 12
Ridde, V., & Guichard, A. (2011) · Global Health Promotion · doi:10.1177/1757975911412400 · Cited by 10
Realism in evidence based medicine: interpreting the randomised controlled trial
Connelly, J. (2004) · Journal of Health Organization and Management · doi:10.1108/14777260410538861 · Cited by 10
Grocott, P., Cowley, S., & Richardson, A. M. (2002) · Evaluation · doi:10.1177/135638902401462457 · Cited by 7
Rodriguez, E., & Mead, J. P. (1997) · Evaluation and Program Planning · doi:10.1016/s0149-7189(96)00051-1 · Cited by 4
Research works printed but not linked: 4
- Connell JP, 1995, New Approaches To Evaluating Community Initiatives. Concepts, Methods, and Contexts
- Kernick D, 2005, British Journal of Medical Practice, 55, 908
- Pawson R, 1997, Realistic Evaluation
- Weiss C, 1995, New Approaches to Evaluating Comprehensive Community Initiatives: Concepts, Methods and Contexts
Cited by
Sorted by citations
What’s in a mechanism? Development of a key concept in realist evaluation
Dalkin, S. M., Greenhalgh, J., Jones, D., Cunningham, B., & Lhussier, M. (2015) · Implementation Science · doi:10.1186/s13012-015-0237-x · Cited by 896
RAMESES II reporting standards for realist evaluations
Wong, G., Westhorp, G., Manzano, A., Greenhalgh, J., Jagosh, J., & Greenhalgh, T. (2016) · BMC Medicine · doi:10.1186/s12916-016-0643-1 · Cited by 880
“Further, as noted in Item 5 above, the evaluation question(s), scope and design may evolve over the course of the evaluation [19].”
BackgroundInfluentialThe craft of interviewing in realist evaluation
Manzano, A. (2016) · Evaluation · doi:10.1177/1356389016638615 · Cited by 691
Using theory of change to design and evaluate public health interventions: a systematic review
Breuer, E., Lee, L., De Silva, M., & Lund, C. (2015) · Implementation Science · doi:10.1186/s13012-016-0422-6 · Cited by 425
“Despite some fundamental differences in their theoretical underpinnings, many of these approaches are used interchangeably or together [18, 20].”
BackgroundInfluentialJagosh, J. (2019) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-031816-044451 · Cited by 346
Metadata
- Authors
- Marchal, B., van Belle, S., van Olmen, J., Hoerée, T., & Kegels, G.
- Affiliations
- ITM-A, Belgium
- BTC, Belgium
- Venue
- Evaluation
- Publisher
- SAGE Publications
- Published
- 2012-04
- Volume
- 18
- Issue
- 2
- Pages
- 192-212
- ISSN
- 1356-3890 (ISSN-L)
- 1461-7153
- OpenAlex
- W2077589521
- Cited by
- 476
- References
- 92
- Citations by year
- 2026: 22 · 2025: 29 · 2024: 41 · 2023: 29 · 2022: 51 · 2021: 64 · 2020: 31 · 2019: 40 · 2018: 37 · 2017: 28 · 2016: 31 · 2015: 34 · 2014: 24 · 2013: 12 · 2012: 3
- FWCI
- 30.83
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Evaluation (publishedVersion)
- Topics
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- realist evaluation, health systems research, mechanisms, context, intervention, theory of change, theory-based evaluation, methodological recommendations, conceptual clarity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
