The DOI matches its Crossref record; 95 of the 97 references it cites were found in a registry or an index.
Development of methodological guidance, publication standards and training materials for realist and meta-narrative reviews: the RAMESES (Realist And Meta-narrative Evidence Syntheses – Evolving Standards) project
Open accessAbstract
Background: There is growing interest in theory-driven, qualitative and mixed-method approaches to systematic review, such as realist and meta-narrative review. These approaches offer the potential to expand the knowledge base in policy-relevant areas. However, the quality of such reviews can be difficult to assess. Objectives: The aim of this project was to produce methodological guidance, publication standards and training resources for those seeking to undertake realist and/or meta-narrative reviews. Methods/design: We (1) collated and summarised existing literature on the principles of good practice in realist and meta-narrative systematic reviews; (2) considered the extent to which these principles had been followed by published and in-progress reviews, thereby identifying how rigour may have been lost and how existing methods could be improved; (3) used an online Delphi method with an interdisciplinary panel of experts from academia and policy, to produce a draft set of methodological steps and publication standards; (4) produced training materials with learning objectives linked to these steps; (5) refined these standards and training materials prospectively on real reviews in progress, capturing methodological and other challenges as they arose; (6) synthesised expert input, evidence review and real-time problem analysis into more definitive guidance and standards; and (7) disseminated outputs to audiences in academia and policy. Results: An important element of this study was the establishment of an e-mail mailing list to bring together researches in the field ( www.jiscmail.ac.uk/RAMESES ). Our literature review identified 35 and nine realist and meta-narrative reviews respectively. Analysis and discussion within the project team produced a summary of the published literature, and common questions and challenges into briefing materials for the Delphi panel, comprising 37 and 33 members (for realist and meta-narrative reviews respectively). Within three rounds this panel had reached a consensus on 19 (realist) and 20 (meta-narrative) key publication standards, with an overall response rate of 90% and 91% respectively. The Realist And Meta-narrative Evidence Syntheses – Evolving Standards (RAMESES) publication standards for realist syntheses and meta-narrative reviews were published in open-access journals and quickly became highly accessed. The RAMESES quality standards and training materials drew together the following sources of data: (1) personal expertise as researchers and trainers; (2) data from the Delphi panels; (3) feedback from participants at training sessions we ran; and (4) comments made on RAMESES mailing list. The quality standards and training materials are freely available online ( www.ramesesproject.org ). Discussion: The production of these standards and guidance drew on multiple sources of knowledge and expertise, and a high degree of a consensus was achieved despite ongoing debate among researchers about the overall place of these methodologies in the secondary research toolkit. As with all secondary research methods, guidance on quality assurance and uniform reporting is an important step towards improving quality and consistency of studies. We anticipate that as more reviews are undertaken, further refinement will be needed to the publication and quality standards and training materials. Limitations: The project’s outputs are not definitive and in the future updating and further development is likely to be needed. Conclusion: An initial set of publication standards, quality standards and training materials have been produced for researchers, users and funders of realist or meta-narrative reviews. As realist and meta-narrative reviews are relatively new approaches to evidence synthesis, methodological development is needed for both review approaches. Funding: The National Institute for Health Research Health Services and Delivery Research programme.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“The realist approach followed processes set out by Pawson et al. and the Realist and Meta-narrative Evidence Synthesis: Evolving Standards (RAMESES) [29, 32].”
From Do hospital-to-home transitions work for older adults with multiple long-term conditions including dementia? A realist review (2025) · cites this paper as [32]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · BMJ · doi:10.1136/bmj.b2700 · Cited by 17,648
Methods for the thematic synthesis of qualitative research in systematic reviews
Thomas, J., & Harden, A. (2008) · BMC Medical Research Methodology · doi:10.1186/1471-2288-8-45 · Cited by 9,312
Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gotzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c869 · Cited by 8,883
Diffusion of Innovations in Service Organizations: Systematic Review and Recommendations
GREENHALGH, T., ROBERT, G., MACFARLANE, F., BATE, P., & KYRIAKIDOU, O. (2004) · The Milbank Quarterly · doi:10.1111/j.0887-378x.2004.00325.x · Cited by 7,448
Self-citationResearch guidelines for the Delphi survey technique
Hasson, F. (2000) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.2000.01567.x · Cited by 5,951
The Delphi Technique: Making Sense of Consensus
Hsu, C.-C., & Sandford, B. A. (2007) · University of Massachusetts Amherst · doi:10.7275/pdz9-th90 · Cited by 4,500
Realist review - a new method of systematic review designed for complex policy interventions
Pawson, R., Greenhalgh, T., Harvey, G., & Walshe, K. (2005) · Journal of Health Services Research & Policy · doi:10.1258/1355819054308530 · Cited by 3,190 · cited 3×
“The quality standards and training materials drew on the following sources of data: (1) personal expertise as researchers and trainers; (2) data from the Delphi panels; (3) feedback from participants at training sessions we ran; and (4) comments made on RAMESES mailing list.”
MethodSelf-citationSynthesising qualitative and quantitative evidence: A review of possible methods
Dixon-Woods, M., Agarwal, S., Jones, D., Young, B., & Sutton, A. (2005) · Journal of Health Services Research & Policy · doi:10.1177/135581960501000110 · Cited by 2,132 · cited 4×
“Two of our initial questions related to these issues: (5) What is the link between realist and meta-narrative review and the policy-making process? How have published reviews been commissioned or sponsored? How have policy-makers been involved in shaping the review? How have they been involved in disseminating and app…”
BackgroundMethodInfluentialGreenhalgh, T., & Peacock, R. (2005) · BMJ · doi:10.1136/bmj.38636.593461.68 · Cited by 2,045
Self-citationMethods for the synthesis of qualitative research: a critical review
Barnett-Page, E., & Thomas, J. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-59 · Cited by 2,042
Research works printed but not linked: 3
- Weiss, 1995, New Approaches to Evaluating Community Initiatives: Concepts, Methods, and Contexts, 65
- Meyer, 2008, Independent Learning Literature Review (Research Report DCSF-RR051)
- McLean, 2010, Behavioral caregiving for adults with traumatic brain injury living in nursing homes: developing a practice model, J Theory Constr Test, 14, 17
Cited by
Munn, Z., Stern, C., Aromataris, E., Lockwood, C., & Jordan, Z. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0468-4 · Cited by 1,164
RAMESES II reporting standards for realist evaluations
Wong, G., Westhorp, G., Manzano, A., Greenhalgh, J., Jagosh, J., & Greenhalgh, T. (2016) · BMC Medicine · doi:10.1186/s12916-016-0643-1 · Cited by 880
“In other words the data gathered needs to contain information that [20, 48]: (1)May be used to develop, support, refute or refine the assignment of the conceptual label of C, M or O”
MethodSelf-citationPollock, A., & Berge, E. (2018) · International Journal of Stroke · doi:10.1177/1747493017743796 · Cited by 530
“Realist And Metanarrative Evidence Syntheses – Evolving Standards (RAMESES) publication standards (126)”
BackgroundOrganizational Health Literacy: Review of Theories, Frameworks, Guides, and Implementation Issues
Farmanova, E., Bonneville, L., & Bouchard, L. (2018) · INQUIRY: The Journal of Health Care Organization, Provision, and Financing · doi:10.1177/0046958018757848 · Cited by 264
“We used an adaptation of the metanarrative review as a means for synthesizing a range of literature.(34) This review provides unique insights into the extent and scope of research and implementation of OHL and discusses recommendations to improve the organization, quality, and delivery of healthcare.”
MethodDada, S., Dalkin, S., Gilmore, B., Hunter, R., & Mukumbang, F. C. (2023) · Research Synthesis Methods · doi:10.1002/jrsm.1630 · Cited by 212
“Within prominent realist review methods papers and guidelines, terms such as credible, plausible, believable, trustworthy and coherent have been used individually or in combination to describe rigour.(16,18,19,21) This difficulty of defining and applying rigour is compounded when using data from different research par…”
BackgroundMethod
Metadata
- Authors
- Wong, G., Greenhalgh, T., Westhorp, G., & Pawson, R.
- Affiliations
- University of London (GB)
- Centre for Primary Care and Public Health, Queen Mary University of London, London, UK
- Community Matters, Mount Torrens, SA, Australia
- University of Leeds (GB)
- Corresponding author
- Wong, Geoff
- Venue
- Health Services and Delivery Research
- Published
- 2014-09
- Volume
- 2
- Issue
- 30
- Pages
- 1-252
- ISSN
- 2050-4349 (ISSN-L)
- 2050-4357
- OpenAlex
- W2072084577
- Cited by
- 276
- References
- 97
- Citations by year
- 2026: 26 · 2025: 40 · 2024: 31 · 2023: 24 · 2022: 36 · 2021: 40 · 2020: 15 · 2019: 20 · 2018: 14 · 2017: 16 · 2016: 8 · 2015: 6
- FWCI
- 2.75
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Health Services and Delivery Research (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Institute for Health Research (10/1008/07)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Keywords
- realist review, meta-narrative review, methodological recommendations, publication standards, training materials, expert consensus, Delphi method, review quality, research training, policy-relevant research, research capacity building
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Development of methodological guidance, publication standards and training materials for realist and meta-narrative reviews: the RAMESES (Realist And Meta-narrative Evidence Syntheses – Evolving Standards) project. https://app.cochat.ai/papers/10.3310/hsdr02300
- The paper
- Wong, G., Greenhalgh, T., Westhorp, G., & Pawson, R. (2014). Development of methodological guidance, publication standards and training materials for realist and meta-narrative reviews: the RAMESES (Realist And Meta-narrative Evidence Syntheses – Evolving Standards) project. Health Services and Delivery Research. https://doi.org/10.3310/hsdr02300
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.