CoChat

The DOI matches its Crossref record; 95 of the 97 references it cites were found in a registry or an index.

Development of methodological guidance, publication standards and training materials for realist and meta-narrative reviews: the RAMESES (Realist And Meta-narrative Evidence Syntheses – Evolving Standards) project

Open access
Wong, Geoff, Greenhalgh, Trish, Westhorp, Gill, Pawson, Ray · 2014 · Health Services and Delivery Research

Abstract

Background: There is growing interest in theory-driven, qualitative and mixed-method approaches to systematic review, such as realist and meta-narrative review. These approaches offer the potential to expand the knowledge base in policy-relevant areas. However, the quality of such reviews can be difficult to assess. Objectives: The aim of this project was to produce methodological guidance, publication standards and training resources for those seeking to undertake realist and/or meta-narrative reviews. Methods/design: We (1) collated and summarised existing literature on the principles of good practice in realist and meta-narrative systematic reviews; (2) considered the extent to which these principles had been followed by published and in-progress reviews, thereby identifying how rigour may have been lost and how existing methods could be improved; (3) used an online Delphi method with an interdisciplinary panel of experts from academia and policy, to produce a draft set of methodological steps and publication standards; (4) produced training materials with learning objectives linked to these steps; (5) refined these standards and training materials prospectively on real reviews in progress, capturing methodological and other challenges as they arose; (6) synthesised expert input, evidence review and real-time problem analysis into more definitive guidance and standards; and (7) disseminated outputs to audiences in academia and policy. Results: An important element of this study was the establishment of an e-mail mailing list to bring together researches in the field ( www.jiscmail.ac.uk/RAMESES ). Our literature review identified 35 and nine realist and meta-narrative reviews respectively. Analysis and discussion within the project team produced a summary of the published literature, and common questions and challenges into briefing materials for the Delphi panel, comprising 37 and 33 members (for realist and meta-narrative reviews respectively). Within three rounds this panel had reached a consensus on 19 (realist) and 20 (meta-narrative) key publication standards, with an overall response rate of 90% and 91% respectively. The Realist And Meta-narrative Evidence Syntheses – Evolving Standards (RAMESES) publication standards for realist syntheses and meta-narrative reviews were published in open-access journals and quickly became highly accessed. The RAMESES quality standards and training materials drew together the following sources of data: (1) personal expertise as researchers and trainers; (2) data from the Delphi panels; (3) feedback from participants at training sessions we ran; and (4) comments made on RAMESES mailing list. The quality standards and training materials are freely available online ( www.ramesesproject.org ). Discussion: The production of these standards and guidance drew on multiple sources of knowledge and expertise, and a high degree of a consensus was achieved despite ongoing debate among researchers about the overall place of these methodologies in the secondary research toolkit. As with all secondary research methods, guidance on quality assurance and uniform reporting is an important step towards improving quality and consistency of studies. We anticipate that as more reviews are undertaken, further refinement will be needed to the publication and quality standards and training materials. Limitations: The project’s outputs are not definitive and in the future updating and further development is likely to be needed. Conclusion: An initial set of publication standards, quality standards and training materials have been produced for researchers, users and funders of realist or meta-narrative reviews. As realist and meta-narrative reviews are relatively new approaches to evidence synthesis, methodological development is needed for both review approaches. Funding: The National Institute for Health Research Health Services and Delivery Research programme.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3310/hsdr02300

Cites

95 of 95 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 3

  1. Weiss, 1995, New Approaches to Evaluating Community Initiatives: Concepts, Methods, and Contexts, 65
  2. Meyer, 2008, Independent Learning Literature Review (Research Report DCSF-RR051)
  3. McLean, 2010, Behavioral caregiving for adults with traumatic brain injury living in nursing homes: developing a practice model, J Theory Constr Test, 14, 17

Cited by

See all 276

Metadata

Authors
Wong, G., Greenhalgh, T., Westhorp, G., & Pawson, R.
Affiliations
  • University of London (GB)
  • Centre for Primary Care and Public Health, Queen Mary University of London, London, UK
  • Community Matters, Mount Torrens, SA, Australia
  • University of Leeds (GB)
Corresponding author
Wong, Geoff
Venue
Health Services and Delivery Research
Published
2014-09
Volume
2
Issue
30
Pages
1-252
ISSN
  • 2050-4349 (ISSN-L)
  • 2050-4357
OpenAlex
W2072084577
Cited by
276
References
97
Citations by year
2026: 26 · 2025: 40 · 2024: 31 · 2023: 24 · 2022: 36 · 2021: 40 · 2020: 15 · 2019: 20 · 2018: 14 · 2017: 16 · 2016: 8 · 2015: 6
FWCI
2.75
Type
journal article
Language
en
Open access
Locations
  • Health Services and Delivery Research (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
National Institute for Health Research (10/1008/07)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Public Administration › Evaluation and Performance Assessment
  • Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
Keywords
realist review, meta-narrative review, methodological recommendations, publication standards, training materials, expert consensus, Delphi method, review quality, research training, policy-relevant research, research capacity building

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.