Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 12 of the 13 references it cites were found in a registry or an index; 4 of the 12 sources its claims cite were checked for support.

SANRA—a scale for the quality assessment of narrative review articles

Open access
Baethge, Christopher, Goldbeck-Wood, Sandra, Mertens, Stephan · 2019 · Research Integrity and Peer Review

Abstract

BACKGROUND: Narrative reviews are the commonest type of articles in the medical literature. However, unlike systematic reviews and randomized controlled trials (RCT) articles, for which formal instruments exist to evaluate quality, there is currently no instrument available to assess the quality of narrative reviews. In response to this gap, we developed SANRA, the Scale for the Assessment of Narrative Review Articles. METHODS: A team of three experienced journal editors modified or deleted items in an earlier SANRA version based on face validity, item-total correlations, and reliability scores from previous tests. We deleted an item which addressed a manuscript's writing and accessibility due to poor inter-rater reliability. The six items which form the revised scale are rated from 0 (low standard) to 2 (high standard) and cover the following topics: explanation of (1) the importance and (2) the aims of the review, (3) literature search and (4) referencing and presentation of (5) evidence level and (6) relevant endpoint data. For all items, we developed anchor definitions and examples to guide users in filling out the form. The revised scale was tested by the same editors (blinded to each other's ratings) in a group of 30 consecutive non-systematic review manuscripts submitted to a general medical journal. RESULTS: Raters confirmed that completing the scale is feasible in everyday editorial work. The mean sum score across all 30 manuscripts was 6.0 out of 12 possible points (SD 2.6, range 1-12). Corrected item-total correlations ranged from 0.33 (item 3) to 0.58 (item 6), and Cronbach's alpha was 0.68 (internal consistency). The intra-class correlation coefficient (average measure) was 0.77 [95% CI 0.57, 0.88] (inter-rater reliability). Raters often disagreed on items 1 and 4. CONCLUSIONS: SANRA's feasibility, inter-rater reliability, homogeneity of items, and internal consistency are sufficient for a scale of six items. Further field testing, particularly of validity, is desirable. We recommend rater training based on the "explanations and instructions" document provided with SANRA. In editorial decision-making, SANRA may complement journal-specific evaluation of manuscripts-pertaining to, e.g., audience, originality or difficulty-and may contribute to improving the standard of non-systematic reviews.

Overview

CoChat Summary

TL;DR

SANRA is a six-item, 0–2 scale (maximum 12) for assessing the quality of narrative review articles, developed and field-tested by three journal editors. In 30 consecutive non-systematic review manuscripts, it showed feasible use, sufficient internal consistency and inter-rater reliability, and a modest association with editorial acceptance.

Key findings

  • Across 30 manuscripts (90 ratings), the mean SANRA sum score was 6.0 out of 12 (SD 2.6, range 1–12, median 6).
  • Internal consistency was Cronbach’s alpha 0.68; corrected item-total correlations ranged from 0.33 (item 3) to 0.58 (item 6).
  • Inter-rater reliability was an intra-class correlation coefficient (average measure) of 0.77 [95% CI 0.57, 0.88]; raters often disagreed on items 1 and 4.
  • Average SANRA scores were modestly associated with editorial acceptance (mean 6.6, SD 1.9, n = 17) versus rejection (mean 5.1, SD 2.1, n = 13), point biserial correlation 0.37 (t = 2.09, df 28; two-sided p = 0.046).

Methods

  • SANRA was developed between 2010 and 2017 by three experienced editors at Deutsches Ärzteblatt; the revised six-item version deleted an item on writing and accessibility due to poor inter-rater reliability.
  • The revised scale was tested on 30 consecutive non-systematic review manuscripts submitted to Deutsches Ärzteblatt in 2015; three editor-raters scored them independently and blinded to each other’s ratings.
  • Analyses included descriptive statistics, Cronbach’s alpha, corrected item-total correlations, average-measure intra-class correlation using a two-way random effects model (consistency definition), and a post hoc point biserial correlation between SANRA sum scores and acceptance/rejection.

Limitations

  • The paper states the main limitation is sample size: 30 manuscripts was a compromise between representativeness, power, and feasibility, and took about 6 months to sample.
  • The authors state that the scale’s authors were also the test-raters, so inter-rater reliability may be lower in groups less familiar with SANRA; the study also did not assess test-retest reliability, and results are study-specific.
  • The paper acknowledges restricted content validity because SANRA is limited to six items and is a critical appraisal tool rather than a reporting guideline; criterion validity evidence is post hoc and weak, and no established cut-offs exist for comparing manuscripts across settings.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s41073-019-0064-8

Cites

12 of 12 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 SANRA - Scale
Fig. 1 SANRA - ScaleCC BY · PubMed Central
Fig. 2 SANRA—explanations and instructions document
Fig. 2 SANRA—explanations and instructions documentCC BY · PubMed Central

Verification

Claims the paper makes

3 Valid · 0 Not valid · 5 Not checked · 27 Own results and interpretations

14 more claims

Create a free account to see all 35 claims and to verify the 3 references still open.

See all claims — free account
  • Probably not a substantial claim. 11
  • It may state two separate claims as one. 11

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

10 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Show all 13 references

Research works printed but not linked: 3

  1. Higgins JPT, Green S. (eds.). Cochrane Handbook for Systematic Reviews and Interventions. Version 5.1.0. The Cochrane Collaboration 2011, section 1.2.2., www.handbook.cochrane.org , retrieved on Oct 31, 2018.
  2. Baethge C, Mertens S, Goldbeck-Wood S. Development of a quality score for the assessment of non-systematic review articles (SANRA). In: Poster, Seventh International Congress on Peer Review and Biomedical Publication. USA: Chicago, Illinois; 2013.
  3. Baethge C, Goldbeck-Wood S, Mertens S. A scale for the assessment of non-systematic review articles (SANRA). In: Poster, Eighth International Congress on Peer and Biomedical Publication. USA: Chicago, Illinois; 2017.

Cited by

See all 4,334

Metadata

Authors
Baethge, C., Goldbeck-Wood, S., & Mertens, S.
Affiliations
  • University of Cologne (DE)
  • BMJ Group (United Kingdom) (GB)
Corresponding author
Baethge, Christopher
Venue
Research Integrity and Peer Review
Published
2019-03-26
Volume
4
Issue
1
Pages
5-5
ISSN
2058-8615 (ISSN-L)
OpenAlex
W2922577842
Cited by
3,469
References
13
Citations by year
2026: 1823 · 2025: 551 · 2024: 379 · 2023: 309 · 2022: 218 · 2021: 152 · 2020: 32 · 2019: 4
FWCI
20.57
Type
journal article
Language
en
Open access
Licence
  • cc-by
  • creativecommons.org/licenses/by/4.0/ (tdm)
Locations
  • Research Integrity and Peer Review (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, other-oa)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Kölner Universitäts PublikationsServer (Universität zu Köln) (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Library and Information Sciences › Academic Writing and Publishing
  • Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
Keywords
quality assessment scale, manuscript review, inter-rater reliability, level of evidence, scale development

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: SANRA—a scale for the quality assessment of narrative review articles. https://app.cochat.ai/papers/10.1186/s41073-019-0064-8
The paper
Baethge, C., Goldbeck-Wood, S., & Mertens, S. (2019). SANRA—a scale for the quality assessment of narrative review articles. Research Integrity and Peer Review. https://doi.org/10.1186/s41073-019-0064-8

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.