CoChat

Improving the quality and reporting of systematic reviews⋆

Eccleston, Christopher, Moore, R. Andrew, Derry, Sheena, Bell, Rae F., McQuay, Henry · 2010 · European Journal of Pain
Cited by 31

The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.

Abstract

Establishing the evidence for any treatment offered to patients suffering from chronic pain is a primary goal of clinical pain research. Understanding how confident to be about the claims made by individual researchers in individual trials has been greatly enhanced by developments in systematic review and meta-analysis. However, systematic reviews are themselves subject to bias and can sometimes be dangerously wrong (Derry et al., 2006; Gerber et al., 2007). In this issue of the European Journal of Pain Van de Wetering et al. (2010) report a systematic review that falls foul of some of the common pitfalls and problems in the undertaking and reporting of systematic reviews in pain (Moore et al., in press). The evidence synthesis they offer is, unfortunately, flawed. This paper raises the opportunity to consider issues of quality in systematic reviews that are submitted for publication in the European Journal of Pain. Here we discuss the common problems faced by authors in trying to manage the biases inherent in primary studies that can undermine the credibility and utility of any review. Further, we focus on the particular problems posed by reviews of non-pharmacological interventions, and last we address the specific problems of this particular study. There are multiple threats to the credibility of a review, but most can be controlled to some degree by authors in the process of selection of studies for inclusion. Systematic reviews, however well constructed, are undermined by the inclusion of low quality studies. We have now a fairly good understanding of what leads to low quality. Common biases of treatment intervention studies include 'allocation bias', the favouring of certain patients to certain treatments, which can be controlled by the use of a randomization with a concealed allocation procedure. Influencing the outcomes of those allocated to each/any treatment can be minimized by blinding patients, therapists, and investigators to the allocation. Size is important (Moore et al., 1998). Ensuring that each trial or review has sufficient numbers of patients and trial events to ensure power is essential. When the number of events is small, chance has a large effect (Flather et al., 1997). Simulation studies in pain converge on a safe event n of 200 (Counsell et al., 1994). We have recently argued that the inclusion of trials with fewer than 50 patients in each arm carries a high risk of bias (Moore et al., 2010). Perhaps most critical to the credibility of a review is careful consideration of the validity of the question posed, and the assumptions made in data synthesis. There are a number of threats to validity when combining pain trials. . First, a lack of common definition of pain condition can significantly undermine statistical methods of data synthesis if different conditions are combined inappropriately. Similarly, the uncritical combination of treatments considered a class, or similar, can introduce significant uncontrollable heterogeneity. . Second, choice of relevant endpoints in pain trials has become critical. Recent IMMPACT criteria have stressed the importance of inclusion of functional, global, and psychosocial outcomes in addition to pain outcomes, especially for chronic pain (Dworkin et al., 2005). For pain outcomes, there is now a growing consensus that the overemphasis on average data in pain trials is likely to be misleading (Dougados et al., 2007). ACTINPAIN and PaPaS now concur with both IMMPACT (www.immpact.org) and OMERACT (www.intermed.med.uottawa.caresearchomeract) that the report of change in pain over baseline is clinically relevant, with 30% improvement being considered moderate benefit, and 50% improvement considered substantial benefit. Although patients would, we recognize, like more (O'Brien et al., 2010). . A third threat to the validity of any review relates to how those patients who report adverse effects of treatment, and/or those who withdraw from a study, are statistically managed. Adverse events should be reported and summarized across trials, and where possible a number needed to harm (NNH) should be reported (Moore et al., 2010). The safest method of imputing missing data from patients who do not complete pain trials (for example, due to lack of efficacy or intolerable adverse events) is not currently known, although carrying the baseline observation forward is likely to be more conservative than the more common practice of carrying forward the last observation (Beunckens et al., 2005). Trials of surgery, physical, or psychological therapy share features that should be taken into consideration for evidence synthesis. Most critical is the lack of separation between the manufacturing of the treatment and its delivery. Typically, the treatment is manufactured in the moment. The operator or therapist must adapt the technique (within certain boundaries) to meet the contingencies of the environment, including patient variables. Blinding operators to treatment allocati

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.ejpain.2010.05.015

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Cited by

See all 31

Metadata

Authors
Eccleston, C., Moore, R. A., Derry, S., Bell, R. F., & McQuay, H.
Venue
European Journal of Pain
Publisher
Wiley
Published
2010-08
Volume
14
Issue
7
Pages
667-669
ISSN
  • 1090-3801 (ISSN-L)
  • 1532-2149
OpenAlex
W2092257260
Cited by
31
References
5
FWCI
1.54
Type
journal article
Language
en
Open access
closed

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.