CoChat
Sign Up
Cited by 337Meta-analysis and systematic reviewsPharmaceutical industry and healthcareStatistical Methods in Epidemiology

The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.

Publication Bias: The Elephant in the Review

Open access
Dalton, Jarrod E., Bolen, Shari D., Mascha, Edward J.·2016·Anesthesia & Analgesia
DOI 10.1213/ane.0000000000001596PMC PMC5482177
Open this paper in CoChat — free account

Abstract

Systematic reviews and meta-analyses serve an important role in summarizing the results of multiple investigations, particularly, rigorously designed trials aimed at the estimation of specific treatment effects. It is widely believed that, when properly conducted, results of meta-analyses can be stronger than single investigations because of increased sample sizes and diversity of settings. Proper conduct of these investigations, however, is simultaneously difficult and labor intensive. Clearly, the conclusion reached by a particular meta-analysis or systematic review depends on many methodologic factors, including choice of search criteria, choice of literature sources, study inclusion and exclusion criteria, procedures for abstracting treatment effects, and, importantly, the extent to which things like confounding might differentially affect the results from included studies. One key factor that may affect the conclusions reached by many such reviews is the hidden elephant of publication bias. In this issue of Anesthesia & Analgesia, Hedin et al1 provide an assessment of the extent to which systematic reviews and meta-analyses reported in major anesthesia journals include evaluations of publication bias. Describing publication bias as the tendency to publish “only results that are statistically or clinically significant,” they found that, among 207 systematic reviews meeting inclusion criteria, only 114 (55%) discussed it and 89 (43%) evaluated it. Furthermore, they found that only 68 (33%) of the reviews reported following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (a number that may be artificially low as an estimate of the proportion following reporting guidelines in these studies, given that guidelines other than PRISMA were predominant before 2009), which clearly recommend the assessment of publication bias as a means for avoiding situations in which “[t]he absence of information from some studies may pose a serious threat to the validity of a review.”2 Although the prevalence of publication bias in major anesthesia journals has been described previously,3 a dedicated evaluation of current practice in assessing and reporting publication bias within systematic reviews and meta-analyses in these journals was both lacking and needed. The finding that the prevalence of evaluating publication bias may be as low as 43% is troubling. Because publication bias, however, is difficult to evaluate among reviews of 10 or fewer studies (because of lack of power) and among reviews of nonrandomized studies (because of confounding issues), perhaps a more relevant denominator might be the 25 studies that involved meta-analysis of >10 randomized trials. The prevalence of evaluating publication bias using the denominator of systematic reviews containing meta-analyses would then be much greater. In our experience, the degree to which publication bias may affect results is lower among this subset. Indeed, based on our review of Table 4, it appears to us that a rather small number (say, 2–5) of the included reviews had meaningful differences in pooled effect estimates after accounting for publication bias. Regardless of this point, however, publication bias does exist and can lead to large differences in conclusions when not adequately assessed and addressed. Hedin et al briefly distinguished systematic review and meta-analyses in the title and in the Methods section but subsequently referred to all articles as “systematic reviews” in the Results and Discussion sections. Systematic reviews are a reproducible method for identifying and collating all empirical evidence on a specific topic. These may or may not include meta-analyses, which is a quantitative method for combining individual studies to estimate an overall treatment effect. The distinction is important when evaluating publication bias because systematic reviews by definition do not typically contain an assessment of overall treatment effect. We cannot verify the authors’ claims that the proportion of anesthesia reviews evaluating publication bias is lower than that of other medical fields or that the prevalence of publication bias is greater than that of other medical fields, because their estimates are not restricted to reviews that included meta-analyses. Appreciating the mechanisms by which publication bias and other forms of reporting bias might arise is a prerequisite to minimizing their ultimate impact on our interpretation of the literature and correspondingly needs to be thoroughly understood by the anesthesia research community. Such mechanisms include confirmation bias (selective preference for new results that agree with previous evidence), improper study design (eg, lack of power for detecting meaningful differences, improper specification of the relevant patient population for the intervention of interest), improper hypothesis testing practice (discontinuation of the research/manuscript development proc

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1213/ane.0000000000001596

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 9 references

Cited by

Sorted by citations

Items 301–337 of 337

Related

See all

Metadata

Authors
Dalton, J. E., Bolen, S. D., & Mascha, E. J.
Affiliations
MetroHealth (US)
Corresponding author
  • Dalton, Jarrod E.
  • Mascha, Edward J.
ORCID
Venue
Anesthesia & Analgesia
Publisher
Ovid Technologies (Wolters Kluwer Health)
Published
2016-10
Volume
123
Issue
4
Pages
812-813
ISSN
0003-2999 (ISSN-L)
DOI
10.1213/ane.0000000000001596
PMC
PMC5482177
OpenAlex
W2520224500
Cited by
337
References
9
Citations by year
2026: 35 · 2025: 64 · 2024: 81 · 2023: 47 · 2022: 40 · 2021: 34 · 2020: 16 · 2019: 11 · 2018: 4 · 2017: 2 · 2016: 1
Type
journal article
Language
en
Open access
Locations
  • Anesthesia & Analgesia (publishedVersion)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
Funders
  • National Institutes of Health (KL2TR000440)
  • National Center for Advancing Translational Sciences
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Health Sciences › Nursing › Issues, ethics and legal aspects › Pharmaceutical industry and healthcare
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
Keywords
publication bias, systematic reviews, meta-analysis, reporting bias, PRISMA guidelines, reporting quality, anesthesiology journals, publication practices, confounding, reporting guidelines adherence, research funding conflicts of interest, editorial policy
MeSH terms
Meta-Analysis as Topic, Publication Bias

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review