Publication Bias: The Elephant in the Review

Abstract

Systematic reviews and meta-analyses serve an important role in summarizing the results of multiple investigations, particularly, rigorously designed trials aimed at the estimation of specific treatment effects. It is widely believed that, when properly conducted, results of meta-analyses can be stronger than single investigations because of increased sample sizes and diversity of settings. Proper conduct of these investigations, however, is simultaneously difficult and labor intensive. Clearly, the conclusion reached by a particular meta-analysis or systematic review depends on many methodologic factors, including choice of search criteria, choice of literature sources, study inclusion and exclusion criteria, procedures for abstracting treatment effects, and, importantly, the extent to which things like confounding might differentially affect the results from included studies. One key factor that may affect the conclusions reached by many such reviews is the hidden elephant of publication bias. In this issue of Anesthesia & Analgesia, Hedin et al1 provide an assessment of the extent to which systematic reviews and meta-analyses reported in major anesthesia journals include evaluations of publication bias. Describing publication bias as the tendency to publish “only results that are statistically or clinically significant,” they found that, among 207 systematic reviews meeting inclusion criteria, only 114 (55%) discussed it and 89 (43%) evaluated it. Furthermore, they found that only 68 (33%) of the reviews reported following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (a number that may be artificially low as an estimate of the proportion following reporting guidelines in these studies, given that guidelines other than PRISMA were predominant before 2009), which clearly recommend the assessment of publication bias as a means for avoiding situations in which “[t]he absence of information from some studies may pose a serious threat to the validity of a review.”2 Although the prevalence of publication bias in major anesthesia journals has been described previously,3 a dedicated evaluation of current practice in assessing and reporting publication bias within systematic reviews and meta-analyses in these journals was both lacking and needed. The finding that the prevalence of evaluating publication bias may be as low as 43% is troubling. Because publication bias, however, is difficult to evaluate among reviews of 10 or fewer studies (because of lack of power) and among reviews of nonrandomized studies (because of confounding issues), perhaps a more relevant denominator might be the 25 studies that involved meta-analysis of >10 randomized trials. The prevalence of evaluating publication bias using the denominator of systematic reviews containing meta-analyses would then be much greater. In our experience, the degree to which publication bias may affect results is lower among this subset. Indeed, based on our review of Table 4, it appears to us that a rather small number (say, 2–5) of the included reviews had meaningful differences in pooled effect estimates after accounting for publication bias. Regardless of this point, however, publication bias does exist and can lead to large differences in conclusions when not adequately assessed and addressed. Hedin et al briefly distinguished systematic review and meta-analyses in the title and in the Methods section but subsequently referred to all articles as “systematic reviews” in the Results and Discussion sections. Systematic reviews are a reproducible method for identifying and collating all empirical evidence on a specific topic. These may or may not include meta-analyses, which is a quantitative method for combining individual studies to estimate an overall treatment effect. The distinction is important when evaluating publication bias because systematic reviews by definition do not typically contain an assessment of overall treatment effect. We cannot verify the authors’ claims that the proportion of anesthesia reviews evaluating publication bias is lower than that of other medical fields or that the prevalence of publication bias is greater than that of other medical fields, because their estimates are not restricted to reviews that included meta-analyses. Appreciating the mechanisms by which publication bias and other forms of reporting bias might arise is a prerequisite to minimizing their ultimate impact on our interpretation of the literature and correspondingly needs to be thoroughly understood by the anesthesia research community. Such mechanisms include confirmation bias (selective preference for new results that agree with previous evidence), improper study design (eg, lack of power for detecting meaningful differences, improper specification of the relevant patient population for the intervention of interest), improper hypothesis testing practice (discontinuation of the research/manuscript development proc