23. Role of Meta-analysis in Interpreting the Scientific Literature

Abstract

the pRoblem of infoRmal "expeRt" assessment of ReseaRch findingsScientiStS often deal With vast amounts of data, and the ability to summarize this information effectively is a major asset.Therefore, researchers make a considerable effort to acquire the necessary statistical skills to rigorously analyze each empirical data set that they collect.The same thoroughness should occur when writing up work for publication, which ideally requires synthesis of the scientific literature for each question that is answered (i.e.statistical test conducted) to place the results in context.This synthesis is a real challenge.One thing that makes it challenging for ecologists and evolutionary biologists to stay up to date with research findings is that, in so doing, they usually try to place their own results in a much broader context.This means that they often do not confine their frame of reference to studies of the same species, taxon, or ecosystem.There is, not unexpectedly, evidence that those working on a taxon studied by relatively few researchers are more likely to cite studies of other more popular study systems.For example, herpetologists more often cite studies of other classes of vertebrate than those studying mammals or birds cite reptilian studies (Bonnet et al. 2002; see also Taborsky 2009).In ecology and evolutionary biology, one reason for the inclusion of citations following specific research results is to illustrate the extent to which the author's results agree or disagree with the findings of others.Arguably, the most common way researchers assess the "level of agreement" is to consult reviews, including meta-analyses, to reach a general conclusion (e.g., most studies report a positive finding), which they can then claim their own study supports or contradicts.The validity of any assessment of general findings depends on the rigor of the review used to inform this judgment.If the review is a meta-analysis, one can be fairly certain that the confidence intervals for the mean trend provide a reasonable quantitative summary of the available studies.If it is a narrative review, then the potential for a subjective bias on the part of the reviewer, including reliance on "expert opinion," which is surprisingly often flawed or erroneous (Surowiecki 2004), is more worrisome.The general conclusions drawn have a strong bearing on how research findings are presented (i.e., whether we describe results as refuting or supporting a general trend), and therefore have a major impact on the future direction of a researcher's own work and that of their colleagues (e.g., few researchers will ask a question if publications repeatedly state that we already know the answer).