Comprehensive evaluations of the adverse effects of drugs: importance of appropriate study selection and data sources

Abstract

While systematic reviews and meta-analyses are at the top of the evidence hierarchy, most of the methodology has focused on assessing treatment benefit. Hence, we propose a structured framework for the initial steps of searching and identifying relevant data sources so that adverse effects can be evaluated in a comprehensive, unbiased manner. The unique methodological challenges stem from the difficulties of addressing diverse outcomes encompassing common, mild symptoms to rare, fatal events. Retrieval of the most appropriate studies should be specifically tailored to fit the nature of the adverse effects, according to the primary objective and study question. In our framework, the structure of the review takes different forms depending on whether the main aim is on scoping/hypothesis generation, or evaluating statistically the magnitude of risk (hypothesis testing), or clarifying characteristics and risk factors of the adverse effect. The wide range of data sources covering adverse effects all have distinct strengths and limitations, and selection of appropriate sources depends on characteristics of the adverse effect (e.g. background incidence and effect size of the drug, clinical presentation, time of onset after drug exposure). Reviewers need to retrieve particular study designs that are most likely to yield robust data on the adverse effects of interest, rather than rely on studies that cannot reliably detect adverse effects, and may yield ‘false negatives’. Type II errors (a particular problem when evaluating rare adverse effects) can lull us into a false sense of security (e.g. wrongly concluding that there was no significant difference in harm between drug and control, with the drug erroneously judged as safe). Given the rapid rate at which methodological improvements occur, this proposed framework is by no means definitive, but aims to stimulate further debate and discussion amongst the pharmacoepidemiological and systematic review communities to reach a common consensus on the best methods.