Selective outcome reporting and the effectiveness of psychotherapies for depression

Abstract

Only 40% of trials of psychotherapies for depression publish­ed between 2015 and 2018 were prospectively registered, and discrepancies between publications and protocols were noted for 76% of registered trials1. It is often assumed that such divergences are the result of intentionally favoring statistically significant findings ("selective reporting"). However, discrepancies could be due to other reasons, such as justified protocol amendments, logistic difficulties or carelessness. A survey of trials published in high-impact clinical psychology journals over four years2 identified 27 prospectively registered trials, of which only 13 with a clearly specified primary outcome­ measure and time of assessment. Among these 13 trials, four contained protocol deviations favoring significant findings (for two others this was impossible to adjudicate). However, it is difficult to reliably estimate the prevalence and impact of selective reporting from investigations of such small cohorts of trials. Therefore, we examined differences in effectiveness associated with selective reporting across a complete cohort of prospectively registered trials of psychotherapies for depression. We conducted a pre-registered survey (PROSPERO: CRD42019136130) of all randomized trials comparing psychological interventions to control conditions for adult depression which started enrollment after July 1, 2005, when journal registration mandates became widespread3. We selected trials from a regularly updated meta-analysis of psychotherapies for depression (https://osf.io/825c6/), using the most recent update (January 1, 2020). We identified matching registrations from the publication, key word searches in public registries, or, failing these, by contact with investigators. Registration was considered prospective if it occurred within one month of enrollment start. For prospectively registered trials with a pre-specified outcome measure and assessment time point, we examined changes in primary depression outcomes between registries and publications. Potential discrepancies included4: a) omission of registered primary outcome (non-­reporting); b) addition of new, not registered, primary outcome; c) downgrading of registered primary outcome to secondary; d) upgrading of secondary registered outcome to primary; e) assessment time point changes; f) analysis method changes. Selective reporting was adjudicated for a) or b), and, for other discrepancies, on the basis of the judgement of two independent researchers. Effect sizes were computed as standardized mean differences (SMDs) between intervention and control for primary depression outcomes at post-treatment or the time point specified as primary, using data from publications. For event data (e.g., response, remission), we computed odds ratios and converted them into SMDs5. We pooled effect sizes separately for trials with and without selective reporting, using robust variance estimation with weights from a random effects model, small sample adjustment and an assumed correlation between all pairs of observed effects sizes of 0.86. Analyses were run in Stata/SE 16.1. We found that, out of 353 randomized controlled trials in the cohort, 185 commenced enrollment after July 2005. Of these, 142 (77%, 95% CI: 70%-83%) were registered. Seventy-five trials (40%, 95% CI: 33%-48%) were registered prospectively, 11 of which (15%, 95% CI: 8%-25%) without specifying outcome measures or assessment time points. Fifty-one trials (68%, 95% CI: 56%-78%) were rated as free from selective reporting. Discrepancies between registries and reports were identified for 19/75 (25%, 95% CI: 16%-37%) trials, of which 13 (17%, 95% CI: 10%-28%) were judged as involving selective reporting. For six trials with an omitted registered primary outcome, we queried primary investigators and received four replies, all explaining that the outcome measure had been dropped out before starting data collection. The summary effect size was –0.81 (95% CI: –1.25 to –0.38, tau2=0.22) for trials with selective reporting, and –0.54 (95% CI: –0.65 to –0.43, tau2=0.10) for trials without. When analyses were limited to outcomes registered as primary, the effect size in trials with selective reporting was slightly reduced to –0.75 (95% CI: –1.21 to –0.29). Conversely, excluding the six trials that omitted a registered primary outcome led to a considerably reduced effect size for trials with selective reporting (SMD=–0.51, 95% CI: –0.83 to –0.19), closely resembling that of trials without selective reporting. Similarly, excluding the four trials with an added non-registered primary outcome led to a reduced estimate (SMD=–0.62, 95% CI: –1.00 to –0.24) in trials with selective reporting. Finally, analyses restricted to self-report and unblinded measures showed a substantially increased effect size for trials with selective reporting (SMD=–1.02, 95% CI: –1.66 to –0.38), but minimal changes in the effect size for trials without selective reporting (SMD