Effect Size Estimation in Neuroimaging

Abstract

A central goal of translational neuroimaging is to establish robust links between brain measures and clinical outcomes.Success hinges on the development of brain biomarkers with large effect sizes.With large enough effects, a measure may be diagnostic of outcomes at the individual patient level.Surprisingly, however, standard brain-mapping analyses are not designed to estimate or optimize the effect sizes of brain-outcome relationships, and estimates are often biased.Here, we review these issues and how to estimate effect sizes in neuroimaging research.Effect size is a unit-free description of the strength of an effect, independent of sample size.Examples include Cohen d, Pearson r, and number needed to treat.1,2 For a given sample size (N), these can be converted to a t or z score (eg, Cohen d is t/[N] 1/2 ).But t, z, F, and P values are sample size dependent and relate to the presence of an effect (statistical significance), not its magnitude.By contrast, effect size describes a finding's practical significance, which determines its clinical importance.This is an important distinction because small effects can reach statistical significance given a large enough sample, even if they are unlikely to be of practical importance or replicable across diverse samples.3 Traditional neuroimaging studies are not designed to estimate effect sizes.A typical analysis tests for effects at each of 50 000 to 350 000 brain voxels.Post hoc effect sizes are selectively reported for a small subset of significant voxels.This practice creates bias, making effect size estimates larger than their true values.4 It is like a mediocre golfer who plays 5000 holes over the course of his career but only reports his 10 best holes.Bias is introduced because the best performance, selected post hoc, is not representative of expected performance.The Figure shows a simulation in which the true effect size in a set of voxels is d = 0.5.Once noise is added and a statistical test (t test) is conducted across 30 individuals, all significant voxels have an estimated effect size greater than the true effect.Why does this occur?Voxels tend to be significant if they show a true effect and have noise that favors the hypothesis.Correcting for multiple comparisons reduces false positives but actually increases this optimistic bias.6 As statistical thresholds become more stringent, an increasingly small subset of tests with favorable noise will reach significance, making the estimated post hoc effect size grow.In sum, conducting a large number of tests inher-