Behavior, sensitivity, and power of activation likelihood estimation characterized by massive empirical simulation

“Moreover, it may be discussed whether indeed something like “the” true location of an effect exists, given recent accounts of distributed coding and representation (Bzdok et al., 2015; Haxby, 2012; Kriegeskorte, 2009; Rissman and Wagner, 2012).”

“In turn, a higher number of significant voxels at the target location allow for more false positive voxels when the false positive rate is controlled (Genovese et al., 2002).”

“…topics from cognitive (Chase et al., 2015; Molenberghs et al., 2012), affective (Kohn et al., 2014; Lamm et al., 2011) and motor (King et al., 2014; Yuan and Brown, 2015) neuroscience to the neurobiology of neurological (Herz et al., 2014; Rehme et al., 2012) and psychiatric (Bludau et al., 2015;…”

“Particularly ALE has been applied to a wide range of different topics from cognitive (Chase et al., 2015; Molenberghs et al., 2012), affective (Kohn et al., 2014; Lamm et al., 2011) and motor (King et al., 2014; Yuan and Brown, 2015) neuroscience to the neurobiology of neurological (Herz et al.,…”

“More specifically, this is quite likely to happen if one experiment is based on a substantially larger N than the remaining ones and hence features a tighter Gaussian leading to higher voxel-wise probabilities around the reported foci (Eickhoff et al., 2009b).”

“Empirically, it has repeatedly been noted that significant effects may be largely driven by even a single experiment if the total number of experiments is relatively low (Bzdok et al., 2011; Raemaekers et al., 2007; Turkeltaub et al., 2012; Wager et al., 2007).”

“Such structured integration is enabled by community-wide standards of spatial normalization (Amunts et al., 2014; Evans et al., 2012; Mazziotta et al., 2001) and the convention of reporting peak locations in stereotaxic reference spaces.”

“a) Family-wise error correction on the voxel level controls the chance of observing a given Z-value if foci were randomly distributed (Eickhoff et al., 2012).”

“, 2012) and psychiatric (Bludau et al., 2015; Chan et al., 2011; Goodkind et al., 2015) disorders.”

“The more worrisome pattern, however, is found for FDR correction (Laird et al., 2005).”

“Searches for studies investigating major brain disorders such as schizophrenia, depression, dementia, stroke or Parkinson’s disease likewise revealed many hundred clinical neuroimaging papers.”

“This is consistent behavior in traditional brain imaging where cluster-level inference typically exhibits superior power to voxel-level inference (Friston et al., 1996), though this difference must be considered in light of cluster-inference's reduced spatial specificity relative to voxel-level inference.”

“…contribution of individual studies Empirically, it has repeatedly been noted that significant effects may be largely driven by even a single experiment if the total number of experiments is relatively low (Bzdok et al., 2011; Raemaekers et al., 2007; Turkeltaub et al., 2012; Wager et al., 2007).”

“Moreover, it may be discussed whether indeed something like “the” true location of an effect exists, given recent accounts of distributed coding and representation (Bzdok et al., 2015; Haxby, 2012; Kriegeskorte, 2009; Rissman and Wagner, 2012).”

“The first pool contained 15 hand-coded datasets (Bzdok et al., 2012; Cieslik et al., 2015; Hardwick et al., 2013; Kohn et al., 2014; Rottschy et al., 2012).”

“Rather, multi-level kernel density analysis (MKDA (Kober and Wager, 2010; Nee et al., 2007; Wager et al., 2009)) and signed difference map analysis (SDM (Palaniyappan et al., 2012; Radua and Mataix-Cols, 2012; Radua et al., 2010)) have likewise been frequently used to investigate convergence across…”

“…meta-analyses, i.e., studies integrating previous neuroimaging findings on a particular type of mental process or task, there is a growing interest in using meta-analytic methods for the assessment of task-based interaction patterns (Eickhoff et al., 2010; Fox et al., 2014; Laird et al., 2013).”

“The first pool contained 15 hand-coded datasets (Bzdok et al., 2012; Cieslik et al., 2015; Hardwick et al., 2013; Kohn et al., 2014; Rottschy et al., 2012).”

“The first pool contained 15 hand-coded datasets (Bzdok et al., 2012; Cieslik et al., 2015; Hardwick et al., 2013; Kohn et al., 2014; Rottschy et al., 2012).”

“Furthermore, the substantial analytic flexibility of neuroimaging pipelines also leads to low reliability and reproducibility of neuroimaging findings (Carp, 2012b; Glatard et al., 2015; Wager et al., 2009).”

“As previously pointed out, voxel-wise FDR correction, the type considered here and almost exclusively used in ALE analysis is not appropriate for inference on topological features such as regions of significant convergence of a smooth dataset (Chumbley and Friston, 2009).”

“Empirically, it has repeatedly been noted that significant effects may be largely driven by even a single experiment if the total number of experiments is relatively low (Bzdok et al., 2011; Raemaekers et al., 2007; Turkeltaub et al., 2012; Wager et al., 2007).”

“…contribution of individual studies Empirically, it has repeatedly been noted that significant effects may be largely driven by even a single experiment if the total number of experiments is relatively low (Bzdok et al., 2011; Raemaekers et al., 2007; Turkeltaub et al., 2012; Wager et al., 2007).”

“They provide robust insights into the location of psychological and pathological effects (Kober and Wager, 2010; Nickl-Jockschat et al., 2015; Radua and Mataix-Cols, 2012; Schilbach et al., 2012; Yarkoni et al., 2011).”

“Furthermore, the substantial analytic flexibility of neuroimaging pipelines also leads to low reliability and reproducibility of neuroimaging findings (Carp, 2012b; Glatard et al., 2015; Wager et al., 2009).”

“Moreover, it may be discussed whether indeed something like “the” true location of an effect exists, given recent accounts of distributed coding and representation (Bzdok et al., 2015; Haxby, 2012; Kriegeskorte, 2009; Rissman and Wagner, 2012).”

“Moreover, it may be discussed whether indeed something like “the” true location of an effect exists, given recent accounts of distributed coding and representation (Bzdok et al., 2015; Haxby, 2012; Kriegeskorte, 2009; Rissman and Wagner, 2012).”

“…meta-analyses, i.e., studies integrating previous neuroimaging findings on a particular type of mental process or task, there is a growing interest in using meta-analytic methods for the assessment of task-based interaction patterns (Eickhoff et al., 2010; Fox et al., 2014; Laird et al., 2013).”

“…functional imaging using Positron-Emission Tomography (PET) and in particular functional Magnetic Resonance Imaging (fMRI) have provided ample information about the location of cognitive, sensory and motor processes in the human brain (Bandettini, 2012; Poldrack, 2012; Rosen and Savoy, 2012).”

“It may hence come as no surprise that brain regions reported to be associated with a given psychological process may be found almost anywhere in the brain (Cieslik et al., 2015; Nee et al., 2007).”

“…et al., 2012), affective (Kohn et al., 2014; Lamm et al., 2011) and motor (King et al., 2014; Yuan and Brown, 2015) neuroscience to the neurobiology of neurological (Herz et al., 2014; Rehme et al., 2012) and psychiatric (Bludau et al., 2015; Chan et al., 2011; Goodkind et al., 2015) disorders.”

“The fundamental idea behind these types of location-based meta-analyses is to assess which regions are co-active above chance given activation in a predefined region of interest as a marker of functional connectivity between them (Robinson et al., 2010; Rottschy et al., 2013).”

“For more than two decades, functional imaging using PositronEmission Tomography (PET) and in particular functional Magnetic Resonance Imaging (fMRI) have provided ample information about the location of cognitive, sensory and motor processes in the human brain (Bandettini, 2012; Poldrack, 2012; Rosen and Savoy, 2012).”

“…analysis (MKDA (Kober and Wager, 2010; Nee et al., 2007; Wager et al., 2009)) and signed difference map analysis (SDM (Palaniyappan et al., 2012; Radua and Mataix-Cols, 2012; Radua et al., 2010)) have likewise been frequently used to investigate convergence across functional and structural…”

“Rather, multi-level kernel density analysis (MKDA (Kober and Wager, 2010; Nee et al., 2007; Wager et al., 2009)) and signed difference map analysis (SDM (Palaniyappan et al., 2012; Radua and Mataix-Cols, 2012; Radua et al., 2010)) have likewise been frequently used to investigate convergence across…”

“They provide robust insights into the location of psychological and pathological effects (Kober and Wager, 2010; Nickl-Jockschat et al., 2015; Radua and Mataix-Cols, 2012; Schilbach et al., 2012; Yarkoni et al., 2011).”

“Particularly ALE has been applied to a wide range of different topics from cognitive (Chase et al., 2015; Molenberghs et al., 2012), affective (Kohn et al.”

“There is no question that results obtained throughout all realizations of the jackknife procedure must be considered robustly present in the dataset and not dependent on any one experiment (Amanzio et al., 2013; Palaniyappan et al., 2012).”

“The notion of effect-size, however, is somewhat tricky in ALE meta-analyses, as in contrast to classical behavioral meta-analyses neuroimaging meta-analyses assess spatial convergence (Eickhoff et al., 2009b; Kober and Wager, 2010; Radua and Mataix-Cols, 2012).”

“Empirically, it has repeatedly been noted that significant effects may be largely driven by even a single experiment if the total number of experiments is relatively low (Bzdok et al., 2011; Raemaekers et al., 2007; Turkeltaub et al., 2012; Wager et al., 2007).”

“, 2014; Yuan and Brown, 2015) neuroscience to the neurobiology of neurological (Herz et al., 2014; Rehme et al., 2012) and psychiatric (Bludau et al.”

“…a cluster-extent threshold that controls the chance of observing a cluster of that size if foci were randomly distributed (Eickhoff et al., 2012; Woo et al., 2014). c) Correction for multiple comparisons using the false-discovery rate (Laird et al., 2005; Wager et al., 2007) has also been…”

“In fact, ALE is in stark contrast to behavioral “effect-size” meta-analyses by not considering the size of the reported effects (Cheung et al., 2012; Jones, 1995) but rather their location and associated spatial uncertainty.”

“…meta-analyses, i.e., studies integrating previous neuroimaging findings on a particular type of mental process or task, there is a growing interest in using meta-analytic methods for the assessment of task-based interaction patterns (Eickhoff et al., 2010; Fox et al., 2014; Laird et al., 2013).”

“…has been applied to a wide range of different topics from cognitive (Chase et al., 2015; Molenberghs et al., 2012), affective (Kohn et al., 2014; Lamm et al., 2011) and motor (King et al., 2014; Yuan and Brown, 2015) neuroscience to the neurobiology of neurological (Herz et al., 2014; Rehme et…”

“Particularly ALE has been applied to a wide range of different topics from cognitive (Chase et al., 2015; Molenberghs et al., 2012), affective (Kohn et al.”

“…functional imaging using Positron-Emission Tomography (PET) and in particular functional Magnetic Resonance Imaging (fMRI) have provided ample information about the location of cognitive, sensory and motor processes in the human brain (Bandettini, 2012; Poldrack, 2012; Rosen and Savoy, 2012).”

“Furthermore, the substantial analytic flexibility of neuroimaging pipelines also leads to low reliability and reproducibility of neuroimaging findings (Carp, 2012b; Glatard et al., 2015; Wager et al., 2009).”

“…et al., 2012), affective (Kohn et al., 2014; Lamm et al., 2011) and motor (King et al., 2014; Yuan and Brown, 2015) neuroscience to the neurobiology of neurological (Herz et al., 2014; Rehme et al., 2012) and psychiatric (Bludau et al., 2015; Chan et al., 2011; Goodkind et al., 2015) disorders.”

“…states requires operationalization of the experiment, involving choices on the nature of psychological task, the displayed stimuli, the timing and arrangement of individual trials, their modeling in the statistical analysis and the assessed contrast (Carp, 2012a; Rottschy et al., 2012).”

“Some brain regions like the anterior insula (Kurth et al., 2010; Yarkoni et al., 2011) and the posteriormedial frontal cortex (Muller et al., 2015) are more often activated than, for instance, the lateral temporal cortex.”

“Rather, multi-level kernel density analysis (MKDA (Kober and Wager, 2010; Nee et al., 2007; Wager et al., 2009)) and signed difference map analysis (SDM (Palaniyappan et al., 2012; Radua and Mataix-Cols, 2012; Radua et al., 2010)) have likewise been frequently used to investigate convergence across…”

“Derfuss and Mar estimated that until 2007 almost 8,000 neuroimaging studies had been published (Derrfuss and Mar, 2009).”

“In fact, ALE is in stark contrast to behavioral “effect-size” meta-analyses by not considering the size of the reported effects (Cheung et al., 2012; Jones, 1995) but rather their location and associated spatial uncertainty.”

“They provide robust insights into the location of psychological and pathological effects (Kober and Wager, 2010; Nickl-Jockschat et al., 2015; Radua and Mataix-Cols, 2012; Schilbach et al., 2012; Yarkoni et al., 2011).”

“Additionally, the measured signals are indirect hemodynamic proxies rather than direct measures of neuronal activity (Logothetis and Wandell, 2004).”

“In this paper, we addressed two pressing questions related to ALE meta-analysis: i) Which thresholding method is most appropriate to perform statistical inference? ii) Which sample size, i.e., number of experiments, is needed to perform robust meta-analyses?”

“Meta-Analytic Co-Activation Modeling (MACM) thus provides a complementary aspect of long-range integration relative to measures such as resting-state functional connectivity or structural covariance (Clos et al., 2014; Hardwick et al., 2015).”

“…topics from cognitive (Chase et al., 2015; Molenberghs et al., 2012), affective (Kohn et al., 2014; Lamm et al., 2011) and motor (King et al., 2014; Yuan and Brown, 2015) neuroscience to the neurobiology of neurological (Herz et al., 2014; Rehme et al., 2012) and psychiatric (Bludau et al., 2015;…”

“The first pool contained 15 hand-coded datasets (Bzdok et al., 2012; Cieslik et al., 2015; Hardwick et al., 2013; Kohn et al., 2014; Rottschy et al., 2012).”

“, 2011) and motor (King et al., 2014; Yuan and Brown, 2015) neuroscience to the neurobiology of neurological (Herz et al.”

“They provide robust insights into the location of psychological and pathological effects (Kober and Wager, 2010; Nickl-Jockschat et al., 2015; Radua and Mataix-Cols, 2012; Schilbach et al., 2012; Yarkoni et al., 2011).”

“Domain-specific ALE (as well as domain-by-disorder analyses) was performed for contrasts with at least 20 experiments (31), and these are reported in the online data supplement (see Figures S12 and S13 and Tables S19 and S20).”

“Moreover, the limited number of typical rumination/distraction studies provides equivocal statistical power for meta-analysis (Eickhoff et al., 2016).”

“The SDM method is based on the positive features from existing peak probability methods for meta-analysis, such as activation likelihood estimation (ALE, Eickhoff et al., 2016) or multilevel kernel density analysis (MKDA, Wager, Lindquist, & Kaplan, 2007).”

“Because of the small number of studies, we combined loss anticipation and feedback (88), and this may have diluted effects.”

“(20), the minimum number of experiments required for robust ALE analysis is 17.”

“, scanner make and model, scanning parameters), and processing choices [13,20,61].”

“A false-discovery rate (FDR) threshold, which has often been used in previous meta-analyses, was not used as this method is highly susceptible to false-positives (Chumbley & Friston, 2009; Eickhoff et al., 2012, 2016).”

“Subparcellation of the dorsomedial prefrontal cortex has further specified that the observed cluster of convergence comprises two smaller contiguous regions, with the left strongly connected to the fronto-parietal network and the right strongly connected to the salience network (23, 43) (see Figure S21 in the online s…”

“The cluster-wise family-wise error (FWE) correction was employed to correct for multiple comparisons at a significance threshold of p .05 (Eickhoff et al., 2016).”

“Furthermore, we probed for convergence among task-fMRI studies when focusing only on specific cognitive domains and found a sufficient number of experiments (> 17) (Eickhoff et al., 2016) for inhibitory control and attention, but also performed a joint analyses on the remaining cognitive tasks (memory, timing, reasoni…”

“…of different study designs and limited sample size, no quantitative meta-analysis was performed on the reviewed studies, which did not meet the requirements ensuring robust results from an Activation Likelihood Estimation coordinate-based meta-analysis (Eickhoff et al., 2016; Muller et al., 2018).”

“Finally, as recommended by a recent simulation study [Eickhoff et al., 2016], the resulting P-values were thresholded at P 0.05, cluster-level family-wise error (cFWE) r 5893 r corrected (cluster-forming threshold at voxel-level P 0.001) and transformed into Z-scores for display.”

“Approximately 17-20 experimental datasets are needed for ALE to be adequately powered to detect small effects, and ensure results are not driven by single experiments (47).”

“this significance testing procedure was found beneficial for quantitative meta-analysis experiments based on the ALE algorithm in a recent systematic evaluation (Eickhoff et al. 2016).”

“…statistical maps were thresholded at p 0.05 using a cluster-level correction for multiple comparisons and a cluster forming threshold at p 0.001 (Eickhoff et al., 2017), rather than false discovery rate that is not appropriate for inference on topological features (Eickhoff et al., 2016).”

“These analyses confirmed that our results meet criteria for robust, unbiased results reflecting “true” convergence [Eickhoff et al., 2016].”

“The authors of the ALE algorithm advise to employ a voxel- or cluster-level family-wise error correction (FWE) [28].”

“The resulting individual coactivation profiles were then compared between voxels to identify clusters of voxels with similar coactivation patterns (Eickhoff et al. 2016b).”

“05), an approach which provides the best compromise between sensitivity and specificity (Eickhoff et al., 2016).”

“Adank (2012b) reports a cluster in this region for language production in a meta-analysis based on 21 studies (we had 15); recent recommendations are for 20 studies to enable adequate power in cluster-based meta-analysis (Eickhoff et al., 2016).”

“…if about one-third of the included experiments show the effect of interest (corresponding to an effect size of 0.33), 17 experiments are sufficient to achieve a desired power of 80% using the cFWE correction, though, in general, at least about 20 experiments are recommended (Eickhoff et al., 2016).”

“The number of studies included in most of these meta-analyses is in line with the recent prescriptions for ALE (Eickhoff et al., 2016; Muller et al., 2018), suggesting a minimum of 17 studies to ensure that results would not be driven by single experiments (see also Wu et al., 2019).”

“It has been shown that voxel-wise FDR correction is not an optimal method and increases the chance of reporting the spurious clusters (Chumbley & Friston, 2009; Eickhoff et al., 2016).”

“…twofold: first, given the large variations in experimental designs and phenotype in view of an already rather small number of included studies, additional ALE meta-analyses tackling these questions would be unfeasible for the sake of statistical reliability and robustness [Eickhoff et al., 2016].”

“These two parameters are pivotal for the implementation of the ALE method and the identification of the effect (Eickhoff et al., 2016).”

“05) and a minimum cluster size of 200 mm3 as previously recommended (Eickhoff et al., 2016).”

“…and linguistic complexity—because those are the most commonly used paradigms and thus have a sufficiently large number of neuroimaging studies to assess patterns of brain activation (Eickhoff, Bzdok, Laird, Kurth, & Fox, 2012; Eickhoff, Laird, Fox, Lancaster, & Fox, 2017; Eickhoff et al., 2016).”

“001) (Eickhoff et al., 2016) according with the recent guidelines for coordinate-based meta-analysis (Muller et al.”

“Analyses with a higher number of included experiments can have a higher power (Eickhoff et al., 2016).”

“, 2016) as well as an influx of articles standardizing rigorous procedures for meta-analyses (Eickhoff et al., 2016; Müller et al., 2018) suggest that an update is warranted.”

“Diagnosis-specific ALE analyses were only performed for schizophrenia (number of studies: 31) and MDD (number of studies: 22); the data for all other disorders were insufficient as the minimum number of studies for robust results is currently considered to be 17 [40].”

“…(Muller et al., 2018) that requires all included experiments use the same search coverage (i.e., the same brain coverage, EEG and ERP components, neurocognitive domains) and that there were a sufficient number of studies included in the analysis (i.e., >17–20 experiments; Eickhoff et al., 2016).”

“This average non-linear contribution to the ALE value was computed from the ratio of the ALE values at the location of the cluster with and without each contributing experiment (Eickhoff et al., 2016).”

“A minimum of 17–20 experiments is currently recommended to ensure the robustness of each cluster and sufficient statistical power (Eickhoff et al., 2016, 2017).”

“In line with recent recommendations for the application of cluster-based correction methods (Eickhoff et al., 2016) an initial cluster forming threshold of p < .”

“Finally, it has to be noted that we did not include contrasts for reciprocity decisions in the one-shot TG, distrust decisions in the iterative TG and decisions to propose offers in the one-shot UG, because we could not find enough published studies reporting contrasts for these decisions to reach a reasonable number…”

“For example, based on a recent simulation study (Eickhoff et al., 2016), it is known that the results of a meta-analysis with a low number of papers could be driven by few experiments (as evidenced by our analysis, see Supporting Information Figure S5).”

“…likelihood estimation (ALE; performed with GingerALE software), multi-level kernel density analysis (MKDA) and signed differential mapping (SDM) (Eickhoff et al., 2016; Radua & Mataix-Cols, 2012; Turkel-taub, Eden, Jones, & Zeffiro, 2002; Wager, Lindquist, Nichols, Kober, & Van Snellenberg,…”

“The cluster-wise FWE-corrected threshold was preferred in accordance with a recent simulation study by Eickhoff et al. (2016).”

“In the case of this analysis, we chose a p .05 for cluster-level family-wise error correction (i.e. the cluster-level inference option on GingerALE), which provides the best compromise between sensitivity and specificity (Eickhoff et al., 2016).”

“First, it has been proposed that at least 20 experiments should be included in a meta-analysis [72].”

“However, a minimum amount of 20 experiments is usually thought to be sufficient to resolve this issue (Eickhoff et al., 2016), so that analyses based on large databases, as the one used here, should not be so much biased as to produce invalid results.”

“Of note, there was not enough experiment per task to perform a statistically sound CBMA [27].”

“05 corrected with Family-Wise Error (FWE), and 5000 permutation tests to correct for multiple comparisons [49].”

“In fact, both our ALE analyses of pain in depression and depression in pain likely are low in statistical power [23] and should be interpreted with caution.”

“In addition, the average non-linear contribution of each experiment for each cluster was calculated from the fraction of the ALE values at the cluster with and without the experiment in question [Eickhoff et al., 2016].”

“g-HR = 6 experiments; c-HR = 4; RDSZ = 7; ChSZ = 7; Table S3) was insufficient to achieve robust ALE estimates (Eickhoff et al., 2016).”

“However, we could not investigate those methodological impacts because the number of studies in some subcategories (e.g. PET studies) fell below the minimum number required to control the excessive contribution of individual experiments (Eickhoff et al., 2016).”

“Although there are no available methods for performing robustness analyses with GingerALE, simulations of ALE analyses have been performed to test sensitivity, ensuing cluster sizes, number of incidental clusters, and statistical power (Eickhoff et al., 2016).”

“experiments (Eickhoff et al., 2016), we normalized all MACM maps to a sum of one.”

“…that the more conservative cluster-level FWE correction employed in the present analysis was recommended following massive empirical simulation (Eickhoff et al., 2016), we are confident that the presently reported ROIs reflect the most reliable meta-analytic outcomes available based on the…”