The DOI matches its Crossref record; 46 of the 48 references it cites were found in a registry or an index.
Assessing robustness against potential publication bias in Activation Likelihood Estimation (ALE) meta-analyses for fMRI
Open accessAbstract
The importance of integrating research findings is incontrovertible and procedures for coordinate-based meta-analysis (CBMA) such as Activation Likelihood Estimation (ALE) have become a popular approach to combine results of fMRI studies when only peaks of activation are reported. As meta-analytical findings help building cumulative knowledge and guide future research, not only the quality of such analyses but also the way conclusions are drawn is extremely important. Like classical meta-analyses, coordinate-based meta-analyses can be subject to different forms of publication bias which may impact results and invalidate findings. The file drawer problem refers to the problem where studies fail to get published because they do not obtain anticipated results (e.g. due to lack of statistical significance). To enable assessing the stability of meta-analytical results and determine their robustness against the potential presence of the file drawer problem, we present an algorithm to determine the number of noise studies that can be added to an existing ALE fMRI meta-analysis before spatial convergence of reported activation peaks over studies in specific regions is no longer statistically significant. While methods to gain insight into the validity and limitations of results exist for other coordinate-based meta-analysis toolboxes, such as Galbraith plots for Multilevel Kernel Density Analysis (MKDA) and funnel plots and egger tests for seed-based d mapping, this procedure is the first to assess robustness against potential publication bias for the ALE algorithm. The method assists in interpreting meta-analytical results with the appropriate caution by looking how stable results remain in the presence of unreported information that may differ systematically from the information that is included. At the same time, the procedure provides further insight into the number of studies that drive the meta-analytical results. We illustrate the procedure through an example and test the effect of several parameters through extensive simulations. Code to generate noise studies is made freely available which enables users to easily use the algorithm when interpreting their results.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“By pooling together numerous studies, ALE also allows for an increased power to detect true effects (Acar et al., 2018).”
From Bridging phenomenology and neural mechanisms of inner speech: ALE meta-analysis on egocentricity and spontaneity in a dual-mechanistic framework (2023) · cites this paper as Acar et al., 2018
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722 · cited 1×
“The Egger regression test estimates funnel plot asymmetry and can therefore be used as an indicator for the presence of publication bias [32].”
MethodDuval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099 · cited 1×
“In classical meta-analysis techniques (based on summarizing effect sizes), this is for instance possible with the trim and fill method [41], that looks at the asymmetry present in a funnel plot and tries to fill the gap.”
MethodThe file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 3×
“The original FSN [21] represents the required number of findings with an effect of zero to change the conclusions of the meta-analysis under the assumption that these findings are still in the “file drawer”.”
BackgroundInfluentialSterne, J. A. C., Sutton, A. J., Ioannidis, J. P. A., Terrin, N., Jones, D. R., Lau, J., Carpenter, J., Rucker, G., Harbord, R. M., Schmid, C. H., et al. (2011) · BMJ · doi:10.1136/bmj.d4002 · Cited by 7,709
Some Methods for Strengthening the Common χ 2 Tests
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001616 · Cited by 3,718
What we can do and what we cannot do with fMRI
Logothetis, N. K. (2008) · Nature · doi:10.1038/nature06976 · Cited by 3,612 · cited 1×
“Functional magnetic resonance imaging (fMRI) continues to contribute greatly to the knowledge about the location of cognitive functions in the brain [1].”
BackgroundPublication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368
Eickhoff, S. B., Laird, A. R., Grefkes, C., Wang, L. E., Zilles, K., & Fox, P. T. (2009) · Human Brain Mapping · doi:10.1002/hbm.20718 · Cited by 1,924 · cited 1×
“We select a meta-analysis on finger tapping [7,38] from the BrainMap database that was used to demonstrate the working of GingerALE [15–17].”
MethodSelf-citationPublication bias in meta-analysis: Prevention, assessment and adjustments
Viechtbauer, W. (2007) · Psychometrika · doi:10.1007/s11336-006-1450-y · Cited by 1,508 · cited 3×
“We illustrate the FSN with an example from the book “Publication Bias in Meta-Analysis” [20].”
BackgroundMethodInfluentialActivation Likelihood Estimation meta-analysis revisited
Eickhoff, S., Bzdok, D., Laird, A., Kurth, F., & Fox, P. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2011.09.017 · Cited by 1,448
Self-citation
Flagged first, then by citations
2 of the 48 references couldn’t be matched to a known paper.
Research works printed but not linked: 3
- J Radua, 2012, Meta-analytic methods for neuroimaging data explained, Biology of Mood & Anxiety Disorders, 2, 1
- SG Costafreda, 2012, Parametric coordinate-based meta-analysis: Valid effect size meta-analysis of studies with differing statistical thresholds, Journal of Neuroscience Methods, 210, 291, 10.1016/j.jneumeth.2012.07.016 · DOI
- A Laird, 2008, Brainmap: The social evolution of a functional neuroimaging database, Neuroinformatics, 65
Cited by
Sorted by citations
Items 101–148 of 148
Dadario, N. B., Tanglay, O., Stafford, J. F., Davis, E. J., Young, I. M., Fonseka, R. D., Briggs, R. G., Yeung, J. T., Teo, C., & Sughrue, M. E. (2023) · Brain and Behavior · doi:10.1002/brb3.2945 · Cited by 7
“While numerous revisions of this method have been previously described, the ALE is subject to possible publica- tion bias like most meta-analyses and may require additional analyses to control for unreported information (Acar et al., 2018; Eickhoff et al., 2012).”
MethodLiloia, D., Rocca, P., Brasso, C., Tanaka, M., Manuello, J., Crocetta, A., Duca, S., Costa, T., & Cauda, F. (2026) · Psychological Medicine · doi:10.1017/s0033291726103511 · Cited by 4
“To address this issue, we assessed the robustness of our findings via the fail-safe technique adapted for the neuroimaging meta-analytic environment (Acar, Seurinck, Eickhoff, & Moerkerke, 2018).”
Presti, S. L., Bonavita, M., Piga, V., Lozito, S., Doricchi, F., & Lasaponara, S. (2025) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2025.106153 · Cited by 5
“Specifically, we examined the correlation between sample size and number of foci and performed a "Fail-Safe N (FSN)" analysis (Acar et al., 2018) for each cluster resulting from the meta-analyses to confirm that our results were not unduly influenced by a small number of studies/contrasts or the…”
InfluentialQi, M., Wang, Y.-Z., Zhang, Y., Feng, Y.-C., & Liu, B. (2025) · Quantitative Imaging in Medicine and Surgery · doi:10.21037/qims-24-916 · Cited by 4
“Fail-safe N (FSN) analysis proposed by Acar et al. was used to assess robustness against potential publication bias.”
Rediscovering the musician’s brain: a systematic review and meta-analysis
Criscuolo, A., Pando-Naude, V., Bonetti, L., Vuust, P., & Brattico, E. (2021) · doi:10.1101/2021.03.12.434473 · Cited by 4
Distinct prefrontal lateralization in placebo and reappraisal mechanisms: An ALE meta-analysis
Monachesi, B., Pisanu, E., Chiffi, D., Rumiati, R., & Grecucci, A. (2025) · NeuroImage · doi:10.1016/j.neuroimage.2025.121459 · Cited by 3
Watanuki, S. (2025) · Frontiers in Neuroscience · doi:10.3389/fnins.2024.1443578 · Cited by 2
Shu, Y., Zhang, Q., Zheng, Z.-L., & Hou, Y. (2025) · Journal of Neural Transplantation and Plasticity · doi:10.1155/np/7885279 · Cited by 3
“Moreover, the ALE meta-analysis, which constructs probability distributions, is more strongly in fl uenced by studies that report a larger number of activation foci [77].”
Muehlhan, M., Spindler, C., Nowaczynski, S., Buchner, C., Fascher, M., & Trautmann, S. (2024) · Journal of Neurochemistry · doi:10.1111/jnc.16104 · Cited by 3
“This upper boundary was determined by the number of studies contributing to an ALE cluster and, according to Acar et al. ( 2018 ), should be set such that at least 5–10% of the original studies must still contribute to the effect of a cluster.”
InfluentialComstock, L. B. (2024) · Brain and Language · doi:10.1016/j.bandl.2023.105377 · Cited by 7
Acar, F., Maumet, C., Heuten, T., Vervoort, M., Bossier, H., Seurinck, R., & Moerkerke, B. (2023) · Neuroinformatics · doi:10.1007/s12021-023-09643-5 · Cited by 2
“(Acar et al., 2018; Eickhoff et al., 2009; Turkeltaub et al., 2012; Eickhoff et al., 2012).”
Self-citationHenderson, J., Mari, T., Hewitt, D., Newton‐Fenner, A., Giesbrecht, T., Marshall, A., Stančák, A., & Fallon, N. B. (2023) · Brain and Behavior · doi:10.1002/brb3.3264 · Cited by 6
“However, a leave-one-out analysis was conducted to assess the sensitivity of results (Supporting Information 2; Acar et al., 2018), which demonstrated that clusters in the right SII and left SI/IPL were stable across all 13 studies.”
Default Mode Network spatial configuration varies across task domainsPreprint
Mancuso, L., Cavuoti-Cabanillas, S., Liloia, D., Manuello, J., Buzi, G., Duca, S., Cauda, F., & Costa, T. (2021) · bioRxiv · doi:10.1101/2021.03.17.435799 · Cited by 3
“To take into account the file-drawer effect, a Fail-Safe procedure was implemented (Acar et al., 2018).”
MethodDetermining the Conclusiveness of Systematic Review Evidence
Ban, J.-W., Madsen, T., Robinson, K. A., & Lund, H. (2026) · Annals of Internal Medicine · doi:10.7326/annals-25-02790 · Cited by 4
Dang, A., Wang, D., Towne, J. M., Seshadri, S. S., Habes, M., & Fox, P. T. (2025) · Human Brain Mapping · doi:10.1002/hbm.70402 · Cited by 2
Hu, Y., Liu, T., Song, S., Qin, K., & Chan, W. (2022) · Journal of Cognitive Psychology · doi:10.1080/20445911.2022.2143788 · Cited by 2
“Although the SDM method was less widely used compared with the activation likelihood estimation (ALE, Acar et al., 2018; Eickhoff et al., 2012; Turkeltaub et al., 2002), SDM-PSI has some improvements over ALE.”
MethodA Meta-Analysis of Factors Related to Job Satisfaction: Focused on Korean Nurses
Yoon, S., & Kim, Y. A. (2022) · Iranian Journal of Public Health · doi:10.18502/ijph.v51i5.9413 · Cited by 2
Mapping the functional neurocognitive correlates of loss aversion: A coordinate-based meta-analysis
Arioli, M., Colletti, M. P. M., Cattaneo, Z., & Canessa, N. (2026) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2026.106937 · Cited by 1
Evaluating the methodological quality of coordinate-based meta-analyses: the qual-CBMA checklist
Raduà, J., Fortea, L., Albajes‐Eizagirre, A., Eickhoff, C. R., Fullana, M. A., Fusar‐Poli, P., Mataix‐Cols, D., Shin, J. I., Solanes, A., Tahmasian, M., et al. (2026) · Frontiers in Neuroimaging · doi:10.3389/fnimg.2026.1753543 · Cited by 1
Self-citationNorrman, H. N., Huang, Y.-T., Westeinde, A. V., van Leeuwen, T. V., Fransson, P., Bölte, S., & Neufeld, J. (2026) · Biological Psychiatry Global Open Science · doi:10.1016/j.bpsgos.2026.100760 · Cited by 1
“As null results cannot be represented in an ALE (35), we have reported them descriptively.”
InfluentialBaten, C., Keller, A. S., Miller, C. H., & Sacchet, M. D. (2026) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2026.106709 · Cited by 1
Demchenko, I., Sousa-Ho, R. L., Baltazar, V. A., Tailor, I., Roy, N., Schweizer, T. A., Kennedy, S. H., Boutet, A., Lozano, A. M., Dunlop, K., et al. (2026) · Translational Psychiatry · doi:10.1038/s41398-026-04006-5 · Cited by 1
“To assess robustness against publication bias, an adapted Fail-Safe N (FSN) was computed for each ALE run [50].”
Lee, Z. L., Li, J. J., Siew, S. K. H., Billaud, C. H. A., & Yu, J. (2025) · GeroScience · doi:10.1007/s11357-025-01850-z · Cited by 1
Driving factors of auditory category learning success
Wang, N., & Feng, G. (2025) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2025.106435 · Cited by 2
Lányi, O., Zahemszky, D., Wenning, A. S., Engh, M., Molnár, Z., Horvath, A., Hegyi, P., & Csukly, G. (2025) · Biological Psychiatry: Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2025.05.017 · Cited by 1
Carraturo, G., Pando-Naude, V., Costa, M., Vuust, P., Bonetti, L., & Brattico, E. (2023) · bioRxiv · doi:10.1101/2023.03.16.532764 · Cited by 1
“To address potential publication bias and assess the robustness of the results, the Fail-Safe N analysis (FSN) was performed (Acar et al., 2018).”
Appetitive and aversive cue reactivities differentiate biotypes of alcohol drinkers
Chen, Y., & Li, C. (2022) · bioRxiv · doi:10.1101/2022.07.31.502197 · Cited by 1
“We also performed a “Fail-Safe N (FSN)” analysis to evaluate potential publication bias (Acar et al., 2018).”
MethodWatershed Brain Regions for Characterizing Brand Equity-Related Mental Processes
Watanuki, S. (2021) · Brain Science · doi:10.3390/brainsci11121619 · Cited by 1
“The ALE method [41], the most popular qualitative meta-analysis method [42], was also adopted.”
MethodNeural Correlates of Moral Judgment in Dilemma Versus Non‐Dilemma Contexts
Fu, J., Song, D., Tolomeo, S., & Yu, R. (2026) · Brain and Behavior · doi:10.1002/brb3.71864
“Moreover, to assess the robustness of the ALE findings against potential publication bias, we conducted a Fail-Safe N (FSN) analysis following Acar et al. ( 2018 ).”
InfluentialBonacina, G., Wagels, L., Habel, U., Wilkin-Krug, L. C. M., O’Leary, A., & Boccadoro, S. (2026) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2026.106998
Zuo, X., Li, G., Wang, Z., Peng, P., Dong, C., & Wang, S. (2026) · iScience · doi:10.1016/j.isci.2026.117610 · Cited by 2
Fascher, M., Krasowski, K., & Muehlhan, M. (2026) · Translational Psychiatry · doi:10.1038/s41398-026-04458-9
“Each model included a single predictor or one two-way interaction, and no model contained more than two predictors, in order to minimise the risk of over fi tting under conditions of limited sample size.”
InfluentialUnveiling the Neural Correlates of Incidental Memory: a Meta-analysis of fMRI Studies
Piero, A. D., Teghil, A., Guariglia, C., & Casagrande, M. (2026) · Neuropsychology Review · doi:10.1007/s11065-026-09708-7
How are grammatical and arithmetic processing related in the brain? An ALE meta-analysis
Viesel-Nordmeyer, N., Ziegler, J. C., & Prado, J. (2026) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2026.106925
Zhou, J.-X., Xu, Y., Li, T., & Yang, X. (2026) · Human Brain Mapping · doi:10.1002/hbm.70594
“This method models activation foci from independent neuroimaging studies as three-dimensional Gaussian probability distributions, enabling probabilistic spatial convergence analysis through iterative kernel-based smoothing and statistical thresholding (Acar et al. 2018).”
McCurry, K. L., Brown, V. M., Bansal, V., Goyal, S., Casas, B. R., & Chiu, P. H. (2026) · Biological Psychiatry · doi:10.1016/j.biopsych.2026.06.027
How visual attention underpins reading: Converging evidence from a meta-analysis of fMRI studies
Gao, Y., Zhu, B., & Zhou, W. (2026) · NeuroImage · doi:10.1016/j.neuroimage.2026.122114
Brain Structural Abnormalities in Addiction Disorders: Transdiagnostic Meta‐Analyses
Banaja, D. M. A., Hernandez, E. T., Franklin, C. G., Fox, P. M., Okafor, C. N., Ginsburg, B. C., Salinas, F. S., Moring, J. C., & Fox, P. T. (2026) · Human Brain Mapping · doi:10.1002/hbm.70551
“Importantly, unlike classical effect-size meta-analyses, ALE is agnostic to reported test statistics; coordinates are modeled as spatial probability distributions weighted by sample size rather than by the statistical significance of individual findings (Acar et al. 2018).”
Lin, J., He, Y., Zhang, S., Mo, L., Kuang, C., & Chen, Y. (2026) · Brain Structure and Function · doi:10.1007/s00429-026-03127-3
“First, the pres-ent ALE algorithm cannot assess effect sizes that existed in the included studies, and this deficit might lead to overes - timating effects on the final conclusion (Acar et al. 2018).”
Functional Neural Architecture of Working Memory in Musicians: An ALE Meta‐Analysis and Review
Wolff, L., Quan, Y.-X., Thompson, W. F., & Baumann, O. (2026) · Wiley Interdisciplinary Reviews: Cognitive Science · doi:10.1002/wcs.70036
“Thirdly, a Fail-Safe N analysis (FSN: Acar et al. 2018) was conducted to assess the influence of unpublished studies on the “file drawer effect,” whereby positive results are overrepresented in published literature.”
InfluentialYu, X.-C., Ran, W., Chen, G., Chen, T., Cui, J., Wang, Y., & Chan, R. C. K. (2026) · NeuroImage · doi:10.1016/j.neuroimage.2026.121948 · Cited by 1
Inhibitory control in problematic usage of the internet: An ALE meta-analysis
Li, X., Yang, J., Turel, O., Zhang, S., & He, Q. (2026) · NeuroImage · doi:10.1016/j.neuroimage.2026.121859
Sawada, Y., Edgar, E. V., Ibrahim, K., Reid, L., Polack, R. G., Bloch, M. H., & Crowley, M. J. (2026) · Cerebral Cortex · doi:10.1093/cercor/bhag031
Veneziani, G., Giraldi, E., Mazza, M., Aceto, P., Terrizzi, E., CASALE, A. D., & Lai, C. (2026) · Developmental Psychobiology · doi:10.1002/dev.70146
Baten, C., Zamora, G., Klassen, A. M., Rackley, J., Nimarko, A. F., Woo, E., Keller, A. S., Hamilton, J. P., Hedges, D. W., Sacchet, M. D., et al. (2026) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2026.01.001 · Cited by 1
“This number is then compared with a minimum FSN (mFSN), which is based on the number of estimated unpublished null findings, to determine whether each cluster is robust to publication bias (32).”
Xue, R., Li, J., & Yang, H. (2025) · Cerebral Cortex · doi:10.1093/cercor/bhae493 · Cited by 1
Masharipov, R., Nikolaev, Y., Korotkov, A., Didur, M., Cherednichenko, D., & Kireev, M. (2021) · bioRxiv · doi:10.1101/2021.06.02.446711
“This publishing bias is also known as the ‘file-drawer problem’ (Rosenthal, 1979; Ioannidis et al., 2014; De Winter and Dodou, 2015; for evidence in fMRI studies, see Jennings and Van Horn, 2012; Acar et al., 2018; David et al., 2018; Samartsidis et al., 2020).”
BackgroundIs migraine associated to brain anatomical alterations? New data and coordinate-based meta-analysis
Masson, R., Demarquay, G., Meunier, D., Lévêque, Y., Hannoun, S., Bidet-Caulet, A., & Caclin, A. (2020) · doi:10.1101/2020.02.18.20024554
Metadata
- Authors
- Acar, F., Seurinck, R., Eickhoff, S. B., & Moerkerke, B.
- Affiliations
- Ghent University (BE)
- Forschungszentrum Jülich (DE)
- Corresponding author
- Acar, Freya
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2018-11-30
- Volume
- 13
- Issue
- 11
- Pages
- e0208177
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC6267999
- OpenAlex
- W2902120147
- Cited by
- 154
- References
- 48
- Citations by year
- 2026: 22 · 2025: 19 · 2024: 18 · 2023: 28 · 2022: 27 · 2021: 19 · 2020: 13 · 2019: 7
- FWCI
- 7.44
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Ghent University Academic Bibliography (Ghent University) (submittedVersion, other-oa)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- JuSER (Forschungszentrum Jülich) (submittedVersion, other-oa)
- Funders
- Fonds Wetenschappelijk Onderzoek (G.0149.14)
- National Institute of Mental Health (R01-MH074457)
- Horizon 2020 (7202070 (HBP SGA1))
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Life Sciences › Neuroscience › Cognitive Neuroscience › Functional Brain Connectivity Studies
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- publication bias, file drawer problem, activation likelihood estimation, coordinate-based meta-analysis, noise study, robustness analysis, spatial convergence
- MeSH terms
- Algorithms, Brain, Brain Mapping, Humans, Magnetic Resonance Imaging, Meta-Analysis as Topic, Likelihood Functions, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
