Estimating the prevalence of missing experiments in a neuroimaging meta‐analysis
Open accessThe DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
Coordinate‐based meta‐analyses (CBMA) allow researchers to combine the results from multiple functional magnetic resonance imaging experiments with the goal of obtaining results that are more likely to generalize. However, the interpretation of CBMA findings can be impaired by the file drawer problem, a type of publication bias that refers to experiments that are carried out but are not published. Using foci per contrast count data from the BrainMap database, we propose a zero‐truncated modeling approach that allows us to estimate the prevalence of nonsignificant experiments. We validate our method with simulations and real coordinate data generated from the Human Connectome Project. Application of our method to the data from BrainMap provides evidence for the existence of a file drawer effect, with the rate of missing experiments estimated as at least 6 per 100 reported. The R code that we used is available at https://osf.io/ayhfv/ .
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Large-scale automated synthesis of human functional neuroimaging data.
Yarkoni, T., Poldrack, R. A., Nichols, T. E., Van Essen, D. C., & Wager, T. D. (2011) · Nature Methods · doi:10.1038/nmeth.1635 · Cited by 3,686 · cited 1×
“org(5) (Yarkoni et al., 2011), though note Neurosynth does not report foci per contrast but per paper).”
BackgroundSelf-citationGeneralized additive models for location, scale and shape.
Rigby, R. A., & Stasinopoulos, D. M. (2005) · Journal of the Royal Statistical Society Series C: Applied Statistics · doi:10.1111/j.1467-9876.2005.00510.x · Cited by 3,228 · cited 2×
“Summaries of the BrainMap subsamples A-E data for each level of context can be found in A. Parameter estimation is done under the generalized additive models for location scale and shape (GAMLSS) framework of Rigby and Stasinopoulos.21 The fitting is done in R22,§ with the gamlss library.23 Confidence intervals are ob…”
MethodEickhoff, S. B., Laird, A. R., Grefkes, C., Wang, L. E., Zilles, K., & Fox, P. T. (2009) · Human Brain Mapping · doi:10.1002/hbm.20718 · Cited by 1,924 · cited 1×
“(2018) develop a robustness check for the activation likelihood estimation algorithm (Eickhoff et al., 2009), which is based on the fail-safe N (Rosenthal, 1979; Iyengar and Greenhouse, 1988) that is, the minimum number of unpublished studies required to overturn the outcome of meta-analysis.”
Self-citationEmpirical assessment of effect of publication bias on meta-analyses
Sutton, A. J. (2000) · BMJ · doi:10.1136/bmj.320.7249.1574 · Cited by 1,398
ALE meta-analysis: controlling the false discovery rate and performing statistical contrasts
Laird, A. R., Fox, P. M., Price, C. J., Glahn, D. C., Uecker, A. M., Lancaster, J. L., Turkeltaub, P. E., Kochunov, P., & Fox, P. T. (2005) · Human Brain Mapping · doi:10.1002/hbm.20136 · Cited by 868
Self-citationVoxel-wise meta-analysis of grey matter changes in obsessive-compulsive disorder
Radua, J., & Mataix-Cols, D. (2009) · British Journal of Psychiatry · doi:10.1192/bjp.bp.108.055046 · Cited by 863
Meta-analysis, funnel plots and sensitivity analysis
Copas, J. (2000) · Biostatistics · doi:10.1093/biostatistics/1.3.247 · Cited by 503 · cited 1×
“, 1997; Duval and Tweedie, 2000), weight functions (Larose and Dey, 1998; Copas and Jackson, 2004) or sensitivity analyses (Copas and Shi, 2000).”
BackgroundMeta-analysis of functional neuroimaging data: current and future directions
Wager, T. D., Lindquist, M., & Kaplan, L. (2007) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nsm015 · Cited by 434
Hung, Y., Gaillard, S. L., Yarmak, P., & Arsalidou, M. (2018) · Human Brain Mapping · doi:10.1002/hbm.24232 · Cited by 212
Gender differences in working memory networks: a BrainMap meta-analysis
Hill, A. C., Laird, A. R., & Robinson, J. L. (2014) · Biological Psychology · doi:10.1016/j.biopsycho.2014.06.008 · Cited by 182
Self-citation
Cited by
Gray, J. P., Müller, V. I., Eickhoff, S. B., & Fox, P. T. (2020) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2019.19050560 · Cited by 449
Self-citationAcar, F., Seurinck, R., Eickhoff, S. B., & Moerkerke, B. (2018) · PLOS ONE · doi:10.1371/journal.pone.0208177 · Cited by 154
Pratts, J., Pobric, G., & Yao, B. (2023) · NeuroImage · doi:10.1016/j.neuroimage.2023.120399 · Cited by 34
“Given that there is an estimated upper bound of 30 unpublished studies with null findings per 100 published neuroimaging studies investigating language (Samartsidis et al., 2020), we re-analysed the unbalanced pooled dataset ( N = 22) with up to 7 additional null studies (30”
BackgroundLiloia, D., Rocca, P., Brasso, C., Tanaka, M., Manuello, J., Crocetta, A., Duca, S., Costa, T., & Cauda, F. (2026) · Psychological Medicine · doi:10.1017/s0033291726103511 · Cited by 4
“Based on a recent simulation study (Samartsidis et al., 2020), estimating a 6% rate of missing experiments in the BrainMap database, we retested our analyses by introducing an equivalent percentage of noise (i.e. random simulated coordinates of variation) to our datasets of no interest (Figure 1D).”
Metadata
- Authors
- Samartsidis, P., Montagna, S., Laird, A. R., Fox, P. T., Johnson, T. D., & Nichols, T. E.
- Venue
- Research Synthesis Methods
- Publisher
- Wiley
- Published
- 2020-09-27
- Volume
- 11
- Issue
- 6
- Pages
- 866-883
- ISSN
- 1759-2879 (ISSN-L)
- 1759-2887
- OpenAlex
- W3081646839
- Cited by
- 56
- References
- 12
- FWCI
- 4.27
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.