The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
On the Plurality of (Methodological) Worlds: Estimating the Analytic Flexibility of fMRI Experiments
Open accessAbstract
How likely are published findings in the functional neuroimaging literature to be false? According to a recent mathematical model, the potential for false positives increases with the flexibility of analysis methods. Functional MRI (fMRI) experiments can be analyzed using a large number of commonly used tools, with little consensus on how, when, or whether to apply each one. This situation may lead to substantial variability in analysis outcomes. Thus, the present study sought to estimate the flexibility of neuroimaging analysis by submitting a single event-related fMRI experiment to a large number of unique analysis procedures. Ten analysis steps for which multiple strategies appear in the literature were identified, and two to four strategies were enumerated for each step. Considering all possible combinations of these strategies yielded 6,912 unique analysis pipelines. Activation maps from each pipeline were corrected for multiple comparisons using five thresholding approaches, yielding 34,560 significance maps. While some outcomes were relatively consistent across pipelines, others showed substantial methods-related variability in activation strength, location, and extent. Some analysis decisions contributed to this variability more than others, and different decisions were associated with distinct patterns of variability across the brain. Qualitative outcomes also varied with analysis parameters: many contrasts yielded significant activation under some pipelines but not others. Altogether, these results reveal considerable flexibility in the analysis of fMRI experiments. This observation, when combined with mathematical simulations linking analytic flexibility with elevated false positive rates, suggests that false positive results may be more prevalent than expected in the literature. This risk of inflated false positive rates may be mitigated by constraining the flexibility of analytic choices or by abstaining from selective analysis reporting.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040296 · Cited by 23,020
Tzourio-Mazoyer, N., Landeau, B., Papathanassiou, D., Crivello, F., Etard, O., Delcroix, N., Mazoyer, B., & Joliot, M. (2002) · NeuroImage · doi:10.1006/nimg.2001.0978 · Cited by 16,319 · cited 1×
“All ROIs were defined using the Automatic Anatomical Labeling (AAL) atlas (Tzourio-Mazoyer et al., 2002).”
MethodWhy Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Power, J. D., Barnes, K. A., Snyder, A. Z., Schlaggar, B. L., & Petersen, S. E. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2011.10.018 · Cited by 7,429
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Thresholding of statistical maps in functional neuroimaging using the false discovery rate
Genovese, C. R., Lazar, N. A., & Nichols, T. (2002) · NeuroImage · doi:10.1006/nimg.2001.1037 · Cited by 4,923
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 7×
“Furthermore, recent empirical (Ioannidis, 2005a) and mathematical modeling studies (Ioannidis, 2005b) argue that the true incidence of false positives may far exceed the nominal rate in the broader scientific literature.”
BackgroundInfluentialMovement‐Related effects in fMRI time‐series
Friston, K. J., Williams, S., Howard, R., Frackowiak, R. S. J., & Turner, R. (1996) · Magnetic Resonance in Medicine · doi:10.1002/mrm.1910350312 · Cited by 3,774
Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD Initiative.
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2003) · Annals of Internal Medicine · doi:10.7326/0003-4819-138-1-200301070-00010 · Cited by 3,219
Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., & de Vet, H. C. W. (2003) · BMJ · doi:10.1136/bmj.326.7379.41 · Cited by 3,215
Cited by
MEG and EEG data analysis with MNE-Python
Gramfort, A. (2013) · Frontiers in Neuroscience · doi:10.3389/fnins.2013.00267 · Cited by 3,784
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
Cognitive Reappraisal of Emotion: A Meta-Analysis of Human Neuroimaging Studies
Buhle, J. T., Silvers, J. A., Wager, T. D., Lopez, R., Onyemekwu, C., Kober, H., Weber, J., & Ochsner, K. N. (2014) · Cerebral Cortex · doi:10.1093/cercor/bht154 · Cited by 1,896
Scanning the horizon: towards transparent and reproducible neuroimaging research
Poldrack, R. A., Baker, C. I., Durnez, J., Gorgolewski, K. J., Matthews, P. M., Munafò, M. R., Nichols, T. E., Poline, J.-B., Vul, E., & Yarkoni, T. (2017) · Nature Reviews Neuroscience · doi:10.1038/nrn.2016.167 · Cited by 1,410
Variability in the analysis of a single neuroimaging dataset by many teams
(2020) · doi:10.1038/s41586-020-2314-9 · Cited by 1,097
Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them
Flake, J. K., & Fried, E. I. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920952393 · Cited by 969
“Note that inherent flexibility in quantifying raw data is not isolated to surveys and abounds in others approaches to measurement, such as neuroimaging (Carp, 2012).”
BackgroundTen simple rules for neuroimaging meta-analysis
Müller, V. I., Cieslik, E. C., Laird, A. R., Fox, P. T., Radua, J., Mataix-Cols, D., Tench, C. R., Yarkoni, T., Nichols, T. E., Turkeltaub, P. E., et al. (2018) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2017.11.012 · Cited by 805
Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results
Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E., Bahník, Š., Bai, F., Bannard, C., Bonnier, E., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245917747646 · Cited by 784
Eickhoff, S. B., Nichols, T. E., Laird, A. R., Hoffstaedter, F., Amunts, K., Fox, P. T., Bzdok, D., & Eickhoff, C. R. (2016) · NeuroImage · doi:10.1016/j.neuroimage.2016.04.072 · Cited by 683
“Furthermore, the substantial analytic flexibility of neuroimaging pipelines also leads to low reliability and reproducibility of neuroimaging findings (Carp, 2012b; Glatard et al., 2015; Wager et al., 2009).”
Establishment of Best Practices for Evidence for Prediction: A Review
Poldrack, R. A., Huckins, G., & Varoquaux, G. (2020) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2019.3671 · Cited by 738
Best practice guidance for linear mixed-effects models in psychological science
Meteyard, L., & Davies, R. A. I. (2020) · Journal of Memory and Language · doi:10.1016/j.jml.2020.104092 · Cited by 698
Building a Science of Individual Differences from fMRI
Dubois, J., & Adolphs, R. (2016) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2016.03.014 · Cited by 628
The Academic, Economic and Societal Impacts of Open Access: An Evidence-Based Review
Tennant, J. P., Waldner, F., Jacques, D. C., Masuzzo, P., Collister, L. B., & Hartgerink, C. H. J. (2016) · F1000Research · doi:10.12688/f1000research.8460.3 · Cited by 560
Methods for cleaning the BOLD fMRI signal
Caballero-Gaudes, C., & Reynolds, R. C. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2016.12.018 · Cited by 542
Standardizing workflows in imaging transcriptomics with the abagen toolbox
Markello, R. D., Arnatkeviciute, A., Poline, J.-B., Fulcher, B. D., Fornito, A., & Misic, B. (2021) · eLife · doi:10.7554/elife.72129 · Cited by 529
Making big data open: data sharing in neuroimaging
Poldrack, R. A., & Gorgolewski, K. J. (2014) · Nature Neuroscience · doi:10.1038/nn.3818 · Cited by 425
Deficient approaches to human neuroimaging
Stelzer, J., Lohmann, G., Mueller, K., Buschmann, T., & Turner, R. (2014) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2014.00462 · Cited by 412
How Bayes factors change scientific practice
Dienes, Z. (2016) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2015.10.003 · Cited by 401
Chen, X., Lu, B., & Yan, C. (2018) · Human Brain Mapping · doi:10.1002/hbm.23843 · Cited by 340
Toward open sharing of task-based fMRI data: the OpenfMRI project
Poldrack, R. A., Barch, D. M., Mitchell, J. P., Wager, T. D., Wagner, A. D., Devlin, J. T., Cumba, C., Koyejo, O., & Milham, M. P. (2013) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2013.00012 · Cited by 342
Comparing spatial null models for brain maps
Markello, R. D., & Misic, B. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.118052 · Cited by 342
Benchmarking functional connectome-based predictive models for resting-state fMRI
Dadi, K., Rahim, M., Abraham, A., Chyzhyk, D., Milham, M., Thirion, B., & Varoquaux, G. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.02.062 · Cited by 333
Analysis of task-based functional MRI data preprocessed with fMRIPrep
Esteban, O., Ciric, R., Finc, K., Blair, R. W., Markiewicz, C. J., Moodie, C. A., Kent, J. D., Goncalves, M., DuPre, E., Gomez, D. E. P., et al. (2020) · Nature Protocols · doi:10.1038/s41596-020-0327-3 · Cited by 315
Chase, H. W., Boudewyn, M. A., Carter, C. S., & Phillips, M. L. (2020) · Molecular Psychiatry · doi:10.1038/s41380-019-0499-9 · Cited by 286
Progress and challenges in probing the human brain
Poldrack, R. A., & Farah, M. J. (2015) · Nature · doi:10.1038/nature15692 · Cited by 280
A Hitchhiker's Guide to Functional Magnetic Resonance Imaging
Soares, J. M., Magalhães, R., Moreira, P. S., Sousa, A., Ganz, E., Sampaio, A., Alves, V., Marques, P., & Sousa, N. (2016) · Frontiers in Neuroscience · doi:10.3389/fnins.2016.00515 · Cited by 272
Implementation errors in the GingerALE Software: Description and recommendations
Eickhoff, S. B., Laird, A. R., Fox, P. M., Lancaster, J. L., & Fox, P. T. (2017) · Human Brain Mapping · doi:10.1002/hbm.23342 · Cited by 269
Ten common statistical mistakes to watch out for when writing or reviewing a manuscript
Makin, T. R., & Orban de Xivry, J.-J. (2019) · eLife · doi:10.7554/elife.48175 · Cited by 238
Big data, open science and the brain: lessons learned from genomics
Choudhury, S., Fishman, J. R., McGowan, M. L., & Juengst, E. T. (2014) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2014.00239 · Cited by 160
“…a succession of controversies in the neuroimaging community regarding statistical shortcomings of certain findings and limited reproducibility of others, owing in part to the plurality of data analysis methods (Jabbi et al., 2009; Vul et al., 2009; Margulies, 2011; Carp, 2012; Poline et al., 2012).”
BackgroundMeasuring neural representations with fMRI: practices and pitfalls
Davis, T., & Poldrack, R. A. (2013) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.12156 · Cited by 132
Progress toward openness, transparency, and reproducibility in cognitive neuroscience
Gilmore, R. O., Diaz, M. T., Wyble, B. A., & Yarkoni, T. (2017) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.13325 · Cited by 95
Event-related fMRI studies of false memory: an activation likelihood estimation meta-analysis
Kurkela, K. A., & Dennis, N. A. (2016) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2015.12.006 · Cited by 90
MRI predictors of pharmacotherapy response in major depressive disorder
Gerlach, A. R., Karim, H. T., Peciña, M., Ajilore, O., Taylor, W. D., Butters, M. A., & Andreescu, C. (2022) · NeuroImage: Clinical · doi:10.1016/j.nicl.2022.103157 · Cited by 57
McCarthy, P., Benuskova, L., & Franz, E. A. (2014) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2014.00301 · Cited by 48
Reporting details of neuroimaging studies on individual traits prediction: A literature survey
Yeung, A. W. K., More, S., Wu, J., & Eickhoff, S. B. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119275 · Cited by 36
Shafer, A. T., Beason-Held, L., An, Y., Williams, O. A., Huo, Y., Landman, B. A., Caffo, B. S., & Resnick, S. M. (2021) · Neurobiology of Aging · doi:10.1016/j.neurobiolaging.2021.03.013 · Cited by 16
Addressing selective reporting of experiments through predefined exclusion criteria
Neves, K., & Amaral, O. B. (2020) · eLife · doi:10.7554/elife.56626 · Cited by 15
Forks over knives: Predictive inconsistency in criminal justice algorithmic risk assessment tools
Greene, T., Shmueli, G., Fell, J., Lin, C.-F., & Liu, H.-W. (2022) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/rssa.12966 · Cited by 15
Estimating the replicability of Brazilian biomedical science
Initiative, T. B. R., Amaral, O. B., Carneiro, C. F. D., Neves, K., Sampaio, A. P. W., Gomes, B. V., Abreu, M. B. D., Tan, P. B., Mota, G. P. S., Goulart, R. N., et al. (2025) · bioRxiv · doi:10.1101/2025.04.02.645026 · Cited by 13
“Preregistering analysis methods poses fewer difficulties when data is relatively simple, as in our experiments; nevertheless, more data-intensive methods can involve steps of data filtering or complex analytical decisions (e.g. Carp, 2012) that may be equally hard to specify in advance.”
A standardized framework to test event-based experiments
Lepauvre, A., Hirschhorn, R., Bendtz, K., Mudrik, L., & Melloni, L. (2024) · Behavior Research Methods · doi:10.3758/s13428-024-02508-y · Cited by 9
Metadata
- Authors
- Carp, J.
- Venue
- Frontiers in Neuroscience
- Publisher
- Frontiers Media SA
- Published
- 2012
- Volume
- 6
- Pages
- 149-149
- ISSN
- 1662-453X (ISSN-L)
- 1662-4548
- OpenAlex
- W2156451368
- Cited by
- 381
- References
- 56
- FWCI
- 5.22
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.