The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
On the Plurality of (Methodological) Worlds: Estimating the Analytic Flexibility of fMRI Experiments
Open accessAbstract
How likely are published findings in the functional neuroimaging literature to be false? According to a recent mathematical model, the potential for false positives increases with the flexibility of analysis methods. Functional MRI (fMRI) experiments can be analyzed using a large number of commonly used tools, with little consensus on how, when, or whether to apply each one. This situation may lead to substantial variability in analysis outcomes. Thus, the present study sought to estimate the flexibility of neuroimaging analysis by submitting a single event-related fMRI experiment to a large number of unique analysis procedures. Ten analysis steps for which multiple strategies appear in the literature were identified, and two to four strategies were enumerated for each step. Considering all possible combinations of these strategies yielded 6,912 unique analysis pipelines. Activation maps from each pipeline were corrected for multiple comparisons using five thresholding approaches, yielding 34,560 significance maps. While some outcomes were relatively consistent across pipelines, others showed substantial methods-related variability in activation strength, location, and extent. Some analysis decisions contributed to this variability more than others, and different decisions were associated with distinct patterns of variability across the brain. Qualitative outcomes also varied with analysis parameters: many contrasts yielded significant activation under some pipelines but not others. Altogether, these results reveal considerable flexibility in the analysis of fMRI experiments. This observation, when combined with mathematical simulations linking analytic flexibility with elevated false positive rates, suggests that false positive results may be more prevalent than expected in the literature. This risk of inflated false positive rates may be mitigated by constraining the flexibility of analytic choices or by abstaining from selective analysis reporting.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040296 · Cited by 23,020
Tzourio-Mazoyer, N., Landeau, B., Papathanassiou, D., Crivello, F., Etard, O., Delcroix, N., Mazoyer, B., & Joliot, M. (2002) · NeuroImage · doi:10.1006/nimg.2001.0978 · Cited by 16,319
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Power, J. D., Barnes, K. A., Snyder, A. Z., Schlaggar, B. L., & Petersen, S. E. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2011.10.018 · Cited by 7,429
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Thresholding of statistical maps in functional neuroimaging using the false discovery rate
Genovese, C. R., Lazar, N. A., & Nichols, T. (2002) · NeuroImage · doi:10.1006/nimg.2001.1037 · Cited by 4,923
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 7×
“Furthermore, recent empirical (Ioannidis, 2005a) and mathematical modeling studies (Ioannidis, 2005b) argue that the true incidence of false positives may far exceed the nominal rate in the broader scientific literature.”
BackgroundInfluentialMovement‐Related effects in fMRI time‐series
Friston, K. J., Williams, S., Howard, R., Frackowiak, R. S. J., & Turner, R. (1996) · Magnetic Resonance in Medicine · doi:10.1002/mrm.1910350312 · Cited by 3,774
Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD Initiative.
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2003) · Annals of Internal Medicine · doi:10.7326/0003-4819-138-1-200301070-00010 · Cited by 3,219
Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., & de Vet, H. C. W. (2003) · BMJ · doi:10.1136/bmj.326.7379.41 · Cited by 3,215
Towards Complete and Accurate Reporting of Studies of Diagnostic Accuracy: The STARD Initiative
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., & de Vet, H. C. W. (2003) · Radiology · doi:10.1148/radiol.2261021292 · Cited by 3,215
Inhibition and the right inferior frontal cortex
Aron, A. R., Robbins, T. W., & Poldrack, R. A. (2004) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2004.02.010 · Cited by 3,197
Temporal Autocorrelation in Univariate Linear Modeling of FMRI Data
Woolrich, M. W., Ripley, B. D., Brady, M., & Smith, S. M. (2001) · NeuroImage · doi:10.1006/nimg.2001.0931 · Cited by 2,852
Moher, D., Schulz, K. F., & Altman, D. G. (2001) · BMC Medical Research Methodology · doi:10.1186/1471-2288-1-2 · Cited by 2,755
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2007) · Preventive Medicine · doi:10.1016/j.ypmed.2007.08.012 · Cited by 2,698
Moher, D. (2001) · JAMA · doi:10.1001/jama.285.15.1987 · Cited by 2,545
Murphy, K., Birn, R. M., Handwerker, D. A., Jones, T. B., & Bandettini, P. A. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2008.09.036 · Cited by 2,387
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
REST: A Toolkit for Resting-State Functional Magnetic Resonance Imaging Data Processing
Song, X.-W., Dong, Z.-Y., Long, X.-Y., Li, S.-F., Zuo, X.-N., Zhu, C.-Z., He, Y., Yan, C.-G., & Zang, Y.-F. (2011) · PLoS ONE · doi:10.1371/journal.pone.0025031 · Cited by 1,872
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845
Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546
Contradicted and Initially Stronger Effects in Highly Cited Clinical Research
Ioannidis, J. P. A. (2005) · JAMA · doi:10.1001/jama.294.2.218 · Cited by 1,465
Moher, D., Schulz, K. F., Altman, D. G., & for the CONSORT Group* (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-134-8-200104170-00011 · Cited by 1,426
Type I and Type II error concerns in fMRI research: re-balancing the scale
Lieberman, M. D., & Cunningham, W. A. (2009) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nsp052 · Cited by 1,342
Controlling the familywise error rate in functional neuroimaging: a comparative review
Nichols, T., & Hayasaka, S. (2003) · Statistical Methods in Medical Research · doi:10.1191/0962280203sm341ra · Cited by 1,218
Aron, A. R., Behrens, T. E., Smith, S., Frank, M. J., & Poldrack, R. A. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.0519-07.2007 · Cited by 982 · cited 3×
“Further details on sample characteristics, task specifications, and imaging acquisition are given in the original report of these data (Aron et al., 2007).”
MethodResultPlint, A. C., Moher, D., Morrison, A., Schulz, K., Altman, D. G., Hill, C., & Gaboury, I. (2006) · Medical Journal of Australia · doi:10.5694/j.1326-5377.2006.tb00557.x · Cited by 872
Weissenbacher, A., Kasess, C., Gerstl, F., Lanzenberger, R., Moser, E., & Windischberger, C. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2009.05.005 · Cited by 776
Comparison of Registered and Published Primary Outcomes in Randomized Controlled Trials
Mathieu, S. (2009) · JAMA · doi:10.1001/jama.2009.1242 · Cited by 730
Handbook of Functional MRI Data Analysis
Poldrack, R. A., Mumford, J. A., & Nichols, T. E. (2011) · Cambridge University Press eBooks · doi:10.1017/cbo9780511895029 · Cited by 699
Guidelines for reporting an fMRI study
Poldrack, R. A., Fletcher, P. C., Henson, R. N., Worsley, K. J., Brett, M., & Nichols, T. E. (2008) · NeuroImage · doi:10.1016/j.neuroimage.2007.11.048 · Cited by 550
The case for open computer programs
Ince, D. C., Hatton, L., & Graham-Cumming, J. (2012) · Nature · doi:10.1038/nature10836 · Cited by 535
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2019) · CHANCE · doi:10.1080/09332480.2019.1579573 · Cited by 463
The secret lives of experiments: Methods reporting in the fMRI literature
Carp, J. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.07.004 · Cited by 457
Self-citationSimmons, J., Nelson, L., & Simonsohn, U. (2011) · PsycEXTRA Dataset · doi:10.1037/e519702015-014 · Cited by 454
Slice-timing effects and their correction in functional MRI
Sladky, R., Friston, K. J., Tröstl, J., Cunnington, R., Moser, E., & Windischberger, C. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2011.06.078 · Cited by 453
Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research
Chan, A.-W. (2004) · Canadian Medical Association Journal · doi:10.1503/cmaj.1041086 · Cited by 444
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2015) · Methodological issues and strategies in clinical research (4th ed.). · doi:10.1037/14805-033 · Cited by 395
Purdon, P. L., & Weisskoff, R. M. (1998) · Human Brain Mapping · doi:10.1002/(sici)1097-0193(1998)6:4<239::aid-hbm4>3.0.co;2-4 · Cited by 323
Excess Significance Bias in the Literature on Brain Volume Abnormalities
Ioannidis, J. P. A. (2011) · Archives of General Psychiatry · doi:10.1001/archgenpsychiatry.2011.28 · Cited by 299
ROC Analysis of Statistical Methods Used in Functional MRI: Individual Subjects
Skudlarski, P., Constable, R. T., & Gore, J. C. (1999) · NeuroImage · doi:10.1006/nimg.1999.0402 · Cited by 260
Evaluating the consistency and specificity of neuroimaging data using meta-analysis
Wager, T. D., Lindquist, M. A., Nichols, T. E., Kober, H., & Van Snellenberg, J. X. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2008.10.061 · Cited by 242
Motion or activity: their role in intra- and inter-subject variation in fMRI
Lund, T. E., Nørgaard, M. D., Rostrup, E., Rowe, J. B., & Paulson, O. B. (2005) · NeuroImage · doi:10.1016/j.neuroimage.2005.02.021 · Cited by 221
Automatic independent component labeling for artifact removal in fMRI
Tohka, J., Foerde, K., Aron, A. R., Tom, S. M., Toga, A. W., & Poldrack, R. A. (2008) · NeuroImage · doi:10.1016/j.neuroimage.2007.10.013 · Cited by 217
John, L., Loewenstein, G., & Prelec, D. (2012) · PsycEXTRA Dataset · doi:10.1037/e632032012-001 · Cited by 177
Variability in fMRI: A re‐examination of inter‐session differences
Smith, S. M., Beckmann, C. F., Ramnani, N., Woolrich, M. W., Bannister, P. R., Jenkinson, M., Matthews, P. M., & McGonigle, D. J. (2005) · Human Brain Mapping · doi:10.1002/hbm.20080 · Cited by 174
Optimizing the order of operations for movement scrubbing: Comment on Power et al.
Carp, J. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2011.12.061 · Cited by 172
Self-citationA Study of Analysis Parameters That Influence the Sensitivity of Event-Related fMRI Analyses
Hopfinger, J. B., Büchel, C., Holmes, A. P., & Friston, K. J. (2000) · NeuroImage · doi:10.1006/nimg.2000.0549 · Cited by 150
An Empirical Comparison of SPM Preprocessing Parameters to the Analysis of fMRI Data
Della‐Maggiore, V., Chau, W., Peres‐Neto, P. R., & McIntosh, A. R. (2002) · NeuroImage · doi:10.1006/nimg.2002.1113 · Cited by 142
Strother, S. C., Conte, S. L., Hansen, L. K., Anderson, J., Zhang, J., Pulapura, S. K., & Rottenberg, D. A. (2004) · NeuroImage · doi:10.1016/j.neuroimage.2004.07.022 · Cited by 134
Churchill, N. W., Oder, A., Abdi, H., Tam, F., Lee, W., Thomas, C. G., Ween, J. E., Graham, S. J., & Strother, S. C. (2011) · Human Brain Mapping · doi:10.1002/hbm.21238 · Cited by 124
Churchill, N. W., Yourganov, G., Oder, A., Tam, F., Graham, S. J., & Strother, S. C. (2012) · PLoS ONE · doi:10.1371/journal.pone.0031147 · Cited by 51
Contradicted and Initially Stronger Effects in Highly Cited Clinical Research
Ioannidis, J. P. A. (2005) · ACC Current Journal Review · doi:10.1016/j.accreview.2005.09.021 · Cited by 41
Poline, J., Strother, S. C., Dehaene‐Lambertz, G., Egan, G. F., & Lancaster, J. L. (2006) · Human Brain Mapping · doi:10.1002/hbm.20268 · Cited by 37
PHYCAA: Data-driven measurement and removal of physiological noise in BOLD fMRI
Churchill, N. W., Yourganov, G., Spring, R., Rasmussen, P. M., Lee, W., Ween, J. E., & Strother, S. C. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2011.08.021 · Cited by 31
Glover (2000) · doi:10.1002/1522-2594(200007)44:1<162::aid-mrm23>3.0.co;2-e
Cited by
MEG and EEG data analysis with MNE-Python
Gramfort, A. (2013) · Frontiers in Neuroscience · doi:10.3389/fnins.2013.00267 · Cited by 3,784
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
Cognitive Reappraisal of Emotion: A Meta-Analysis of Human Neuroimaging Studies
Buhle, J. T., Silvers, J. A., Wager, T. D., Lopez, R., Onyemekwu, C., Kober, H., Weber, J., & Ochsner, K. N. (2014) · Cerebral Cortex · doi:10.1093/cercor/bht154 · Cited by 1,896
Scanning the horizon: towards transparent and reproducible neuroimaging research
Poldrack, R. A., Baker, C. I., Durnez, J., Gorgolewski, K. J., Matthews, P. M., Munafò, M. R., Nichols, T. E., Poline, J.-B., Vul, E., & Yarkoni, T. (2017) · Nature Reviews Neuroscience · doi:10.1038/nrn.2016.167 · Cited by 1,410
Variability in the analysis of a single neuroimaging dataset by many teams
(2020) · doi:10.1038/s41586-020-2314-9 · Cited by 1,097
Metadata
- Authors
- Carp, J.
- Venue
- Frontiers in Neuroscience
- Publisher
- Frontiers Media SA
- Published
- 2012
- Volume
- 6
- Pages
- 149-149
- ISSN
- 1662-453X (ISSN-L)
- 1662-4548
- OpenAlex
- W2156451368
- Cited by
- 381
- References
- 56
- FWCI
- 5.22
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.