Cited by 340Functional Brain Connectivity StudiesAdvanced MRI Techniques and ApplicationsAdvanced Neuroimaging Techniques and Applications

Reproducibility of R-fMRI metrics on the impact of different strategies for multiple comparison correction and sample sizes

Open access
Chen, Xiao, Lu, Bin, Yan, Chao‐Gan·2018·Human Brain Mapping

Published version of a preprint

Abstract

Concerns regarding reproducibility of resting‐state functional magnetic resonance imaging (R‐fMRI) findings have been raised. Little is known about how to operationally define R‐fMRI reproducibility and to what extent it is affected by multiple comparison correction strategies and sample size. We comprehensively assessed two aspects of reproducibility, test–retest reliability and replicability, on widely used R‐fMRI metrics in both between‐subject contrasts of sex differences and within‐subject comparisons of eyes‐open and eyes‐closed (EOEC) conditions. We noted permutation test with Threshold‐Free Cluster Enhancement (TFCE), a strict multiple comparison correction strategy, reached the best balance between family‐wise error rate (under 5%) and test–retest reliability/replicability (e.g., 0.68 for test–retest reliability and 0.25 for replicability of amplitude of low‐frequency fluctuations (ALFF) for between‐subject sex differences, 0.49 for replicability of ALFF for within‐subject EOEC differences). Although R‐fMRI indices attained moderate reliabilities, they replicated poorly in distinct datasets (replicability < 0.3 for between‐subject sex differences, < 0.5 for within‐subject EOEC differences). By randomly drawing different sample sizes from a single site, we found reliability, sensitivity and positive predictive value (PPV) rose as sample size increased. Small sample sizes (e.g., < 80 [40 per group]) not only minimized power (sensitivity < 2%), but also decreased the likelihood that significant results reflect “true” effects (PPV < 0.26) in sex differences. Our findings have implications for how to select multiple comparison correction strategies and highlight the importance of sufficiently large sample sizes in R‐fMRI studies to enhance reproducibility. Hum Brain Mapp 39:300–318, 2018 . © 2017 Wiley Periodicals, Inc.

Integrity signals

Registry

Crossref · 10.1002/hbm.23843

Cites

72 of 72 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The multiple comparison problem was accounted for by using threshold-free cluster enhancement, except for graph measures where false discovery rate with p < 0.05 was used.

    “The multiple comparison problem was accounted for by using a threshold-free cluster enhancement (TFCE) method (Chen et al., 2018; Smith & Nichols, 2009), except for the graph measures, where we used the false discovery rate (FDR) method with p < 0.05.”
  • TFCE can assess statistical significance at the voxel and ROI level without requiring an arbitrary threshold, thus providing unbiased results.

    “TFCE’s corrections were employed for raw and ROI-to-ROI connectivity because they can assess statistical significance at the voxel and ROI level without requiring one to set an arbitrary threshold, thus providing unbiased results (Chen et al., 2018; Smith & Nichols, 2009).”
  • TFCE is more sensitive to both focal and peripheral effects than classical correction methods, reaching the best balance between family-wise error rates and replicability.

    “Additionally, TFCE is more sensitive to both focal and peripheral effects than classical correction methods, reaching the best balance between family-wise error (FWE) rates and replicability (Chen et al., 2018).”

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Chen, X., Lu, B., & Yan, C.
Affiliations
  • Institute of Psychology, Chinese Academy of Sciences (CN)
  • Chinese Academy of Sciences (CN)
Corresponding author
Yan, Chao‐Gan
Venue
Human Brain Mapping
Publisher
Wiley
Published
2017-10-11
Volume
39
Issue
1
Pages
300-318
ISSN
  • 1065-9471 (ISSN-L)
  • 1097-0193
OpenAlex
W2762154136
Cited by
340
References
72
Citations by year
2026: 29 · 2025: 36 · 2024: 27 · 2023: 44 · 2022: 53 · 2021: 56 · 2020: 40 · 2019: 33 · 2018: 21 · 2017: 1
FWCI
14.73
Type
journal article
Language
en
Open access
Locations
  • Human Brain Mapping (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
  • Institutional Repository of Institute of Psychology, Chinese Academy of Sciences (Institute of Psychology, Chinese Academy of Sciences) (submittedVersion)
Funders
National Natural Science Foundation of China (81671774, 81630031)
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Functional Brain Connectivity Studies
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Advanced MRI Techniques and Applications
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Advanced Neuroimaging Techniques and Applications
Keywords
resting-state fMRI, reproducibility, multiple testing correction, permutation tests, threshold-free cluster enhancement, test-retest reliability, replicability, sample size, amplitude of low-frequency fluctuation, sex differences, positive predictive value, statistical power
MeSH terms
Brain, Brain Mapping, Female, Humans, Magnetic Resonance Imaging, Male, Rest, Sex Characteristics, Visual Perception, Reproducibility of Results, Sample Size, Young Adult

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review