CoChat
Sign Up
PreprintCited by 32Statistical Methods and Bayesian InferenceSpatial and Panel Data AnalysisFunctional Brain Connectivity Studies

Application of the hierarchical bootstrap to multi-level data in neuroscience

Open access
Saravanan, Varun, Berman, Gordon J, Sober, Samuel J·2020·arXiv
DOI 10.48550/arxiv.2007.07797arXiv 2007.07797PMC PMC7906290
Open this paper in CoChat — free account

Abstract

A common feature in many neuroscience datasets is the presence of hierarchical data structures, most commonly recording the activity of multiple neurons in multiple animals across multiple trials. Accordingly, the measurements constituting the dataset are not independent, even though the traditional statistical analyses often applied in such cases (e.g., Students t-test) treat them as such. The hierarchical bootstrap has been shown to be an effective tool to accurately analyze such data and while it has been used extensively in the statistical literature, its use is not widespread in neuroscience - despite the ubiquity of hierarchical datasets. In this paper, we illustrate the intuitiveness and utility of this approach to analyze hierarchically nested datasets. We use simulated neural data to show that traditional statistical tests can result in a false positive rate of over 45%, even if the Type-I error rate is set at 5%. While summarizing data across non-independent points (or lower levels) can potentially fix this problem, this approach greatly reduces the statistical power of the analysis. The hierarchical bootstrap, when applied sequentially over the levels of the hierarchical structure, keeps the Type-I error rate within the intended bound and retains more statistical power than summarizing methods. We conclude by demonstrating the effectiveness of the method in two real-world examples, first analyzing singing data in male Bengalese finches (Lonchura striata var. domestica) and second quantifying changes in behavior under optogenetic control in flies (Drosophila melanogaster).

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2007.07797

Cites

42 of 42 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Even after accounting for overdispersion, some models may still yield inflated or deflated type I error rates, therefore requiring statistical testing via a resampling procedure.

    “Second, even after accounting for overdispersion, some models may still yield inflated or deflated type I error rates (not observed in our examples of Fig. 3), therefore requiring statistical testing via a resampling procedure (Ives, 2015; Saravanan et al., 2020; Szöcs & Schäfer, 2015; Warton et al., 2016; Warton & Hui, 2011)”

    From Violating the normality assumption may be the lesser of two evils (2021) · cites this paper as Saravanan et al., 2020

    Section 3.1.3 (p. 12) reports elevated false-positive rates for LMMs and the hierarchical bootstrap at very small group sizes; section 3.1.2 (p. 11) reports a below-5% rate for the bootstrap with independent data. Section 3.1.1 (p. 8) uses bootstrapping to test whether rates differ from 5%. The paper does not establish that these deviations persist after accounting for overdispersion or that a resampling procedure is required. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • The probably riskiest assumption to violate (in terms of producing type I errors) is the assumption of independence of data points.

    “The probably riskiest assumption to violate (in terms of producing type I errors) is the assumption of independence of data points (Forstmeier et al., 2017; Kass et al., 2016; Saravanan et al., 2020)”

    From Violating the normality assumption may be the lesser of two evils (2021) · cites this paper as Saravanan et al., 2020

    The Discussion (p. 19) calls independence a critical assumption, and Results §3.1.1 (p. 8) report that treating clustered data as independent raised the traditional method’s false-positive rate well above the expected 5%. The paper does not compare violations of independence with violations of other assumptions, so it does not establish that independence is the riskiest assumption to violate. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 21

  1. [3][3] Arceneaux K, Nickerson DW. Modeling certainty with clustered data: A comparison of methods. Political analysis 2009;17(2):177-190. -
  2. [4][4] Musca SC, Kamiejski R, Nugier A, Méot A, Er-Rafiy A, Brauer M. Data with hierarchical structure: impact of intraclass correlation and sample size on Type-I error. Frontiers in Psychology 2011;2:74. -
  3. [7][7] Aarts E, Dolan CV, Verhage M, van der Sluis S. Multilevel analysis quantifies variation in the experimental effect while optimizing power and preventing false positives. BMC neuroscience 2015;16(1):94. -
  4. [8][8] Arlet M, Jubin R, Masataka N, Lemasson A. Grooming-at-a-distance by exchanging calls in non-human primates. Biology letters 2015;11(10):20150711. -
  5. [11][11] Pleil KE, Helms CM, Sobus JR, Daunais JB, Grant KA, Kash TL. Effects of chronic alcohol consumption on neuronal function in the non-human primate BNST. Addiction biology 2016;21(6):1151-1167. -
  6. [14][14] Huang FL. Using cluster bootstrapping to analyze nested data with a few clusters. Educational and psychological measurement 2018;78(2):297-318. -
  7. [15][15] Seedorff M, Oleson J, McMurray B. Maybe maximal: Good enough mixed models optimize power while controlling Type I error. PsyArXiv 2019;. -
  8. [21][21] HardenJJ. A bootstrap method for conducting statistical inference with clustered data. State Politics & Policy Quarterly 2011;11(2):223-246. -
  9. [23][23] Thai HT, Mentré F, Holford NH, Veyrat-Follet C, Comets E. A comparison of bootstrap approaches for estimating uncertainty of parameters in linear mixed-effects models. Pharmaceutical statistics 2013;12(3):129-140. -
  10. [25][25] AdamsDC,GurevitchJ,RosenbergMS. Resamplingtestsformeta-analysisofecologicaldata. Ecology1997;78(4):12771283. -
  11. [30][30] Snijders TA. Multilevel analysis. Springer; 2011. -
  12. [31][31] HoxJJ, Moerbeek M, Van de Schoot R. Multilevel analysis: Techniques and applications. Routledge; 2017. -
  13. [35][35] Davison AC, Hinkley DV. Bootstrap methods and their application, vol. 1. Cambridge university press; 1997. -
  14. [37][37] Felsenstein J, Kishino H. Is there something wrong with the bootstrap on phylogenies? A reply to Hillis and Bull. Systematic Biology 1993;42(2):193-200. -
  15. [38][38] Shimodaira H. An approximately unbiased test of phylogenetic tree selection. Systematic biology 2002;51(3):492-508. -
  16. [44][44] TianLY,BrainardMS. Discretecircuitssupportgeneralizedversuscontext-specificvocallearninginthesongbird. Neuron 2017;96(5):1168-1177. -
  17. [48][48] Heffner RA, Butler MJ, Reilly CK. Pseudoreplication revisited. Ecology 1996;77(8):2558-2562. -
  18. [51][51] BaayenRH,DavidsonDJ,BatesDM. Mixed-effectsmodelingwithcrossedrandomeffectsforsubjectsanditems. Journal of memory and language 2008;59(4):390-412. -
  19. [53][53] Barr DJ, Levy R, Scheepers C, Tily HJ. Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of memory and language 2013;68(3):255-278. -
  20. [65][65] Baker M. 1,500 scientists lift the lid on reproducibility. Nature News 2016;533(7604):452. -
  21. [67][67] Gerlai R. Reproducibility and replicability in zebrafish behavioral neuroscience research. Pharmacology Biochemistry and Behavior 2019;178:30-38.

Cited by

Sorted by citations

Related

Authors
Saravanan, V., Berman, G. J., & Sober, S. J.
Affiliations
Emory University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
Volume
3
Issue
5
DOI
10.48550/arxiv.2007.07797
arXiv
2007.07797
PMC
PMC7906290
OpenAlex
W3043477088
Cited by
26
References
42
Citations by year
2026: 1 · 2025: 3 · 2024: 4 · 2023: 5 · 2022: 7 · 2021: 2 · 2020: 4
FWCI
3.17
Type
journal article
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Science Foundation (1456912)
  • Research Corporation for Science Advancement (25999)
  • National Institutes of Health
  • National Institute of Mental Health (NIMHR01, MH115831, MH115831-01, R01MH115831)
  • National Institute of Neurological Disorders and Stroke (R01NS084844, F31NS100406, NS084844)
  • National Institute of Biomedical Imaging and Bioengineering (EB022872, R01EB022872)
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
  • Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Spatial and Panel Data Analysis
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Functional Brain Connectivity Studies
Keywords
multilevel data, nested data structures, resampling, statistical inference, neuroscience data analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review