Application of the Hierarchical Bootstrap to Multi-Level Data in Neuroscience
Open accessAbstract
A bstract: A common feature in many neuroscience datasets is the presence of hierarchical data structures, most commonly recording the activity of multiple neurons in multiple animals across multiple trials. Accordingly, the measurements constituting the dataset are not independent, even though the traditional statistical analyses often applied in such cases (e.g. Student’s t-test) treat them as such. The hierarchical bootstrap has been shown to be an effective tool to accurately analyze such data and while it has been used extensively in the statistical literature, its use is not widespread in neuroscience - despite the ubiquity of hierarchical datasets. In this paper, we illustrate the intuitiveness and utility of this approach to analyze hierarchically nested datasets. We use simulated neural data to show that traditional statistical tests can result in a false positive rate of over 45%, even if the Type-I error rate is set at 5%. While summarizing data across non-independent points (or lower levels) can potentially fix this problem, this approach greatly reduces the statistical power of the analysis. The hierarchical bootstrap, when applied sequentially over the levels of the hierarchical structure, keeps the Type-I error rate within the intended bound and retains more statistical power than summarizing methods. We conclude by demonstrating the effectiveness of the method in two real-world examples, first analyzing singing data in male Bengalese finches ( Lonchura striata var. domestica ) and second quantifying changes in behavior under optogenetic control in flies ( Drosophila melanogaster ).
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505 · see citing works
Mixed-effects modeling with crossed random effects for subjects and items
Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008) · Journal of Memory and Language · doi:10.1016/j.jml.2007.12.005 · Cited by 8,540 · see citing works
Pseudoreplication and the Design of Ecological Field Experiments
Hurlbert, S. H. (1984) · Ecological Monographs · doi:10.2307/1942661 · Cited by 8,377 · see citing works
The Jackknife, the Bootstrap and Other Resampling Plans
Efron, B. (1982) · Society for Industrial and Applied Mathematics eBooks · doi:10.1137/1.9781611970319 · Cited by 8,120 · see citing works
The ASA Statement on <i>p</i> -Values: Context, Process, and Purpose
Wasserstein, R. L., & Lazar, N. A. (2016) · The American Statistician · doi:10.1080/00031305.2016.1154108 · Cited by 6,306 · see citing works
1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209 · see citing works
Sufficient Sample Sizes for Multilevel Modeling
Maas, C. J. M., & Hox, J. J. (2005) · Methodology · doi:10.1027/1614-2241.1.3.86 · Cited by 3,582 · see citing works
Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations
Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016) · European Journal of Epidemiology · doi:10.1007/s10654-016-0149-3 · Cited by 3,200 · see citing works
Erroneous analyses of interactions in neuroscience: A problem of significance
Nieuwenhuis, S., Forstmann, B. U., & Wagenmakers, E.-J. (2011) · Nature Neuroscience · doi:10.1038/nn.2886 · Cited by 864 · see citing works
Standard Errors and Sample Sizes for Two-Level Research
Snijders, T. A. B., & Bosker, R. J. (1993) · Journal of Educational Statistics · doi:10.3102/10769986018003237 · Cited by 487 · cited 2× · see citing works
“Given the relationship between the number of points within a cluster to the Design Effect (DEFF; seeDesign Effect inMaterials andMethods), we would expect the false positive rate to increase with the number of neurons per subject in the traditional analysis but not using the other methods [30, 36, 1].”
BackgroundMethodThe problem of pseudoreplication in neuroscientific studies: is it affecting your analysis?
Lazic, S. E. (2010) · BMC Neuroscience · doi:10.1186/1471-2202-11-5 · Cited by 381 · see citing works
Data with Hierarchical Structure: Impact of Intraclass Correlation and Sample Size on Type-I Error
Musca, S. C., Kamiejski, R., Nugier, A., Méot, A., Er-Rafiy, A., & Brauer, M. (2011) · Frontiers in Psychology · doi:10.3389/fpsyg.2011.00074 · Cited by 259 · see citing works
Optogenetic dissection of descending behavioral control in Drosophila
Cande, J., Namiki, S., Qiu, J., Korff, W., Card, G. M., Shaevitz, J. W., Stern, D. L., & Berman, G. J. (2018) · eLife · doi:10.7554/elife.34275 · Cited by 163 · see citing works
Self-citationHeffner, R. A., Butler, M. J., & Reilly, C. K. (1996) · Ecology · doi:10.2307/2265754 · Cited by 139 · see citing works
Using cluster bootstrapping to analyze nested data with a few clusters
Huang, F. L. (2018) · Educational and Psychological Measurement · doi:10.1177/0013164416678980 · Cited by 107 · cited 3× · see citing works
“The hierarchical bootstrap [16, 17, 18, 19] is a statistical method that has been applied successfully to awide variety of clustered datasets, including census and polling results, educational and psychological data, and phylogenetic tree information [20, 21, 14].”
BackgroundMethod
Flagged first, then by citations
Cited by
Sorted by citations
Violating the normality assumption may be the lesser of two evils
Knief, U., & Forstmeier, W. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01587-5 · Cited by 654 · see citing works
A visual pathway for skylight polarization processing in Drosophila
Hardcastle, B. J., Omoto, J. J., Kandimalla, P., Nguyen, B.-C. M., Keleş, M. F., Boyd, N. K., Hartenstein, V., & Frye, M. A. (2021) · eLife · doi:10.7554/elife.63225 · Cited by 111 · see citing works
A size principle for recruitment of Drosophila leg motor neurons
Azevedo, A. W., Dickinson, E. S., Gurung, P., Venkatasubramanian, L., Mann, R. S., & Tuthill, J. C. (2020) · eLife · doi:10.7554/elife.56754 · Cited by 87 · see citing works
Munc13 supports fusogenicity of non-docked vesicles at synapses with disrupted active zones.
Tan, C., de Nola, G., Qiao, C., Imig, C., Born, R. T., Brose, N., & Kaeser, P. S. (2022) · eLife · doi:10.7554/elife.79077 · Cited by 17 · see citing works
- Authors
- Saravanan, V., Berman, G. J., & Sober, S. J.
- Affiliations
- Emory University (US)
- Corresponding author
- Sober, Samuel J.
- ORCID
- Saravanan, Varun
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2019-10-28
- OpenAlex
- W2981896893
- Cited by
- 170
- References
- 15
- Citations by year
- 2026: 12 · 2025: 35 · 2024: 42 · 2023: 32 · 2022: 19 · 2021: 14 · 2020: 14 · 2019: 2
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Licence
- cc-by
- Locations
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
- Funders
- National Science Foundation (1456912)
- Simons Foundation
- Research Corporation for Science Advancement (25999)
- National Institutes of Health
- National Institute of Mental Health (NIMHR01, MH115831, MH115831-01, R01MH115831)
- National Institute of Neurological Disorders and Stroke (R01NS084844, F31NS100406, NS084844)
- National Institute of Biomedical Imaging and Bioengineering (EB022872, R01EB022872)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Keywords
- multilevel data, hierarchical data structures, type I error rate, false positive rate, statistical power, optogenetic control, Drosophila behavior
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
