SLEAP: Multi-animal pose tracking
Open accessThe DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
The desire to understand how the brain generates and patterns behavior has driven rapid methodological innovation to quantify and model natural animal behavior. This has led to important advances in deep learning-based markerless pose estimation that have been enabled in part by the success of deep learning for computer vision applications. Here we present SLEAP (Social LEAP Estimates Animal Poses), a framework for multi-animal pose tracking via deep learning. This system is capable of simultaneously tracking any number of animals during social interactions and across a variety of experimental conditions. SLEAP implements several complementary approaches for dealing with the problems inherent in moving from single-to multi-animal pose tracking, including configurable neural network architectures, inference techniques, and tracking algorithms, enabling easy specialization and tuning for particular experimental conditions or performance requirements. We report results on multiple datasets of socially interacting animals (flies, bees, and mice) and describe how dataset-specific properties can be leveraged to determine the best configuration of SLEAP models. Using a high accuracy model (<2.8 px error on 95% of points), we were able to track two animals from full size 1024 × 1024 pixel frames at up to 320 FPS. The SLEAP framework comes with a sophisticated graphical user interface, multi-platform support, Colab-based GPU-free training and inference, and complete tutorials available, in addition to the datasets, at sleap.ai .
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields
Cao, Z., Simon, T., Wei, S.-E., & Sheikh, Y. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.143 · Cited by 7,539
Applications for deep learning in ecology
Christin, S., Hervet, É., & Lecomte, N. (2019) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13256 · Cited by 724
Computing receptive fields of convolutional neural networks
Araujo, A., Norris, W., & Sim, J. (2019) · Distill · doi:10.23915/distill.00021 · Cited by 312
Numerical Coordinate Regression with Convolutional Neural Networks
Nibali, A., He, Z., Morgan, S., & Prendergast, L. (2018) · arXiv · doi:10.48550/arxiv.1801.07372 · Cited by 186 · cited 1×
“We leverage integral regression [25] to compute real-valued offsets by taking the weighted average of the local patch of the confidence maps around each grid-aligned peak.”
Method
Cited by
Multi-animal pose estimation, identification and tracking with DeepLabCut
Lauer, J., Zhou, M., Ye, S., Menegas, W., Schneider, S., Nath, T., Rahman, M. M., Di Santo, V., Soberanes, D., Feng, G., et al. (2022) · Nature Methods · doi:10.1038/s41592-022-01443-0 · Cited by 647
Anipose: A toolkit for robust markerless 3D pose estimation
Karashchuk, P., Rupp, K. L., Dickinson, E. S., Walling-Bell, S., Sanders, E., Azim, E., Brunton, B. W., & Tuthill, J. C. (2021) · Cell Reports · doi:10.1016/j.celrep.2021.109730 · Cited by 324
Quantifying behavior to understand the brain
Pereira, T. D., Shaevitz, J. W., & Murthy, M. (2020) · Nature Neuroscience · doi:10.1038/s41593-020-00734-z · Cited by 309
Self-citationWalter, T., & Couzin, I. D. (2021) · eLife · doi:10.7554/elife.64000 · Cited by 286
Organization of neural circuits underlying social behavior: A consideration of the medial amygdala
Raam, T., & Hong, W. (2021) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2021.02.008 · Cited by 134
Metadata
- Authors
- Pereira, T. D., Tabris, N., Li, J., Ravindranath, S., Papadoyannis, E. S., Wang, Z. Y., Turner, D. M., McKenzie-Smith, G., Kocher, S. D., Falkner, A. L., Shaevitz, J. W., & Murthy, M.
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2020-09-02
- OpenAlex
- W3082044205
- Cited by
- 87
- References
- 5
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.