CoChat
Sign Up
Cited by 218Human Pose and Action RecognitionMemory and Neural MechanismsHuman Motion and Animation

Identifying behavioral structure from deep variational embeddings of animal motion

Open access
Luxem, Kevin, Mocellin, Petra, Fuhrmann, Falko, Kürsch, Johannes, Miller, Stephanie R., Palop, Jorge J., Remy, Stefan, Bauer, Pavol·2022·Communications Biology
DOI 10.1038/s42003-022-04080-7PMC PMC9674640

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Quantification and detection of the hierarchical organization of behavior is a major challenge in neuroscience. Recent advances in markerless pose estimation enable the visualization of high-dimensional spatiotemporal behavioral dynamics of animal motion. However, robust and reliable technical approaches are needed to uncover underlying structure in these data and to segment behavior into discrete hierarchically organized motifs. Here, we present an unsupervised probabilistic deep learning framework that identifies behavioral structure from deep variational embeddings of animal motion (VAME). By using a mouse model of beta amyloidosis as a use case, we show that VAME not only identifies discrete behavioral motifs, but also captures a hierarchical representation of the motif’s usage. The approach allows for the grouping of motifs into communities and the detection of differences in community-specific motif usage of individual mouse cohorts that were undetectable by human visual observation. Thus, we present a robust approach for the segmentation of animal motion that is applicable to a wide range of experimental setups, models and conditions without requiring supervised or a-priori human interference.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s42003-022-04080-7

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 52

  1. [1]Krakauer JW Ghazanfar AA Gomez-Marin A MacIver MA Poeppel D Neuroscience needs behavior: correcting a reductionist bias Neuron 2017 93 480 490 10.1016/j.neuron.2016.12.041 28182904 · DOI · Paper page
  2. [2]Nilsson, S. R. O. et al. Simple behavioral analysis (simba) – an open source toolkit for computer classification of complex social behaviors in experimental animals. Preprint at https://www.biorxiv.org/content/10.1101/2020.04.19.049452v2 (2020). · DOI
  3. [3]Segalin C The mouse action recognition system (mars) software pipeline for automated analysis of social behaviors in mice eLife 2021 10 e63720 10.7554/eLife.63720 34846301 PMC8631946 · DOI
  4. [4]Bohnslav JP Deepethogram, a machine learning pipeline for supervised behavior classification from raw pixels eLife 2021 10 e63377 10.7554/eLife.63377 34473051 PMC8455138 · DOI
  5. [5]Gomez-Marin A Paton JJ Kampff AR Costa RM Mainen ZF Big behavioral data: psychology, ethology and the foundations of neuroscience Nat. Neurosci. 2014 17 1455 1462 10.1038/nn.3812 25349912 · DOI
  6. [6]Anderson DJ Perona P Toward a science of computational ethology Neuron 2014 84 18 31 10.1016/j.neuron.2014.09.005 25277452 · DOI
  7. [7]Wiltschko AB Mapping sub-second structure in mouse behavior Neuron 2015 88 1121 1135 10.1016/j.neuron.2015.11.031 26687221 PMC4708087 · DOI · Paper page
  8. [8]Berman GJ Measuring behavior across scales BMC Biol. 2018 16 23 10.1186/s12915-018-0494-7 29475451 PMC5824583 · DOI
  9. [9]Brown AEX de Bivort B Ethology as a physical science Nat. Phys. 2018 14 653 657 10.1038/s41567-018-0093-0 · DOI
  10. [11]Berman GJ Choi DM Bialek W Shaevitz JW Mapping the stereotyped behaviour of freely moving fruit flies J. R. Soc. Interface 2014 11 20140672 10.1098/rsif.2014.0672 25142523 PMC4233753 · DOI · Paper page
  11. [12]Hsu AI Yttri EA B-soid, an open-source unsupervised algorithm for identification and fast prediction of behaviors Nat. Commun. 2021 12 5188 10.1038/s41467-021-25420-x 34465784 PMC8408193 · DOI
  12. [13]Mathis A DeepLabCut: markerless pose estimation of user-defined body parts with deep learning Nat. Neurosci. 2018 21 1281 1289 10.1038/s41593-018-0209-y 30127430 · DOI · Paper page
  13. [14]Pereira TD Sleap: a deep learning system for multi-animal pose tracking Nat. Methods 2022 19 4 PMC9007740 35379947
  14. [15]Graving JM Deepposekit, a software toolkit for fast and robust animal pose estimation using deep learning eLife 2019 8 e47994 10.7554/eLife.47994 31570119 PMC6897514 · DOI · Paper page
  15. [16]Shi C Learning disentangled behavior embeddings Neural Inf. Process. Syst. 2021 34 22562 22573
  16. [17]Kingma, D. P. & Welling, M. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, ICLR (2014).
  17. [18]Rezende, D. J., Mohamed, S. & Wierstra, D. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st International Conference on Machine Learning, vol. 31, 1278–1286 (2014).
  18. [19]Bengio Y Courville A Vincent P Representation learning: a review and new perspectives IEEE Trans. Pattern Anal. Mach. Intell. 2013 35 1798 1828 10.1109/TPAMI.2013.50 23787338 · DOI · Paper page
  19. [20]Kuehne H Richard A Gall J A hybrid rnn-hmm approach for weakly supervised temporal action segmentation IEEE Trans. Pattern Anal. Mach. Intell. 2020 42 765 779 10.1109/TPAMI.2018.2884469 30582525 · DOI
  20. [21]Chung J A recurrent latent variable model for sequential data Adv. Neural Inf. Process. Syst. 2015 28 2980 2988
  21. [22]Chen, X. et al. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems, vol. 29, 2172–2180 (2016).
  22. [23]Higgins, I. et al. beta-vae: Learning basic visual concepts with a constrained variational framework. In 5th International Conference on Learning Representations, ICLR (2017).
  23. [24]Jiang, Z., Zheng, Y., Tan, H., Tang, B. & Zhou, H. Variational deep embedding: an unsupervised and generative approach to clustering. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, vol. 26, 1965–1972 (2017).
  24. [25]Pereira, J. & Silveira, M. Learning representations from healthcare time series data for unsupervised anomaly detection. In 2019 IEEE International Conference on Big Data and Smart Computing (BigComp), 1–7 (2019).
  25. [26]Ma, Q., Zheng, J., Li, S. & Cottrell, G. W. Learning representations for time series clustering. In Advances in Neural Information Processing Systems, vol. 32, 3776–3786 (2019).
  26. [27]Jankowsky JL Mutant presenilins specifically elevate the levels of the 42 residue beta-amyloid peptide in vivo: evidence for augmentation of a 42-specific gamma secretase Hum. Mol. Genet. 2004 13 159 170 10.1093/hmg/ddh019 14645205 · DOI · Paper page
  27. [28]Schuster M Paliwal KK Bidirectional recurrent neural networks IEEE Trans. Signal Process. 1997 45 2673 2681 10.1109/78.650093 · DOI · Paper page
  28. [29]Cho, K. et al. Learning phrase representations using rnn encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, 1724–1734 (2014).
  29. [31]Rabiner L Juang B-H An introduction to hidden Markov models IEEE ASSP Mag. 1986 3 4 16 10.1109/MASSP.1986.1165342 · DOI · Paper page
  30. [32]Huang H Characterization of AD-like phenotype in aged APPSwe/PS1dE9 mice Age (Dordr., Neth.) 2016 38 303 322 10.1007/s11357-016-9929-7 PMC5061676 27439903 · DOI
  31. [35]Janus C Flores AY Xu G Borchelt DR Behavioral abnormalities in APPSwe/PS1dE9 mouse model of AD-like pathology: comparative analysis across multiple behavioral domains Neurobiol. Aging 2015 36 2519 2532 10.1016/j.neurobiolaging.2015.05.010 26089165 · DOI
  32. [36]Giovannetti, E. A. et al. Restoring memory by optogenetic synchronization of hippocampal oscillations in an alzheimer’s disease mouse model. Preprint at https://www.biorxiv.org/content/10.1101/363820v1 (2018). · DOI
  33. [37]Webster SJ Bachstetter AD Van Eldik LJ Comprehensive behavioral characterization of an app/ps-1 double knock-in mouse model of alzheimer’s disease Alzheimers Res. Ther. 2013 5 28 10.1186/alzrt182 23705774 PMC3706792 · DOI
  34. [38]Biallosterski BT Changes in voiding behavior in a mouse model of alzheimer’s disease Front. Aging Neurosci. 2015 7 160 10.3389/fnagi.2015.00160 26379542 PMC4548184 · DOI
  35. [39]Chaudhuri R Gerçek B Pandey B Peyrache A Fiete I The intrinsic attractor manifold and population dynamics of a canonical cognitive circuit across waking and sleep Nat. Neurosci. 2019 22 1512 1520 10.1038/s41593-019-0460-x 31406365 · DOI
  36. [40]Rubin A Revealing neural correlates of behavior without behavioral measurements Nat. Commun. 2019 10 4745 10.1038/s41467-019-12724-2 31628322 PMC6802184 · DOI
  37. [42]Zong W Large-scale two-photon calcium imaging in freely moving mice Cell 2022 185 1240–1256.e30 10.1016/j.cell.2022.02.017 35305313 PMC8970296 · DOI
  38. [43]Xu S Behavioral state coding by molecularly defined paraventricular hypothalamic cell type ensembles Science 2020 370 eabb2494 10.1126/science.abb2494 33060330 PMC11938375 · DOI
  39. [45]Luxem, K., Fuhrmann, F., Remy, S. & Bauer, P. Hierarchical network analysis of behavior and neuronal population activity. In Conference on Cognitive Computational Neuroscience (2019).
  40. [47]Batty, E. et al. BehaveNet: nonlinear embedding and Bayesian neural decoding of behavioral videos. In Advances in Neural Information Processing Systems, vol. 32, 15680–15691 (2019).
  41. [48]Tishby, N. & Zaslavsky, N. Deep learning and the information bottleneck principle. Preprint at https://arxiv.org/abs/1503.02406 (2015). · arXiv
  42. [49]Günel S DeepFly3D, a deep learning-based approach for 3D limb and appendage tracking in tethered, adult Drosophila eLife 2019 8 e48571 10.7554/eLife.48571 31584428 PMC6828327 · DOI · Paper page
  43. [50]Sarkar, I. et al. Evaluation of deep lift pose models for 3d rodent pose estimation based on geometrically triangulated data. In CVPR 2021 Workshop CV4animals (2021).
  44. [51]Dunn TW Geometric deep learning enables 3d kinematic profiling across species and environments Nat. Methods 2021 18 564 573 10.1038/s41592-021-01106-6 33875887 PMC8530226 · DOI
  45. [52]Sun, J. J. et al. Task programming: Learning data efficient behavior representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2876–2885 (2021). PMC9766046 36544482
  46. [53]Whiteway MR Partitioning variability in animal behavioral videos using semi-supervised variational autoencoders PLOS Comput. Biol. 2021 17 1 50 10.1371/journal.pcbi.1009439 PMC8489729 34550974 · DOI
  47. [54]Sun, J. J. et al. Self-supervised keypoint discovery in behavioral videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2161–2170 (2022). PMC9829414 36628357
  48. [55]Luxem, K. et al. Open-source tools for behavioral video analysis: setup, methods, and development. Preprint at https://arxiv.org/ftp/arxiv/papers/2204/2204.02842.pdf (2022). PMC10036114 36951911
  49. [56]Jankowsky JL Co-expression of multiple transgenes in mouse CNS: a comparison of strategies Biomol. Eng. 2001 17 157 165 10.1016/S1389-0344(01)00067-3 11337275 · DOI
  50. [57]Hochreiter S Schmidhuber J Long short-term memory Neural Comput. 1997 9 1735 1780 10.1162/neco.1997.9.8.1735 9377276 · DOI · Paper page
  51. [58]Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR, vol. 3 (2015).
  52. [59]Paszke, A. et al. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop (2017).

Cited by

Sorted by citations

Authors
Luxem, K., Mocellin, P., Fuhrmann, F., Kürsch, J., Miller, S. R., Palop, J. J., Remy, S., & Bauer, P.
Affiliations
  • German Center for Neurodegenerative Diseases (DE)
  • Gladstone Institutes (US)
Corresponding author
Remy, Stefan
ORCID
Venue
Communications Biology
Publisher
Springer Science and Business Media LLC
Published
2022-11-18
Volume
5
Issue
1
Pages
1267-1267
ISSN
2399-3642 (ISSN-L)
DOI
10.1038/s42003-022-04080-7
PMC
PMC9674640
OpenAlex
W4309473115
Cited by
218
References
6
Citations by year
2026: 42 · 2025: 49 · 2024: 47 · 2023: 26 · 2022: 24 · 2021: 23 · 2020: 5 · 2019: 1
FWCI
11.94
Type
journal article
Language
en
Open access
Licence
Locations
  • Communications Biology (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
  • Research Square (acceptedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Informationszentrum Lebenswissenschaften (ZB MED) (submittedVersion, cc-by)
  • DZNE Pub (submittedVersion, cc-by)
Funders
  • Deutsche Forschungsgemeinschaft (SFB-1436, SFB1089)
  • National Institute on Aging (P01AG073082)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
  • Physical Sciences › Engineering › Control and Systems Engineering › Human Motion and Animation
Keywords
markerless pose estimation, community detection, behavioral dynamics
MeSH terms
Animals, Behavior, Animal, Humans, Motion, Neurosciences

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review