CoChat
Sign Up
Cited by 228Human Pose and Action RecognitionNeurobiology and Insect Physiology ResearchCell Image Analysis Techniques

Keypoint-MoSeq: parsing behavior by linking point tracking to pose dynamics

Open access
Weinreb, Caleb, Pearl, Jonah E., Lin, Sherry, Osman, Mohammed Abdal Monium, Zhang, Libby, Annapragada, Sidharth, Conlin, Eli, Hoffmann, Red +9 more·2024·Nature Methods
DOI 10.1038/s41592-024-02318-2PMC PMC11245396
Open this paper in CoChat — free account

Abstract

Keypoint tracking algorithms can flexibly quantify animal movement from videos obtained in a wide variety of settings. However, it remains unclear how to parse continuous keypoint data into discrete actions. This challenge is particularly acute because keypoint data are susceptible to high-frequency jitter that clustering algorithms can mistake for transitions between actions. Here we present keypoint-MoSeq, a machine learning-based platform for identifying behavioral modules (‘syllables’) from keypoint data without human supervision. Keypoint-MoSeq uses a generative model to distinguish keypoint noise from behavior, enabling it to identify syllables whose boundaries correspond to natural sub-second discontinuities in pose dynamics. Keypoint-MoSeq outperforms commonly used alternative clustering methods at identifying these transitions, at capturing correlations between neural activity and behavior and at classifying either solitary or social behaviors in accordance with human annotations. Keypoint-MoSeq also works in multiple species and generalizes beyond the syllable timescale, identifying fast sniff-aligned movements in mice and a spectrum of oscillatory behaviors in fruit flies. Keypoint-MoSeq, therefore, renders accessible the modular structure of behavior through standard video recordings.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41592-024-02318-2

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 48

  1. [1]Tinbergen, N. The Study of Instinct (Clarendon Press, 1951).
  2. [2]Dawkins, R. In Growing Points in Ethology (Bateson, P. P. G. & Hinde, R. A. eds.) Chap 1 (Cambridge University Press, 1976).
  3. [3]Baerends GP The functional organization of behaviour Anim. Behav. 1976 24 726 738 10.1016/S0003-3472(76)80002-4 · DOI
  4. [4]Pereira TD SLEAP: a deep learning system for multi-animal pose tracking Nat. Methods 2022 19 486 495 10.1038/s41592-022-01426-1 35379947 PMC9007740 · DOI
  5. [5]Mathis A DeepLabCut: markerless pose estimation of user-defined body parts with deep learning Nat. Neurosci. 2018 21 1281 1289 10.1038/s41593-018-0209-y 30127430 · DOI · Paper page
  6. [6]Sun JJ Self-supervised keypoint discovery in behavioral videos Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. 2022 2022 2161 2170 36628357 PMC9829414
  7. [7]Graving JM DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning eLife 2019 8 e47994 10.7554/eLife.47994 31570119 PMC6897514 · DOI · Paper page
  8. [8]Mathis A Schneider S Lauer J Mathis MW A primer on motion capture with deep learning: principles, pitfalls, and perspectives Neuron 2020 108 44 65 10.1016/j.neuron.2020.09.017 33058765 · DOI · Paper page
  9. [10]Anderson DJ Perona P Toward a science of computational ethology Neuron 2014 84 18 31 10.1016/j.neuron.2014.09.005 25277452 · DOI
  10. [11]Pereira TD Shaevitz JW Murthy M Quantifying behavior to understand the brain Nat. Neurosci. 2020 23 1537 1549 10.1038/s41593-020-00734-z 33169033 PMC7780298 · DOI
  11. [12]Hsu AI Yttri EA B-SOiD, an open-source unsupervised algorithm for identification and fast prediction of behaviors Nat. Commun. 2021 12 5188 10.1038/s41467-021-25420-x 34465784 PMC8408193 · DOI
  12. [13]Luxem K Identifying behavioral structure from deep variational embeddings of animal motion Commun. Biol. 2022 5 1267 10.1038/s42003-022-04080-7 36400882 PMC9674640 · DOI · Paper page
  13. [14]Marques JC Lackner S Félix R Orger MB Structure of the Zebrafish locomotor repertoire revealed with unsupervised behavioral clustering Curr. Biol. 2018 28 181 195 10.1016/j.cub.2017.12.002 29307558 · DOI · Paper page
  14. [15]Todd JG Kain JS de Bivort BL Systematic exploration of unsupervised methods for mapping behavior Phys. Biol. 2017 14 015002 10.1088/1478-3975/14/1/015002 28166059 · DOI
  15. [16]Wiltschko AB Mapping sub-second structure in mouse behavior Neuron 2015 88 1121 1135 10.1016/j.neuron.2015.11.031 26687221 PMC4708087 · DOI · Paper page
  16. [17]Berman, G. J., Choi, D. M., Bialek, W. & Shaevitz, J. W. Mapping the stereotyped behaviour of freely moving fruit flies. J. R. Soc. Interface10.1098/rsif.2014.0672 (2014). PMC4233753 25142523
  17. [18]Batty, E. et al. BehaveNet: nonlinear embedding and Bayesian neural decoding of behavioral videos. in Advances in Neural Information Processing Systems 32 (eds H. Larochelle et al.) 15706–15717 (Curran Associates, 2019).
  18. [19]Costacurta, J. C. et al. Distinguishing discrete and continuous behavioral variability using warped autoregressive HMMs. in Advances in Neural Information Processing Systems 35 (eds S. Koyejo et al.) 23838–23850 (Curran Associates, 2022).
  19. [20]Jia Y Selfee, self-supervised features extraction of animal behaviors eLife 2022 11 e76218 10.7554/eLife.76218 35708244 PMC9296132 · DOI
  20. [21]Findley TM Sniff-synchronized, gradient-guided olfactory search by freely moving mice eLife 2021 10 e58523 10.7554/eLife.58523 33942713 PMC8169121 · DOI
  21. [22]Markowitz JE Spontaneous behaviour is structured by reinforcement without explicit reward Nature 2023 614 108 117 10.1038/s41586-022-05611-2 36653449 PMC9892006 · DOI
  22. [23]Markowitz JE The striatum organizes 3D behavior via moment-to-moment action selection Cell 2018 174 44 58 10.1016/j.cell.2018.04.019 29779950 PMC6026065 · DOI
  23. [24]Wiltschko AB Revealing the structure of pharmacobehavioral space through motion sequencing Nat. Neurosci. 2020 10.1038/s41593-020-00706-3 32958923 PMC7606807 · DOI
  24. [25]Lin, S. et al. Characterizing the structure of mouse behavior using motion sequencing. Preprint at https://arxiv.org/abs/2211.08497 (2022). PMC11552546 38926589 · arXiv
  25. [26]Wu, A. et al. Deep Graph Pose: a semi-supervised deep graphical model for improved animal pose tracking. in Proceedings of the 34th International Conference on Neural Information Processing Systems (Curran Associates, 2020).
  26. [27]Murphy, K. P. Machine Learning (MIT Press, 2012).
  27. [28]Linderman, S. et al. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics Vol. 54 (eds Aarti, S. et al.) 914–922 (PMLR, Proceedings of Machine Learning Research, 2017).
  28. [29]Zhang, L., Dunn, T., Marshall, J., Olveczky, B. & Linderman, S. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics Vol. 130 (eds Banerjee Arindam & Fukumizu Kenji) 2800–2808 (PMLR, Proceedings of Machine Learning Research, 2021).
  29. [30]Klibaite U Deep phenotyping reveals movement phenotypes in mouse neurodevelopmental models Mol. Autism 2022 13 12 10.1186/s13229-022-00492-8 35279205 PMC8917660 · DOI · Paper page
  30. [31]Bohnslav JP DeepEthogram, a machine learning pipeline for supervised behavior classification from raw pixels eLife 2021 10 e63377 10.7554/eLife.63377 34473051 PMC8455138 · DOI
  31. [32]Sun, J. J. et al. Caltech mouse social interactions (CalMS21) dataset. 10.22002/D1.1991 (2021). · DOI
  32. [33]Ye, S., Mathis, A. & Mathis, M. W. Panoptic animal pose estimators are zero-shot performers. Preprint at https://arxiv.org/abs/2203.07436 (2022). · arXiv
  33. [34]Marshall JD Continuous whole-body 3D kinematic recordings across the rodent behavioral repertoire Neuron 2021 109 420 437 10.1016/j.neuron.2020.11.016 33340448 PMC7864892 · DOI
  34. [35]Moore JD Hierarchy of orofacial rhythms revealed through whisking and breathing Nature 2013 497 205 210 10.1038/nature12076 23624373 PMC4159559 · DOI
  35. [36]Kurnikova A Moore JD Liao S-M Deschênes M Kleinfeld D Coordination of orofacial motor actions into exploratory behavior by rat Curr. Biol. 2017 27 688 696 10.1016/j.cub.2017.01.013 28216320 PMC5653531 · DOI
  36. [37]McAfee SS Minimally invasive highly precise monitoring of respiratory rhythm in the mouse using an epithelial temperature probe J. Neurosci. Methods 2016 263 89 94 10.1016/j.jneumeth.2016.02.007 26868731 PMC4801653 · DOI
  37. [39]Pereira TD Fast animal pose estimation using deep neural networks Nat. Methods 2019 16 117 125 10.1038/s41592-018-0234-5 30573820 PMC6899221 · DOI
  38. [40]Dan, B. et al. Lightning Pose: improved animal pose estimation via semi-supervised learning, Bayesian ensembling, and cloud-native open-source tools. Preprint at bioRxiv10.1101/2023.04.28.538703 (2023). PMC12087009 38918605
  39. [41]Batty, E. et al. In NeurIPS vol. 32 (eds H. Wallach et al.) (Curran Associates, 2019).
  40. [42]Berman GJ Bialek W Shaevitz JW Predictability and hierarchy in Drosophila behavior Proc. Natl Acad. Sci. USA 2016 113 11943 11948 10.1073/pnas.1607601113 27702892 PMC5081631 · DOI
  41. [43]Berman GJ Measuring behavior across scales BMC Biol. 2018 16 23 10.1186/s12915-018-0494-7 29475451 PMC5824583 · DOI
  42. [44]Zhou, Z., et al. UNet++: a nested U-net architecture for medical image segmentation. in (eds Stoyanov, D. et al.) Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support. DLMIA ML-CDS 2018. Lecture Notes in Computer Science, vol 11045, 3–11 (Springer International Publishing, 2018). 10.1007/978-3-030-00889-5_1 PMC7329239 32613207 · DOI
  43. [45]Sun, K., Xiao, B., Liu, D. & Wang, J. Deep high-resolution representation learning for human pose estimation. in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 5686–5696 (2019).
  44. [46]Nath T Using DeepLabCut for 3D markerless pose estimation across species and behaviors Nat. Protoc. 2019 14 2152 2176 10.1038/s41596-019-0176-0 31227823 · DOI
  45. [47]Ye, S. et al. SuperAnimal pretrained pose estimation models for behavioral analysis. Preprint at https://arxiv.org/abs/2203.07436 (2023). PMC11192880 38906853 · arXiv
  46. [48]Ackerson GA Fu K-S On state estimation in switching environments IEEE Trans. Autom. Control. 1970 15 10 17 10.1109/TAC.1970.1099359 · DOI
  47. [50]Andreella A Finos L Procrustes analysis for high-dimensional data Psychometrika 2022 87 1422 1438 10.1007/s11336-022-09859-5 35583747 PMC9636303 · DOI
  48. [52]Weinreb, C. et al. Keypoint-MoSeq: parsing behavior by linking point tracking to pose dynamics. Zenodo10.5281/zenodo.10636983 (2024). 38997595

Cited by

Sorted by citations

Authors
Weinreb, C., Pearl, J. E., Lin, S., Osman, M. A. M., Zhang, L., Annapragada, S., Conlin, E., Hoffmann, R., Makowska, S., Gillis, W. F., Jay, M., Ye, S., Mathis, A., Mathis, M. W., Pereira, T., Linderman, S. W., & Datta, S. R.
Affiliations
  • Harvard University (US)
  • Stanford University (US)
  • École Polytechnique Fédérale de Lausanne (CH)
  • Salk Institute for Biological Studies (US)
Corresponding author
  • Linderman, Scott W.
  • Datta, Sandeep Robert
ORCID
Venue
Nature Methods
Publisher
Springer Science and Business Media LLC
Published
2024-07
Volume
21
Issue
7
Pages
1329-1339
ISSN
  • 1548-7091 (ISSN-L)
  • 1548-7105
DOI
10.1038/s41592-024-02318-2
PMC
PMC11245396
OpenAlex
W4400580347
Cited by
228
References
2
Citations by year
2026: 93 · 2025: 97 · 2024: 27 · 2023: 9 · 2022: 2
FWCI
30.30
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Methods (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion)
Funders
U.S. Department of Health & Human Services | National Institutes of Health (RF1AG073625, R01NS114020, U24NS109520, F31NS113385, F31NS122155)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
  • Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurobiology and Insect Physiology Research
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Biophysics › Cell Image Analysis Techniques
Keywords
keypoint tracking, generative models, mice, fruit flies, video-based behavior analysis
MeSH terms
Machine Learning, Algorithms, Animals, Behavior, Animal, Drosophila melanogaster, Humans, Male, Movement, Video Recording, Mice

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review