Mapping Sub-Second Structure in Mouse Behavior
Open accessAbstract
Complex animal behaviors are likely built from simpler modules, but their systematic identification in mammals remains a significant challenge. Here we use depth imaging to show that 3D mouse pose dynamics are structured at the sub-second timescale. Computational modeling of these fast dynamics effectively describes mouse behavior as a series of reused and stereotyped modules with defined transition probabilities. We demonstrate this combined 3D imaging and machine learning method can be used to unmask potential strategies employed by the brain to adapt to the environment, to capture both predicted and previously hidden phenotypes caused by genetic or neural manipulations, and to systematically expose the global structure of behavior within an experiment. This work reveals that mouse body language is built from identifiable components and is organized in a predictable fashion; deciphering this language establishes an objective framework for characterizing the influence of environmental cues, genes and neural activity on behavior.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
MoSeq and keypoint-MoSeq use AR-HMM, whereas VAME uses deep recurrent neural networks and variational autoencoders, providing two well-established complementary approaches.
“MoSeq6 and keypoint-MoSeq35 use a statistical framework known as autoregressive hidden Markov model (AR-HMM),6,50,51 whereas VAME1,7 uses deep recurrent neural networks and variational autoencoders, providing two well-established complementary approaches.7,52,53”
From Machine learning reveals prominent spontaneous behavioral changes and treatment efficacy in humanized and transgenic Alzheimer's disease models (2024) · cites this paper as [6]
Recent developments in behavioral analysis permit sophisticated examination of behavioral phenotypes based on unsupervised machine learning.
“Finally, recent developments in behavioral analysis [25][26][27] permit sophisticated examination of behavioral phenotypes based on unsupervised machine learning.”
From Projection-specific deficits in synaptic transmission in adult Sapap3-knockout mice (2020) · cites this paper as [25]
MoSeq is a machine learning-based approach but not per se designed to replace experimenter scoring of behavior, since it automatically segments behavior into dynamical regimes (syllables) which may not fit intuitive description by humans.
“MoSeq is also a machine learning-based approach, but not per se designed to replace experimenter scoring of behavior, since it automatically segments behavior into dynamical regimes, i.e., syllables, which may not fit intuitive description by humans [25].”
From Projection-specific deficits in synaptic transmission in adult Sapap3-knockout mice (2020) · cites this paper as [25]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970 · see citing works
The Elements of Statistical Learning: Data Mining, Inference, and Prediction
Lu, Z. Q. J. (2010) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2010.00646_6.x · Cited by 12,904 · see citing works
The Integrative Action of the Nervous System.
Sherrington, C. (1907) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-190734120-00021 · Cited by 4,136 · see citing works
The problem of serial order in behavior
Lashley, K. S. (1951) · MPG.PuRe (Max Planck Society) · Cited by 2,734 · see citing works
Rowan, W., & Tinbergen, N. (1952) · Bird-Banding · doi:10.2307/4510368 · Cited by 2,715 · see citing works
Chemotaxis in Escherichia coli analysed by Three-dimensional Tracking
Berg, H. C., & Brown, D. A. (1972) · Nature · doi:10.1038/239500a0 · Cited by 2,239 · see citing works
The intrinsic factors in the act of progression in the mammal.
Brown, T. G. (1911) · Proceedings of the Royal Society of London Series B Containing Papers of a Biological Character · doi:10.1098/rspb.1911.0077 · Cited by 1,215 · see citing works
Cerebral hemisphere regulation of motivated behavior
Swanson, L. W. (2000) · Brain Research · doi:10.1016/s0006-8993(00)02905-x · Cited by 846 · see citing works
A circuit for navigation in <i>Caenorhabditis elegans</i>
Gray, J. M., Hill, J. J., & Bargmann, C. I. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0409009101 · Cited by 811 · see citing works
JAABA: Interactive machine learning for automatic annotation of animal behavior
Kabra, M., Robie, A. A., Rivera-Alba, M., Branson, S., & Branson, K. (2013) · Nature Methods · doi:10.1038/nmeth.2281 · Cited by 644 · see citing works
The construction of movement by the spinal cord
Tresch, M. C., Saltiel, P., & Bizzi, E. (1999) · Nature Neuroscience · doi:10.1038/5721 · Cited by 620 · see citing works
The case for and against muscle synergies
Tresch, M. C., & Jarc, A. (2009) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2009.09.002 · Cited by 568 · see citing works
Mapping the stereotyped behaviour of freely moving fruit flies
Berman, G. J., Choi, D. M., Bialek, W., & Shaevitz, J. W. (2014) · Journal of The Royal Society Interface · doi:10.1098/rsif.2014.0672 · Cited by 565 · see citing works
The Fundamental Role of Pirouettes in Caenorhabditis elegans Chemotaxis
Pierce-Shimomura, J. T., Morse, T. M., & Lockery, S. R. (1999) · Journal of Neuroscience · doi:10.1523/jneurosci.19-21-09557.1999 · Cited by 536 · see citing works
Dimensionality and Dynamics in the Behavior of C. elegans
Stephens, G. J., Johnson-Kerner, B., Bialek, W., & Ryu, W. S. (2008) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000028 · Cited by 509 · see citing works
Motor primitives in vertebrates and invertebrates
Flash, T., & Hochner, B. (2005) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2005.10.011 · Cited by 508 · see citing works
Toward a Science of Computational Ethology
Anderson, D. J., & Perona, P. (2014) · Neuron · doi:10.1016/j.neuron.2014.09.005 · Cited by 496 · see citing works
Songs to syntax: the linguistics of birdsong
Berwick, R. C., Okanoya, K., Beckers, G. J. L., & Bolhuis, J. J. (2011) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2011.01.002 · Cited by 467 · see citing works
The Mouse Genome Database (MGD): facilitating mouse as a model for human biology and disease
Eppig, J. T., Blake, J. A., Bult, C. J., Kadin, J. A., Richardson, J. E., & The Mouse Genome Database Group (2014) · Nucleic Acids Research · doi:10.1093/nar/gku967 · Cited by 358 · see citing works
Cortico-cortical projections in mouse visual cortex are functionally target specific
Glickfeld, L. L., Andermann, M. L., Bonin, V., & Reid, R. C. (2013) · Nature Neuroscience · doi:10.1038/nn.3300 · Cited by 293 · see citing works
Automated home-cage behavioural phenotyping of mice
Jhuang, H., Garrote, E., Yu, X., Khilnani, V., Poggio, T., Steele, A. D., & Serre, T. (2010) · Nature Communications · doi:10.1038/ncomms1064 · Cited by 274 · see citing works
The EthoVision video tracking system—A tool for behavioral phenotyping of transgenic mice
Spink, A., Tegelenbosch, R. A. J., Buma, M., & Noldus, L. P. J. J. (2001) · Physiology & Behavior · doi:10.1016/s0031-9384(01)00530-3 · Cited by 271 · see citing works
Wallace, K. J., & Rosen, J. B. (2000) · Behavioral Neuroscience · doi:10.1037/0735-7044.114.5.912 · Cited by 264 · see citing works
Computerized video analysis of social interactions in mice
de Chaumont, F., Coura, R., Serreau, P., Cressant, A., Chabout, J., Granon, S., & Olivo‐Marín, J. (2012) · Nature Methods · doi:10.1038/nmeth.1924 · Cited by 237 · see citing works
Weissbrod, A., Shapiro, A., Vasserman, G., Edry, L., Dayan, M., Yitzhaky, A., Hertzberg, L., Feinerman, O., & Kimchi, T. (2013) · Nature Communications · doi:10.1038/ncomms3018 · Cited by 235 · see citing works
New perspectives on spinal motor systems
Bizzi, E., Tresch, M. C., Saltiel, P., & d’Avella, A. (2000) · Nature reviews. Neuroscience · doi:10.1038/35039000 · Cited by 230 · see citing works
Discovery of brainwide neural-behavioral maps via multiscale unsupervised structure learning
Vogelstein, J. T., Park, Y., Ohyama, T., Kerr, R. A., Truman, J. W., Priebe, C. E., & Zlatic, M. (2014) · Science · doi:10.1126/science.1250298 · Cited by 218 · see citing works
André, É., Conquet, F., Steinmayr, M., Stratton, S. C., Porciatti, V., & Becker-André, M. (1998) · The EMBO Journal · doi:10.1093/emboj/17.14.3867 · Cited by 214 · see citing works
Natural syntax rules control action sequence of rats
Berridge, K. C., Fentress, J. C., & Parr, H. (1987) · Behavioural Brain Research · doi:10.1016/0166-4328(87)90242-7 · Cited by 202 · see citing works
Garrity, P. A., Goodman, M. B., Samuel, A. D., & Sengupta, P. (2010) · Genes & Development · doi:10.1101/gad.1953710 · Cited by 197 · see citing works
Components and patterns in the behaviour of the nematode Caenorhabditis elegans
Croll, N. A. (1975) · Journal of Zoology · doi:10.1111/j.1469-7998.1975.tb03191.x · Cited by 183 · see citing works
Blackith, R. E., & Colgan, P. (1979) · Biometrics · doi:10.2307/2529956 · Cited by 175 · see citing works
Statistical discrimination of natural modes of motion in rat exploratory behavior
Drai, D., Benjamini, Y., & Golani, I. (2000) · Journal of Neuroscience Methods · doi:10.1016/s0165-0270(99)00194-6 · Cited by 164 · see citing works
TMT-induced autonomic and behavioral changes and the neural basis of its processing
Fendt, M., Endres, T., Lowry, C. A., Apfelbach, R., & McGregor, I. S. (2005) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2005.04.018 · Cited by 158 · see citing works
Grammar of a Movement Sequence in Inbred Mice
Fentress, J. C., & STILWELL, F. P. (1973) · Nature · doi:10.1038/244052a0 · Cited by 118 · see citing works
Basal ganglia neural mechanisms of natural movement sequences
Aldridge, J. W., Berridge, K. C., & Rosen, A. R. (2004) · Canadian Journal of Physiology and Pharmacology · doi:10.1139/y04-061 · Cited by 107 · see citing works
Behavioral phenotyping of rodents.
Crawley, J. N. (2003) · PubMed · Cited by 102 · see citing works
Long-range Order in Canary Song
Markowitz, J. E., Ivie, E. J., Kligler, L., & Gardner, T. J. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003052 · Cited by 99 · see citing works
Gómez-Marín, À., Partoune, N., Stephens, G. J., & Louis, M. (2012) · PLoS ONE · doi:10.1371/journal.pone.0041642 · Cited by 81 · see citing works
Linked Control of Syllable Sequence and Phonology in Birdsong
Wohlgemuth, M. J., Sober, S. J., & Brainard, M. S. (2010) · Journal of Neuroscience · doi:10.1523/jneurosci.2690-10.2010 · Cited by 79 · see citing works
Liu, H., Kim, S., Fu, Y., Wu, X., Ng, L., Swaroop, A., & Forrest, D. (2013) · Nature Communications · doi:10.1038/ncomms2793 · Cited by 71 · see citing works
Masana, M. I., Sumaya, I. C., Becker-André, M., & Dubocovich, M. L. (2007) · American Journal of Physiology-Regulatory, Integrative and Comparative Physiology · doi:10.1152/ajpregu.00687.2006 · Cited by 68 · see citing works
From Modes to Movement in the Behavior of Caenorhabditis elegans
Stephens, G. J., Johnson-Kerner, B., Bialek, W., & Ryu, W. S. (2010) · PLoS ONE · doi:10.1371/journal.pone.0013914 · Cited by 65 · see citing works
Yang, T.-H. O., Tsai, M., Yen, C., & Lin, T. (2011) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2011.07.019 · Cited by 50 · see citing works
Garcia, V. A., Crispim, C. F., & Marino–Neto, J. (2010) · Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society · doi:10.1109/iembs.2010.5627131 · Cited by 11 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Spontaneous behaviors drive multidimensional, brainwide activity
Stringer, C., Pachitariu, M., Steinmetz, N., Reddy, C. B., Carandini, M., & Harris, K. D. (2019) · Science · doi:10.1126/science.aav7893 · Cited by 1,510 · see citing works
Neural circuit mechanisms of social behavior
Chen, P., & Hong, W. (2018) · Neuron · doi:10.1016/j.neuron.2018.02.026 · Cited by 527 · see citing works
DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning
Graving, J. M., Chae, D., Naik, H., Li, L., Koger, B., Costelloe, B. R., & Couzin, I. D. (2019) · eLife · doi:10.7554/elife.47994 · Cited by 547 · see citing works
Deep learning tools for the measurement of animal behavior in neuroscience
Mathis, M. W., & Mathis, A. (2020) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2019.10.008 · Cited by 450 · see citing works
Computational Neuroethology: A Call to Action
Datta, S. R., Anderson, D. J., Branson, K., Perona, P., & Leifer, A. (2019) · Neuron · doi:10.1016/j.neuron.2019.09.038 · Cited by 453 · see citing works
Self-citationHypothalamic CRH neurons orchestrate complex behaviours after stress
Füzesi, T., Daviu, N., Wamsteeker Cusulin, J. I., Bonin, R. P., & Bains, J. S. (2016) · Nature Communications · doi:10.1038/ncomms11937 · Cited by 316 · see citing works
Structure of the Zebrafish Locomotor Repertoire Revealed with Unsupervised Behavioral Clustering
Marques, J. C., Lackner, S., Félix, R., & Orger, M. B. (2018) · Current Biology · doi:10.1016/j.cub.2017.12.002 · Cited by 245 · see citing works
Behavioral methods to study anxiety in rodents
Lezak, K. R., Missig, G., & Carlezon Jr, W. A. (2017) · Dialogues in Clinical Neuroscience · doi:10.31887/dcns.2017.19.2/wcarlezon · Cited by 290 · see citing works
Mice alternate between discrete strategies during perceptual decision-making
Ashwood, Z. C., Roy, N. A., Stone, I. R., Laboratory, T. I. B., Urai, A. E., Churchland, A. K., Pouget, A., & Pillow, J. W. (2022) · Nature Neuroscience · doi:10.1038/s41593-021-01007-z · Cited by 229 · see citing works
A Primer on Motion Capture with Deep Learning: Principles, Pitfalls, and Perspectives
Mathis, A., Schneider, S., Lauer, J., & Mathis, M. W. (2020) · Neuron · doi:10.1016/j.neuron.2020.09.017 · Cited by 231 · see citing works
The basal ganglia control the detailed kinematics of learned motor skills
Dhawale, A. K., Wolff, S. B. E., Ko, R., & Ölveczky, B. P. (2021) · Nature Neuroscience · doi:10.1038/s41593-021-00889-3 · Cited by 194 · see citing works
Algorithms for Olfactory Search across Species
Baker, K. L., Dickinson, M., Findley, T. M., Gire, D. H., Louis, M., Suver, M. P., Verhagen, J. V., Nagel, K. I., & Smear, M. C. (2018) · The Journal of Neuroscience · doi:10.1523/jneurosci.1668-18.2018 · Cited by 177 · see citing works
Efficient cortical coding of 3D posture in freely behaving rats
Mimica, B., Dunn, B. A., Tombaz, T., Bojja, V. P. T. N. C. S., & Whitlock, J. R. (2018) · Science · doi:10.1126/science.aau2013 · Cited by 124 · see citing works
RORβ Spinal Interneurons Gate Sensory Transmission during Locomotion to Secure a Fluid Walking Gait
Koch, S. C., Del Barrio, M. G., Dalet, A., Gatto, G., Günther, T., Zhang, J., Seidler, B., Saur, D., Schüle, R., & Goulding, M. (2017) · Neuron · doi:10.1016/j.neuron.2017.11.011 · Cited by 102 · see citing works
Measuring behavior in the home cage: study design, applications, challenges, and perspectives
Grieco, F., Bernstein, B. J., Biemans, B., Bikovski, L., Burnett, C. J., Cushman, J. D., van Dam, E. A., Fry, S. A., Richmond-Hacham, B., Homberg, J. R., et al. (2021) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2021.735387 · Cited by 109 · see citing works
Tastekin, I., Khandelwal, A., Tadres, D., Fessner, N. D., Truman, J. W., Zlatic, M., Cardona, A., & Louis, M. (2018) · eLife · doi:10.7554/elife.38740 · Cited by 78 · see citing works
Clemens, J., Coen, P., Roemschied, F. A., Pereira, T. D., Mazumder, D., Aldarondo, D. E., Pacheco, D. A., & Murthy, M. (2018) · Current Biology · doi:10.1016/j.cub.2018.06.011 · Cited by 83 · see citing works
Decision-making behaviors: weighing ethology, complexity, and sensorimotor compatibility
Juavinett, A. L., Erlich, J. C., & Churchland, A. K. (2018) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2017.11.001 · Cited by 82 · see citing works
Spinal Interneurons as Gatekeepers to Neuroplasticity after Injury or Disease
Zholudeva, L. V., Abraira, V. E., Satkunendrarajah, K., McDevitt, T. C., Goulding, M. D., Magnuson, D. S. K., & Lane, M. A. (2021) · The Journal of Neuroscience · doi:10.1523/jneurosci.1654-20.2020 · Cited by 74 · see citing works
Self-citationDynamic structure of locomotor behavior in walking fruit flies
Katsov, A. Y., Freifeld, L., Horowitz, M., Kuehn, S., & Clandinin, T. R. (2017) · eLife · doi:10.7554/elife.26410 · Cited by 43 · see citing works
From motor planning to execution: a sensorimotor loop perspective
Bizzi, E., & Ajemian, R. (2020) · Journal of Neurophysiology · doi:10.1152/jn.00715.2019 · Cited by 49 · see citing works
Behavioral decomposition reveals rich encoding structure employed across neocortex in rats
Mimica, B., Tombaz, T., Battistin, C., Fuglstad, J. G., Dunn, B. A., & Whitlock, J. R. (2023) · Nature Communications · doi:10.1038/s41467-023-39520-3 · Cited by 45 · see citing works
Projection-specific deficits in synaptic transmission in adult Sapap3-knockout mice
Hadjas, L. C., Schartner, M. M., Cand, J., Creed, M. C., Pascoli, V., Lüscher, C., & Simmler, L. D. (2020) · Neuropsychopharmacology · doi:10.1038/s41386-020-0747-3 · Cited by 46 · see citing works
Miller, S. R., Luxem, K., Lauderdale, K., Nambiar, P., Honma, P. S., Ly, K. K., Bangera, S., Bullock, M., Shin, J., Kaliss, N., et al. (2024) · Cell Reports · doi:10.1016/j.celrep.2024.114870 · Cited by 24 · see citing works
“MoSeq 6 and keypoint-MoSeq 35 use a statistical framework known as autoregressive hidden Markov model (AR-HMM), 6,50,51 whereas VAME 1,7 uses deep recurrent neural networks and variational autoencoders, providing two well-established complementary approaches.”
Schut, E. H. S., Alonso, A., Smits, S., Khamassi, M., Samanta, A., Negwer, M., Kasri, N. N., Navarro Lobato, I., & Genzel, L. (2020) · Neurobiology of Learning and Memory · doi:10.1016/j.nlm.2020.107265 · Cited by 8 · see citing works
- Authors
- Wiltschko, A. B., Johnson, M. J., Iurilli, G., Peterson, R. E., Katon, J. M., Pashkovski, S. L., Abraira, V. E., Adams, R. P., & Datta, S. R.
- Affiliations
- Harvard University (US)
- Corresponding author
- Datta, Sandeep Robert
- ORCID
- Venue
- Neuron
- Publisher
- Elsevier BV
- Published
- 2015-12
- Volume
- 88
- Issue
- 6
- Pages
- 1121-1135
- ISSN
- 0896-6273 (ISSN-L)
- PMC
- PMC4708087
- OpenAlex
- W2219858099
- Cited by
- 760
- References
- 45
- Citations by year
- 2026: 76 · 2025: 80 · 2024: 112 · 2023: 98 · 2022: 81 · 2021: 91 · 2020: 71 · 2019: 58 · 2018: 40 · 2017: 40 · 2016: 11
- FWCI
- 26.28
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- Neuron (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Rutgers University Community Repository (Rutgers University) (submittedVersion)
- Funders
- National Institutes of Health (1dp2od007109-01)
- National Institute on Deafness and Other Communication Disorders (R01DC011558, R011DC011558)
- National Institute of Neurological Disorders and Stroke (U01NS094191)
- Eunice Kennedy Shriver National Institute of Child Health and Human Development (U54HD090255, P30HD018655, P30HD-18655)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Life Sciences › Neuroscience › Cognitive Neuroscience › Motor Control and Adaptation
- Keywords
- mouse behavior, transition probabilities, computational modeling of behavior, depth imaging, behavioral phenotyping, genetic engineering, neural manipulation, environmental cues
- MeSH terms
- Machine Learning, Animals, Behavior, Animal, Computer Simulation, Kinesics, Male, Mice, Inbred C57BL, Mice, Transgenic, Imaging, Three-Dimensional, Mice, Optogenetics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
