DeepLabCut: markerless pose estimation of user-defined body parts with deep learning
Open accessAbstract
Quantifying behavior is crucial for many applications in neuroscience. Videography provides easy methods for the observation and recording of animal behavior in diverse settings, yet extracting particular aspects of a behavior for further analysis can be highly time consuming. In motor control studies, humans or other animals are often marked with reflective markers to assist with computer-based tracking, but markers are intrusive, and the number and location of the markers must be determined a priori. Here we present an efficient method for markerless pose estimation based on transfer learning with deep neural networks that achieves excellent results with minimal training data. We demonstrate the versatility of this framework by tracking various body parts in multiple species across a broad collection of behaviors. Remarkably, even when only a small number of frames are labeled (~200), the algorithm achieves excellent tracking performance on test frames that is comparable to human accuracy. Using a deep learning approach to track user-defined body parts during various behaviors across multiple species, the authors show that their toolbox, called DeepLabCut, can achieve human accuracy with only a few hundred frames of training data.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recent developments in behavioral analysis permit sophisticated examination of behavioral phenotypes based on unsupervised machine learning.
“Finally, recent developments in behavioral analysis [25][26][27] permit sophisticated examination of behavioral phenotypes based on unsupervised machine learning.”
From Projection-specific deficits in synaptic transmission in adult Sapap3-knockout mice (2020) · cites this paper as [26]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · see citing works
Fiji: an open-source platform for biological-image analysis
Schindelin, J., Arganda-Carreras, I., Frise, E., Kaynig, V., Longair, M., Pietzsch, T., Preibisch, S., Rueden, C., Saalfeld, S., Schmid, B., et al. (2012) · Nature Methods · doi:10.1038/nmeth.2019 · Cited by 66,890 · see citing works
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,989 · see citing works
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653 · see citing works
(2016) · Cited by 11,679 · see citing works
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570 · see citing works
Biomechanics and motor control of human movement
Winter, D. A. (1990) · Choice Reviews Online · doi:10.5860/choice.28-2135 · Cited by 8,440 · see citing works
Adaptive background mixture models for real-time tracking
Stauffer, C., & Grimson, W. E. L. (2003) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.1999.784637 · Cited by 6,925 · see citing works
The co-ordination and regulation of movements
Bernshteĭn, N. A. (1967) · Brain Research · doi:10.1016/0006-8993(69)90278-9 · Cited by 6,389 · see citing works
Biomechanics and Motor Control of Human Movement
Winter, D. A. (2009) · Wiley eBooks · doi:10.1002/9780470549148 · Cited by 6,112 · see citing works
On aims and methods of ethology
Tinbergen, N. (1963) · Zeitschrift für Tierpsychologie · doi:10.1111/j.1439-0310.1963.tb01161.x · Cited by 4,551 · see citing works
DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., & Darrell, T. (2013) · arXiv · doi:10.48550/arxiv.1310.1531 · Cited by 4,019 · see citing works
The co-ordination and regulation of movements
Foley, J. (1968) · Journal of the Neurological Sciences · doi:10.1016/0022-510x(68)90166-4 · Cited by 3,431 · see citing works
Beyond the Kalman filter particle filters for tracking applications
Ristić, B., Arulampalam, S., & Gordon, N. (2004) · Cited by 3,400 · see citing works
DeepPose: Human Pose Estimation via Deep Neural Networks
Toshev, A., & Szegedy, C. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.214 · Cited by 3,382 · see citing works
2D Human Pose Estimation: New Benchmark and State of the Art Analysis
Andriluka, M., Pishchulin, L., Gehler, P., & Schiele, B. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.471 · Cited by 2,699 · see citing works
Pictorial Structures for Object Recognition
Felzenszwalb, P. F., & Huttenlocher, D. P. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000042934.15159.49 · Cited by 2,167 · see citing works
Neuroscience Needs Behavior: Correcting a Reductionist Bias
Krakauer, J. W., Ghazanfar, A. A., Gomez-Marin, A., MacIver, M. A., & Poeppel, D. (2017) · Neuron · doi:10.1016/j.neuron.2016.12.041 · Cited by 1,358 · see citing works
DeepCut: Joint Subset Partition and Labeling for Multi Person Pose Estimation
Pishchulin, L., Insafutdinov, E., Tang, S., Andres, B., Andriluka, M., Gehler, P., & Schiele, B. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.533 · Cited by 1,090 · see citing works
idTracker: tracking individuals in a group by automatic identification of unmarked animals
Pérez‐Escudero, A., Vicente-Page, J., Hinz, R. C., Arganda, S., & de Polavieja, G. G. (2014) · Nature Methods · doi:10.1038/nmeth.2994 · Cited by 775 · see citing works
Integration of optogenetics with complementary methodologies in systems neuroscience
Kim, C. K., Adhikari, A., & Deisseroth, K. (2017) · Nature reviews. Neuroscience · doi:10.1038/nrn.2017.15 · Cited by 690 · see citing works
Detect to Track and Track to Detect
Feichtenhofer, C., Pinz, A., & Zisserman, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.330 · Cited by 595 · see citing works
Dollár, P., Welinder, P., & Perona, P. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5540094 · Cited by 527 · see citing works
Toward a Science of Computational Ethology
Anderson, D. J., & Perona, P. (2014) · Neuron · doi:10.1016/j.neuron.2014.09.005 · Cited by 496 · see citing works
Automated image-based tracking and its application in ecology
Dell, A. I., Bender, J. A., Branson, K., Couzin, I. D., de Polavieja, G. G., Noldus, L. P. J. J., Pérez-Escudero, A., Perona, P., Straw, A. D., Wikelski, M., et al. (2014) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2014.05.004 · Cited by 492 · see citing works
Neuronal Representation of Social Information in the Medial Amygdala of Awake Behaving Mice
Li, Y., Mathis, A., Grewe, B. F., Osterhout, J. A., Ahanonu, B., Schnitzer, M. J., Murthy, V. N., & Dulac, C. (2017) · Cell · doi:10.1016/j.cell.2017.10.015 · Cited by 291 · see citing works
Self-citationConvolutional Pose MachinesPreprint
Wei, S.-E., Ramakrishna, V., Kanade, T., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.00134 · Cited by 288 · see citing works
Big behavioral data: psychology, ethology and the foundations of neuroscience
Gómez-Marín, À., Paton, J. J., Kampff, A. R., Costa, R. M., & Mainen, Z. F. (2014) · Nature Neuroscience · doi:10.1038/nn.3812 · Cited by 286 · see citing works
A hitchhiker's guide to behavioral analysis in laboratory rodents
Sousa, N., Almeida, O. F. X., & Wotjak, C. T. (2006) · Genes Brain & Behavior · doi:10.1111/j.1601-183x.2006.00228.x · Cited by 273 · see citing works
Decoding Complete Reach and Grasp Actions from Local Primary Motor Cortex Populations
Vargas-Irwin, C. E., Shakhnarovich, G., Yadollahpour, P., Mislow, J. M. K., Black, M. J., & Donoghue, J. P. (2010) · Journal of Neuroscience · doi:10.1523/jneurosci.5443-09.2010 · Cited by 270 · see citing works
ArtTrack: Articulated Multi-Person Tracking in the Wild
Insafutdinov, E., Andriluka, M., Pishchulin, L., Tang, S., Levinkov, E., Andres, B., & Schiele, B. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.142 · Cited by 246 · see citing works
Cortex commands the performance of skilled movement
Guo, J.-Z., Graves, A., Guo, W. W., Zheng, J.-H., Lee, A., Rodríguez-González, J., Li, N., Macklin, J., Phillips, J. W., Mensh, B. D., et al. (2015) · eLife · doi:10.7554/elife.10774 · Cited by 239 · see citing works
Computerized video analysis of social interactions in mice
de Chaumont, F., Coura, R., Serreau, P., Cressant, A., Chabout, J., Granon, S., & Olivo‐Marín, J. (2012) · Nature Methods · doi:10.1038/nmeth.1924 · Cited by 238 · see citing works
Wenger, N., Moraud, E. M., Raspopovic, S., Bonizzato, M., DiGiovanna, J., Musienko, P., Morari, M., Micera, S., & Courtine, G. (2014) · Science Translational Medicine · doi:10.1126/scitranslmed.3008325 · Cited by 209 · see citing works
Somatosensory Cortex Plays an Essential Role in Forelimb Motor Adaptation in Mice
Mathis, M. W., Mathis, A., & Uchida, N. (2017) · Neuron · doi:10.1016/j.neuron.2017.02.049 · Cited by 193 · see citing works
Self-citationMeasuring behavior across scales
Berman, G. J. (2018) · BMC Biology · doi:10.1186/s12915-018-0494-7 · Cited by 184 · see citing works
Machado, A. S., Darmohray, D. M., Fayad, J., Marques, H. G., & Carey, M. R. (2015) · eLife · doi:10.7554/elife.07892 · Cited by 183 · see citing works
SEE: a tool for the visualization and analysis of rodent exploratory behavior
Drai, D., & Golani, I. (2001) · Neuroscience & Biobehavioral Reviews · doi:10.1016/s0149-7634(01)00022-7 · Cited by 149 · see citing works
Ota, N., Gahr, M. L., & Soma, M. (2015) · Scientific Reports · doi:10.1038/srep16614 · Cited by 139 · see citing works
Machine vision methods for analyzing social interactions
Robie, A. A., Seagraves, K. M., Egnor, S. E. R., & Branson, K. M. (2017) · Journal of Experimental Biology · doi:10.1242/jeb.142281 · Cited by 123 · see citing works
Rats track odour trails accurately using a multi-layered strategy with near-optimal sampling
Khan, A. G., Sarangi, M., & Bhalla, U. (2012) · Nature Communications · doi:10.1038/ncomms1712 · Cited by 122 · see citing works
DeeperCut: A Deeper, Stronger, and Faster Multi-Person Pose Estimation ModelPreprint
Insafutdinov, E., Pishchulin, L., Andres, B., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.03170 · Cited by 113 · see citing works
Ben‐Shaul, Y. (2017) · BMC Biology · doi:10.1186/s12915-017-0377-3 · Cited by 93 · see citing works
A 3D-Video-Based Computerized Analysis of Social and Sexual Interactions in Rats
Matsumoto, J., Urakawa, S., Takamura, Y., Malcher‐Lopes, R., Hori, E., Tomaz, C. A. B., Ono, T., & Nishijo, H. (2013) · PLoS ONE · doi:10.1371/journal.pone.0078460 · Cited by 86 · see citing works
Gómez-Marín, À., Partoune, N., Stephens, G. J., & Louis, M. (2012) · PLoS ONE · doi:10.1371/journal.pone.0041642 · Cited by 81 · see citing works
Uhlmann, V., Ramdya, P. P., Delgado-Gonzalo, R., Benton, R., & Unser, M. A. (2017) · PLoS ONE · doi:10.1371/journal.pone.0173433 · Cited by 41 · see citing works
A Markerless 3D Computerized Motion Capture System Incorporating a Skeleton Model for Monkeys
Nakamura, T., Matsumoto, J., Nishimaru, H., Bretas, R. V., Takamura, Y., Hori, E., Ono, T., & Nishijo, H. (2016) · PLoS ONE · doi:10.1371/journal.pone.0166154 · Cited by 38 · see citing works
Superpixels based marker tracking vs. hue thresholding in rodent biomechanics application
Maghsoudi, O. H., Tabrizi, A. V., Robertson, B., et al. (2017) · 2018 52nd Asilomar Conference on Signals, Systems, and Computers · doi:10.1109/acssc.2017.8335168 · Cited by 24 · see citing works
Capturing Motion and Depth Before Cinematography
Wade, N. J. (2015) · Journal of the History of the Neurosciences · doi:10.1080/0964704x.2015.1070029 · Cited by 17 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
High-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450 · see citing works
Spontaneous behaviors drive multidimensional, brainwide activity
Stringer, C., Pachitariu, M., Steinmetz, N., Reddy, C. B., Carandini, M., & Harris, K. D. (2019) · Science · doi:10.1126/science.aav7893 · Cited by 1,510 · see citing works
Deep learning for cellular image analysis
Moen, E., Bannon, D., Kudo, T., Graf, W., Covert, M., & Van Valen, D. (2019) · Nature Methods · doi:10.1038/s41592-019-0403-1 · Cited by 1,225 · see citing works
A deep learning framework for neuroscience
Richards, B. A., Lillicrap, T. P., Beaudoin, P., Bengio, Y., Bogacz, R., Christensen, A., Clopath, C., Costa, R. P., de Berker, A., Ganguli, S., et al. (2019) · Nature Neuroscience · doi:10.1038/s41593-019-0520-2 · Cited by 1,027 · see citing works
Perspectives in machine learning for wildlife conservation
Tuia, D., Kellenberger, B., Beery, S., Costelloe, B. R., Zuffi, S., Risse, B., Mathis, A., Mathis, M. W., van Langevelde, F., Burghardt, T., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-27980-y · Cited by 740 · see citing works
Self-citationDeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning
Graving, J. M., Chae, D., Naik, H., Li, L., Koger, B., Costelloe, B. R., & Couzin, I. D. (2019) · eLife · doi:10.7554/elife.47994 · Cited by 547 · see citing works
Oxytocin, Neural Plasticity, and Social Behavior
Froemke, R. C., & Young, L. J. (2021) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-102320-102847 · Cited by 487 · see citing works
Deep learning tools for the measurement of animal behavior in neuroscience
Mathis, M. W., & Mathis, A. (2020) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2019.10.008 · Cited by 450 · see citing works
Self-citationComputational Neuroethology: A Call to Action
Datta, S. R., Anderson, D. J., Branson, K., Perona, P., & Leifer, A. (2019) · Neuron · doi:10.1016/j.neuron.2019.09.038 · Cited by 453 · see citing works
Anipose: A toolkit for robust markerless 3D pose estimation
Karashchuk, P., Rupp, K. L., Dickinson, E. S., Walling-Bell, S., Sanders, E., Azim, E., Brunton, B. W., & Tuthill, J. C. (2021) · Cell Reports · doi:10.1016/j.celrep.2021.109730 · Cited by 324 · see citing works
Hippocampal CA2 sharp-wave ripples reactivate and promote social memory
Oliva, A., Fernández-Ruiz, A., Leroy, F., & Siegelbaum, S. A. (2020) · Nature · doi:10.1038/s41586-020-2758-y · Cited by 268 · see citing works
Internal state dynamics shape brainwide activity and foraging behaviour
Marques, J. C., Li, M., Schaak, D., Robson, D. N., & Li, J. M. (2020) · Nature · doi:10.1038/s41586-019-1858-z · Cited by 243 · see citing works
The future of rodent models in depression research
Gururajan, A., Reif, A., Cryan, J. F., & Slattery, D. A. (2019) · Nature Reviews Neuroscience · doi:10.1038/s41583-019-0221-6 · Cited by 269 · see citing works
Dopamine release in the nucleus accumbens core signals perceived saliency
Kutlu, M. G., Zachry, J. E., Melugin, P. R., Cajigas, S. A., Chevee, M. F., Kelly, S. J., Kutlu, B., Tian, L., Siciliano, C. A., & Calipari, E. S. (2021) · Current Biology · doi:10.1016/j.cub.2021.08.052 · Cited by 238 · see citing works
A Primer on Motion Capture with Deep Learning: Principles, Pitfalls, and Perspectives
Mathis, A., Schneider, S., Lauer, J., & Mathis, M. W. (2020) · Neuron · doi:10.1016/j.neuron.2020.09.017 · Cited by 231 · see citing works
Self-citationAn action potential initiation mechanism in distal axons for the control of dopamine release
Liu, C., Cai, X., Ritzau-Jost, A., Kramer, P. F., Li, Y., Khaliq, Z. M., Hallermann, S., & Kaeser, P. S. (2022) · Science · doi:10.1126/science.abn0532 · Cited by 213 · see citing works
The basal ganglia control the detailed kinematics of learned motor skills
Dhawale, A. K., Wolff, S. B. E., Ko, R., & Ölveczky, B. P. (2021) · Nature Neuroscience · doi:10.1038/s41593-021-00889-3 · Cited by 194 · see citing works
Magnetoelectric materials for miniature, wireless neural stimulation at therapeutic frequencies
Singer, A., Dutta, S., Lewis, E., Chen, Z., Chen, J. C., Verma, N., Avants, B., Feldman, A. K., O’Malley, J., Beierlein, M., et al. (2020) · Neuron · doi:10.1016/j.neuron.2020.05.019 · Cited by 202 · see citing works
Solié, C., Girard, B., Righetti, B., Tapparel, M., & Bellone, C. (2022) · Nature Neuroscience · doi:10.1038/s41593-021-00972-9 · Cited by 186 · see citing works
Brainstem neurons that command mammalian locomotor asymmetries
Cregg, J. M., Leiras, R., Montalant, A., Wanken, P., Wickersham, I. R., & Kiehn, O. (2020) · Nature Neuroscience · doi:10.1038/s41593-020-0633-7 · Cited by 154 · see citing works
Three-D Safari: Learning to Estimate Zebra Pose, Shape, and Texture From Images “In the Wild”
Zuffi, S., Kanazawa, A., Berger-Wolf, T., & Black, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00546 · Cited by 139 · see citing works
“[20] use deep learning to track animal keypoints defined by a user on infrared images captured in the lab.”
BackgroundThe manifold structure of limb coordination in walking Drosophila
DeAngelis, B. D., Zavatone-Veth, J. A., & Clark, D. A. (2019) · eLife · doi:10.7554/elife.46409 · Cited by 104 · see citing works
“In Drosophila, explorations of VNC circuits are increasingly made possible by automated behavioral tracking (Branson et al., 2009; Mathis et al., 2018; Pereira et al., 2019; Williamson et al., 2018), by preparations to visualize neural activity in the VNC (Chen et al.”
MethodResultInfluentialTwo brain pathways initiate distinct forward walking programs in Drosophila
Bidaye, S. S., Laturney, M., Chang, A. K., Liu, Y., Bockemühl, T., Büschges, A., & Scott, K. (2020) · Neuron · doi:10.1016/j.neuron.2020.07.032 · Cited by 109 · see citing works
Wireless closed-loop optogenetics across the entire dorsoventral spinal cord in mice
Kathe, C., Michoud, F., Schönle, P., Rowald, A., Brun, N., Ravier, J., Furfaro, I., Paggi, V., Kim, K., Soloukey, S., et al. (2022) · Nature Biotechnology · doi:10.1038/s41587-021-01019-x · Cited by 119 · see citing works
Functional diversity for body actions in the mesencephalic locomotor region
Ferreira-Pinto, M. J., Kanodia, H., Falasconi, A., Sigrist, M., Esposito, M. S., & Arber, S. (2021) · Cell · doi:10.1016/j.cell.2021.07.002 · Cited by 105 · see citing works
Measuring behavior in the home cage: study design, applications, challenges, and perspectives
Grieco, F., Bernstein, B. J., Biemans, B., Bikovski, L., Burnett, C. J., Cushman, J. D., van Dam, E. A., Fry, S. A., Richmond-Hacham, B., Homberg, J. R., et al. (2021) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2021.735387 · Cited by 109 · see citing works
Control of orienting movements and locomotion by projection-defined subsets of brainstem V2a neurons
Usseglio, G., Gatier, E., Heuzé, A., Hérent, C., & Bouvier, J. (2020) · Current Biology · doi:10.1016/j.cub.2020.09.014 · Cited by 87 · see citing works
Hippocampo-cortical circuits for selective memory encoding, routing, and replay
Harvey, R. E., Robinson, H. L., Liu, C., Oliva, A., & Fernandez-Ruiz, A. (2023) · Neuron · doi:10.1016/j.neuron.2023.04.015 · Cited by 101 · see citing works
The neuromechanics of animal locomotion: From biology to robotics and back
Ramdya, P., & Ijspeert, A. J. (2023) · Science Robotics · doi:10.1126/scirobotics.adg0279 · Cited by 92 · see citing works
A size principle for recruitment of Drosophila leg motor neurons
Azevedo, A. W., Dickinson, E. S., Gurung, P., Venkatasubramanian, L., Mann, R. S., & Tuthill, J. C. (2020) · eLife · doi:10.7554/elife.56754 · Cited by 87 · see citing works
A cognitive process occurring during sleep is revealed by rapid eye movements
Senzai, Y., & Scanziani, M. (2022) · Science · doi:10.1126/science.abp8852 · Cited by 88 · see citing works
NeuroMechFly, a neuromechanical model of adult Drosophila melanogaster
Lobato-Rios, V., Ramalingasetty, S. T., Özdil, P. G., Arreguit, J., Ijspeert, A. J., & Ramdya, P. (2022) · Nature Methods · doi:10.1038/s41592-022-01466-7 · Cited by 88 · see citing works
Creatures Great and SMAL: Recovering the Shape and Motion of Animals from Video
Biggs, B., Roddick, T., Fitzgibbon, A., & Cipolla, R. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-20873-8_1 · Cited by 89 · see citing works
Anpilov, S., Shemesh, Y., Eren, N., Harony-Nicolas, H., Benjamin, A., Dine, J., Oliveira, V. E. M., Forkosh, O., Karamihalev, S., Hüttl, R.-E., et al. (2020) · Neuron · doi:10.1016/j.neuron.2020.05.028 · Cited by 89 · see citing works
State-dependent pupil dilation rapidly shifts visual feature selectivity
Franke, K., Willeke, K. F., Ponder, K., Galdamez, M., Zhou, N., Muhammad, T., Patel, S., Froudarakis, E., Reimer, J., Sinz, F. H., et al. (2022) · Nature · doi:10.1038/s41586-022-05270-3 · Cited by 85 · see citing works
Distributed control of motor circuits for backward walking in Drosophila
Feng, K., Sen, R., Minegishi, R., Dübbert, M., Bockemühl, T., Büschges, A., & Dickson, B. J. (2020) · Nature Communications · doi:10.1038/s41467-020-19936-x · Cited by 69 · see citing works
Generating parallel representations of position and identity in the olfactory system
Taisz, I., Donà, E., Münch, D., Bailey, S. N., Morris, B. J., Meechan, K. I., Stevens, K. M., Varela-Martínez, I., Gkantia, M., Schlegel, P., et al. (2023) · Cell · doi:10.1016/j.cell.2023.04.038 · Cited by 79 · see citing works
Neural circuit mechanisms for steering control in <i>walking Drosophila</i>
Rayshubskiy, A., Holtz, S. L., Bates, A. S., Vanderbeck, Q. X., Capdevila, L. S., Rockwell, V., & Wilson, R. I. (2020) · doi:10.1101/2020.04.04.024703 · Cited by 64 · see citing works
Ascending neurons convey behavioral state to integrative sensory and action selection brain regions
Chen, C.-L., Aymanns, F., Minegishi, R., Matsuda, V. D. V., Talabot, N., Günel, S., Dickson, B. J., & Ramdya, P. (2023) · Nature Neuroscience · doi:10.1038/s41593-023-01281-z · Cited by 58 · see citing works
AcinoSet: A 3D Pose Estimation Dataset and Baseline Models for Cheetahs in the Wild
Joska, D., Clark, L., Muramatsu, N., Jericevich, R., Nicolls, F., Mathis, A., Mathis, M. W., & Patel, A. (2021) · 2021 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48506.2021.9561338 · Cited by 66 · see citing works
Self-citationFujiwara, T., Brotas, M., & Chiappe, M. E. (2022) · Neuron · doi:10.1016/j.neuron.2022.04.008 · Cited by 59 · see citing works
Behaviorally relevant decision coding in primary somatosensory cortex neurons
Buetfering, C., Zhang, Z., Pitsiani, M., Smallridge, J., Boven, E., McElligott, S., & Häusser, M. (2022) · Nature Neuroscience · doi:10.1038/s41593-022-01151-0 · Cited by 59 · see citing works
Neural circuit mechanisms underlying context-specific halting in Drosophila
Sapkal, N., Mancini, N., Kumar, D. S., Spiller, N., Murakami, K., Vitelli, G., Bargeron, B., Maier, K., Eichler, K., Jefferis, G. S. X. E., et al. (2024) · Nature · doi:10.1038/s41586-024-07854-7 · Cited by 54 · see citing works
The spatial and temporal structure of neural activity across the fly brain
Schaffer, E. S., Mishra, N., Whiteway, M. R., Li, W., Vancura, M. B., Freedman, J., Patel, K. B., Voleti, V., Paninski, L., Hillman, E. M. C., et al. (2023) · Nature Communications · doi:10.1038/s41467-023-41261-2 · Cited by 53 · see citing works
Basal ganglia-spinal cord pathway that commands locomotor gait asymmetries in mice
Cregg, J. M., Sidhu, S. K., Leiras, R., & Kiehn, O. (2024) · Nature Neuroscience · doi:10.1038/s41593-024-01569-8 · Cited by 48 · see citing works
Behavioral decomposition reveals rich encoding structure employed across neocortex in rats
Mimica, B., Tombaz, T., Battistin, C., Fuglstad, J. G., Dunn, B. A., & Whitlock, J. R. (2023) · Nature Communications · doi:10.1038/s41467-023-39520-3 · Cited by 45 · see citing works
A rise-to-threshold process for a relative-value decision
Vijayan, V., Wang, F., Wang, K., Chakravorty, A., Adachi, A., Akhlaghpour, H., Dickson, B. J., & Maimon, G. (2023) · Nature · doi:10.1038/s41586-023-06271-6 · Cited by 49 · see citing works
Projection-specific deficits in synaptic transmission in adult Sapap3-knockout mice
Hadjas, L. C., Schartner, M. M., Cand, J., Creed, M. C., Pascoli, V., Lüscher, C., & Simmler, L. D. (2020) · Neuropsychopharmacology · doi:10.1038/s41386-020-0747-3 · Cited by 46 · see citing works
Shishov, N., Elabd, K., Komisar, V., Chong, H., & Robinovitch, S. N. (2021) · PLOS ONE · doi:10.1371/journal.pone.0258923 · Cited by 42 · see citing works
Flexible neural control of transition points within the egg-laying behavioral sequence in Drosophila
Cury, K. M., & Axel, R. (2023) · Nature Neuroscience · doi:10.1038/s41593-023-01332-5 · Cited by 40 · see citing works
Self-citationNewton, K. C., Kacev, D., Nilsson, S. R. O., Saettele, A. L., Golden, S. A., & Sheets, L. (2023) · Communications Biology · doi:10.1038/s42003-023-04449-2 · Cited by 36 · see citing works
A Whole-Body Musculoskeletal Model of the Mouse
Tata Ramalingasetty, S., Danner, S. M., Arreguit, J., Markin, S. N., Rodarie, D., Kathe, C., Courtine, G., Rybak, I. A., & Ijspeert, A. J. (2021) · IEEE Access · doi:10.1109/access.2021.3133078 · Cited by 35 · see citing works
Boundary vector cells in the goldfish central telencephalon encode spatial information
Cohen, L., Vinepinsky, E., Donchin, O., & Segev, R. (2023) · PLOS Biology · doi:10.1371/journal.pbio.3001747 · Cited by 37 · see citing works
Sexual differentiation of neural mechanisms of stress sensitivity during puberty
Wright, E. C., Luo, P. X., Zakharenkov, H. C., Serna Godoy, A., Lake, A. A., Prince, Z. D., Sekar, S., Culkin, H. I., Ramirez, A. V., Dwyer, T., et al. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2306475120 · Cited by 34 · see citing works
Naguib, S., Lopez-Lee, C., Torres, E. R., Lee, S.-I., Zhu, J., Zhu, D., Ye, P., Norman, K., Zhao, M., Wong, M. Y., et al. (2025) · Immunity · doi:10.1016/j.immuni.2025.05.023 · Cited by 32 · see citing works
Deep imaging in the brainstem reveals functional heterogeneity in V2a neurons controlling locomotion
Schwenkgrub, J., Harrell, E. R., Bathellier, B., & Bouvier, J. (2020) · Science Advances · doi:10.1126/sciadv.abc6309 · Cited by 27 · see citing works
Rigney, N., Zbib, A., de Vries, G. J., & Petrulis, A. (2022) · Journal of Neuroendocrinology · doi:10.1111/jne.13083 · Cited by 29 · see citing works
Neural circuit mechanisms for steering control in walking Drosophila
Rayshubskiy, A., Holtz, S. L., Bates, A. S., Vanderbeck, Q. X., Serratosa Capdevila, L., Rockwell, V., & Wilson, R. (2025) · eLife · doi:10.7554/elife.102230 · Cited by 27 · see citing works
Interpretable video-based tracking and quantification of Parkinsonism clinical motor states
Deng, D., Ostrem, J. L., Nguyen, V., Cummins, D. D., Sun, J., Pathak, A., Little, S., & Abbasi-Asl, R. (2024) · npj Parkinson's Disease · doi:10.1038/s41531-024-00742-x · Cited by 25 · see citing works
Naguib, S., Lopez-Lee, C., Torres, E. R., Lee, S.-I., Zhu, J., Zhu, D., Ye, P., Norman, K., Zhao, M., Wong, M. Y., et al. (2024) · doi:10.1101/2024.04.25.591140 · Cited by 25 · see citing works
Miller, S. R., Luxem, K., Lauderdale, K., Nambiar, P., Honma, P. S., Ly, K. K., Bangera, S., Bullock, M., Shin, J., Kaliss, N., et al. (2024) · Cell Reports · doi:10.1016/j.celrep.2024.114870 · Cited by 24 · see citing works
Goedhoop, J., Arbab, T., & Willuhn, I. (2023) · The Journal of Neuroscience · doi:10.1523/jneurosci.1527-22.2023 · Cited by 24 · see citing works
Abbas, W., & Masip Rodo, D. (2019) · Sensors · doi:10.3390/s19153274 · Cited by 21 · see citing works
The role of a population of descending neurons in the optomotor response in flying <i>Drosophila</i>
Palmer, E. H., Omoto, J. J., & Dickinson, M. H. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.12.05.519224 · Cited by 10 · see citing works
Coexistence of state, choice, and sensory integration coding in barrel cortex LII/III
Gardères, P.-M., Le Gal, S., Rousseau, C., Mamane, A., Ganea, D. A., & Haiss, F. (2024) · Nature Communications · doi:10.1038/s41467-024-49129-9 · Cited by 7 · see citing works
Parameterizing animal sounds and motion with animal-attached tags to study acoustic communication
Casoli, M., Johnson, M., McHugh, K. A., Wells, R. S., & Tyack, P. L. (2022) · Behavioral Ecology and Sociobiology · doi:10.1007/s00265-022-03154-0 · Cited by 8 · see citing works
Asano, Y., Sasaki, D., Ikoma, Y., & Matsui, K. (2024) · Neuroscience Research · doi:10.1016/j.neures.2023.11.008 · Cited by 9 · see citing works
Schut, E. H. S., Alonso, A., Smits, S., Khamassi, M., Samanta, A., Negwer, M., Kasri, N. N., Navarro Lobato, I., & Genzel, L. (2020) · Neurobiology of Learning and Memory · doi:10.1016/j.nlm.2020.107265 · Cited by 8 · see citing works
Michaels, R., Barreira, T. V., Robinovitch, S. N., Sosnoff, J. J., & Moon, Y. (2025) · Scientific Reports · doi:10.1038/s41598-025-85934-y · Cited by 9 · see citing works
Neural circuit mechanisms underlying context-specific halting in <i>Drosophila</i>
Sapkal, N., Mancini, N., Kumar, D. S., Spiller, N., Murakami, K., Vitelli, G., Bargeron, B., Maier, K., Eichler, K., Jefferis, G. S. X. E., et al. (2023) · doi:10.1101/2023.09.25.559438 · Cited by 54 · see citing works
Subspace communication in the hippocampal–retrosplenial axis
Gonzalez, J., Vöröslakos, M., Aykan, D., Soto, N., Nitzan, N., Swanson, R., Karadas, M., Chen, Z. S., & Buzsáki, G. (2026) · Nature · doi:10.1038/s41586-026-10481-z · Cited by 7 · see citing works
Plastic vasomotion entrainment
Sasaki, D., Imai, K., Ikoma, Y., & Matsui, K. (2024) · eLife · doi:10.7554/elife.93721.3 · Cited by 6 · see citing works
Klingelhöfer, D., Braun, M., Dröge, J., Groneberg, D. A., & Brüggmann, D. (2025) · Globalization and Health · doi:10.1186/s12992-025-01128-1 · Cited by 6 · see citing works
Neural circuit mechanisms for steering control in walking Drosophila
Rayshubskiy, A., Holtz, S. L., Bates, A., Vanderbeck, Q. X., Capdevila, L. S., & Wilson, R. I. (2024) · doi:10.7554/elife.102230.1 · Cited by 5 · see citing works
Transferring Dense Pose to Proximal Animal Classes
Sanakoyeu, A., Khalidov, V., McCarthy, M. S., Vedaldi, A., & Neverova, N. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00528 · Cited by 13 · see citing works
Revealing abrupt transitions from goal-directed to habitual behavior
Moore, S., Wang, Z., Zhu, Z., Wang, J., Sun, R., Lee, Y. A., Charles, A., & Kuchibhotla, K. V. (2026) · Nature Communications · doi:10.1038/s41467-026-71048-0 · Cited by 3 · see citing works
Evaluating the play fighting of rats: A sociological perspective
Ham, J. R., Pellis, S. M., Pellis, V. C., & Achterberg, E. J. M. (2026) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2026.110836
Infraslow histaminergic dynamics govern priming states to gate moment-to-moment memory accessibility
Morishita, Y., Takamura, Y., Nishimura, K., Yokoi, Y., Ishihama, Y., Idutsu, R., Ono, M., Matsumoto, R., Hitora-Imamura, N., Minami, M., et al. (2026) · Neuron · doi:10.1016/j.neuron.2026.05.019
Circuit mechanisms for disentangled representations and generalization in the hippocampus
Tang, W., Chang, H., Liu, C., Perez-Hernandez, S., Zheng, W. Y., Park, J., Karaba, L. A., Oliva, A., & Fernandez-Ruiz, A. (2026) · Nature Neuroscience · doi:10.1038/s41593-026-02491-x
- Authors
- Mathis, A., Mamidanna, P., Cury, K. M., Abe, T., Murthy, V. N., Mathis, M. W., & Bethge, M.
- Affiliations
- Harvard University (US)
- University of Tübingen (DE)
- Mortimer B. Zuckerman Mind Brain Behavior Institute (US)
- Baylor College of Medicine (US)
- Corresponding author
- Mathis, Mackenzie Weygandt
- ORCID
- Venue
- Nature Neuroscience
- Published
- 2018-08-20
- Volume
- 21
- Issue
- 9
- Pages
- 1281-1289
- ISSN
- 1097-6256 (ISSN-L)
- 1546-1726
- OpenAlex
- W2887114371
- Semantic Scholar
- 6001895c2fcf69528d01306e9d293d9d2a4cc67b
- Cited by
- 5,187
- References
- 51
- Citations by year
- 2026: 875 · 2025: 1081 · 2024: 927 · 2023: 770 · 2022: 619 · 2021: 501 · 2020: 275 · 2019: 118 · 2018: 15
- FWCI
- 113.66
- Type
- journal article
- Language
- en
- Open access
- green
- infoscience.epfl.ch
- Licence
- cc-by-nc-nd
- www.springer.com/tdm (tdm)
- Locations
- Nature Neuroscience (publishedVersion)
- PubMed (publishedVersion)
- Max Planck Digital Library (submittedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Surveillance and Tracking Methods
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- markerless pose estimation, body part tracking, DeepLabCut, transfer learning, deep neural network, animal behavior quantification, video tracking, limited training data, motor control
- MeSH terms
- Deep Learning, Algorithms, Animals, Behavior, Behavior, Animal, Drosophila melanogaster, Humans, Male, Mice, Inbred C57BL, Nerve Net, Odorants, Posture, Psychomotor Performance, Transfer, Psychology, Video Recording, Neural Networks, Computer, Mice
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review