DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning
Open accessPublished version of a preprint
Abstract
Quantitative behavioral measurements are important for answering questions across scientific disciplines—from neuroscience to ecology. State-of-the-art deep-learning methods offer major advances in data quality and detail by allowing researchers to automatically estimate locations of an animal’s body parts directly from images or videos. However, currently available animal pose estimation methods have limitations in speed and robustness. Here, we introduce a new easy-to-use software toolkit, DeepPoseKit, that addresses these problems using an efficient multi-scale deep-learning model, called Stacked DenseNet, and a fast GPU-based peak-detection algorithm for estimating keypoint locations with subpixel precision. These advances improve processing speed >2x with no loss in accuracy compared to currently available methods. We demonstrate the versatility of our methods with multiple challenging animal pose estimation tasks in laboratory and field settings—including groups of interacting individuals. Our work reduces barriers to using advanced tools for measuring behavior and has broad applicability across the behavioral sciences.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The high level of control afforded by laboratory settings facilitated the rapid development of such tools, many of which are now being adopted for use in field studies of free-moving animals in complex natural environments.
“The high level of control (e.g., of lighting conditions, sensor calibration, and environment) afforded by laboratory settings facilitated the rapid development of such tools, many of which are now being adopted for use in field studies of free-moving animals in complex natural environments75,142.”
From Perspectives in machine learning for wildlife conservation (2022) · cites this paper as [75]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
Self-citationFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · see citing works
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566 · see citing works
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834 · see citing works
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755 · see citing works
Labaca-Castro, R. (2023) · Springer eBooks · doi:10.1007/978-3-658-40442-0_9 · Cited by 21,100 · see citing works
Xception: Deep Learning with Depthwise Separable Convolutions
Chollet, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.195 · Cited by 20,380 · see citing works
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2644615 · Cited by 19,631 · see citing works
GAN（Generative Adversarial Nets）
淳司, 柴. (2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,268 · see citing works
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00068 · Cited by 14,623 · see citing works
Fully convolutional networks for semantic segmentation
Shelhamer, E., Long, J., & Darrell, T. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2572683 · Cited by 11,898 · see citing works
Perceptual Losses for Real-Time Style Transfer and Super-Resolution
Johnson, J., Alahi, A., & Fei-Fei, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_43 · Cited by 11,283 · see citing works
Genome engineering using the CRISPR-Cas9 system
Ran, F. A., Hsu, P. D., Wright, J., Agarwala, V., Scott, D. A., & Zhang, F. (2013) · Nature Protocols · doi:10.1038/nprot.2013.143 · Cited by 11,133 · see citing works
A Stochastic Approximation Method
Robbins, H., & Monro, S. (1951) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729586 · Cited by 9,643 · see citing works
<i>Stan</i> : A Probabilistic Programming Language
Carpenter, B., Gelman, A., Hoffman, M. D., Lee, D., Goodrich, B., Betancourt, M., Brubaker, M., Guo, J., Li, P., & Riddell, A. (2017) · Journal of Statistical Software · doi:10.18637/jss.v076.i01 · Cited by 7,754 · see citing works
Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields
Cao, Z., Simon, T., Wei, S.-E., & Sheikh, Y. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.143 · Cited by 7,539 · see citing works
The new frontier of genome engineering with CRISPR-Cas9
Doudna, J. A., & Charpentier, E. (2014) · Science · doi:10.1126/science.1258096 · Cited by 7,062 · see citing works
Kuhn, M., & Johnson, K. (2013) · doi:10.1007/978-1-4614-6849-3 · Cited by 6,788 · see citing works
Stacked Hourglass Networks for Human Pose Estimation
Newell, A., Yang, K., & Deng, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46484-8_29 · Cited by 5,634 · see citing works
DeepLabCut: markerless pose estimation of user-defined body parts with deep learning
Mathis, A., Mamidanna, P., Cury, K. M., Abe, T., Murthy, V. N., Mathis, M. W., & Bethge, M. (2018) · Nature Neuroscience · doi:10.1038/s41593-018-0209-y · Cited by 5,187 · see citing works
WaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911 · see citing works
Algorithms for the Assignment and Transportation Problems
Munkres, J. R. (1957) · Journal of the Society for Industrial and Applied Mathematics · doi:10.1137/0105003 · Cited by 4,001 · see citing works
Duane, S., Kennedy, A. D., Pendleton, B. J., & Roweth, D. (1987) · Physics Letters B · doi:10.1016/0370-2693(87)91197-x · Cited by 3,920 · see citing works
What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
Kendall, A., & Gal, Y. (2017) · arXiv · doi:10.48550/arxiv.1703.04977 · Cited by 3,088 · see citing works
Conditional Image Generation with PixelCNN DecodersPreprint
van den Oord, A., Kalchbrenner, N., Vinyals, O., Espeholt, L., Graves, A., & Kavukcuoglu, K. (2016) · Neural Information Processing Systems · doi:10.48550/arxiv.1606.05328 · Cited by 3,066 · see citing works
The No-U-turn sampler: adaptively setting path lengths in Hamiltonian Monte CarloPreprint
Homan, M. D., & Gelman, A. (2014) · arXiv (Cornell University) · Cited by 2,964 · see citing works
Speed/Accuracy Trade-Offs for Modern Convolutional Object Detectors
Huang, J., Rathod, V., Sun, C., Zhu, M., Korattikara, A., Fathi, A., Fischer, I., Wojna, Z., Song, Y., Guadarrama, S., et al. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.351 · Cited by 2,743 · see citing works
2D Human Pose Estimation: New Benchmark and State of the Art Analysis
Andriluka, M., Pishchulin, L., Gehler, P., & Schiele, B. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.471 · Cited by 2,699 · see citing works
The No-U-Turn Sampler: Adaptively Setting Path Lengths in Hamiltonian Monte CarloPreprint
Hoffman, M. D., & Gelman, A. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1111.4246 · Cited by 2,575 · see citing works
Visualizing the Loss Landscape of Neural NetsPreprint
Li, H., Xu, Z., Taylor, G., Studer, C., & Goldstein, T. (2017) · Neural Information Processing Systems · doi:10.48550/arxiv.1712.09913 · Cited by 2,497 · see citing works
Stochastic Estimation of the Maximum of a Regression Function
Kiefer, J., & Wolfowitz, J. (1952) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729392 · Cited by 2,200 · see citing works
Self-Normalizing Neural NetworksPreprint
Klambauer, G., Unterthiner, T., Mayr, A., & Hochreiter, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.02515 · Cited by 2,031 · see citing works
Efficient subpixel image registration algorithms
Guizar‐Sicairos, M., Thurman, S. T., & Fienup, J. R. (2008) · Optics Letters · doi:10.1364/ol.33.000156 · Cited by 1,916 · see citing works
The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation
Jégou, S., Drozdzal, M., Vázquez, D., Romero, A., & Bengio, Y. (2017) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.156 · Cited by 1,762 · see citing works
Terrestrial animal tracking as an eye on life and planet
Kays, R., Crofoot, M. C., Jetz, W., & Wikelski, M. (2015) · Science · doi:10.1126/science.aaa2478 · Cited by 1,638 · see citing works
Conditional Image Generation with PixelCNN DecodersPreprint
van den Oord, A., Kalchbrenner, N., Vinyals, O., et al. (2016) · arXiv (Cornell University) · Cited by 1,586 · see citing works
Using DeepLabCut for 3D markerless pose estimation across species and behaviors
Nath, T., Mathis, A., Chen, A. C., Patel, A., Bethge, M., & Mathis, M. W. (2019) · Nature Protocols · doi:10.1038/s41596-019-0176-0 · Cited by 1,476 · see citing works
Neuroscience Needs Behavior: Correcting a Reductionist Bias
Krakauer, J. W., Ghazanfar, A. A., Gomez-Marin, A., MacIver, M. A., & Poeppel, D. (2017) · Neuron · doi:10.1016/j.neuron.2016.12.041 · Cited by 1,358 · see citing works
Content-aware image restoration: pushing the limits of fluorescence microscopy
WEIGERT, M. G., Schmidt, U., Boothe, T., Müller, A., Dibrov, A., Jain, A., Wilhelm, B. G., Schmidt, D., Broaddus, C., Culley, S., et al. (2018) · Nature Methods · doi:10.1038/s41592-018-0216-7 · Cited by 1,210 · see citing works
DeeperCut: A Deeper, Stronger, and Faster Multi-person Pose Estimation Model
Insafutdinov, E., Pishchulin, L., Andres, B., Andriluka, M., & Schiele, B. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46466-4_3 · Cited by 1,191 · see citing works
Visualizing the Loss Landscape of Neural Nets
Li, H., Xu, Z., Taylor, G., Studer, C., & Goldstein, T. (2017) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-b-000461393 · Cited by 1,148 · see citing works
Hierarchical group dynamics in pigeon flocks
Nagy, M., Ákos, Z., Biro, D., & Vicsek, T. (2010) · Nature · doi:10.1038/nature08891 · Cited by 1,053 · see citing works
Automatic early stopping using cross validation: quantifying the criteria
Prechelt, L. (1998) · Neural Networks · doi:10.1016/s0893-6080(98)00010-0 · Cited by 995 · see citing works
The American Association for the Advancement of Science
Haultain, A. (1889) · Nature · doi:10.1038/040556a0 · Cited by 944 · see citing works
idTracker: tracking individuals in a group by automatic identification of unmarked animals
Pérez‐Escudero, A., Vicente-Page, J., Hinz, R. C., Arganda, S., & de Polavieja, G. G. (2014) · Nature Methods · doi:10.1038/nmeth.2994 · Cited by 775 · see citing works
Mapping Sub-Second Structure in Mouse Behavior
Wiltschko, A. B., Johnson, M. J., Iurilli, G., Peterson, R. E., Katon, J. M., Pashkovski, S. L., Abraira, V. E., Adams, R. P., & Datta, S. R. (2015) · Neuron · doi:10.1016/j.neuron.2015.11.031 · Cited by 760 · see citing works
Deep Convolutional Inverse Graphics NetworkPreprint
Kulkarni, T. D., Whitney, W., Kohli, P., & Tenenbaum, J. B. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1503.03167 · Cited by 708 · see citing works
Fast animal pose estimation using deep neural networks
Pereira, T., Aldarondo, D. E., Willmore, L., Kislin, M., Wang, S. S. ‐., Murthy, M., & Shaevitz, J. W. (2018) · Nature Methods · doi:10.1038/s41592-018-0234-5 · Cited by 659 · see citing works
Mapping the stereotyped behaviour of freely moving fruit flies
Berman, G. J., Choi, D. M., Bialek, W., & Shaevitz, J. W. (2014) · Journal of The Royal Society Interface · doi:10.1098/rsif.2014.0672 · Cited by 565 · see citing works
Shared decision-making drives collective movement in wild baboons
Strandburg‐Peshkin, A., Farine, D. R., Couzin, I. D., & Crofoot, M. C. (2015) · Science · doi:10.1126/science.aaa5099 · Cited by 563 · see citing works
Self-citationPoseTrack: A Benchmark for Human Pose Estimation and Tracking
Andriluka, M., Iqbal, U., Insafutdinov, E., Pishchulin, L., Milan, A., Gall, J., & Schiele, B. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00542 · Cited by 501 · see citing works
Toward a Science of Computational Ethology
Anderson, D. J., & Perona, P. (2014) · Neuron · doi:10.1016/j.neuron.2014.09.005 · Cited by 496 · see citing works
Automated image-based tracking and its application in ecology
Dell, A. I., Bender, J. A., Branson, K., Couzin, I. D., de Polavieja, G. G., Noldus, L. P. J. J., Pérez-Escudero, A., Perona, P., Straw, A. D., Wikelski, M., et al. (2014) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2014.05.004 · Cited by 492 · see citing works
Rosenthal, S. B., Twomey, C. R., Hartnett, A. T., Wu, H. S., & Couzin, I. D. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1420068112 · Cited by 474 · see citing works
Self-citationVisual sensory networks and effective information transfer in animal groups
Strandburg‐Peshkin, A., Twomey, C. R., Bode, N. W. F., Kao, A. B., Katz, Y., Ioannou, C. C., Rosenthal, S. B., Torney, C. J., Wu, H. S., Levin, S. A., et al. (2013) · Current Biology · doi:10.1016/j.cub.2013.07.059 · Cited by 430 · see citing works
Self-citationThe Striatum Organizes 3D Behavior via Moment-to-Moment Action Selection
Markowitz, J. E., Gillis, W. F., Beron, C. C., Neufeld, S. Q., Robertson, K., Bhagat, N. D., Peterson, R. E., Peterson, E., Hyun, M., Linderman, S. W., et al. (2018) · Cell · doi:10.1016/j.cell.2018.04.019 · Cited by 422 · see citing works
idtracker.ai: tracking all individuals in small or large collectives of unmarked animals
Romero-Ferrero, F., Bergomi, M. G., Hinz, R. C., Heras, F. J. H., & de Polavieja, G. G. (2019) · Nature Methods · doi:10.1038/s41592-018-0295-5 · Cited by 417 · see citing works
3D Menagerie: Modeling the 3D Shape and Pose of Animals
Zuffi, S., Kanazawa, A., Jacobs, D. W., & Black, M. J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.586 · Cited by 381 · see citing works
Adversarial PoseNet: A Structure-Aware Convolutional Network for Human Pose Estimation
Chen, Y., Shen, C., Wei, X.-S., Liu, L., & Yang, J. J. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.137 · Cited by 367 · see citing works
Consistent Individual Differences Drive Collective Behavior and Group Functioning of Schooling Fish
Jolles, J. W., Boogert, N. J., Sridhar, V. H., Couzin, I. D., & Manica, A. (2017) · Current Biology · doi:10.1016/j.cub.2017.08.004 · Cited by 363 · see citing works
Self-citation(1992) · Cited by 316 · see citing works
Multi-Scale Structure-Aware Network for Human Pose Estimation
Ke, L., Chang, M., Qi, H., & Lyu, S. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01216-8_44 · Cited by 270 · see citing works
Mendes, C. S., Bartos, I., Akay, T., Márka, S., & Mann, R. S. (2013) · eLife · doi:10.7554/elife.00231 · Cited by 247 · see citing works
Virtual reality for freely moving animals
Stowers, J. R., Hofbauer, M., Bastien, R., Griessner, J., Higgins, P., Farooqui, S., Fischer, R. M., Nowikovsky, K., Haubensak, W. E., Couzin, I. D., et al. (2017) · Nature Methods · doi:10.1038/nmeth.4399 · Cited by 229 · see citing works
Self-citationPoseTrack: Joint Multi-person Pose Estimation and Tracking
Iqbal, U., Milan, A., & Gall, J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.495 · Cited by 227 · see citing works
Composing graphical models with neural networks for structured representations and fast inferencePreprint
Johnson, M. J., Duvenaud, D., Wiltschko, A. B., Datta, S. R., & Adams, R. P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.06277 · Cited by 224 · see citing works
Behavioral idiosyncrasy reveals genetic control of phenotypic variability
Ayroles, J. F., Buchanan, S. M., O’Leary, C., Skutt-Kakaria, K., Grenier, J. K., Clark, A. G., Hartl, D. L., & de Bivort, B. L. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1503830112 · Cited by 218 · see citing works
Predictability and hierarchy in Drosophila behavior
Berman, G. J., Bialek, W., & Shaevitz, J. W. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1607601113 · Cited by 204 · see citing works
Behavioural individuality in clonal fish arises despite near-identical rearing conditions
Bierbach, D., Laskowski, K., & Wolf, M. (2017) · Nature Communications · doi:10.1038/ncomms15361 · Cited by 193 · see citing works
Brown, A. E., Yemini, E., Grundy, L. J., Jucikas, T., & Schafer, W. R. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1211447110 · Cited by 186 · see citing works
Measuring behavior across scales
Berman, G. J. (2018) · BMC Biology · doi:10.1186/s12915-018-0494-7 · Cited by 184 · see citing works
Context-dependent hierarchies in pigeons
Nagy, M., Vásárhelyi, G., Pettit, B., Roberts-Mariani, I., Vicsek, T., & Biro, D. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1305552110 · Cited by 178 · see citing works
Günel, S., Rhodin, H., Morales, D., Campagnolo, J., Ramdya, P., & Fua, P. (2019) · eLife · doi:10.7554/elife.48571 · Cited by 167 · see citing works
Composing graphical models with neural networks for structured representations and fast inference
Johnson, M., Duvenaud, D., Wiltschko, A. B., Adams, R. P., & Datta, S. R. (2016) · neural information processing systems · Cited by 165 · see citing works
From local collective behavior to global migratory patterns in white storks
Flack, A., Nagy, M., Fiedler, W., Couzin, I. D., & Wikelski, M. (2018) · Science · doi:10.1126/science.aap7781 · Cited by 164 · see citing works
Self-citationOptogenetic dissection of descending behavioral control in Drosophila
Cande, J., Namiki, S., Qiu, J., Korff, W., Card, G. M., Shaevitz, J. W., Stern, D. L., & Berman, G. J. (2018) · eLife · doi:10.7554/elife.34275 · Cited by 163 · see citing works
Ethology as a physical science
Brown, A. E., & de Bivort, B. L. (2018) · Nature Physics · doi:10.1038/s41567-018-0093-0 · Cited by 163 · see citing works
Strandburg‐Peshkin, A., Farine, D. R., Crofoot, M. C., & Couzin, I. D. (2017) · eLife · doi:10.7554/elife.19505 · Cited by 155 · see citing works
Self-citationAn open-source platform for analyzing and sharing worm-behavior data
Javer, A., Currie, M., Lee, C. W., Hokanson, J., Li, K., Martineau, C. N., Yemini, E., Grundy, L. J., Li, C., Ch'ng, Q.-L., et al. (2018) · Nature Methods · doi:10.1038/s41592-018-0112-1 · Cited by 151 · see citing works
Phototactic personality in fruit flies and its suppression by serotonin and white
Kain, J. S., Stokes, C., & de Bivort, B. (2012) · Proceedings of the National Academy of Sciences of the United States of America · doi:10.1073/pnas.1211988109 · Cited by 145 · see citing works
Incorporating Second-Order Functional Knowledge for Better Option PricingPreprint
Bélisle, F., Bengio, Y., Dugas, C., García, R., & Nadeau, C. (2002) · RePEc: Research Papers in Economics · Cited by 140 · see citing works
BEEtag: A Low-Cost, Image-Based Tracking System for the Study of Animal Behavior and Locomotion
Crall, J. D., Gravish, N., Mountcastle, A. M., & Combes, S. A. (2015) · PLoS ONE · doi:10.1371/journal.pone.0136487 · Cited by 139 · see citing works
Three-D Safari: Learning to Estimate Zebra Pose, Shape, and Texture From Images “In the Wild”
Zuffi, S., Kanazawa, A., Berger-Wolf, T., & Black, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00546 · Cited by 139 · see citing works
Roy, A. G., Conjeti, S., Navab, N., & Wachinger, C. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.03.042 · Cited by 137 · see citing works
FlyMAD: rapid thermogenetic control of neuronal activity in freely walking Drosophila
Bath, D. E., Stowers, J. R., Hörmann, D., Poehlmann, A., Dickson, B. J., & Straw, A. D. (2014) · Nature Methods · doi:10.1038/nmeth.2973 · Cited by 133 · see citing works
Using DeepLabCut for 3D markerless pose estimation across species and behaviorsPreprint
Nath, T., Mathis, A., Chen, A. C., et al. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/476531 · Cited by 132 · see citing works
Leg-tracking and automated behavioural classification in Drosophila
Kain, J. S., Stokes, C. R., Gaudry, Q., Song, X., Foley, J., Wilson, R. I., & de Bivort, B. L. (2013) · Nature Communications · doi:10.1038/ncomms2908 · Cited by 92 · see citing works
Emergence of long timescales and stereotyped behaviors in Caenorhabditis elegans
Stephens, G. J., de Mesquita, M. B., Ryu, W. S., & Bialek, W. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1007868108 · Cited by 89 · see citing works
VideoFlow: A Flow-Based Generative Model for Video
Kumar, M. N. A., Babaeizadeh, M., Erhan, D., Finn, C., Levine, S., Dinh, L., & Kingma, D. (2019) · Cited by 84 · see citing works
Adaptive, locally linear models of complex dynamics
Costa, A. C., Ahamed, T., & Stephens, G. J. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1813476116 · Cited by 74 · see citing works
Drones Flying High as New Tool for Field Biologists
Schiffman, R. (2014) · Science · doi:10.1126/science.344.6183.459 · Cited by 66 · see citing works
Deep Neural Network-Based Cooperative Visual Tracking Through Multiple Micro Aerial Vehicles
Price, E., Lawless, G., Ludwig, R., Martinovic, I., Bulthoff, H. H., Black, M. J., & Ahmad, A. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2850224 · Cited by 64 · see citing works
Movement science needs different pose tracking algorithms
Seethapathi, N., Wang, S., Saluja, R., Blohm, G., & Kording, K. P. (2019) · arXiv · doi:10.48550/arxiv.1907.10226 · Cited by 64 · see citing works
Systematic exploration of unsupervised methods for mapping behavior
Todd, J. G., Kain, J. S., & de Bivort, B. L. (2017) · Physical Biology · doi:10.1088/1478-3975/14/1/015002 · Cited by 62 · see citing works
Boenisch, F., Rosemann, B., Wild, B., Dormagen, D. M., Wario, F., & Landgraf, T. (2018) · Frontiers in Robotics and AI · doi:10.3389/frobt.2018.00035 · Cited by 59 · see citing works
An unsupervised method for quantifying the behavior of paired animals
Klibaite, U., Berman, G. J., Cande, J., Stern, D. L., & Shaevitz, J. W. (2017) · Physical Biology · doi:10.1088/1478-3975/aa5c50 · Cited by 55 · see citing works
Building Efficient ConvNets using Redundant Feature PruningPreprint
Ayinde, B. O., & Żurada, J. M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.07653 · Cited by 47 · see citing works
Werkhoven, Z., Rohrsen, C., Qin, C., Brembs, B., & de Bivort, B. (2019) · bioRxiv · doi:10.1371/journal.pone.0224243 · Cited by 47 · see citing works
Multi-Scale Supervised Network for Human Pose Estimation
Ke, L., Qi, H., Chang, M., & Lyu, S. (2018) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2018.8451114 · Cited by 46 · see citing works
Uhlmann, V., Ramdya, P. P., Delgado-Gonzalo, R., Benton, R., & Unser, M. A. (2017) · PLoS ONE · doi:10.1371/journal.pone.0173433 · Cited by 41 · see citing works
Simple, Distributed, and Accelerated Probabilistic ProgrammingPreprint
Tran, D., Hoffman, M. D., Moore, D., Suter, C., Vasudevan, S., Radul, A., Johnson, M., & Saurous, R. A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.02091 · Cited by 40 · see citing works
Akhund-Zade, J., Ho, S., O’Leary, C., & de Bivort, B. (2019) · Journal of Experimental Biology · doi:10.1242/jeb.202234 · Cited by 39 · see citing works
Alisch, T., Crall, J. D., Kao, A. B., Zucker, D., & de Bivort, B. L. (2018) · eLife · doi:10.7554/elife.37166 · Cited by 38 · see citing works
Physics-as-Inverse-Graphics: Unsupervised Physical Parameter Estimation from VideoPreprint
Jaques, M., Burke, M., & Hospedales, T. M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.11169 · Cited by 35 · see citing works
Markerless Outdoor Human Motion Capture Using Multiple Autonomous Micro Aerial Vehicles
Saini, N., Price, E., Tallamraju, R., Enficiaud, R., Ludwig, R., Martinovic, I., Ahmad, A., & Black, M. J. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00091 · Cited by 35 · see citing works
On the inference speed and video-compression robustness of DeepLabCutPreprint
Mathis, A., & Warren, R. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/457242 · Cited by 34 · see citing works
Versace, E., Caffini, M., Werkhoven, Z., & de Bivort, B. (2019) · Scientific Reports · doi:10.1038/s41598-020-61410-7 · Cited by 29 · see citing works
Paired fruit flies synchronize behavior: Uncovering social interactions in Drosophila melanogaster
Klibaite, U., & Shaevitz, J. W. (2020) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008230 · Cited by 28 · see citing works
Automatic localization and decoding of honeybee markers using deep convolutional neural networksPreprint
Wild, B., Sixt, L., & Landgraf, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.04557 · Cited by 20 · see citing works
ANALYSE MATHÉMATIQUE. – Méthodc générale pour la résolution des systèmes d'équations simultanées
Cauchy, A.-L. (2009) · Cambridge University Press eBooks · doi:10.1017/cbo9780511702396.063 · Cited by 16 · see citing works
Francisco, F. A., Nührenberg, P., & Jordan, A. L. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/571232 · Cited by 5 · see citing works
pinpoint: behavioral tracking using 2D barcode tags v0.0.1-alpha
Graving, J. M. (2017) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1008969 · Cited by 5 · see citing works
Self-citationExample Datasets for DeepPoseKit
Graving, J. M., Chae, D., Naik, H., et al. (2019) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.3366907 · Cited by 1 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 4
- Berman GJ Choi DM Bialek W Shaevitz JW 2014a Mapping the structure of drosophilid behavior bioRxiv 10.1101/002873 · DOI
- Goodfellow I Bengio Y Courville A 2016 Deep Learning MIT press
- Klibaite U Shaevitz JW 2019 Interacting fruit flies synchronize behavior bioRxiv 10.1101/545483 PMC7567355 33021989 · DOI
- Sabour S Frosst N Hinton GE 2017 Dynamic routing between capsules Advances in Neural Information Processing Systems 3856 3866 https://papers.nips.cc/paper/6975-dynamic-routing-between-capsules
Cited by
Sorted by citations
Perspectives in machine learning for wildlife conservation
Tuia, D., Kellenberger, B., Beery, S., Costelloe, B. R., Zuffi, S., Risse, B., Mathis, A., Mathis, M. W., van Langevelde, F., Burghardt, T., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-27980-y · Cited by 740 · see citing works
Self-citationDeep learning tools for the measurement of animal behavior in neuroscience
Mathis, M. W., & Mathis, A. (2020) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2019.10.008 · Cited by 450 · see citing works
Computational Neuroethology: A Call to Action
Datta, S. R., Anderson, D. J., Branson, K., Perona, P., & Leifer, A. (2019) · Neuron · doi:10.1016/j.neuron.2019.09.038 · Cited by 453 · see citing works
Anipose: A toolkit for robust markerless 3D pose estimation
Karashchuk, P., Rupp, K. L., Dickinson, E. S., Walling-Bell, S., Sanders, E., Azim, E., Brunton, B. W., & Tuthill, J. C. (2021) · Cell Reports · doi:10.1016/j.celrep.2021.109730 · Cited by 324 · see citing works
A Primer on Motion Capture with Deep Learning: Principles, Pitfalls, and Perspectives
Mathis, A., Schneider, S., Lauer, J., & Mathis, M. W. (2020) · Neuron · doi:10.1016/j.neuron.2020.09.017 · Cited by 231 · see citing works
Neural circuits regulating prosocial behaviors
Walsh, J. J., Christoffel, D. J., & Malenka, R. C. (2023) · Neuropsychopharmacology · doi:10.1038/s41386-022-01348-8 · Cited by 72 · see citing works
AcinoSet: A 3D Pose Estimation Dataset and Baseline Models for Cheetahs in the Wild
Joska, D., Clark, L., Muramatsu, N., Jericevich, R., Nicolls, F., Mathis, A., Mathis, M. W., & Patel, A. (2021) · 2021 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48506.2021.9561338 · Cited by 66 · see citing works
Carlson, J., & Harris, K. (2020) · PLoS Biology · doi:10.1371/journal.pbio.3000860 · Cited by 61 · see citing works
Towards image-based animal tracking in natural environments using a freely moving camera
Haalck, L., Mangan, M., Webb, B., & Risse, B. (2020) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2019.108455 · Cited by 41 · see citing works
Interpretable video-based tracking and quantification of Parkinsonism clinical motor states
Deng, D., Ostrem, J. L., Nguyen, V., Cummins, D. D., Sun, J., Pathak, A., Little, S., & Abbasi-Asl, R. (2024) · npj Parkinson's Disease · doi:10.1038/s41531-024-00742-x · Cited by 25 · see citing works
- Authors
- Graving, J. M., Chae, D., Naik, H., Li, L., Koger, B., Costelloe, B. R., & Couzin, I. D.
- Affiliations
- University of Konstanz (DE)
- Princeton University (US)
- Corresponding author
- Graving, Jacob M
- Couzin, Iain D
- Venue
- eLife
- Publisher
- eLife Sciences Publications, Ltd
- Published
- 2019-10-01
- Volume
- 8
- ISSN
- 2050-084X (ISSN-L)
- PMC
- PMC6897514
- OpenAlex
- W2941352015
- Cited by
- 547
- References
- 117
- Citations by year
- 2026: 51 · 2025: 81 · 2024: 84 · 2023: 90 · 2022: 94 · 2021: 89 · 2020: 50 · 2019: 8
- FWCI
- 26.96
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- eLife (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- KOPS (University of Konstanz) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) (submittedVersion, other-oa)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- Funders
- National Science Foundation (IOS-1355061)
- Office of Naval Research (N00014-09-1-1074, N00014-14-1-0635)
- Army Research Office (W911NG-11-1-0385, W911NF14-1-0431)
- Deutsche Forschungsgemeinschaft (DFG Centre of Excellence 2117)
- University of Konstanz (Zukunftskolleg Investment Grant)
- Ministry of Science, Research and Art Baden-Württemberg (The Strukture-und Innovations fonds fur die Forschung of the State of Baden-Wurttemberg)
- Max Planck Society
- Horizon 2020 Framework Programme (Marie Sklodowska-Curie grant agreement No. 748549)
- Nvidia (GPU Grant)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Keywords
- animal pose estimation, multi-scale deep learning, keypoint detection, sub-pixel precision, behavioral assessment, animal behavior analysis, group behavior, software toolkit
- MeSH terms
- Deep Learning, Algorithms, Animals, Behavior, Animal, Drosophila melanogaster, Grasshoppers, Locomotion, Software, Equidae, Computational Biology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
