3D Menagerie: Modeling the 3D Shape and Pose of Animals
Abstract
There has been significant work on learning realistic, articulated, 3D models of the human body. In contrast, there are few such models of animals, despite many applications. The main challenge is that animals are much less cooperative than humans. The best human body models are learned from thousands of 3D scans of people in specific poses, which is infeasible with live animals. Consequently, we learn our model from a small set of 3D scans of toy figurines in arbitrary poses. We employ a novel part-based shape model to compute an initial registration to the scans. We then normalize their pose, learn a statistical shape model, and refine the registrations and the model together. In this way, we accurately align animal scans from different quadruped families with very different shapes and poses. With the registration to a common template we learn a shape space representing animals including lions, cats, dogs, horses, cows and hippos. Animal shapes can be sampled from the model, posed, animated, and fit to data. We demonstrate generalization by fitting it to images of real animals including species not seen in training.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recent work defines the SMAL model, a 3D articulated shape model for a set of quadruped families.
“Recent work defines the SMAL (Skinned Multi-Animal Linear) model, a 3D articulated shape model for a set of quadruped families107.”
From Perspectives in machine learning for wildlife conservation (2022) · cites this paper as [107]
Cites
(2008) · Journal of Machine Learning Research · Cited by 33,051 · see citing works
A morphable model for the synthesis of 3D faces
Blanz, V., & Vetter, T. R. (1999) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/311535.311556 · Cited by 4,695 · see citing works
Thompson, D. W. (2019) · Princeton University Press eBooks · doi:10.1515/9780691183978-018 · Cited by 4,179 · see citing works
Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., & J, B. M. (2015) · ACM Transactions on Graphics · doi:10.1145/2816795.2818013 · Cited by 3,734 · see citing works
Self-citationRepresentation and recognition of the spatial organization of three-dimensional shapes
Marr, D., & Nishihara, H. K. (1978) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.1978.0020 · Cited by 2,159 · see citing works
Keep It SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image
Bogo, F., Kanazawa, A., Lassner, C., Gehler, P., Romero, J., & Black, M. J. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46454-1_34 · Cited by 1,502 · see citing works
Self-citationSCAPE: shape completion and animation of people
Anguelov, D., Srinivasan, P., Koller, D., Thrun, S., Rodgers, J., & Davis, J. (2005) · ACM Transactions on Graphics · doi:10.1145/1073204.1073207 · Cited by 1,489 · see citing works
As-Rigid-As-Possible Surface Modeling
Sorkine‐Hornung, O., & Alexa, M. (2007) · doi:10.2312/sgp/sgp07/109-116 · Cited by 804 · see citing works
Recovering non-rigid 3D shape from image streams
Bregler, C., Hertzmann, A., & Biermann, H. (2002) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2000.854941 · Cited by 777 · see citing works
Numerical Geometry of Non-Rigid Shapes
Bronstein, A. M., Bronstein, M. M., & Kimmel, R. (2008) · Texts and monographs in computer science · doi:10.1007/978-0-387-73301-2 · Cited by 767 · see citing works
Statistical methods for tomographic image reconstruction
Geman, S., & McClure, D. E. (1987) · Medical Entomology and Zoology · Cited by 586 · see citing works
Unite the People: Closing the Loop Between 3D and 2D Human Representations
Lassner, C., Romero, J., Kiefel, M., Bogo, F., Black, M. J., & Gehler, P. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.500 · Cited by 533 · see citing works
Self-citationOpenDR: An Approximate Differentiable Renderer
Loper, M., & Black, M. J. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10584-0_11 · Cited by 458 · see citing works
Self-citationA Statistical Model of Human Pose and Body Shape
Hasler, N., Stoll, C., Sunkel, M., Rosenhahn, B., & Seidel, H. (2009) · Computer Graphics Forum · doi:10.1111/j.1467-8659.2009.01373.x · Cited by 419 · see citing works
Home 3D body scans from noisy image and range data
Weiss, A., Hirshberg, D. A., & Black, M. J. (2011) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2011.6126465 · Cited by 306 · see citing works
Self-citationRobinette, K. M., Blackwell, S. U., Daanen, H. A. M., et al. (2002) · doi:10.21236/ada406704 · Cited by 268 · see citing works
Category-specific object reconstruction from a single image
Kar, A., Tulsiani, S., Carreira, J., & Malik, J. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298807 · Cited by 263 · see citing works
Coregistration: Simultaneous Alignment and Modeling of Articulated 3D Shape
Hirshberg, D. A., Loper, M., Rachlin, E., & Black, M. J. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33783-3_18 · Cited by 173 · see citing works
Self-citationWhat Shape Are Dolphins? Building 3D Morphable Models from 2D Images
Cashman, T. J., & Fitzgibbon, A. W. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.68 · Cited by 143 · see citing works
The stitched puppet: A graphical model of 3D human shape and pose
Zuffi, S., & Black, M. J. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298976 · Cited by 136 · see citing works
Self-citationLearning an efficient model of hand shape variation from depth images
Khamis, S., Taylor, J., Shotton, J., Keskin, C., Izadi, S., & Fitzgibbon, A. W. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298869 · Cited by 115 · see citing works
Building models of animals from video
Ramanan, D., Forsyth, D. A., & Barnard, K. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.155 · Cited by 101 · see citing works
Tensor-Based Human Body Modeling
Chen, Y., Liu, Z., & Zhang, Z. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.21 · Cited by 95 · see citing works
Allen, B. L., Curless, B., Popović, Z., & Hertzmann, A. (2006) · doi:10.5555/1218064.1218084 · Cited by 94 · see citing works
Inferring 3D Shapes and Deformations from Single Views
Chen, Y., Kim, T., & Cipolla, R. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15558-1_22 · Cited by 62 · see citing works
Learning 3D Deformation of Animals from 2D Images
Kanazawa, A., Kovalsky, S. Z., Basri, R., & Jacobs, D. W. (2016) · Computer Graphics Forum · doi:10.1111/cgf.12838 · Cited by 41 · see citing works
Self-citationBalloon Shapes: Reconstructing and Deforming Objects with Volume from Images
Vicente, S., & Agapito, L. (2013) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2013.37 · Cited by 40 · see citing works
Favreau, L., Revéret, L., Depraz, C., & Cani, M. (2004) · Computer animation and simulation · doi:10.1145/1028523.1028560 · Cited by 36 · see citing works
Component-Wise Modeling of Articulated Objects
Ntouskos, V., Sanzari, M., Cafaro, B., Nardi, F., Natola, F., Pirri, F., & Ruiz, M. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.268 · Cited by 18 · see citing works
Behavior Discovery and Alignment of Articulated Object Classes from Unstructured Video
Pero, L. D., Ricco, S., Sukthankar, R., & Ferrari, V. (2016) · International Journal of Computer Vision · doi:10.1007/s11263-016-0939-9 · Cited by 18 · see citing works
Animated 3D Creatures from Single-view Video by Skeletal Sketching
Reinert, B., Ritschel, T., & Seidel, H. (2016) · MPG.PuRe (Max Planck Society) · doi:10.5555/3076132.3076159 · Cited by 10 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Perspectives in machine learning for wildlife conservation
Tuia, D., Kellenberger, B., Beery, S., Costelloe, B. R., Zuffi, S., Risse, B., Mathis, A., Mathis, M. W., van Langevelde, F., Burghardt, T., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-27980-y · Cited by 740 · see citing works
Self-citationDeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning
Graving, J. M., Chae, D., Naik, H., Li, L., Koger, B., Costelloe, B. R., & Couzin, I. D. (2019) · eLife · doi:10.7554/elife.47994 · Cited by 547 · see citing works
Deep learning tools for the measurement of animal behavior in neuroscience
Mathis, M. W., & Mathis, A. (2020) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2019.10.008 · Cited by 450 · see citing works
A Primer on Motion Capture with Deep Learning: Principles, Pitfalls, and Perspectives
Mathis, A., Schneider, S., Lauer, J., & Mathis, M. W. (2020) · Neuron · doi:10.1016/j.neuron.2020.09.017 · Cited by 231 · see citing works
Three-D Safari: Learning to Estimate Zebra Pose, Shape, and Texture From Images “In the Wild”
Zuffi, S., Kanazawa, A., Berger-Wolf, T., & Black, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00546 · Cited by 139 · see citing works
“Existing methods require manual annotation of the test images with keypoints and/or segmentation [33, 34] or clean imaging conditions where automatic segmentation is possible [5].”
MethodInfluentialSelf-citationWho Left the Dogs Out? 3D Animal Reconstruction with Expectation Maximization in the Loop
Biggs, B., Boyne, O., Charles, J., Fitzgibbon, A., & Cipolla, R. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58621-8_12 · Cited by 107 · see citing works
NeuroMechFly, a neuromechanical model of adult Drosophila melanogaster
Lobato-Rios, V., Ramalingasetty, S. T., Özdil, P. G., Arreguit, J., Ijspeert, A. J., & Ramdya, P. (2022) · Nature Methods · doi:10.1038/s41592-022-01466-7 · Cited by 88 · see citing works
Creatures Great and SMAL: Recovering the Shape and Motion of Animals from Video
Biggs, B., Roddick, T., Fitzgibbon, A., & Cipolla, R. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-20873-8_1 · Cited by 89 · see citing works
Birds of a Feather: Capturing Avian Shape Models from Images
Wang, Y., Kolotouros, N., Daniilidis, K., & Badger, M. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01450 · Cited by 28 · see citing works
Transferring Dense Pose to Proximal Animal Classes
Sanakoyeu, A., Khalidov, V., McCarthy, M. S., Vedaldi, A., & Neverova, N. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00528 · Cited by 13 · see citing works
- Authors
- Zuffi, S., Kanazawa, A., Jacobs, D. W., & Black, M. J.
- Affiliations
- Istituto di Matematica Applicata e Tecnologie Informatiche (IT)
- University of Maryland, College Park (US)
- Max Planck Institute for Intelligent Systems (DE)
- Venue
- 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2017-07
- Pages
- 5524-5532
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2555859288
- Semantic Scholar
- 870a7d08978d39cd43973a1100abf7e30fd94cbd
- Cited by
- 381
- References
- 31
- Citations by year
- 2026: 35 · 2025: 41 · 2024: 56 · 2023: 75 · 2022: 73 · 2021: 41 · 2020: 33 · 2019: 17 · 2018: 9 · 2017: 1
- FWCI
- 24.70
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Engineering › Computational Mechanics › 3D Shape Modeling and Analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Processing and 3D Reconstruction
- Keywords
- statistical shape models, pose normalization, animal pose estimation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
