Trajectory Space: A Dual Representation for Nonrigid Structure from Motion
The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Abstract
Existing approaches to nonrigid structure from motion assume that the instantaneous 3D shape of a deforming object is a linear combination of basis shapes. These bases are object dependent and therefore have to be estimated anew for each video sequence. In contrast, we propose a dual approach to describe the evolving 3D structure in trajectory space by a linear combination of basis trajectories. We describe the dual relationship between the two approaches, showing that they both have equal power for representing 3D structure. We further show that the temporal smoothness in 3D trajectories alone can be used for recovering nonrigid structure from a moving camera. The principal advantage of expressing deforming 3D structure in trajectory space is that we can define an object independent basis. This results in a significant reduction in unknowns and corresponding stability in estimation. We propose the use of the Discrete Cosine Transform (DCT) as the object independent basis and empirically demonstrate that it approaches Principal Component Analysis (PCA) for natural motions. We report the performance of the proposed method, quantitatively using motion capture data, and qualitatively on several video sequences exhibiting nonrigid motions, including piecewise rigid motion, partially nonrigid motion (such as a facial expressions), and highly nonrigid motion (such as a person walking or dancing).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Multiple View Geometry in Computer Vision
Hartley, R., & Zisserman, A. (2004) · doi:10.1017/cbo9780511811685 · Cited by 21,070
Fundamentals of digital image processing
Jain, A. K. (1988) · Cited by 7,111
Ahmed, N., Natarajan, T., & Rao, K. R. (1974) · IEEE Transactions on Computers · doi:10.1109/t-c.1974.223784 · Cited by 5,156
Fundamentals of digital image processing
(1989) · Computer Vision, Graphics, and Image Processing · doi:10.1016/0734-189x(89)90041-8 · Cited by 4,677
Visual perception of biological motion and a model for its analysis
Johansson, G. (1973) · Perception & Psychophysics · doi:10.3758/bf03212378 · Cited by 4,462
Shape and motion from image streams under orthography: a factorization method
Tomasi, C., & Kanade, T. (1992) · International Journal of Computer Vision · doi:10.1007/bf00129684 · Cited by 2,878
Self-citationMultiple View Geometry in Computer Vision (2nd ed)
Hartley, R. I., & Zisserman, A. (2003) · Cambridge University Press eBooks · Cited by 2,761
A computer algorithm for reconstructing a scene from two projections
Longuet-Higgins, H. C. (1981) · Nature · doi:10.1038/293133a0 · Cited by 2,383
Discrete Cosine Transform: Algorithms, Advantages, Applications
Rao, K. R., Yip, P. C. Y., & Britaňák, V. (1990) · Cited by 1,862
Calculating the Singular Values and Pseudo-Inverse of a Matrix
Golub, G., & Kahan, W. (1965) · Journal of the Society for Industrial and Applied Mathematics Series B Numerical Analysis · doi:10.1137/0702016 · Cited by 1,786 · cited 1×
“The shape and trajectory eigenvalues and eigenvectors are related by the following equations [ 45 ]:”
MethodMa, Y., Soatto, S., Košecká, J., & Sastry, S. S. (2004) · Interdisciplinary Applied Mathematics · doi:10.1007/978-0-387-21779-6 · Cited by 893
Recovering non-rigid 3D shape from image streams
Bregler, C., Hertzmann, A., & Biermann, H. (2002) · Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662) · doi:10.1109/cvpr.2000.854941 · Cited by 754
A Multibody Factorization Method for Independently Moving Objects
Costeira, J. P., & Kanade, T. (1998) · International Journal of Computer Vision · doi:10.1023/a:1008000628999 · Cited by 661
Self-citationRao, K. R., & Yip, P. C. Y. (1990) · Elsevier eBooks · doi:10.1016/b978-0-08-092534-9.50007-2 · Cited by 659
Faugeras, O., Luong, Q.-T., & Papadopoulo, T. (2001) · The MIT Press eBooks · doi:10.7551/mitpress/3259.001.0001 · Cited by 608
Nonrigid Structure-from-Motion: Estimating Shape and Motion with Hierarchical Priors
Torresani, L., Hertzmann, A., & Bregler, C. (2008) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.70752 · Cited by 456 · cited 8×
“This allows us to handle more nonrigidity of deformation than state-of-art methods such as [5] and [ 6 ].”
MethodResultInfluentialFace Recognition Using the Discrete Cosine Transform
Hafed, Z. M., & Levine, M. D. (2001) · International Journal of Computer Vision · doi:10.1023/a:1011183429707 · Cited by 426 · cited 1×
“The optimality of DCT for Markov models has been exploited in numerous applications, such as transform coding of speech signals [46], speech recognition [47], spectral document ranking [48], image compression [3], and face recognition [ 49 ].”
MethodMotion texture: a two-level statistical model for character motion synthesis
Li, Y., Wang, T., & Shum, H.-Y. (2002) · Proceedings of the 29th annual conference on Computer graphics and interactive techniques · doi:10.1145/566570.566604 · Cited by 392 · cited 1×
“In [ 50 ], Li and Wang model the motion capture data sets with a first-order Markov chain for synthesizing human motion.”
MethodMaximizing Rigidity: The Incremental Recovery of 3-D Structure from Rigid and Nonrigid Motion
Ullman, S. (1984) · Perception · doi:10.1068/p130255 · Cited by 293 · cited 1×
“Within the computer vision community, the design of algorithms to reconstruct nonrigidly deforming structure began in earnest in the 1980s, investigating constraints like maximal rigidity, isometry, symmetry, and linear representations in low dimensions in [ 18 ], [19], [20], [21].”
MethodAdaptive transform coding of speech signals
Zelinski, R., & Noll, P. (1977) · IEEE Transactions on Acoustics, Speech, and Signal Processing · doi:10.1109/tassp.1977.1162974 · Cited by 259 · cited 1×
“The optimality of DCT for Markov models has been exploited in numerous applications, such as transform coding of speech signals [ 46 ], speech recognition [47], spectral document ranking [48], image compression [3], and face recognition [49].”
Tracking and modeling non-rigid objects with rank constraints
Torresani, L., Yang, D. B., Alexander, E. J., & Bregler, C. (2005) · Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001 · doi:10.1109/cvpr.2001.990515 · Cited by 226
Nonrigid Structure from Motion in Trajectory Space
Akhter, I., Sheikh, Y. A., Khan, S., & Takeo Kanade, A. (2018) · Figshare · doi:10.1184/r1/6556985 · Cited by 196
Self-citationMotion texture: a two-level statistical model for character motion synthesis
Li, Y., Wang, T., & Shum, H.-Y. (2002) · ACM Transactions on Graphics · doi:10.1145/566654.566604 · Cited by 187
Morphable 3D models from video
Brand, M. (2001) · Cited by 177
Motion segmentation with missing data using powerfactorization and GPCA
Vidal, R., & Hartley, R. (2004) · Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004. · doi:10.1109/cvpr.2004.1315180 · Cited by 176
Capturing and animating occluded cloth
White, R., Crane, K., & Forsyth, D. A. (2007) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/1275808.1276420 · Cited by 165 · cited 1×
“For quantitative evaluation of the reconstruction accuracy, we use five different actions from the CMU motion capture database (drinking, pickup, yoga, stretch, and dance actions), shark, and face sequence available on the project website of Torresani et al. [7] and pants sequence by White et al. [ 53 ].”
MethodMorphable 3D models from video
Brand, W. (2005) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2001.990997 · Cited by 155
A Closed-Form Solution to Non-Rigid Shape and Motion Recovery
Xiao, J., Chai, J., & Kanade, T. (2006) · International Journal of Computer Vision · doi:10.1007/s11263-005-3962-9 · Cited by 143
Self-citationLearning Non-Rigid 3D Shape from 2D Motion
Torresani, L., Hertzmann, A., & Bregler, C. (2003) · Cited by 139
Coarse-to-fine low-rank structure-from-motion
Bartoli, A., Gay‐Bellile, V., Castellani, U., Peyras, J., Olsen, S. I., & Sayd, P. (2008) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2008.4587694 · Cited by 133
A Direct Method for 3D Factorization of Nonrigid Motion Observed in 2D
Brand, M. (2005) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2005.23 · Cited by 121
Capturing and animating occluded cloth
White, R., Crane, K., & Forsyth, D. A. (2007) · ACM Transactions on Graphics · doi:10.1145/1276377.1276420 · Cited by 103
In defense of orthonormality constraints for nonrigid structure from motion
Akhter, I., Sheikh, Y. A., & Khan, S. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2009.5206620 · Cited by 92
Self-citationArticulated Structure from Motion by Factorization
Tresadern, P., & Reid, I. D. (2005) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2005.75 · Cited by 88
Wolf, L., & Shashua, A. (2002) · International Journal of Computer Vision · doi:10.1023/a:1014855311993 · Cited by 75
Trilinear tensor: The fundamental construct of multiple-view geometry and its applications
Shashua, A. (1997) · Lecture notes in computer science · doi:10.1007/bfb0017868 · Cited by 63
Re-thinking non-rigid structure from motion
Rabaud, V., & Belongie, S. (2008) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2008.4587679 · Cited by 62 · cited 1×
“Recently, Rabaud and Belongie [34] proposed nonlinear subspace reduction by imposing locally linear subspace compaction on nonrigid structure [ 39 ].”
BackgroundAutomatic Kinematic Chain Building from Feature Trajectories of Articulated Objects
Yan, J., & Pollefeys, M. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 1 (CVPR'06) · doi:10.1109/cvpr.2006.66 · Cited by 60
A factorization approach to structure from motion with shape priors
Bue, A. D. (2008) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2008.4587708 · Cited by 60
Non-rigid structure from motion using ranklet-based tracking and non-linear optimization
Bue, A. D., Smeraldi, F., & Agapito, L. (2006) · Image and Vision Computing · doi:10.1016/j.imavis.2005.10.004 · Cited by 57 · cited 1×
“In fact, most previous results consider deformations which have a large rigid component, such as talking-head videos or the motion of a swimming shark [2], [5], [6], [7], [8], [ 9 ], [10], [11].”
A DCT-based fast signal subspace technique for robust speech recognition
Huang, J., & Zhao, Y. (2000) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.876314 · Cited by 37 · cited 1×
“The optimality of DCT for Markov models has been exploited in numerous applications, such as transform coding of speech signals [46], speech recognition [ 47 ], spectral document ranking [48], image compression [3], and face recognition [49].”
BackgroundLinear embeddings in non-rigid structure from motion
Rabaud, V., & Belongie, S. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2009.5206628 · Cited by 25
Linear motion estimation for systems of articulated planes
Datta, A., Sheikh, Y. A., & Takeo Kanade, A. (2008) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2008.4587579 · Cited by 18
Self-citation
Cited by
Lifting from the Deep: Convolutional 3D Pose Estimation from a Single Image
Tome, D., Russell, C., & Agapito, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.603 · Cited by 504
Sparseness Meets Deepness: 3D Human Pose Estimation from Monocular Video
Zhou, X., Zhu, M., Leonardos, S., Derpanis, K. G., & Daniilidis, K. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.537 · Cited by 413
Dense Variational Reconstruction of Non-rigid Surfaces from Monocular Video
Garg, R., Roussos, A., & Agapito, L. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.168 · Cited by 210
Ionasec, R. I., Voigt, I., Georgescu, B., Wang, Y., Houle, H., Vega-Higuera, F., Navab, N., & Comaniciu, D. (2010) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2010.2048756 · Cited by 194
Metadata
- Authors
- Akhter, I., Sheikh, Y., Khan, S., & Kanade, T.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2011-07
- Volume
- 33
- Issue
- 7
- Pages
- 1442-1456
- ISSN
- 0162-8828 (ISSN-L)
- OpenAlex
- W2102813107
- Cited by
- 217
- References
- 43
- FWCI
- 16.22
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.