The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
Real-time non-rigid reconstruction using an RGB-D camera
Abstract
We present a combined hardware and software solution for markerless reconstruction of non-rigidly deforming physical objects with arbitrary shape in real-time . Our system uses a single self-contained stereo camera unit built from off-the-shelf components and consumer graphics hardware to generate spatio-temporally coherent 3D models at 30 Hz. A new stereo matching algorithm estimates real-time RGB-D data. We start by scanning a smooth template model of the subject as they move rigidly. This geometric surface prior avoids strong scene assumptions, such as a kinematic human skeleton or a parametric shape model. Next, a novel GPU pipeline performs non-rigid registration of live RGB-D data to the smooth template using an extended non-linear as-rigid-as-possible (ARAP) framework. High-frequency details are fused onto the final mesh using a linear deformation model. The system is an order of magnitude faster than state-of-the-art methods, while matching the quality and robustness of many offline algorithms. We show precise real-time reconstructions of diverse scenes, including: large deformations of users' heads, hands, and upper bodies; fine-scale wrinkles and folds of skin and clothing; and non-rigid interactions performed by users on flexible objects such as toys. We demonstrate how acquired models can be used for many interactive scenarios, including re-texturing, online performance capture and preview, and real-time shape and motion re-targeting.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A morphable model for the synthesis of 3D faces
Blanz, V., & Vetter, T. (1999) · Proceedings of the 26th annual conference on Computer graphics and interactive techniques - SIGGRAPH '99 · doi:10.1145/311535.311556 · Cited by 4,710
KinectFusion: Real-time dense surface mapping and tracking
Newcombe, R. A., Fitzgibbon, A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A. J., Kohi, P., Shotton, J., & Hodges, S. (2011) · 2011 10th IEEE International Symposium on Mixed and Augmented Reality · doi:10.1109/ismar.2011.6092378 · Cited by 4,008
Self-citationKinectFusion: real-time 3D reconstruction and interaction using a moving depth camera
Izadi, S., Kim, D., Hilliges, O., Molyneaux, D., Newcombe, R., Kohli, P., Shotton, J., Hodges, S., Freeman, D., Davison, A., et al. (2011) · Proceedings of the 24th annual ACM symposium on User interface software and technology · doi:10.1145/2047196.2047270 · Cited by 2,110
Self-citationDeformation transfer for triangle meshes
Sumner, R. W., & Popović, J. (2004) · ACM Transactions on Graphics · doi:10.1145/1015706.1015736 · Cited by 1,052
Real-time 3D reconstruction at scale using voxel hashing
Nießner, M., Zollhöfer, M., Izadi, S., & Stamminger, M. (2013) · ACM Transactions on Graphics · doi:10.1145/2508363.2508374 · Cited by 1,002
Self-citationEfficient model-based 3D tracking of hand articulations using Kinect
Oikonomidis, I., Kyriazis, N., & Argyros, A. (2011) · Procedings of the British Machine Vision Conference 2011 · doi:10.5244/c.25.101 · Cited by 874
As-Rigid-As-Possible Surface Modeling
Sorkine‐Hornung, O., & Alexa, M. (2007) · doi:10.2312/sgp/sgp07/109-116 · Cited by 778
PatchMatch Stereo - Stereo Matching with Slanted Support Windows
Bleyer, M., Rhemann, C., & Rother, C. (2011) · Procedings of the British Machine Vision Conference 2011 · doi:10.5244/c.25.14 · Cited by 736
Improved Computation for Levenberg–Marquardt Training
Wilamowski, B. M., & Yu, H. (2010) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2010.2045657 · Cited by 622
Articulated mesh animation from multi-view silhouettes
Vlasic, D., Baran, I., Matusik, W., & Popović, J. (2008) · ACM Transactions on Graphics · doi:10.1145/1360612.1360696 · Cited by 610
On Linear Variational Surface Deformation Methods
Botsch, M., & Sorkine, O. (2008) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2007.1054 · Cited by 582
Performance capture from sparse multi-view video
de Aguiar, E., Stoll, C., Theobalt, C., Ahmed, N., Seidel, H.-P., & Thrun, S. (2008) · ACM Transactions on Graphics · doi:10.1145/1360612.1360697 · Cited by 535
Self-citationScanning 3D Full Human Bodies Using Kinects
Tong, J., Zhou, J., Liu, L., Pan, Z., & Yan, H. (2012) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2012.56 · Cited by 498
Realtime performance-based facial animation
Weise, T., Bouaziz, S., Li, H., & Pauly, M. (2011) · ACM SIGGRAPH 2011 papers · doi:10.1145/1964921.1964972 · Cited by 460
Surface Capture for Performance-Based Animation
Starck, J., & Hilton, A. (2007) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.2007.68 · Cited by 452
Embedded deformation for shape manipulation
Sumner, R. W., Schmid, J., & Pauly, M. (2007) · ACM SIGGRAPH 2007 papers · doi:10.1145/1275808.1276478 · Cited by 449
Global Correspondence Optimization for Non‐Rigid Registration of Depth Scans
Li, H., Sumner, R. W., & Pauly, M. (2008) · Computer Graphics Forum · doi:10.1111/j.1467-8659.2008.01282.x · Cited by 399
Realtime performance-based facial animation
Weise, T., Bouaziz, S., Li, H., & Pauly, M. (2011) · ACM Transactions on Graphics · doi:10.1145/2010324.1964972 · Cited by 356
3D shape regression for real-time facial animation
Cao, C., Weng, Y., Lin, S., & Zhou, K. (2013) · ACM Transactions on Graphics · doi:10.1145/2461912.2462012 · Cited by 348
Embedded deformation for shape manipulation
Sumner, R. W., Schmid, J., & Pauly, M. (2007) · ACM Transactions on Graphics · doi:10.1145/1276377.1276478 · Cited by 337
Robust single-view geometry and motion reconstruction
Li, H., Adams, B., Guibas, L. J., & Pauly, M. (2009) · ACM Transactions on Graphics · doi:10.1145/1618452.1618521 · Cited by 334
Home 3D body scans from noisy image and range data
Weiss, A., Hirshberg, D., & Black, M. J. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126465 · Cited by 306
Realtime facial animation with on-the-fly correctives
Li, H., Yu, J., Ye, Y., & Bregler, C. (2013) · ACM Transactions on Graphics · doi:10.1145/2461912.2462019 · Cited by 282
The Vitruvian manifold: Inferring dense correspondences for one-shot human pose estimation
Taylor, J., Shotton, J., Sharp, T., & Fitzgibbon, A. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247664 · Cited by 270
Self-citationDynamic shape capture using multi-view photometric stereo
Vlasic, D., Peers, P., Baran, I., Debevec, P., Popović, J., Rusinkiewicz, S., & Matusik, W. (2009) · ACM Transactions on Graphics · doi:10.1145/1618452.1618520 · Cited by 238
High-quality passive facial performance capture using anchor frames
Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P., Gotsman, C., Sumner, R. W., & Gross, M. (2011) · ACM SIGGRAPH 2011 papers · doi:10.1145/1964921.1964970 · Cited by 238
High-quality passive facial performance capture using anchor frames
Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P., Gotsman, C., Sumner, R. W., & Gross, M. (2011) · ACM Transactions on Graphics · doi:10.1145/2010324.1964970 · Cited by 223
Accurate realtime full-body motion capture using a single depth camera
Wei, X., Zhang, P., & Chai, J. (2012) · ACM Transactions on Graphics · doi:10.1145/2366145.2366207 · Cited by 214
Reconstructing detailed dynamic face geometry from monocular video
Garrido, P., Valgaert, L., Wu, C., & Theobalt, C. (2013) · ACM Transactions on Graphics · doi:10.1145/2508363.2508380 · Cited by 204
Self-citationDeformation transfer for triangle meshes
Sumner, R. W., & Popović, J. (2004) · ACM SIGGRAPH 2004 Papers · doi:10.1145/1186562.1015736 · Cited by 203
Face/Off: live facial puppetry
Weise, T., Li, H., Van Gool, L., & Pauly, M. (2009) · Proceedings of the 2009 ACM SIGGRAPH/Eurographics Symposium on Computer Animation · doi:10.1145/1599470.1599472 · Cited by 194
Articulated mesh animation from multi-view silhouettes
Vlasic, D., Baran, I., Matusik, W., & Popović, J. (2008) · ACM SIGGRAPH 2008 papers · doi:10.1145/1399504.1360696 · Cited by 187
Bradley, D., Popa, T., Sheffer, A., Heidrich, W., & Boubekeur, T. (2008) · ACM Transactions on Graphics · doi:10.1145/1360612.1360698 · Cited by 176
Global non-rigid alignment of 3-D scans
Brown, B. J., & Rusinkiewicz, S. M. (2007) · ACM Transactions on Graphics · doi:10.1145/1276377.1276404 · Cited by 165
Time-of-Flight Sensors in Computer Graphics
Kolb, A., Barth, E., Koch, R., & Larsen, R. (2009) · Eurographics · doi:10.2312/egst.20091064 · Cited by 164
Li, H., Vouga, E., Gudym, A., Luo, L., Barron, J. T., & Gusev, G. (2013) · ACM Transactions on Graphics · doi:10.1145/2508363.2508407 · Cited by 162
Non-rigid Photometric Stereo with Colored Lights
Hernandez, C., Vogiatzis, G., Brostow, G. J., Stenger, B., & Cipolla, R. (2007) · 2007 IEEE 11th International Conference on Computer Vision · doi:10.1109/iccv.2007.4408939 · Cited by 158
Lightweight binocular facial performance capture under uncontrolled lighting
Valgaerts, L., Wu, C., Bruhn, A., Seidel, H., & Theobalt, C. (2012) · ACM Transactions on Graphics · doi:10.1145/2366145.2366206 · Cited by 143
Self-citationEfficient reconstruction of nonrigid shape and motion from real-time 3D scanner data
Wand, M., Adams, B., Ovsjanikov, M., Berner, A., Bokeloh, M., Jenke, P., Guibas, L., Seidel, H.-P., & Schilling, A. (2009) · ACM Transactions on Graphics · doi:10.1145/1516522.1516526 · Cited by 136
MonoFusion: Real-time 3D reconstruction of small scenes with a single web camera
Pradeep, V., Rhemann, C., Izadi, S., Zach, C., Bleyer, M., & Bathiche, S. (2013) · Proceedings - International Symposium on Mixed and Augmented Reality, ISMAR · doi:10.1109/ismar.2013.6671767 · Cited by 131
Self-citationIn-hand scanning with online loop closure
Weise, T., Wismer, T., Leibe, B., & Gool, L. V. (2009) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2009.5457479 · Cited by 117
Performance Capture of Interacting Characters with Handheld Kinects
Ye, G., Liu, Y., Hasler, N., Ji, X., Dai, Q., & Theobalt, C. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33709-3_59 · Cited by 110
Self-citationScalable 3D video of dynamic scenes
Waschbüsch, M., Würmlin, S., Cotting, D., Sadlo, F., & Groß, M. H. (2005) · The Visual Computer · doi:10.1007/s00371-005-0346-7 · Cited by 107
Modeling deformable objects from a single depth camera
Liao, M., Zhang, Q., Wang, H., Yang, R., & Gong, M. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459161 · Cited by 101
Motion capture using joint skeleton tracking and surface estimation
Gall, J., Stoll, C., de Aguiar, E., Theobalt, C., Rosenhahn, B., & Seidel, H. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvprw.2009.5206755 · Cited by 101
Self-citationMitra, N. J., Flöry, S., Ovsjanikov, M., Gelfand, N., Guibas, L. J., & Pottmann, H. (2007) · doi:10.5555/1281991.1282015 · Cited by 101
Personalization and Evaluation of a Real-Time Depth-Based Full Body Tracker
Helten, T., Baak, A., Bharaj, G., Müller, M., Seidel, H., & Theobalt, C. (2013) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2013.44 · Cited by 97
Self-citationFree-form mesh tracking: A patch-based approach
Cagniart, C., Boyer, E., & Ilić, S. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5539814 · Cited by 91
Animation cartography—intrinsic reconstruction of shape and motion
Tevs, A., Berner, A., Wand, M., Ihrke, I., Bokeloh, M., Kerber, J., & Seidel, H. (2012) · ACM Transactions on Graphics · doi:10.1145/2159516.2159517 · Cited by 90
On-set performance capture of multiple actors with a stereo camera
Wu, C., Stoll, C., Valgaerts, L., & Theobalt, C. (2013) · ACM Transactions on Graphics · doi:10.1145/2508363.2508418 · Cited by 85
Self-citationScanning and tracking dynamic objects with commodity depth cameras
Dou, M., Fuchs, H., & Frahm, J. (2013) · Proceedings - International Symposium on Mixed and Augmented Reality, ISMAR · doi:10.1109/ismar.2013.6671769 · Cited by 76
Efficient GPU Data Structures and Methods to Solve Sparse Linear Systems in Dynamics Applications
Weber, D., Bender, J., Schnoes, M., Stork, A., & Fellner, D. W. (2012) · Computer Graphics Forum · doi:10.1111/j.1467-8659.2012.03227.x · Cited by 73
Templateless Quasi-rigid Shape Modeling with Implicit Loop-Closure
Zeng, M., Zheng, J., Cheng, X., & Liu, X. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.26 · Cited by 72
KinÊtre: animating the world with the human body
Chen, J., Izadi, S., & Fitzgibbon, A. W. (2012) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2380116.2380171 · Cited by 65
Self-citationTracking surfaces with evolving topology
Bojsen-Hansen, M., Li, H., & Wojtan, C. (2012) · ACM Transactions on Graphics · doi:10.1145/2185520.2185549 · Cited by 60
Performance Capture from Multi-View Video
Theobalt, C., de Aguiar, E., Stoll, C., Seidel, H., & Thrun, S. (2010) · Geometry and computing · doi:10.1007/978-3-642-12392-4_6 · Cited by 41
Self-citation
Cited by
Face2Face: Real-Time Face Capture and Reenactment of RGB Videos
Thies, J., Zollhofer, M., Stamminger, M., Theobalt, C., & Niessner, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.262 · Cited by 1,747
Self-citationVNect: real-time 3D human pose estimation with a single RGB camera
Mehta, D., Sridhar, S., Sotnychenko, O., Rhodin, H., Shafiei, M., Seidel, H.-P., Xu, W., Casas, D., & Theobalt, C. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073596 · Cited by 1,113
Self-citationNerfies: Deformable Neural Radiance Fields
Park, K., Sinha, U., Barron, J. T., Bouaziz, S., Goldman, D. B., Seitz, S. M., & Martin-Brualla, R. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00581 · Cited by 1,059
DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time
Newcombe, R. A., Fox, D., & Seitz, S. M. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298631 · Cited by 951
Neural volumes: learning dynamic renderable volumes from images
Lombardi, S., Simon, T., Saragih, J., Schwartz, G., Lehrmann, A., & Sheikh, Y. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3323020 · Cited by 693
Metadata
- Authors
- Zollhöfer, M., Nießner, M., Izadi, S., Rehmann, C., Zach, C., Fisher, M., Wu, C., Fitzgibbon, A., Loop, C., Theobalt, C., & Stamminger, M.
- Venue
- ACM Transactions on Graphics
- Published
- 2014-07-27
- Volume
- 33
- Issue
- 4
- Pages
- 1-12
- ISSN
- 0730-0301 (ISSN-L)
- 1557-7368
- OpenAlex
- W2075402943
- Cited by
- 405
- References
- 56
- FWCI
- 37.57
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.