The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
Real-time non-rigid reconstruction using an RGB-D camera
Abstract
We present a combined hardware and software solution for markerless reconstruction of non-rigidly deforming physical objects with arbitrary shape in real-time . Our system uses a single self-contained stereo camera unit built from off-the-shelf components and consumer graphics hardware to generate spatio-temporally coherent 3D models at 30 Hz. A new stereo matching algorithm estimates real-time RGB-D data. We start by scanning a smooth template model of the subject as they move rigidly. This geometric surface prior avoids strong scene assumptions, such as a kinematic human skeleton or a parametric shape model. Next, a novel GPU pipeline performs non-rigid registration of live RGB-D data to the smooth template using an extended non-linear as-rigid-as-possible (ARAP) framework. High-frequency details are fused onto the final mesh using a linear deformation model. The system is an order of magnitude faster than state-of-the-art methods, while matching the quality and robustness of many offline algorithms. We show precise real-time reconstructions of diverse scenes, including: large deformations of users' heads, hands, and upper bodies; fine-scale wrinkles and folds of skin and clothing; and non-rigid interactions performed by users on flexible objects such as toys. We demonstrate how acquired models can be used for many interactive scenarios, including re-texturing, online performance capture and preview, and real-time shape and motion re-targeting.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A morphable model for the synthesis of 3D faces
Blanz, V., & Vetter, T. (1999) · Proceedings of the 26th annual conference on Computer graphics and interactive techniques - SIGGRAPH '99 · doi:10.1145/311535.311556 · Cited by 4,710
KinectFusion: Real-time dense surface mapping and tracking
Newcombe, R. A., Fitzgibbon, A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A. J., Kohi, P., Shotton, J., & Hodges, S. (2011) · 2011 10th IEEE International Symposium on Mixed and Augmented Reality · doi:10.1109/ismar.2011.6092378 · Cited by 4,008
Self-citationKinectFusion: real-time 3D reconstruction and interaction using a moving depth camera
Izadi, S., Kim, D., Hilliges, O., Molyneaux, D., Newcombe, R., Kohli, P., Shotton, J., Hodges, S., Freeman, D., Davison, A., et al. (2011) · Proceedings of the 24th annual ACM symposium on User interface software and technology · doi:10.1145/2047196.2047270 · Cited by 2,110
Self-citationDeformation transfer for triangle meshes
Sumner, R. W., & Popović, J. (2004) · ACM Transactions on Graphics · doi:10.1145/1015706.1015736 · Cited by 1,052
Real-time 3D reconstruction at scale using voxel hashing
Nießner, M., Zollhöfer, M., Izadi, S., & Stamminger, M. (2013) · ACM Transactions on Graphics · doi:10.1145/2508363.2508374 · Cited by 1,002
Self-citationEfficient model-based 3D tracking of hand articulations using Kinect
Oikonomidis, I., Kyriazis, N., & Argyros, A. (2011) · Procedings of the British Machine Vision Conference 2011 · doi:10.5244/c.25.101 · Cited by 874
As-Rigid-As-Possible Surface Modeling
Sorkine‐Hornung, O., & Alexa, M. (2007) · doi:10.2312/sgp/sgp07/109-116 · Cited by 778
PatchMatch Stereo - Stereo Matching with Slanted Support Windows
Bleyer, M., Rhemann, C., & Rother, C. (2011) · Procedings of the British Machine Vision Conference 2011 · doi:10.5244/c.25.14 · Cited by 736
Improved Computation for Levenberg–Marquardt Training
Wilamowski, B. M., & Yu, H. (2010) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2010.2045657 · Cited by 622
Articulated mesh animation from multi-view silhouettes
Vlasic, D., Baran, I., Matusik, W., & Popović, J. (2008) · ACM Transactions on Graphics · doi:10.1145/1360612.1360696 · Cited by 610
Cited by
Face2Face: Real-Time Face Capture and Reenactment of RGB Videos
Thies, J., Zollhofer, M., Stamminger, M., Theobalt, C., & Niessner, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.262 · Cited by 1,747
Self-citationVNect: real-time 3D human pose estimation with a single RGB camera
Mehta, D., Sridhar, S., Sotnychenko, O., Rhodin, H., Shafiei, M., Seidel, H.-P., Xu, W., Casas, D., & Theobalt, C. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073596 · Cited by 1,113
Self-citationNerfies: Deformable Neural Radiance Fields
Park, K., Sinha, U., Barron, J. T., Bouaziz, S., Goldman, D. B., Seitz, S. M., & Martin-Brualla, R. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00581 · Cited by 1,059
DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time
Newcombe, R. A., Fox, D., & Seitz, S. M. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298631 · Cited by 951
Neural volumes: learning dynamic renderable volumes from images
Lombardi, S., Simon, T., Saragih, J., Schwartz, G., Lehrmann, A., & Sheikh, Y. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3323020 · Cited by 693
Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes
Li, Z., Niklaus, S., Snavely, N., & Wang, O. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00643 · Cited by 674
BundleFusion: Real-Time Globally Consistent 3D Reconstruction Using On-the-Fly Surface Reintegration
Dai, A., Nießner, M., Zollhöfer, M., Izadi, S., & Theobalt, C. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3054739 · Cited by 662
Self-citationHoloportation: Virtual 3D Teleportation in Real-time
Orts-Escolano, S., Rhemann, C., Fanello, S., Chang, W., Kowdle, A., Degtyarev, Y., Kim, D., Davidson, P. L., Khamis, S., Dou, M., et al. (2016) · Proceedings of the 29th Annual Symposium on User Interface Software and Technology · doi:10.1145/2984511.2984517 · Cited by 662
Self-citationHigh-quality streamable free-viewpoint video
Collet, A., Chuang, M., Sweeney, P., Gillett, D., Evseev, D., Calabrese, D., Hoppe, H., Kirk, A., & Sullivan, S. (2015) · ACM Transactions on Graphics · doi:10.1145/2766945 · Cited by 573
Masood, M., Nawaz, M., Malik, K. M., Javed, A., Irtaza, A., & Malik, H. (2023) · Applied Intelligence · doi:10.1007/s10489-022-03766-z · Cited by 543
Fusion4D: real-time performance capture of challenging scenes
Dou, M., Khamis, S., Degtyarev, Y., Davidson, P., Fanello, S. R., Kowdle, A., Escolano, S. O., Rhemann, C., Kim, D., Taylor, J., et al. (2016) · ACM Transactions on Graphics · doi:10.1145/2897824.2925969 · Cited by 500
Self-citationVideo Based Reconstruction of 3D People Models
Alldieck, T., Magnor, M., Xu, W., Theobalt, C., & Pons-Moll, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00875 · Cited by 468
Self-citationTretschk, E., Tewari, A., Golyanik, V., Zollhofer, M., Lassner, C., & Theobalt, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01272 · Cited by 432
Self-citationMaskFusion: Real-Time Recognition, Tracking and Reconstruction of Multiple Moving Objects
Runz, M., Buffier, M., & Agapito, L. (2018) · 2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) · doi:10.1109/ismar.2018.00024 · Cited by 415
Real-time expression transfer for facial reenactment
Thies, J., Zollhöfer, M., Nießner, M., Valgaerts, L., Stamminger, M., & Theobalt, C. (2015) · ACM Transactions on Graphics · doi:10.1145/2816795.2818056 · Cited by 369
Self-citationDynamic FAUST: Registering Human Bodies in Motion
Bogo, F., Romero, J., Pons-Moll, G., & Black, M. J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.591 · Cited by 358
Space-time Neural Irradiance Fields for Free-Viewpoint Video
Xian, W., Huang, J.-B., Kopf, J., & Kim, C. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00930 · Cited by 357
Function4D: Real-time Human Volumetric Capture from Very Sparse Consumer RGBD Sensors
Yu, T., Zheng, Z., Guo, K., Liu, P., Dai, Q., & Liu, Y. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00569 · Cited by 355
Zhu, Z., Ng, D. W. H., Park, H. S., & McAlpine, M. C. (2020) · Nature Reviews Materials · doi:10.1038/s41578-020-00235-2 · Cited by 348
Huang, Y., Kaufmann, M., Aksan, E., Black, M. J., Hilliges, O., & Pons-Moll, G. (2018) · ACM Transactions on Graphics · doi:10.1145/3272127.3275108 · Cited by 342
State of the Art on 3D Reconstruction with RGB‐D Cameras
Zollhöfer, M., Stotko, P., Görlitz, A., Theobalt, C., Nießner, M., Klein, R., & Kolb, A. (2018) · Computer Graphics Forum · doi:10.1111/cgf.13386 · Cited by 339
Self-citationState of research in automatic as-built modelling
Pătrăucean, V., Armeni, I., Nahangi, M., Yeung, J., Brilakis, I., & Haas, C. (2015) · Advanced Engineering Informatics · doi:10.1016/j.aei.2015.01.001 · Cited by 328
Learning to Reconstruct People in Clothing From a Single RGB Camera
Alldieck, T., Magnor, M., Bhatnagar, B. L., Theobalt, C., & Pons-Moll, G. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00127 · Cited by 326
Self-citationVolumeDeform: Real-Time Volumetric Non-rigid Reconstruction
Innmann, M., Zollhöfer, M., Nießner, M., Theobalt, C., & Stamminger, M. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46484-8_22 · Cited by 308
Self-citationModeling, learning, perception, and control methods for deformable object manipulation
Yin, H., Varava, A., & Kragic, D. (2021) · Science Robotics · doi:10.1126/scirobotics.abd8803 · Cited by 299
Detailed, Accurate, Human Shape Estimation from Clothed 3D Scan Sequences
Zhang, C., Pujades, S., Black, M., & Pons-Moll, G. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.582 · Cited by 295
3DRegNet: A Deep Neural Network for 3D Point Registration
Pais, G. D., Ramalingam, S., Govindu, V. M., Nascimento, J. C., Chellappa, R., & Miraldo, P. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00722 · Cited by 268
Reconstruction of Personalized 3D Face Rigs from Monocular Video
Garrido, P., Zollhöfer, M., Casas, D., Valgaerts, L., Varanasi, K., Pérez, P., & Theobalt, C. (2016) · ACM Transactions on Graphics · doi:10.1145/2890493 · Cited by 260
Self-citationBundleFusion: Real-Time Globally Consistent 3D Reconstruction Using On-the-Fly Surface Reintegration
Dai, A., Nießner, M., Zollhöfer, M., Izadi, S., & Theobalt, C. (2017) · ACM Transactions on Graphics · doi:10.1145/3054739 · Cited by 258
Self-citationYu, T., Zheng, Z., Guo, K., Zhao, J., Dai, Q., Li, H., Pons-Moll, G., & Liu, Y. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00761 · Cited by 255
A Perspective on Open Challenges in Deformable Object Manipulation
McKennaa, R. P., & Oyekan, J. (2026) · arXiv · doi:10.48550/arxiv.2602.22998 · Cited by 1
“Nonlinear computational models generally struggle to achieve real-time performance, except in cases where explicit integration or hardware acceleration [62] is employed.”
Metadata
- Authors
- Zollhöfer, M., Nießner, M., Izadi, S., Rehmann, C., Zach, C., Fisher, M., Wu, C., Fitzgibbon, A., Loop, C., Theobalt, C., & Stamminger, M.
- Venue
- ACM Transactions on Graphics
- Published
- 2014-07-27
- Volume
- 33
- Issue
- 4
- Pages
- 1-12
- ISSN
- 0730-0301 (ISSN-L)
- 1557-7368
- OpenAlex
- W2075402943
- Cited by
- 405
- References
- 56
- FWCI
- 37.57
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.