KinectFusion: Real-time dense surface mapping and tracking
The DOI matches its Crossref record; 32 of the 34 references it cites were found in a registry or an index.
Abstract
We present a system for accurate real-time mapping of complex and arbitrary indoor scenes in variable lighting conditions, using only a moving low-cost depth camera and commodity graphics hardware. We fuse all of the depth data streamed from a Kinect sensor into a single global implicit surface model of the observed scene in real-time. The current sensor pose is simultaneously obtained by tracking the live depth frame relative to the global model using a coarse-to-fine iterative closest point (ICP) algorithm, which uses all of the observed depth data available. We demonstrate the advantages of tracking against the growing full surface model compared with frame-to-frame tracking, obtaining tracking and mapping results in constant time within room sized scenes with limited drift and high accuracy. We also show both qualitative and quantitative results relating to various aspects of our tracking and mapping system. Modelling of natural scenes, in real-time with only commodity sensor and GPU hardware, promises an exciting step forward in augmented reality (AR), in particular, it allows dense surfaces to be reconstructed in real-time, with a level of detail and robustness beyond any solution yet presented using passive computer vision.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
A method for registration of 3-D shapes
Besl, P. J., & McKay, N. D. (1992) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.121791 · Cited by 18,103 · cited 2×
“The most important class of algorithms have been based on the ICP concept introduced in [3] which poses data alignment as a nonlinear optimisation problem in which correspondences between scans are approximated using the closest pairs of points found between scans at the previous iteration.”
BackgroundMethodMarching cubes: A high resolution 3D surface construction algorithm
Lorensen, W. E., & Cline, H. E. (1987) · Proceedings of the 14th annual conference on Computer graphics and interactive techniques · doi:10.1145/37401.37422 · Cited by 10,104 · cited 1×
“One option is to extract the connected surfaces using a marching cubes type algorithm [18], followed by a standard rasterising rendering pipeline.”
MethodBilateral Filtering for Gray and Color Images
Tomasi, C., & Manduchi, R. (2002) · Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271) · doi:10.1109/iccv.1998.710815 · Cited by 8,177 · cited 1×
“We apply a bilateral filter [27] to the raw depth map to obtain a discontinuity preserved depth map with reduced noise Dk,”
MethodParallel Tracking and Mapping for Small AR Workspaces
Klein, G., & Murray, D. (2007) · 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality · doi:10.1109/ismar.2007.4538852 · Cited by 4,346 · cited 3×
“By using only depth data, our system can work in complete darkness mitigating any issues concerning low light conditions, problematic for passive camera [17, 19, 26] and RGB-D based systems [14].”
BackgroundMethodObject modelling by registration of multiple range images
Chen, Y., & Medioni, G. (1992) · Image and Vision Computing · doi:10.1016/0262-8856(92)90066-c · Cited by 3,154
A volumetric method for building complex models from range images
Curless, B., & Levoy, M. (1996) · Proceedings of the 23rd annual conference on Computer graphics and interactive techniques · doi:10.1145/237170.237269 · Cited by 3,124 · cited 6×
“A related non-parametric representation used in graphics is the signed distance function (SDF) introduced in [7] for the purpose of fusing partial depth scans while mitigating problems related to mesh-based reconstruction algorithms.”
BackgroundMethodInfluentialA Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms
Seitz, S. M., Curless, B., Diebel, J., Scharstein, D., & Szeliski, R. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 1 (CVPR'06) · doi:10.1109/cvpr.2006.19 · Cited by 2,457 · cited 1×
“In computer vision, research on structure from motion (SFM) and multi-view stereo (MVS) has produced many compelling re-sults, in particular accurate camera tracking and sparse reconstructions (e.g. [10]), and increasingly reconstruction of dense surfaces (e.g. [24]).”
BackgroundIterative point matching for registration of free-form curves and surfaces
Zhang, Z. (1994) · International Journal of Computer Vision · doi:10.1007/bf01427149 · Cited by 2,402 · cited 1×
“A number of ICP variants perform early iterations on a subset of possibly corresponding points or operate within a coarse-to-fine scheme [31], speeding up both the data association and final pose optimisation.”
BackgroundKinectFusion: real-time 3D reconstruction and interaction using a moving depth camera
Izadi, S., Kim, D., Hilliges, O., Molyneaux, D., Newcombe, R., Kohli, P., Shotton, J., Hodges, S., Freeman, D., Davison, A., et al. (2011) · Proceedings of the 24th annual ACM symposium on User interface software and technology · doi:10.1145/2047196.2047270 · Cited by 2,110 · cited 1×
“In [16] we discuss all these possibilities in detail.”
BackgroundSelf-citationDTAM: Dense tracking and mapping in real-time
Newcombe, R. A., Lovegrove, S. J., & Davison, A. J. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126513 · Cited by 2,078
Self-citationReal-time simultaneous localisation and mapping with a single camera
Davison (2003) · Proceedings Ninth IEEE International Conference on Computer Vision · doi:10.1109/iccv.2003.1238654 · Cited by 1,733 · cited 3×
“Such ‘monocular SLAM’ systems such as MonoSLAM [8] and the more accurate Parallel Tracking and Mapping (PTAM) system [17] allow researchers to investigate flexible infrastructure-and marker-free AR applications.”
BackgroundObject modeling by registration of multiple range images
Chen, Y., & Medioni, G. (2002) · Proceedings. 1991 IEEE International Conference on Robotics and Automation · doi:10.1109/robot.1991.132043 · Cited by 1,409 · cited 3×
“This allows us to use the fast projective data association algorithm [4] to obtain correspondence and the point-plane metric [5] for pose optimisation.”
BackgroundMethodRGB-D mapping: Using Kinect-style depth cameras for dense 3D modeling of indoor environments
Henry, P., Krainin, M., Herbst, E., Ren, X., & Fox, D. (2012) · The International Journal of Robotics Research · doi:10.1177/0278364911434148 · Cited by 1,161
RGB-D Mapping: Using Depth Cameras for Dense 3D Modeling of Indoor Environments
Henry, P., Krainin, M., Herbst, E., Ren, X., & Fox, D. (2014) · Springer Tracts in Advanced Robotics · doi:10.1007/978-3-642-28572-1_33 · Cited by 809 · cited 2×
“By using only depth data, our system can work in complete darkness mitigating any issues concerning low light conditions, problematic for passive camera [17, 19, 26] and RGB-D based systems [14].”
MethodReal-time 3D model acquisition
Rusinkiewicz, S., Hall-Holt, O., & Levoy, M. (2002) · Proceedings of the 29th annual conference on Computer graphics and interactive techniques · doi:10.1145/566570.566600 · Cited by 576 · cited 2×
“The point-plane error metric in combination with correspondences obtained using projective data association was first demonstrated in a real time modelling system by [23] where frame-toframe tracking was used (with a fixed camera) for depth map alignment.”
BackgroundMethodIncremental mapping of large cyclic environments
Gutmann, J.-S., & Konolige, K. (2003) · Proceedings 1999 IEEE International Symposium on Computational Intelligence in Robotics and Automation. CIRA'99 (Cat. No.99EX375) · doi:10.1109/cira.1999.810068 · Cited by 568 · cited 1×
“Some SLAM algorithms have also made use of depth data alignment and ICP (often referred to in robotics as scan matching), originally in 2D using laser range-finder sensors, to produce effective robot localisation algorithms which can also autonomously map detailed space occupancy in large areas [12].”
MethodParallel Prefix Sum (Scan) with CUDA
Harris, M. (2011) · Cited by 545
Real-time monocular SLAM: Why filter?
Strasdat, H., Montiel, J. M. M., & Davison, A. J. (2010) · 2010 IEEE International Conference on Robotics and Automation · doi:10.1109/robot.2010.5509636 · Cited by 475 · cited 1×
“Compared to filters, this architecture means that an order of magnitude more scene features could be packed into the map [25], and the result was realtime accuracy now comparable to results in off-line reconstruction.”
ResultSelf-citationRegistering multiview range data to create 3D computer objects
Blais, G., & Levine, M. D. (1995) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.400574 · Cited by 421 · cited 3×
“This allows us to use the fast projective data association algorithm [4] to obtain correspondence and the point-plane metric [5] for pose optimisation.”
BackgroundMethodAutomatic camera recovery for closed or open image sequences
Fitzgibbon, A. W., & Zisserman, A. (1998) · Lecture Notes in Computer Science · doi:10.1007/bfb0055675 · Cited by 402 · cited 3×
“[10]), and increasingly reconstruction of dense surfaces (e.”
BackgroundMethodSelf-citationLive dense reconstruction with a single moving camera
Newcombe, R. A., & Davison, A. J. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5539794 · Cited by 366 · cited 3×
“By using only depth data, our system can work in complete darkness mitigating any issues concerning low light conditions, problematic for passive camera [17, 19, 26] and RGB-D based systems [14].”
MethodSelf-citation3D shape scanning with a time-of-flight camera
Cui, Y., Schuon, S., Chan, D., Thrun, S., & Theobalt, C. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5540082 · Cited by 293 · cited 1×
“[6] demonstrate a moving handheld ToF object scanner.”
BackgroundInteractive ray tracing for isosurface rendering
Parker, S. G., Shirley, P., Livnat, Y., Hansen, C. D., & Sloan, P. (1998) · doi:10.5555/288216.288266 · Cited by 249
Sequential Updating of Projective and Affine Structure from Motion
Beardsley, P. A., Zisserman, A., & Murray, D. W. (1997) · International Journal of Computer Vision · doi:10.1023/a:1007923216416 · Cited by 248 · cited 2×
“Early SFM algorithms capable of dealing with a large number of images had either tracked camera motion incrementally, accumulating drift [2], or required off-line optimisation [10] to close loops.”
BackgroundMethodInteractive ray tracing for isosurface rendering
Parker, S., Shirley, P. S., Livnat, Y., Hansen, C. D., & Sloan, P. (1998) · Visualization · doi:10.1109/visual.1998.745713 · Cited by 221 · cited 4×
“This is attractive due to the scene complexity-independent nature of the algorithm [21], a factor that is becoming increasingly important as the requirement for real-time photo-realistic rendering increases [1].”
BackgroundMethodInfluentialA Globally Optimal Algorithm for Robust TV-L1 Range Image Integration
Zach, C., Pock, T., & Bischof, H. (2007) · 2007 IEEE 11th International Conference on Computer Vision · doi:10.1109/iccv.2007.4408983 · Cited by 220 · cited 1×
“For higher fidelity reconstructions, at the cost of extra computation, when depth maps are contaminated with heavytailed noise or outliers, the more recent work of [30] uses the more robust L1 norm on the truncated SDF data term together with a total variation regularisation to obtain globally optimal surface reconstr…”
MethodReal-Time Dense Geometry from a Handheld Camera
Stühmer, J., Gumhold, S., & Cremers, D. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15986-2_2 · Cited by 205 · cited 1×
“PTAM’s handheld camera tracking capability with dense surface MVS-style reconstruction modules, enabling more sophisticated occlusion prediction and surface interaction [19, 26 ].,By using only depth data, our system can work in complete darkness mitigating any issues concerning low light conditions, problematic for p…”
MethodData-Parallel Octrees for Surface Reconstruction
Zhou, K., Gong, M., Huang, X. S., & Guo, B. (2010) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2010.75 · Cited by 182 · cited 1×
“As shown in Figure 4, there is only a relatively thin crust of non truncated values near the surface, and techniques based on octrees [32] could exploit the compressibility of such a function.”
MethodProbabilistic visibility for multi-view stereo
Hernandez, C., Vogiatzis, G., & Cipolla, R. (2007) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2007.383193 · Cited by 106 · cited 2×
“It has been shown in [15], that Bayesian probabilistic inference of the optimal surface reconstruction, under a simple Gaussian noise model on the depth measurements with a surface visibility predicate that every surface point is visible from all sensor viewpoints, results in a simple algorithm of averaging weighted s…”
BackgroundElfes, A., & Matthies, L. H. (1987) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc.1987.272800 · Cited by 79 · cited 1×
“Occupancy mapping has been popular in robotics, and represents space using a grid of cells, within each of which a probability of occupancy is accumulated via Bayesian updates every time a new range scan provides an informative observation [ 9 ].”
Remarks on the implementation of the fast marching method
Rasch, C., & Satzger, T. (2008) · IMA Journal of Numerical Analysis · doi:10.1093/imanum/drm028 · Cited by 20 · cited 1×
“Although efficient algorithms exist for computing the true discrete SDF for a given set of point measurements (complexity is linear in the the number of voxels) [22], sophisticated implementations are required to achieve top performance on GPU hardware, without which real-time computation is not possible for a reasona…”
Efficient estimation of 3D Euclidean distance fields from 2D range images
Frisken, S. F., & Perry, R. N. (2003) · doi:10.1109/swg.2002.1226513 · Cited by 9 · cited 1×
“It has been shown that for points close to the surface, a correction can be applied by scaling the SDF by cos(θ) [11].”
Method
Research works printed but not linked: 4
- weise, 0, Inhand scanning with online loop closure, Proceedings of the IEEE International Workshop on 3D Digital Imaging and Modeling (3DIM) 2009
- wurm, 0, OctoMap: A probabilistic, flexible, and compact 3D map representation for robotic systems, Proceedings of the ICRA Workshop on Best Practice in 3D Perception and Modeling for Mobile Manipulation 2010
- 0, NVIDIA OptiX 2 Ray Tracing Engine
- zach, 0, A globally optimal algorithm for robust TV-L1 range image integration, Proceedings of the International Conference on Computer Vision (ICCV) 2007
Cited by
ORB-SLAM2: An Open-Source SLAM System for Monocular, Stereo, and RGB-D Cameras
Mur-Artal, R., & Tardos, J. D. (2017) · IEEE Transactions on Robotics · doi:10.1109/tro.2017.2705103 · Cited by 6,898
“Note that our SLAM does not perform any fusion like KinectFusion [4] or similar, but the good definition indicates the accuracy of the keyframe poses.”
BackgroundScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes
Dai, A., Chang, A. X., Savva, M., Halber, M., Funkhouser, T., & Niessner, M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.261 · Cited by 4,368
“The RGB-D scanning system must be trivial to use, the data processing must be robust and automatic, the semantic annotations must be crowdsourced, and the flow of data through the system must be handled by a tracking server.”
A benchmark for the evaluation of RGB-D SLAM systems
Sturm, J., Engelhard, N., Endres, F., Burgard, W., & Cremers, D. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6385773 · Cited by 3,976
“Other algorithms focus on fusing depth maps to a coherent 3D model [5].”
BackgroundPast, Present, and Future of Simultaneous Localization and Mapping: Toward the Robust-Perception Age
Cadena, C., Carlone, L., Carrillo, H., Latif, Y., Scaramuzza, D., Neira, J., Reid, I., & Leonard, J. J. (2016) · IEEE Transactions on Robotics · doi:10.1109/tro.2016.2624754 · Cited by 3,596
“attracting increasing attention after the seminal work [183].”
DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation
Park, J. J., Florence, P., Straub, J., Newcombe, R., & Lovegrove, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00025 · Cited by 3,422
“This inherits from the success of fusion approaches that utilize a truncated SDF (TSDF), pioneered in [15, 37], to combine noisy depth maps into a single 3D model.”
MethodSelf-citation
Metadata
- Authors
- Newcombe, R. A., Fitzgibbon, A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A. J., Kohi, P., Shotton, J., & Hodges, S.
- Affiliations
- Imperial College London (GB)
- Microsoft (United States) (US)
- Venue
- 2011 10th IEEE International Symposium on Mixed and Augmented Reality
- Publisher
- IEEE
- Published
- 2011-10
- Pages
- 127-136
- ISSN
- 1554-7868 (ISSN-L)
- OpenAlex
- W1987648924
- Cited by
- 4,008
- References
- 34
- Citations by year
- 2026: 120 · 2025: 200 · 2024: 213 · 2023: 252 · 2022: 231 · 2021: 299 · 2020: 314 · 2019: 357 · 2018: 383 · 2017: 361 · 2016: 322 · 2015: 343 · 2014: 256 · 2013: 243 · 2012: 112
- FWCI
- 378.08
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Proceedings - International Symposium on Mixed and Augmented Reality, ISMAR (publishedVersion)
- Lancaster EPrints (Lancaster University) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Robotics and Sensor-Based Localization
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › 3D Shape Modeling and Analysis
- Physical Sciences › Earth and Planetary Sciences › Geology › 3D Surveying and Cultural Heritage
- Keywords
- KinectFusion, real-time 3D reconstruction, depth camera, Kinect sensor, iterative closest point, pose tracking, indoor scene reconstruction, augmented reality, GPU rendering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.