Visual-Inertial Sensor Fusion: Localization, Mapping and Sensor-to-Sensor Self-calibration
The DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Abstract
Visual and inertial sensors, in combination, are able to provide accurate motion estimates and are well suited for use in many robot navigation tasks. However, correct data fusion, and hence overall performance, depends on careful calibration of the rigid body transform between the sensors. Obtaining this calibration information is typically difficult and time-consuming, and normally requires additional equipment. In this paper we describe an algorithm, based on the unscented Kalman filter, for self-calibration of the transform between a camera and an inertial measurement unit (IMU). Our formulation rests on a differential geometric analysis of the observability of the camera—IMU system; this analysis shows that the sensor-to-sensor transform, the IMU gyroscope and accelerometer biases, the local gravity vector, and the metric scene structure can be recovered from camera and IMU measurements alone. While calibrating the transform we simultaneously localize the IMU and build a map of the surroundings, all without additional hardware or prior knowledge about the environment in which a robot is operating. We present results from simulation studies and from experiments with a monocular camera and a low-cost IMU, which demonstrate accurate estimation of both the calibration parameters and the local scene structure.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589
Golub, G. (2012) · doi:10.56021/9781421407944 · Cited by 34,505
Isidori, A. (1989) · Springer eBooks · doi:10.1007/978-3-662-02581-9 · Cited by 7,869
Isidori, A. (1995) · Communications and Control Engineering · doi:10.1007/978-1-84628-615-5 · Cited by 6,755
Unscented Filtering and Nonlinear Estimation
Julier, S. J., & Uhlmann, J. K. (2004) · Proceedings of the IEEE · doi:10.1109/jproc.2003.823141 · Cited by 6,588
(2012) · SpringerReference · doi:10.1007/springerreference_117570 · Cited by 4,486
Parallel Tracking and Mapping for Small AR Workspaces
Klein, G., & Murray, D. (2007) · 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality · doi:10.1109/ismar.2007.4538852 · Cited by 4,346
Closed-form solution of absolute orientation using unit quaternions
Horn, B. K. P. (1987) · Journal of the Optical Society of America A · doi:10.1364/josaa.4.000629 · Cited by 4,116
MonoSLAM: Real-time single camera SLAM
Davison, A. J., Reid, I. D., Molton, N. D., & Stasse, O. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1049 · Cited by 3,959 · cited 1×
“For example, Davison et al. (2007) describe an extended Kalman filter (EKF)-based system that is able to localize a camera in a room-sized environment.”
BackgroundStrapdown Inertial Navigation Technology
Titterton, D., & Weston, J. (2004) · Institution of Engineering and Technology eBooks · doi:10.1049/pbra017e · Cited by 3,348
Cited by
Keyframe-based visual–inertial odometry using nonlinear optimization
Leutenegger, S., Lynen, S., Bosse, M., Siegwart, R., & Furgale, P. (2015) · The International Journal of Robotics Research · doi:10.1177/0278364914554813 · Cited by 1,723
Robust visual inertial odometry using a direct EKF-based approach
Bloesch, M., Omari, S., Hutter, M., & Siegwart, R. (2015) · 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2015.7353389 · Cited by 917
High-precision, consistent EKF-based visual-inertial odometry
Li, M., & Mourikis, A. I. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913481251 · Cited by 813
“…the EKF Jacobians are computed, even though the IMU’s rotation about gravity (the yaw) is not observable in VIO (see, e.g., (Jones and Soatto, 2011; Kelly and Sukhatme, 2011; Martinelli, 2012)), it appears to be observable in the linearized system model used by the MSCKF, and the same occurs in…”
BackgroundMethodInfluentialUnified temporal and spatial calibration for multi-sensor systems
Furgale, P., Rehder, J., & Siegwart, R. (2013) · 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2013.6696514 · Cited by 807
A Tutorial on Quantitative Trajectory Evaluation for Visual(-Inertial) Odometry
Zhang, Z., & Scaramuzza, D. (2018) · 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8593941 · Cited by 658
A Review on Challenges of Autonomous Mobile Robot and Sensor Fusion Methods
Alatise, M. B., & Hancke, G. P. (2020) · IEEE Access · doi:10.1109/access.2020.2975643 · Cited by 609
Robust Stereo Visual Inertial Odometry for Fast Autonomous Flight
Sun, K., Mohta, K., Pfrommer, B., Watterson, M., Liu, S., Mulgaonkar, Y., Taylor, C. J., & Kumar, V. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2793349 · Cited by 539
Iterated extended Kalman filter based visual-inertial odometry using direct photometric feedback
Bloesch, M., Burri, M., Omari, S., Hutter, M., & Siegwart, R. (2017) · The International Journal of Robotics Research · doi:10.1177/0278364917728574 · Cited by 485
Opportunities and challenges with autonomous micro aerial vehicles
Kumar, V., & Michael, N. (2012) · The International Journal of Robotics Research · doi:10.1177/0278364912455954 · Cited by 472
Keyframe-Based Visual-Inertial SLAM using Nonlinear Optimization
Leutenegger, S., Furgale, P., Rabaud, V., Chli, M., Konolige, K., & Siegwart, R. (2013) · Robotics: Science and Systems IX · doi:10.15607/rss.2013.ix.037 · Cited by 447
Weiss, S., Achtelik, M. W., Lynen, S., Chli, M., & Siegwart, R. (2012) · 2012 IEEE International Conference on Robotics and Automation · doi:10.1109/icra.2012.6225147 · Cited by 417
Visual-Inertial Navigation: A Concise Review
Huang, G. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8793604 · Cited by 391
Using Inertial Sensors for Position and Orientation Estimation
Kok, M., Hol, J. D., & Schön, T. B. (2017) · Foundations and Trends® in Signal Processing · doi:10.1561/2000000094 · Cited by 347
Loianno, G., Brunner, C., McGrath, G., & Kumar, V. (2017) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2016.2633290 · Cited by 305
Scaramuzza, D., Achtelik, M. C., Doitsidis, L., Friedrich, F., Kosmatopoulos, E., Martinelli, A., Achtelik, M. W., Chli, M., Chatzichristofis, S., Kneip, L., et al. (2014) · IEEE Robotics & Automation Magazine · doi:10.1109/mra.2014.2322295 · Cited by 299
Martinelli, A. (2012) · IEEE Transactions on Robotics · doi:10.1109/tro.2011.2160468 · Cited by 279
Yang, Z., & Shen, S. (2017) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2016.2550621 · Cited by 274
Consistency Analysis and Improvement of Vision-aided Inertial Navigation
Hesch, J. A., Kottas, D. G., Bowman, S. L., & Roumeliotis, S. I. (2014) · IEEE Transactions on Robotics · doi:10.1109/tro.2013.2277549 · Cited by 266
Continuous-time batch estimation using temporal basis functions
Furgale, P., Barfoot, T. D., & Sibley, G. (2012) · 2012 IEEE International Conference on Robotics and Automation · doi:10.1109/icra.2012.6225005 · Cited by 259
A UAV system for inspection of industrial facilities
Nikolic, J., Burri, M., Rehder, J., Leutenegger, S., Huerzeler, C., & Siegwart, R. (2013) · 2013 IEEE Aerospace Conference · doi:10.1109/aero.2013.6496959 · Cited by 251
“Commonly, the transformation between a camera and inertial sensors is estimated recursively [10][11], while time delays are determined in a separate process [12].”
BackgroundCamera-IMU-based localization: Observability analysis and consistency improvement
Hesch, J. A., Kottas, D. G., Bowman, S. L., & Roumeliotis, S. I. (2014) · The International Journal of Robotics Research · doi:10.1177/0278364913509675 · Cited by 245
“For example, Mirzaei and Roumeliotis (2008) as well as Kelly and Sukhatme (2011) have studied the observability properties of IMU-camera extrinsic calibration using Lie derivatives (Hermann and Krener 1977).”
BackgroundA Spline-Based Trajectory Representation for Sensor Fusion and Rolling Shutter Cameras
Patron-Perez, A., Lovegrove, S., & Sibley, G. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0811-3 · Cited by 104
On the Consistency of Vision-Aided Inertial Navigation
Kottas, D. G., Hesch, J. A., Bowman, S. L., & Roumeliotis, S. I. (2013) · Springer Tracts in Advanced Robotics · doi:10.1007/978-3-319-00065-7_22 · Cited by 76
Depth enhanced visual-inertial odometry based on Multi-State Constraint Kalman Filter
Pang, F., Chen, Z., Pu, L., & Wang, T. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8205989 · Cited by 8
Metadata
- Authors
- Kelly, J., & Sukhatme, G. S.
- Venue
- The International Journal of Robotics Research
- Publisher
- SAGE Publications
- Published
- 2010-11-05
- Volume
- 30
- Issue
- 1
- Pages
- 56-79
- ISSN
- 0278-3649 (ISSN-L)
- 1741-3176
- OpenAlex
- W2147536420
- Cited by
- 536
- References
- 51
- FWCI
- 51.34
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.