High-precision, consistent EKF-based visual-inertial odometry
Abstract
In this paper, we focus on the problem of motion tracking in unknown environments using visual and inertial sensors. We term this estimation task visual–inertial odometry (VIO), in analogy to the well-known visual-odometry problem. We present a detailed study of extended Kalman filter (EKF)-based VIO algorithms, by comparing both their theoretical properties and empirical performance. We show that an EKF formulation where the state vector comprises a sliding window of poses (the multi-state-constraint Kalman filter (MSCKF)) attains better accuracy, consistency, and computational efficiency than the simultaneous localization and mapping (SLAM) formulation of the EKF, in which the state vector contains the current pose and the features seen by the camera. Moreover, we prove that both types of EKF approaches are inconsistent, due to the way in which Jacobians are computed. Specifically, we show that the observability properties of the EKF’s linearized system models do not match those of the underlying system, which causes the filters to underestimate the uncertainty in the state estimates. Based on our analysis, we propose a novel, real-time EKF-based VIO algorithm, which achieves consistent estimation by (i) ensuring the correct observability properties of its linearized system model, and (ii) performing online estimation of the camera-to-inertial measurement unit (IMU) calibration parameters. This algorithm, which we term MSCKF 2.0, is shown to achieve accuracy and consistency higher than even an iterative, sliding-window fixed-lag smoother, in both Monte Carlo simulations and real-world testing.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ORB-SLAM2 has shown state-of-the-art performance on several visual odometry benchmarks.
“ORB-SLAM2 has shown SoA performance on a variety of VO benchmarks.”
From Unsupervised Deep Visual-Inertial Odometry with Online Error Correction for RGB-D Imagery (2020) · cites this paper as [20]
Cites
Shi, J., & Tomasi (1994) · Proceedings of IEEE Conference on Computer Vision and Pattern Recognition CVPR-94 · doi:10.1109/cvpr.1994.323794 · Cited by 7,724 · cited 1× · see citing works
“For feature extraction the Shi-Tomasi algorithm was used (Shi and Tomasi, 1994), and matching was carried out by normalized cross-correlation.”
MethodEstimation with Applications to Tracking and Navigation: Theory, Algorithms and Software
Bar‐Shalom, Y., Li, X. ‐., & Kirubarajan, T. (2002) · doi:10.1002/0471221279 · Cited by 6,782 · see citing works
Matrix Analysis and Applied Linear Algebra
Meyer, C. (2000) · doi:10.1137/1.9780898719512 · Cited by 3,565 · see citing works
Matrix analysis and applied linear algebra
Meyer, C. D. (2001) · Choice Reviews Online · doi:10.5860/choice.38-5065 · Cited by 3,384 · see citing works
Stochastic models, estimation, and control
(1982) · Mathematics in Science and Engineering/Mathematics in science and engineering · doi:10.1016/s0076-5392(08)x6186-7 · Cited by 3,230 · see citing works
A Multi-State Constraint Kalman Filter for Vision-aided Inertial Navigation
Mourikis, A. I., & Roumeliotis, S. I. (2007) · Proceedings 2007 IEEE International Conference on Robotics and Automation · doi:10.1109/robot.2007.364024 · Cited by 1,951 · see citing works
Self-citationStochastic models, estimation, and control
(1979) · Mathematics in Science and Engineering/Mathematics in science and engineering · doi:10.1016/s0076-5392(08)x6125-9 · Cited by 1,625 · see citing works
Pedestrian Tracking with Shoe-Mounted Inertial Sensors
Foxlin, E. M. (2005) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.2005.140 · Cited by 1,409 · see citing works
Nistér, D., Naroditsky, O., & Bergen, J. R. (2004) · Cited by 1,169 · see citing works
Nistér, D., Naroditsky, O., & Bergen, J. R. (2004) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2004.1315094 · Cited by 1,031 · see citing works
Aided Navigation: GPS with High Rate Sensors
Farrell, J. A. (2008) · Cited by 774 · see citing works
Inverse Depth Parametrization for Monocular SLAM
Civera, J., Davison, A. J., & Montiel, J. M. M. (2008) · IEEE Transactions on Robotics · doi:10.1109/tro.2008.2003276 · Cited by 768 · see citing works
Visual-Inertial Sensor Fusion: Localization, Mapping and Sensor-to-Sensor Self-calibration
Kelly, J. W., & Sukhatme, G. S. (2010) · The International Journal of Robotics Research · doi:10.1177/0278364910382802 · Cited by 537 · see citing works
FrameSLAM: From Bundle Adjustment to Real-Time Visual Mapping
Konolige, K., & Agrawal, M. (2008) · IEEE Transactions on Robotics · doi:10.1109/tro.2008.2004832 · Cited by 529 · see citing works
Real-time monocular SLAM: Why filter?
Strasdat, H., Montiel, J. M. M., & Davison, A. J. (2010) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2010.5509636 · Cited by 485 · see citing works
Lupton, T., & Sukkarieh, S. (2011) · IEEE Transactions on Robotics · doi:10.1109/tro.2011.2170332 · Cited by 463 · see citing works
Mirzaei, F. M., & Roumeliotis, S. I. (2008) · IEEE Transactions on Robotics · doi:10.1109/tro.2008.2004486 · Cited by 446 · see citing works
Visual-inertial navigation, mapping and localization: A scalable real-time causal approach
Jones, E., & Soatto, S. (2011) · The International Journal of Robotics Research · doi:10.1177/0278364910388963 · Cited by 431 · see citing works
Weiß, S., Achtelik, M., Lynen, S., Chli, M., & Siegwart, R. (2012) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2012.6225147 · Cited by 418 · see citing works
Large-Scale Visual Odometry for Rough Terrain
Konolige, K., Agrawal, M., & Solà, J. (2010) · Springer tracts in advanced robotics · doi:10.1007/978-3-642-14743-2_18 · Cited by 373 · see citing works
Martinelli, A. (2011) · IEEE Transactions on Robotics · doi:10.1109/tro.2011.2160468 · Cited by 283 · see citing works
Observability-based Rules for Designing Consistent EKF SLAM Estimators
Huang, G. P., Mourikis, A. I., & Roumeliotis, S. I. (2009) · The International Journal of Robotics Research · doi:10.1177/0278364909353640 · Cited by 276 · see citing works
Self-citationSliding window filter with application to planetary landing
Sibley, G., Matthies, L. H., & Sukhatme, G. S. (2010) · Journal of Field Robotics · doi:10.1002/rob.20360 · Cited by 261 · see citing works
Analysis and improvement of the consistency of extended Kalman filter based SLAM
Huang, G. P., Mourikis, A. I., & Roumeliotis, S. I. (2008) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2008.4543252 · Cited by 222 · see citing works
Self-citationReal-time metric state estimation for modular vision-inertial systems
Weiß, S., & Siegwart, R. (2011) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2011.5979982 · Cited by 221 · see citing works
Augmenting inertial navigation with image-based motion estimation
Roumeliotis, S. I., Johnson, A., & Montgomery, J. F. (2003) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2002.1014441 · Cited by 203 · see citing works
Improving the accuracy of EKF-based visual-inertial odometry
Li, M., & Mourikis, A. I. (2012) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2012.6225229 · Cited by 194 · see citing works
Self-citationReal-Time Computer Vision/DGPS-Aided Inertial Navigation System for Lane-Level Vehicle Navigation
Vu, A., Ramanandan, A., Chen, A., Farrell, J. A., & Barth, M. (2012) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2012.2187641 · Cited by 177 · see citing works
Real-time implementation of airborne inertial-SLAM
Kim, J., & Sukkarieh, S. (2006) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2006.06.006 · Cited by 161 · see citing works
Vision‐aided inertial navigation for pin‐point landing using observations of mapped landmarks
Trawny, N., Mourikis, A. I., Roumeliotis, S. I., Johnson, A., & Montgomery, J. F. (2007) · Journal of Field Robotics · doi:10.1002/rob.20189 · Cited by 158 · see citing works
Self-citationVision-Aided Inertial Navigation for Flight Control
Wu, A. D., Johnson, E. N., & Proctor, A. A. (2005) · Journal of Aerospace Computing Information and Communication · doi:10.2514/1.16038 · Cited by 133 · see citing works
A new approach to vision-aided inertial navigation
Tardif, J.-P., George, M., Laverne, M., Kelly, A., & Stentz, A. (2010) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2010.5651059 · Cited by 119 · see citing works
Inertial Aiding of Inverse Depth SLAM using a Monocular Camera
Piniés, P., Lupton, T., Sukkarieh, S., et al. (2007) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2007.363895 · Cited by 115 · see citing works
Epipolar Constraints for Vision-Aided Inertial Navigation
Diel, D. D., DeBitetto, P. A., & Teller, S. J. (2005) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/acvmot.2005.48 · Cited by 101 · see citing works
Proceedings of the 2018 International Symposium on Experimental Robotics
(2020) · Springer proceedings in advanced robotics · doi:10.1007/978-3-030-33950-0 · Cited by 99 · see citing works
Motion tracking with fixed-lag smoothing: Algorithm and consistency analysis
Dong-Si, T.-C., & Mourikis, A. I. (2011) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2011.5980267 · Cited by 97 · see citing works
Self-citationVisual Odometry System Using Multiple Stereo Cameras and Inertial Measurement Unit
Oskiper, T., Zhu, Z., Samarasekera, S., & Kumar, R. (2007) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2007.383087 · Cited by 92 · see citing works
On the Consistency of Vision-Aided Inertial Navigation
Kottas, D. G., Hesch, J. A., Bowman, S. L., & Roumeliotis, S. I. (2013) · Springer Tracts in Advanced Robotics · doi:10.1007/978-3-319-00065-7_22 · Cited by 76 · see citing works
Combined Visual and Inertial Navigation for an Unmanned Aerial Vehicle
Kelly, J. W., Saripalli, S., & Sukhatme, G. S. (2008) · Springer tracts in advanced robotics · doi:10.1007/978-3-540-75404-6_24 · Cited by 76 · see citing works
Mourikis, A. I., Roumeliotis, S. I., & Burdick, J. W. (2007) · IEEE Transactions on Robotics · doi:10.1109/tro.2007.900610 · Cited by 71 · see citing works
Self-citationTowards Consistent Vision-Aided Inertial Navigation
Hesch, J. A., Kottas, D. G., Bowman, S. L., & Roumeliotis, S. I. (2013) · Springer tracts in advanced robotics · doi:10.1007/978-3-642-36279-8_34 · Cited by 64 · see citing works
A dual-layer estimator architecture for long-term localization
Mourikis, A. I., & Roumeliotis, S. I. (2008) · 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2008.4563131 · Cited by 62 · see citing works
Self-citationMa, J., Susca, S., Bajracharya, M., Matthies, L. H., Malchano, M., & Wooden, D. (2012) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2012.6225132 · Cited by 57 · see citing works
Monocular SLAM for Visual Odometry
Munguía, R., & Grau, A. (2007) · IEEE International Symposium on Intelligent Signal Processing, · doi:10.1109/wisp.2007.4447564 · Cited by 49 · see citing works
State estimation of an underwater robot using visual and inertial information
Shkurti, F., Rekleitis, I. M., Scaccia, M., & Dudek, G. (2011) · 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2011.6094680 · Cited by 47 · see citing works
Consistency of the monocular EKF-SLAM algorithm for three different landmark parametrizations
Solà, J. (2010) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2010.5509518 · Cited by 46 · see citing works
Fast Relative Pose Calibration for Visual and Inertial Sensors
Kelly, J. W., & Sukhatme, G. S. (2009) · Springer tracts in advanced robotics · doi:10.1007/978-3-642-00196-3_59 · Cited by 45 · see citing works
Optimization-Based Estimator Design for Vision-Aided Inertial Navigation
Li, M., & Mourikis, A. I. (2012) · Robotics: Science and Systems · doi:10.15607/rss.2012.viii.031 · Cited by 45 · see citing works
Self-citationBrockers, R., Susca, S., Zhu, D. Q., & Matthies, L. H. (2012) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.919278 · Cited by 34 · see citing works
Inertial aided monocular SLAM for GPS-denied navigation
Kleinert, M., & Schleith, S. (2010) · IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI) · doi:10.1109/mfi.2010.5604453 · Cited by 27 · see citing works
Camera-aided inertial navigation using epipolar points
Zachariah, D., & Jansson, M. (2010) · IEEE/ION Position Location and Navigation Symposium · doi:10.1109/plans.2010.5507246 · Cited by 20 · see citing works
Williams, B. P., Hudson, N., Tweddle, B. E., et al. (2011) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2011.5979997 · Cited by 20 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator
Qin, T., Li, P., & Shen, S. (2018) · IEEE Transactions on Robotics · doi:10.1109/tro.2018.2853729 · Cited by 4,669 · see citing works
LSD-SLAM: Large-scale direct monocular SLAM
Engel, J., Schöps, T., & Cremers, D. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10605-2_54 · Cited by 3,851 · see citing works
Iterated extended Kalman filter based visual-inertial odometry using direct photometric feedback
Bloesch, M., Burri, M., Omari, S., Hutter, M., & Siegwart, R. (2017) · The International Journal of Robotics Research · doi:10.1177/0278364917728574 · Cited by 485 · see citing works
Large-scale direct SLAM for omnidirectional cameras
Caruso, D., Engel, J., & Cremers, D. (2015) · 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2015.7353366 · Cited by 244 · see citing works
Unsupervised Deep Visual-Inertial Odometry with Online Error Correction for RGB-D Imagery
Shamwell, E. J., Lindgren, K., Leung, S., & Nothwang, W. D. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2909895 · Cited by 86 · see citing works
Dense RGB-D-inertial SLAM with map deformations
Laidlow, T., Bloesch, M., Li, W., & Leutenegger, S. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8206591 · Cited by 61 · see citing works
Shamwell, E. J., Leung, S., & Nothwang, W. D. (2018) · 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8593573 · Cited by 41 · see citing works
Graph-based terrain relative navigation with optimal landmark database selection
Steiner, T. J., Brady, T. M., & Hoffman, J. A. (2015) · 2015 IEEE Aerospace Conference · doi:10.1109/aero.2015.7119053 · Cited by 16 · see citing works
Depth enhanced visual-inertial odometry based on Multi-State Constraint Kalman Filter
Pang, F., Chen, Z., Pu, L., & Wang, T. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8205989 · Cited by 8 · see citing works
- Authors
- Li, M., & Mourikis, A. I.
- Affiliations
- University of California, Riverside (US)
- ORCID
- Li, Mingyang
- Venue
- The International Journal of Robotics Research
- Publisher
- SAGE Publications
- Published
- 2013-05
- Volume
- 32
- Issue
- 6
- Pages
- 690-711
- ISSN
- 0278-3649 (ISSN-L)
- 1741-3176
- OpenAlex
- W2056358962
- Cited by
- 813
- References
- 53
- Citations by year
- 2026: 45 · 2025: 64 · 2024: 65 · 2023: 72 · 2022: 64 · 2021: 76 · 2020: 72 · 2019: 99 · 2018: 81 · 2017: 71 · 2016: 27 · 2015: 45 · 2014: 26 · 2013: 6
- FWCI
- 55.76
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- The International Journal of Robotics Research (publishedVersion)
- Topics
- Physical Sciences › Engineering › Aerospace Engineering › Robotics and Sensor-Based Localization
- Physical Sciences › Engineering › Aerospace Engineering › Inertial Sensor and Navigation
- Physical Sciences › Computer Science › Artificial Intelligence › Target Tracking and Data Fusion in Sensor Networks
- Keywords
- visual-inertial odometry, extended Kalman filter, multi-state constraint Kalman filter, estimator consistency, observability analysis, Jacobian computation, real-time visual tracking, uncertainty estimation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
