Unsupervised Deep Visual-Inertial Odometry with Online Error Correction for RGB-D Imagery

May combine two claims.

The online error correction modules correct feedforward errors from intermediate network outputs, preventing catastrophic failures and trajectory divergence.

“The OEC modules correct feedforward errors from intermediate network outputs to prevent catastrophic failures and trajectory divergences.”

• AssumptionRests on an assumption: Real-world vision-aided localization systems will encounter novel sensory states not seen during training.
• AssumptionRests on an assumption: It is impractical to train feedforward networks to handle every foreseeable real-world situation.

May combine two claims.

VIOLearner can run with only monocular imagery and IMU measurements when RGB-D imagery is unavailable, producing unscaled trajectories.

“While our approach is capable of operating with only monocular imagery and inertial measurement unit (IMU) measurements when RGB-D imagery is unavailable (and thus will generate unscaled trajectories; see Section 7.7.1 for more detail)”

May combine two claims.

ORB-SLAM2 Stereo outperforms VIOLearner RGB-D in mean rotation error and in rotation errors for both KITTI sequences 09 and 10.

“However, ORB-SLAM2 Stereo outperforms VIOLearner RGB-D in mean rotation error and rotation errors for both KITTI sequences 09 and 10.”

May combine two claims.

VIOLearner RGB-D outperforms ORB-SLAM2 Stereo on mean translation error and on translation error for KITTI sequences 09 and 10.

“VIOLearner RGB-D outperforms ORB-SLAM2 Stereo on mean translation error and translation error for KITTI sequences 09 and 10.”

May combine two claims.

For KITTI sequence 09, the average variance of pose error between the four hypotheses was 2.7e-5 in x, 3e-6 in y, and 4.69e-4 in z, and the Euclidean error between computed and true translation was 4.06e-4.

“For the ûs generated by each of the four hypothesis pathways in the final level for KITTI sequence 09, the average variance of pose error between the four hypotheses was 2:7eÀ5 in the x-dimension, 3eÀ6 in the y-dimension, 4:69eÀ4 in the z-dimension, and 4:06eÀ4 was the euclidean error between the computed translation and the true translation.”

May combine two claims.

The mean RMSE for Level 3 is 0.042, while the mean RMSE from each individual hypothesis pathway is 0.0375, 0.0555, 0.0262, 0.0303, and 0.0332 (0.0261 when the lowest-error hypothesis is chosen) for KITTI 09 and 0.029, 0.058, 0.034, and 0.037 (0.027 when the lowest-error hypothesis is chosen) for KITTI 10.

“While this reduction in the final layer can be partially attributed to the multi-hypothesis mechanism in Level 4, the mean RMSE for Level 3 is 0.042 while the mean RMSE from each individual hypothesis pathway is 0.0375, 0.0555, 0.0262, 0.0303, 0.0332 (0.0261 when the lowest error hypothesis is chosen) for KITTI 09 and 0.029, 0.058, 0.034, and 0.037 (0.027 when the lowest error hypothesis is chosen”

May combine two claims.

OKVIS outperforms VIOLearner RGB-D in mean rotation error and in rotation error on two of the three individual AirSim sequences.

“However, OKVIS outperforms VIOLearner RGB-D in mean rotation error and rotation error on two of the three individual sequences.”

May combine two claims.

ORB-SLAM2 produced a lower mean rotational error and lower rotational error on each individual AirSim sequence.

“Again, however, ORB-SLAM2 produced a lower mean rotational error and lower rotational error on each individual AirSim sequence.”

May combine two claims.

For the highest fog condition, ORB-SLAM2-S failed to run on any trajectory while VIOLearner outperformed OKVIS-S in translation error.

“For the highest fog condition, ORB-SLAM2-S failed to run on any of the trajectories while VIOLearner outperformed OKVIS-S in translation error.”

May combine two claims.

OKVIS showed robustness to rotation errors under 20 degrees but is unable to handle orientation offsets greater than 30 degrees.

“Contrastingly, OKVIS showed surprising robustness to rotation errors under 20 degrees but is unable to handle orientation offsets greater than 30 degrees.”

May combine two claims.

The z-dimension shows one to two orders of magnitude more variance than the x- and y-dimensions and is the main contributor to hypothesis variance.

“The z-dimension shows 1-2 orders of magnitude more variance compared to the x-and y-dimensions and is the main contributor to hypothesis variance.”

May combine two claims.

In VIOLearner, multi-scale projections help overcome gradient locality during training and also support online error correction at runtime.

“However, in our network, multi-scale projections not only help to overcome gradient locality during training (see [33], [47] for a broader discussion of this well-known issue), but also aid in online error correction at runtime.”

• cites [33] sourceNot checked yet

  Not checked in the free run.

• cites [47] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

VIOLearner outperformed OKVIS-S and ORB-SLAM2-S on the figure-eight trajectory in translation error but was outperformed by OKVIS-S on the rotation-only trajectory.

“VIOLearner still outperformed OKVIS-S and ORB-SLAM2-S on the figure-eight trajectory in translation error but was outperformed by OKVIS-S on the rotation-only trajectory.”

May combine two claims.

VIOLearner learns to estimate camera-IMU extrinsics and IMU intrinsics implicitly from the data.

“VIOLearner implicitly learns to estimate camera-IMU extrinsics and IMU instrinsics directly from the data which means that VIOLearner does not currently translate from one dataset (with a given camera-IMU configuration) to another (with a different camera-IMU configuration).”

• Also stated in: § 7.2.1 KITTI · plain

  “It should also again be noted that while VINet requires camera-IMU extrinsic calibrations, our network is able to implicitly learn this transform from the data itself.”

• Also stated in: § 7.6.2 Imagery · plain

  “As mentioned earlier, VIOLearner implicitly learns the extrinsic parameters of the camera-IMU system as well as IMU intrinsics.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A weighted L1 penalty on bias terms was empirically found to improve training and gradient backpropagation.

“The final loss L by which the network is trained is then simply the sum of the euclidean loss terms for each level plus a weighted L1 penalty over the bias terms which we empirically found to better facilitate training and gradient backpropagation”

May combine two claims.

VIOLearner RGB-D outperforms Zhan et al.'s deep VO approach and traditional VISO2 on KITTI sequences 09 and 10.

“Similarly, VIOLearner RGB-D outperforms the deep VO approach of Zhan et al. [30] and the traditional VISO2 on both sequences.”

• cites [30] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

On AirSim sequences, VIOLearner RGB-D and VIOLearner RGB outperform ORB-SLAM2 Stereo on mean translation error and on translation error for each individual sequence.

“Similarly for the AirSim sequences, VIOLearner RGB-D and VIOLearner RGB outperforms ORB-SLAM2 Stereo on mean translation error as well as translation error on each individual AirSim sequence.”

May combine two claims.

VIOLearner, using only single source-target image pairs, attains lower translation error than recent deep visual odometry methods and than traditional visual-inertial odometry and visual SLAM methods that use image sequences, keyframe bundle adjustment, full bundle adjustment, and loop closure.

“Despite using only single source-target image pairs, VIO-Learner outperforms recent deep VO approaches in translation error as well as traditional VIO and VSLAM approaches that use sequences of images, key-frame based bundle adjustment, and full bundle adjustment and loop closure, respectively.”

• Also stated in: § 8 CONCLUSION AND FUTURE WORK · boosted

  “In this work, we have presented our VIOLearner architecture and demonstrated superior performance against SoA VO, VIO, and even VSLAM approaches that use far more data.”

May combine two claims.

VIOLearner RGB-D has mean translation error 1.78% and mean rotation error 1.23° on KITTI sequences 09 and 10.

“Mean | 1.78 | 1.23 | 2.53 | 1.31 | 2.82 | 1.64 | 21.09 | 7.25 | 12.27 | 3.52 | 6.11 | 2.44 | 2.06 | 1.03”

May combine two claims.

On KITTI, ORB-SLAM2-S had a mean runtime of 89 ms on sequences 09 and 10, while OKVIS-S had a mean runtime of 143 ms.

“On KITTI, ORB-SLAM2-S had a mean runtime of 89 ms on sequences 09 and 10 while OKVIS-S had a mean runtime of 143 ms.”

The quote only partly states this claim.

With an induced IMU orientation offset in the x-dimension, OKVIS suffers catastrophic translation estimation failure after 20-30 sequences depending on the sequence, unlike VIOLearner.

“In contrast to VIOLearner, after 20-30 (depending on the sequence), OKVIS exhibits catastropic failure in translation estimation.”

May combine two claims.

VIOLearner RGB-D has mean translation error 2.57% and mean rotation error 1.19° on KITTI sequences 00, 02, 05, 07, and 08.

“Mean | 2.57 | 1.19 | 1.22 | 0.56 | 2.61 | 1.34 | 4.07 | 2.03 | 36.23 | 4.56 | 17.92 | 2.8 | 2.58 | 1.03”

May combine two claims.

VIOLearner runs in real-time and is faster than ORB-SLAM2-S and OKVIS-S.

“VIOLearner runs in real-time and is faster than ORB-SLAM2-S and OKVIS-S.”

May combine two claims.

ORB-SLAM2-S had a mean runtime of 67 ms per image on AirSim evaluation trajectories 1, 2, and 3, and OKVIS-S had a mean runtime of 122 ms.

“ORB-SLAM2-S had a mean runtime of 67 ms per image on AirSim evaluation trajectories 1, 2, and 3 and OKVIS-S has a mean runtime of 122 ms.”

The quote only partly states this claim.

ORB-SLAM2 failed to complete some medium and heavy fog sequences in the AirSim robustness test.

“1 (Med. Fog) | 8.86 | 16.42 | 6.78 | 5.79 | X | X”

May combine two claims.

VIOLearner RGB-D has mean translation error 4.94% and mean rotation error 10.24° on AirSim sequences 1, 2, and 3.

“Mean | 4.94 | 10.24 | 8.92 | 20.0 | 6.58 | 12.5 | 19.99 | 16.9 | 7.51 | 9.03 | 13.43 |”

The quote only partly states this claim.

Repeated use of the error correction module substantially lowers pose RMSE for KITTI sequences 09 and 10.

“We see a substantial decrease in error after repeated applications of our error correction module.”

• Also stated in: § 8.2 Rotational Error · plain

  “Pose root mean squared error (RMSE) from poses computed by VIOLearner compared to ground truth after each level for KITTI sequences 09 and 10.”

May combine two claims.

For KITTI, VIOLearner (no IMU) outperforms SFMLearner and the network of Zhan et al. but is outperformed by ORB-SLAM-2, which uses full bundle adjustment.

“Compared to other vision-only approaches, for KITTI, VIOLearner (no IMU) outperforms approaches SFMLearner [33] and the network of Zhan et al. [30] and yet is outperformed by ORB-SLAM-2 which uses full bundle adjustment.”

• cites [33] sourceNot checked yet

  Not checked in the free run.

• cites [30] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

On AirSim, the eight-hypothesis networks generally did better than the four-hypothesis networks, while on KITTI the reverse held.

“In AirSim, the eight hypothesis networks generally outperformed the four hypothesis networks (as seen in Table 7) but on KITTI, the opposite was true.”

Classified as probably not a substantial claim.

Recovering scale requires depth or another fixed reference.

“To recover scale, depth or some other fixed reference is needed (see below in Sections 2.1.1 and 2.1.2).”

Probably not a substantial claim: may not pass S3 (load-bearing).

RGB images and scanning LIDAR measurements can be converted to RGB-D imagery when extrinsic and intrinsic sensor calibrations are known.

“Camera RGB images along with measurements from scanning LIDARs can be converted to RGB-D imagery assuming extrinsic and intrinsic sensor calibrations are known (see Section 5.3.1 for more detail).”

Probably not a substantial claim: may not pass S3 (load-bearing).

For stereo imagery, a known metric baseline allows recovery of translational scale and depth, potentially reducing the scale-estimation burden on visual-inertial methods.

“For stereo imagery a metric baseline distance between cameras is known and with this information, translational scale and depth can be recovered, potentially alleviating visualinertial approaches of the full burden of scale estimation.”

May combine two claims.

Accumulated map drift causes orientation error that leads to scale error, and systems cannot always recover from it.

“Accumulated map drift introduces orientation error that leads to scale error from which systems can not always recover.”

May combine two claims.

In VIO, gravity must be separated from measured linear accelerations, which requires correctly estimating the IMU's orientation in the world frame.

“While IMU measurements can help resolve scale ambiguity, gravity must be separated from the measured linear accelerations which means that the orientation of the IMU in the world frame must be correctly estimated [18].”

• cites [18] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

MSCKF has become a standard filtering-based approach to VIO.

“The multi-state constraint Kalman filter (MSCKF) [21] has become a standard for filtering-based approaches to VIO.”

• cites [21] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Localization problems have very low error tolerances, so one mistake in estimating individual poses can cause catastrophic failure and make the entire localization solution unusable.

“In part, this is due to the very low error tolerances of localization problems-a single mistake while estimating the individual poses that constitute a trajectory can cause catastrophic failure and render the entire localization solution unusable.”

May combine two claims.

Large-scale monocular systems for autonomous navigation are uncommon, mainly because of scale drift.

“Successful large-scale monocular systems for autonomous navigation are uncommon, primarily due to scale drift [17].”

• cites [17]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

ORB-SLAM2 has shown state-of-the-art performance on several visual odometry benchmarks.

“ORB-SLAM2 has shown SoA performance on a variety of VO benchmarks.”

• cites [20] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Stereo depth has effective range inversely related to baseline: small baselines give high-error depth for distant scene elements, while large baselines miss close depth and introduce calibration difficulties.

“However, a downside of using stereo cameras to infer depth is that stereo cameras have effective ranges that are inversely proportional to the baseline between the stereo camera pair: small camera baselines produce high-error depth estimates for scene elements further from the cameras while large camera baselines fail to capture depth close to the cameras and introduce calibration difficulties”

• cites [19]Not checked yet

  Not checked in the free run.

May combine two claims.

UnDeepVO is an unsupervised depth and egomotion method that produces properly scaled trajectories, but it uses stereo image pairs for training and cannot be extended to inertial measurements because stereo spatial transformations are unobservable by an IMU.

“Additionally, the network architecture of UnDeepVO cannot be extended to include motion estimates derived from inertial measurements because the spatial transformation between paired images from stereo cameras are unobservable by an IMU (stereo images are recorded simultaneously).”

• cites [35] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The better performance of VIOLearner RGB over VIOLearner RGB-D on Table 2 sequences may result from motion artefacts in the 3D LIDAR data used to create the depth channel for VIOLearner RGB-D.

“This may be due to known motion artefacts that contaminate 3D LIDAR data which was used to generate the depth channel of inputs for VIOLearner RGB-D which does not affect depth learned by VIOLearner RGB.”

May combine two claims.

VINet was robust to temporal and spatial misalignments of an IMU and camera but required extrinsic calibration between them.

“While VINet showed robustness to temporal and spatial misalignments of an IMU and camera, it still required extrinsic calibration parameters between camera and IMU.”

• cites [2] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The multi-hypothesis approach lets the network generate several pathways and effectively sample from an unknown noise distribution.

“This multi-hypothesis approach allows the network to generate several different pathways and effectively sample from an unknown noise distribution.”

Probably not a substantial claim: may not pass S3 (load-bearing).

MSCKF approaches are typically less accurate than optimization-based approaches.

“While its complexity is linear in the number of features being used for egomotion estimation and it was generally more robust compared to optimization-based approaches (as has been recently reported in [22]), MSCKF approaches are typically less accurate in comparison.”

• cites [22] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

KITTI used only software synchronization between sensors, which causes problems for some VIO approaches because imagery and IMU data are not accurately time-synchronized.

“However, KITTI only used software synchronization between sensors which causes issues for some VIO approaches due to the inaccurate time synchronization between imagery and IMU data.”

• cites [1] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

SFMLearner learned unscaled egomotion and depth from RGB imagery without supervision, but could not recover scale and required a sequence of images.

“SFMLearner [33] demonstrated the unsupervised learning of unscaled egomotion and depth from RGB imagery.”

• Also stated in: § 3.2 Learning-Based Methods · plain

  “However, their approach was unable to recover the scale for the depth estimates or, most crucially, the scale of the changes in pose.”

• Also stated in: § 3.2 Learning-Based Methods · plain

  “SFMLearner also required a sequence of images to compute a trajectory.”

• cites [33] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The KITTI dataset contains RGB and depth data over 39.2 km and was ground truthed within 10 cm using an OXTS RT 3003 GPS solution.

“The KITTI dataset [1] included RGB and depth data captured over 39.2 km and was ground truthed with accuracies within 10cm via an OXTS RT 3003 GPS solution.”

• cites [1] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

OKVIS is an optimization-based keyframe visual-inertial SLAM method for monocular and stereo cameras, and it does not include loop closure.

“OKVIS [3] is a optimization-based keyframe visual-inertial SLAM approach for monocular and stereo cameras.”

• Also stated in: § 3.1 Traditional Methods · plain

  “However, it did not include loop closure, unlike the SLAM algorithms with SoA performance.”

• cites [3] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

VINet was trained supervised and required ground-truth pose differences.

“In addition, VINet was trained in a supervised manner and thus required the ground truth pose differences for each exemplar in the training set which are not always readily available.”

• cites [2] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

For Levels 0 to 2, û are computed using downsampled images, so their Jacobian inputs have access to less information and use one fewer convolutional layer each.

“It should however be noted that û for Levels 0 to 2 are computed using downsampled images and thus their Jacobian inputs both have access to less information and use one less convolutional layer each.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The greater motion variability of AirSim MAV trajectories relative to KITTI trajectories may account for why eight-hypothesis networks appeared more useful on AirSim than on KITTI.

“As the MAV Air-Sim trajectories typically exhibited more varied motion compared to KITTI trajectories, this may explain why the eight-hypothesis networks seem to have been better utilized on AirSim compared to KITTI.”

• AssumptionRests on an assumption: AirSim MAV trajectories typically showed more varied motion than KITTI trajectories.

Classified as probably not a substantial claim.

Prior work [48] indicates a positive correlation between input data noise and the advantage of adding hypotheses, which might also apply to input data variance.

“Previous results from [48] suggest a positive correlation between noise in the input data and benefits of additional hypotheses which may also translate to variance in the input data.”

• cites [48] sourceNot checked yet

  Not checked in the free run.