Modelling uncertainty in deep learning for camera relocalization

“Convolutional neural networks have recently been shown to be extremely powerful for generating image representations for classification [12], [13].”

“This has been achieved using SIFT [2] features with FAB-MAP [6] or using convolutional neural network features [7].”

“Additionally, in green we show the output from the first auxiliary pose regressor from the GoogLeNet architecture (see Figure 3 of [13]).”

“These higher level contours and shape cues [14] have been shown to be increasingly invariant to object instantiation parameters.”

“Using standard libraries such as [25] we can obtain an efficient implementation of a Bernoulli Bayesian convolutional neural network.”

“Visual SLAM [1], [4], [5], [9], and relocalization research [10], [11], has largely focused on registering viewpoints using point-based landmark features, such as SIFT [2] or ORB [3].”

“nsider localization uncertainty [1], [5], [11], previous probabilistic algorithms have been proposed. Bayesian approaches include extended Kalman filters and particle filter approaches such as FastSLAM [18]. However these approaches estimate uncertainty from sensor noise models, not the uncertainty of the model to repre…”

“Visual SLAM [1], [4], [5], [9], and relocalization research [10], [11], has largely focused on registering viewpoints using point-based landmark features, such as SIFT [2] or ORB [3].”

“Recent work [23] has extended dropout to approximate Bayesian inference over the network’s weights.”

“The extension of this work by [11], named Bayesian PoseNet, models relocalization uncertainty for the same task.”

“This method to quantify the model uncertainty has been applied into various tasks, such as camera relocalization [12].”

“This is consistent with the finding of [27], which showed epistemic uncertainty and positional error were positively correlated in a camera relocalization task.”

“On the other hand, pose-regression-based methods use deep learning to approximate highly non-linear functions mapping an input image to camera poses [16, 17, 18].”

“Quantifying uncertainty of network predictions is highly desirable in many applications, such as camera relocalization [25], depth uncertainty [4] and photometric uncertainty [68] in SLAM, and optical flow estimation [22].”

“By leveraging the notion of Bayesian networks, Kendall proposed a method for estimating the uncertainty of the predicted pose in [18].”

“Due to PoseNet’s inaccurate estimation for handling scenes with some obstacles, Kendall et al. [22] proposed a Bayesian convolutional neural network to regress the camera 6-DoF pose.”

“was applied to camera localization estimation [40], semantic segmentation [41], and open-set object detection tasks [42].”

“Kendall and Cipolla [18] propose an uncertainty quantification to the output of Kendall et al.”

“Gal’s method for approximating an output probability and thus estimating uncertainty has found a wide range of applications in the literature, including semantic segmentation [23], pose estimation [24], active learning [25] and medical imaging [26].”

“[13] model camera localisation uncertainty by utilising dropout at test time.”

“For this experiment we used PSPnet [85] with the experimental protocol in [26].”

“Some approaches to uncertainty-aware APR include CoordiNet [32], Bayesian PoseNet [22], and Hy-draNet [34].”

“Examples include Posenet [24], CNN/LSTM combination [27], CNN map compression [28], differentiable RANSAC [29], object detection with 6D pose estimation [30], regressing scene’s world space [31], Monte Carlo dropout [32], video-clip relocalization [33], NetVLAD [34], and multimodal color/shape inputs [35].”

“By using Bayesian CNN, the authors extended their work to model precision uncertainty [22].”

“Monte Carlo methods for deep learning model uncertainty estimation is successfully applied to classification, segmentation [11, 21], camera relocalization [12] problems.”

“Using this approach, researchers have increased semantic segmentation performance on images [21] and visual relocalization accuracy [22].”

“In [11][19][20], a CNN is used to directly output a 6 degree of freedom (6DoF) camera pose given a monocular RGB camera image.”

“[16], [17], [18] proposed small variations to the PoseNet [14] architecture to reduce median localization error.”

“This technique has proved to be effective in cases such as increasing visual relocalization accuracy [19] and semantic segmentation performance on images [20].”

“Further modifications to network architecures [8], [9], [10], [11], [12] and different training strategies [17], [14] have been made to improve accuracy and performance.”

“Although the model is robust to dynamic changes of the scene due to its high-level generalization ability, the performance of PoseNet [15] and its variants [13, 14, 26] do not perform sufficiently well for accurate localization.”

“Unlike the previous works [19], [40], [41], our system does not require any pre-processing such as cropping or subtracting the mean for [43] and Pose-Hourglass [52]. the images.”

“Kendall and Cipolla [25] used a Bayesian convolutional neural network to regress the six-DoF camera pose from a single RGB image.”

“In [20], Kendall and Cipolla proposed an approach to improve the accuracy of PoseNet by introducing uncertainty of prediction.”

“Recent advances in machine learning, such as regression forests [16]–[18] and deep convolutional neural networks [8], [19]–[22], may become an alternative.”

“This work was later expanded on with the inclusion of measures of uncertainty into the output [7] using Bayesian deep learning.”

“Initial localization is usually performed by sensor-reading-to-map matching [7], [8], which is frequently referred to as re-localization [9], place recognition [10] or the kidnapped robot prolem [11].”

“Vital tasks undertaken by robotic systems can benefit from deep learning; we have seen promising improvements to camera relocalization[11], [12], [13], de-convolutional networks are being applied to structural change detection over time in SLAM systems[14], object recognition in point clouds and RGB-D images[15], [16].”

“PoseNet [26] is a CNN that learns a mapping from images in an environment to metric camera poses, but it can only operate on the environment on which it was trained.”

“Kendall and Cipolla [40] showed that modeling the uncertainty in camera pose estimates can lead to higher localization performance.”

“PoseNet [109] first applies a convolutional neural network (CNN) for 6DoF camera localization, followed by a series of extensions with uncertainty reasoning [114] and geometric constraint [115] .”

“[19] used a Bayesian convolutional neural network to estimate relocalization uncertainty and regress the pose obtained from the camera.”

“Bayesian PoseNet [7] applies an uncertainty framework to the PoseNet by averaging Monte Carlo dropout samples from the posterior Bernoulli distribution of the Bayesian ConvNet’s weights.”

“Other works tried to address overfitting by averaging over predictions from models with randomly dropped activations [14] or by reducing the dimensionality of the global image encoding with LongShort-Term-Memory (LSTM) layers [36].”

“According to the loss function type, global feature-based regression methods include: (1) fixed Euclidean loss-based methods, which introduce the scaling factor for balancing the position item and orientation item [11], or add Bernoulli distributions to describe the uncertainty of localization [36].”

“Contemporary cutting-edge learning-based approaches for visual localization could be categorized as absolute posture regression (APR) [6], [7], [8], relative posture regression (RPR) [9], [10], and scene coordinates regression (SCoRe) [11], [12], [13], [14], [15].”

“Absolute pose approaches directly output the absolute camera pose [13–18 , 43 ].”

“Following PoseNet and its successor architectures [1][9][20][23][24], we report our results on the Cambridge Landmarks [9] and 7-Scenes [29] datasets with median error values.”

“Finally, we show experimental results of our methods and compare our results with the state-art networks, such as the PoseNet [21] and Bayesian PoseNet [23].”

“After PoseNet, subsequent works [20]–[24] focus on network architecture and loss function to improve accuracy.”

“The advantages of using MC Dropout are that it is scalable for deep neural networks since it does not introduce additional parameters, variational inference and inference procedures can be straightforwardly implemented in deep learning frameworks because this only requires small changes on the training and testing pro…”

“followed PoseNet to apply a Bayesian CNN [5], long-short”

“Following training, dropout layers were maintained to estimate pixelwise mean and variance probability by predicting 20 times to estimate model uncertainty using a Monte Carlo dropout approach (Y. Gal and Ghahramani, 2015; Kendall and Cipolla, 2015; Kendall et al., 2015).”

“[13] A. Kendall and R. Cipolla, “Modelling uncertainty in deep learning for camera relocalization,” in Proc.”

“In [18] authors have formulated the problem of finding the location of a robot as a regression problem where a continuous pose is predicted from an input image.”

“However, there are several major challenges in realizing fusion-based indoor navigation systems: • Mode-dependent feature extraction : In spot-based navigation, recent approaches focus on extracting features using convolutional neural networks (CNN) [23, 24, 43] with instant inputs, e.g., images.”

“The PoseNet architecture is an example of a regression network and estimates a 6-DoF pose from one single image [9].”

“Scene Area or Volume Active Search (w/o) [43] Active Search (w/) [43] PoseNet [23] Bayesian PoseNet [22] Proposed + Improvement(pos,ori)”

“Subsequent works further extend this approach by adding uncertainty of the estimated poses [19] and a learning weighted loss [4].”

“Kendall [19] first demonstrated end-to-end pose estimation using CNNs, eliminating intermediate steps like feature matching [20].”

“To evaluate the proposed method, four different scenes of Cambridge Landmarks dataset [15], [18], [19] are used.”

“An enhancement to the original PoseNet is the Bayesian PoseNet [25] which provides re-localization uncertainty by adding dropout layers after each convolutional layer and improves the PoseNet accuracy by averaging over multiple forward passes.”

“In the literature, the ground truth camera poses of the real images are estimated using SfM approaches, and the images are captured along a trajectory in the indoor or outdoor spaces (Kendall et al., 2015, Walch et al., 2017, Kendall and Cipolla, 2016, Clark et al., 2017).”

“At the opposite side of the spectrum, the six pose parameters are directly obtained from end-to-end CNN regression, thanks to prior training on data associating images with ground truth poses [15, 13, 20].”

“ess memory and has a constant inference time. Multiple improvements to the PoseNet include new loss functions [15], adding Long-Short-Term-Memory (LSTM) layers [33], the geometric re-projection error [14], and additional data sources and sensor measurements [5]. Other proposals extend the Posenet-like networks with au…”

“As for the training process, we train our model from scratch by Adam optimizer [ Kendall and Cipolla, 2016 ] with the batch size of 16 and the initial learning rate of 0.00025, (cid:12) 1 =0.9, (cid:12) 2 =0.999.”

“Scene PN [10] MN† [13] BPN [28] BMDN [16] Ours”

“Uncertainty quantification is a crucial aspect in deep learning [27], with applications in different fields including robotics [28], [29].”

“Recently, with the development of deep learning, the learning-based approaches to global localization have received much at-tentionwithintheroboticscommunity[7],[8].”

“Kendall and Cipolla (2016) successfully applied the uncertainty framework of convolutional neural networks to pose regression problems.”

“This work is extended by adopting a stochastic Dropout [48] in [49], and geometric loss minimizing reprojection errors in [50].”

“Previous researches [1, 2, 3] have shown that adding a dropout layer after each weight layer in a DNN is unnecessary.”

“They have further enhanced their method’s performance by modeling uncertainty and using the Bayesian model [22].”

“PoseNet (Kendall et al., 2015) which is the first end ‐ to ‐ end supervised learning ‐ based pose estimation algorithm, uses the deep CNN to learn the geometrical features and regress poses.”

“[29] used a Bayesian CNN to estimate the uncertainty of the model for the subsequent year, to detect the presence of scenes in the input image, and to improve the positioning accuracy of a large-scale outdoor dataset.”

“These methods are based on GoogleNet architecture[40] which consist of Inception modules, convolutions layers, and linear regressors, called PoseNet [13, 41, 42].”

“The proposed network is evaluated against recent state-of-the-art camera re-localization networks, including single image-based absolute camera pose regression network PoseNet and its variants [20–22,42] among which,”

“We choose a deep neural network structure based on GoogLeNet (Szegedy et al., 2015), owing to its use in past work that adapted it for orientation regression (Kendall and Cipolla, 2016; Kendall et al., 2015).”

“Other variants of PoseNet include: uncertainty in the pose prediction [17] and geometric constraints [5, 18].”

“Loss functions have been considered, based on the scene’s geometry to learn camera pose without any hyper-parameters [20] and use a Bayesian convolutional neural network implementation to get relocalisation uncertainties and improve the accuracy [21].”

“redictions. In computer vision, MC Dropout is the most popular instance of BNNs due to its speed and simplicity. It has been extended to other tasks, e.g., semantic segmentation [24], pose estimation [25]. However, the benets of Dropout are more limited for convolutional layers, where specic architectural design choic…”

“We also present results from other computer vision methods, including PoseNet [Kendall et al. 2015], Bayesian PoseNet [Kendall and Cipolla 2016], and LSTM-Pose [Walch et al. 2017].”

“Therefore, UAV is an effective alternative and the modal parameters can be identi fi ed using the extracted dynamic responses and OMA method (Forster et al., 2014; Kendall and Cipolla,2016; Yoon et al., 2006; Zhang,2000).”

“Subsequently, “Bayesian PoseNet” was proposed [22] to model the uncertainty of pose estimation using Bayesian learning.”

“This method is further improved in [17] and [15] by adding a Dropout layer or using a new loss function.”