Distilling Knowledge From a Deep Pose Regressor Network
Open accessThe DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Abstract
This paper presents a novel method to distill knowledge from a deep pose regressor network for efficient Visual Odometry (VO). Standard distillation relies on ''dark knowledge'' for successful knowledge transfer. As this knowledge is not available in pose regression and the teacher prediction is not always accurate, we propose to emphasize the knowledge transfer only when we trust the teacher. We achieve this by using teacher loss as a confidence score which places variable relative importance on the teacher prediction. We inject this confidence score to the main training task via Attentive Imitation Loss (AIL) and when learning the intermediate representation of the teacher through Attentive Hint Training (AHT) approach. To the best of our knowledge, this is the first work which successfully distill the knowledge from a deep pose regression network. Our evaluation on the KITTI and Malaga dataset shows that we can keep the student prediction close to the teacher with up to 92.95% parameter reduction and 2.12x faster in computation time.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Multiple View Geometry in Computer Vision
Hartley, R., & Zisserman, A. (2004) · doi:10.1017/cbo9780511811685 · Cited by 21,070
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897 · cited 8×
“• We perform extensive experiment on KITTI and Malaga datasets which show that our proposed approach can reduce the number of student parameters by up to 92.95% (2.12× faster) whilst keeping the prediction accuracy very close to that of the teacher.”
BackgroundMethodInfluentialHan, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662
Flownet: Learning optical flow with convolutional networks
Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., Smagt, P. V. D., Cremers, D., & Brox, T. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.316 · Cited by 4,313
FitNets: Hints for Thin Deep Nets
Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.6550 · Cited by 2,859
BinaryConnect: Training Deep Neural Networks with binary weights during propagations
Courbariaux, M., Bengio, Y., & David, J.-P. (2015) · arXiv · doi:10.48550/arxiv.1511.00363 · Cited by 2,180
Hubara, I., Soudry, D., & Yaniv, R. E. (2016) · arXiv · doi:10.48550/arxiv.1602.02505 · Cited by 1,667
Learning both Weights and Connections for Efficient Neural Networks
Han, S., Pool, J., Tran, J., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1506.02626 · Cited by 1,481 · cited 2×
“The pruning can be applied in the individual neuron connection [14, 13] or in convolutional filter itself [21].”
MethodDeep Learning with Limited Numerical Precision
Gupta, S., Agrawal, A., Gopalakrishnan, K., & Narayanan, P. (2015) · arXiv · doi:10.48550/arxiv.1502.02551 · Cited by 1,364 · cited 1×
“The quantization could be applied by using 16-bit or 8-bit representation as proposed by [31, 11].”
MethodSpeeding up Convolutional Neural Networks with Low Rank Expansions
Jaderberg, M., Vedaldi, A., & Zisserman, A. (2014) · Proceedings of the British Machine Vision Conference 2014 · doi:10.5244/c.28.88 · Cited by 1,164
Deepvo: towards end-to-end visual odometry with deep recurrent convolutional neural networks
Wang, S., Clark, R., Wen, H., & Trigoni, N. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989236 · Cited by 970 · cited 4×
“In particular, we employ DNN-based VO methods [33, 34, 37, 38, 26, 1] which replaces the conventional VO pipeline based on multiple-view geometry [15, 27] to DNN-based pipeline which automatically learns useful features for estimating 6 Degree-of-Freedom (DoF) camera poses.”
BackgroundMethodInfluentialSelf-citationCheng, Y., Wang, D., Zhou, P., & Zhang, T. (2018) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2765695 · Cited by 817 · cited 2×
“-1 or 1), binary networks can dramatically reduce both computation time and memory consumption at the cost of significant reduction in accuracy [5].”
BackgroundZhan, H., Garg, R., Weerasekera, C. S., Li, K., Agarwal, H., & Reid, I. M. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00043 · Cited by 705
Unifying distillation and privileged information
Lopez-Paz, D., Bottou, L., Schölkopf, B., & Vapnik, V. (2015) · arXiv · doi:10.48550/arxiv.1511.03643 · Cited by 270 · cited 1×
“KD methods for classification [16, 25, 20, 36, 32, 19, 23] rely solely on the teacher prediction without considering the error made w.”
MethodThe Málaga urban dataset: High-rate stereo and LiDAR in a realistic urban scenario
Blanco-Claraco, J.-L., Moreno-Dueñas, F.-Á., & González-Jiménez, J. (2014) · The International Journal of Robotics Research · doi:10.1177/0278364913507326 · Cited by 262
Bhattacharya, S., & Lane, N. D. (2016) · Proceedings of the 14th ACM Conference on Embedded Networked Sensor Systems CD-ROM · doi:10.1145/2994551.2994564 · Cited by 236
DeepTAM: Deep Tracking and Mapping
Zhou, H., Ummenhofer, B., & Brox, T. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01270-0_50 · Cited by 226
Yao, S., Zhao, Y., Zhang, A., Su, L., & Abdelzaher, T. (2017) · Proceedings of the 15th ACM Conference on Embedded Network Sensor Systems · doi:10.1145/3131672.3131675 · Cited by 188
Cited by
Knowledge Distillation: A Survey
Gou, J., Yu, B., Maybank, S. J., & Tao, D. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-021-01453-z · Cited by 3,931
Deep Learning for Visual Localization and Mapping: A Survey
Chen, C., Wang, B., Lu, C. X., Trigoni, N., & Markham, A. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3309809 · Cited by 97
Self-citationFrom Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networks
Geng, X., Wang, Z., Chen, C., Xu, Q., Xu, K., Jin, C., Gupta, M., Yang, X., Chen, Z., Sabry Aly, M. M., et al. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3394494 · Cited by 26
Metadata
- Authors
- Saputra, M. R. U., Gusmao, P., Almalioglu, Y., Markham, A., & Trigoni, N.
- Venue
- 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2019-10
- Pages
- 263-272
- OpenAlex
- W2965100203
- Cited by
- 117
- References
- 18
- FWCI
- 25.96
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.