Cross-modal adaptation for RGB-D detection
The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
In this paper we propose a technique to adapt convolutional neural network (CNN) based object detectors trained on RGB images to effectively leverage depth images at test time to boost detection performance. Given labeled depth images for a handful of categories we adapt an RGB object detector for a new category such that it can now use depth images in addition to RGB images at test time to produce more accurate detections. Our approach is built upon the observation that lower layers of a CNN are largely task and category agnostic and domain specific while higher layers are largely task and category specific while being domain agnostic. We operationalize this observation by proposing a mid-level fusion of RGB and depth CNNs. Experimental evaluation on the challenging NYUD2 dataset shows that our proposed adaptation technique results in an average 21% relative improvement in detection performance over an RGB-only baseline even when no depth training data is available for the particular category evaluated. We believe our proposed technique will extend advances made in computer vision to RGB-D data leading to improvements in performance at little additional annotation effort.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Selective Search for Object Recognition
Uijlings, J. R. R., van de Sande, K. E. A., Gevers, T., & Smeulders, A. W. M. (2013) · International Journal of Computer Vision · doi:10.1007/s11263-013-0620-5 · Cited by 5,980
Indoor segmentation and support inference from rgbd images
Silberman, N., Hoiem, D., Kohli, P., & Fergus, R. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33715-4_54 · Cited by 5,760
Learning a Similarity Metric Discriminatively, with Application to Face Verification
Chopra, S., Hadsell, R., & LeCun, Y. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.202 · Cited by 4,036
DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., & Darrell, T. (2013) · arXiv · doi:10.48550/arxiv.1310.1531 · Cited by 4,019
Self-citationAdapting visual category models to new domains
Saenko, K., Kulis, B., Fritz, M., & Darrell, T. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15561-1_16 · Cited by 2,847 · cited 1×
“a) Transferring Information Across Tasks: Many methods have been proposed to transfer general information between different data sources for related tasks [39], [30], [17], [23], [14], [6].”
BackgroundSelf-citationEdge Boxes: Locating Object Proposals from Edges
Zitnick, C. L., & Dollár, P. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_26 · Cited by 2,656
Signature Verification Using A "Siamese" Time Delay Neural Network
BROMLEY, J., BENTZ, J. W., BOTTOU, L., GUYON, I., LECUN, Y., MOORE, C., SÄCKINGER, E., & SHAH, R. (1993) · International Journal of Pattern Recognition and Artificial Intelligence · doi:10.1142/s0218001493000339 · Cited by 2,077
Learning Rich Features from RGB-D Images for Object Detection and Segmentation
Gupta, S., Girshick, R., Arbeláez, P., & Malik, J. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10584-0_23 · Cited by 1,528
Self-citationCross Modal Distillation for Supervision Transfer
Gupta, S., Hoffman, J., & Malik, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.309 · Cited by 530 · cited 2×
“([19] and [20] + Fast R-CNN as described in [21]).”
MethodSelf-citationUnderstanding image representations by measuring their equivariance and equivalence
Lenc, K., & Vedaldi, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298701 · Cited by 366
Unsupervised feature learning for 3D scene labeling
Lai, K., Bo, L., & Fox, D. (2014) · 2014 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2014.6907298 · Cited by 317 · cited 2×
“c) Object Detection: Lastly, there has been considerable work on the problem of object detection for RGB-D images [26], [43], [32], [48], [51], [20], [42], [31].”
BackgroundLSDA: Large Scale Detection Through Adaptation
Hoffman, J., Guadarrama, S., Tzeng, E., Hu, R., Donahue, J., Girshick, R., Darrell, T., & Saenko, K. (2014) · arXiv · doi:10.48550/arxiv.1407.5035 · Cited by 284 · cited 12×
“The LSDA [24] model was available only for RGB detection along with an RGB region proposal method (selective Fig.”
MethodInfluentialSelf-citationHolistic Scene Understanding for 3D Object Detection with RGBD Cameras
Lin, D., Fidler, S., & Urtasun, R. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.179 · Cited by 253
How good are detection proposals, really?
Hosang, J., Benenson, R., & Schiele, B. (2014) · Proceedings of the British Machine Vision Conference 2014 · doi:10.5244/c.28.24 · Cited by 215
Cited by
Deep visual domain adaptation: A survey
Wang, M., & Deng, W. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.05.083 · Cited by 2,209
Wang, L., & Yoon, K.-J. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3055564 · Cited by 752
SemanticFusion: Dense 3D semantic mapping with convolutional neural networks
McCormac, J., Handa, A., Davison, A., & Leutenegger, S. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989538 · Cited by 682
Cross Modal Distillation for Supervision Transfer
Gupta, S., Hoffman, J., & Malik, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.309 · Cited by 530
“This can naturally be extended to incorporate annotations from auxiliary categories as explored in [21], but we defer this to future work.”
BackgroundSelf-citationCNNs-Based RGB-D Saliency Detection via Cross-View Transfer and Multiview Fusion
Han, J., Chen, H., Liu, N., Yan, C., & Li, X. (2018) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2017.2761775 · Cited by 423
Metadata
- Authors
- Hoffman, J., Gupta, S., Leong, J., Guadarrama, S., & Darrell, T.
- Venue
- 2016 IEEE International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2016-05
- Pages
- 5032-5039
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W2413439537
- Cited by
- 81
- References
- 17
- FWCI
- 4.35
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.