Outdoor RGBD Instance Segmentation With Residual Regretting Learning
Abstract
Indoor semantic segmentation with RGBD input has received decent progress recently, but studies on instance-level objects in outdoor scenarios meet challenges due to the ambiguity in the acquired outdoor depth map. To tackle this problem, we proposed a residual regretting mechanism, incorporated into current flexible, general and solid instance segmentation framework Mask R-CNN in an end-to-end manner. Specifically, regretting cascade is designed to gradually refine and fully unearth useful information in depth maps, acting in a filtering and backup way. Additionally, embedded by a novel residual connection structure, the regretting module combines RGB and depth branches with pixel-level mask robustly. Extensive experiments on the challenging Cityscapes and KITTI dataset manifest the effectiveness of our residual regretting scheme for handling outdoor depth map. Our approach achieves state-of-the-art performance on RGBD instance segmentation, with 13.4% relative improvement over Mask R-CNN on Cityscapes by depth cue.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Fusing multi-modal data at the feature level is most prevalent, which can be further categorized into early fusion [186], late fusion [185], and fusion at multiple levels [85], [182].
“Among them, fusing multi-modal data at the feature level is most prevalent, which can be further categorized into three groups, i.e., early fusion [186], late fusion [185], and fusion at multiple levels [85], [182].”
From Empowering things with intelligence: a survey of the progress, challenges, and opportunities in artificial intelligence of things (2021) · cites this paper as [186]
Cites
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · see citing works
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834 · see citing works
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684 · see citing works
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939 · see citing works
He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.322 · Cited by 29,137 · see citing works
Girshick, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.169 · Cited by 28,105 · see citing works
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392 · see citing works
Zhao, H., Shi, J., Qi, X., Wang, X., & Jia, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.660 · Cited by 16,438 · see citing works
Self-citationThe Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433 · see citing works
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077 · see citing works
Path Aggregation Network for Instance Segmentation
Liu, S., Qi, L., Qin, H., Shi, J., & Jia, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00913 · Cited by 10,011 · see citing works
Self-citationVision meets robotics: The KITTI dataset
Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913491297 · Cited by 9,913 · see citing works
Eigen, D., & Fergus, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.304 · Cited by 2,862 · see citing works
Learning Rich Features from RGB-D Images for Object Detection and Segmentation
Gupta, S., Girshick, R., Arbeláez, P., & Malik, J. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10584-0_23 · Cited by 1,528 · see citing works
Instance-aware Semantic Segmentation via Multi-task Network Cascades
Dai, J., He, K., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1512.04412 · Cited by 1,273 · see citing works
Instance-Aware Semantic Segmentation via Multi-task Network Cascades
Dai, J., He, K., & Sun, J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.343 · Cited by 1,251 · see citing works
Simultaneous Detection and Segmentation
Hariharan, B., Arbeláez, P., Girshick, R., & Malik, J. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10584-0_20 · Cited by 1,176 · see citing works
Fully Convolutional Instance-Aware Semantic Segmentation
Li, Y., Qi, H., Dai, J., Ji, X., & Wei, Y. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.472 · Cited by 1,116 · see citing works
Deep Watershed Transform for Instance Segmentation
Bai, M., & Urtasun, R. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.305 · Cited by 563 · see citing works
Cross Modal Distillation for Supervision Transfer
Gupta, S., Hoffman, J., & Malik, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.309 · Cited by 530 · cited 2× · see citing works
“In [12], a transfer mechanism to learn rich representation for depth modality is proposed.”
BackgroundCascade Residual Learning: A Two-Stage Convolutional Neural Network for Stereo Matching
Pang, J., Sun, W., Ren, J., Yang, C., & Yan, Q. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.108 · Cited by 499 · see citing works
3D Graph Neural Networks for RGBD Semantic Segmentation
Qi, X., Liao, R., Jia, J., Fidler, S., & Urtasun, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.556 · Cited by 482 · see citing works
Indoor Semantic Segmentation using depth information
Couprie, C., Farabet, C., Najman, L. A., & LeCun, Y. (2013) · HAL · doi:10.48550/arxiv.1301.3572 · Cited by 451 · see citing works
Semantic Labeling of 3D Point Clouds for Indoor Scenes
Koppula, H. S., Anand, A., Joachims, T., & Saxena, A. (2011) · neural information processing systems · Cited by 322 · see citing works
End-to-End Instance Segmentation with Recurrent Attention
Ren, M., & Zemel, R. S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.39 · Cited by 309 · see citing works
SGN: Sequential Grouping Networks for Instance Segmentation
Liu, S., Jia, J., Fidler, S., & Urtasun, R. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.378 · Cited by 281 · see citing works
Self-citationInstance Segmentation by Jointly Optimizing Spatial Embeddings and Clustering Bandwidth
Neven, D., Brabandere, B. D., Proesmans, M., & Gool, L. V. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00904 · Cited by 272 · see citing works
The Layout Consistent Random Field for Recognizing and Segmenting Partially Occluded Objects
Winn, J. M., & Shotton, J. (2006) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2006.305 · Cited by 256 · see citing works
InstanceCut: From Edges to Instances with MultiCut
Kirillov, A., Levinkov, E., Andres, B., Savchynskyy, B., & Rother, C. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.774 · Cited by 251 · see citing works
Pixelwise Instance Segmentation with a Dynamically Instantiated Network
Arnab, A., & Torr, P. H. S. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.100 · Cited by 235 · see citing works
Proposal-Free Network for Instance-Level Object Segmentation
Liang, X., Lin, L., Wei, Y., Shen, X., Yang, J., & Yan, S. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2775623 · Cited by 233 · see citing works
Instance-Level Segmentation for Autonomous Driving with Deep Densely Connected MRFs
Zhang, Z., Fidler, S., & Urtasun, R. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.79 · Cited by 215 · see citing works
Learning Common and Specific Features for RGB-D Semantic Segmentation with Deconvolutional Networks
Wang, J., Wang, Z., Tao, D., See, S. C. W., & Wang, G. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46454-1_40 · Cited by 169 · see citing works
Pixel-Level Encoding and Depth Layering for Instance-Level Semantic Labeling
Uhrig, J., Cordts, M., Franke, U., & Brox, T. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-45886-1_2 · Cited by 154 · see citing works
Bridging Category-level and Instance-level Semantic Image SegmentationPreprint
Wu, Z., Shen, C., & van den Hengel, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.06885 · Cited by 148 · see citing works
Semantic Segmentation of RGBD Images with Mutex Constraints
Deng, Z., Todorović, S., & Latecki, L. J. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.202 · Cited by 80 · see citing works
Beat the MTurkers: Automatic Image Labeling from Weak 3D Supervision
Chen, L.-C., Fidler, S., & Urtasun, R. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.409 · Cited by 75 · see citing works
Recurrent Neural Networks for Semantic Instance Segmentation
Salvador, A., Bellver, M., Campos, V., Baradad, M., Marques, F., Torres, J., & Giro-i-Nieto, X. (2017) · arXiv · doi:10.48550/arxiv.1712.00617 · Cited by 62 · see citing works
BshapeNet: Object detection and instance segmentation with bounding shape masks
Kang, B. R., Lee, H., Park, K., Ryu, H., & Kim, H. Y. (2020) · Pattern Recognition Letters · doi:10.1016/j.patrec.2020.01.024 · Cited by 31 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Zhang, J., & Tao, D. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3039359 · Cited by 774 · see citing works
CalibNet: Dual-Branch Cross-Modal Calibration for RGB-D Salient Instance Segmentation
Pei, J., Jiang, T., Tang, H. Q., Liu, N., Jin, Y., Fan, D.-P., & Heng, P. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3432328 · Cited by 32 · see citing works
Multispectral Video Semantic Segmentation: A Benchmark Dataset and Baseline
Ji, W., Li, J., Bian, C., Zhou, Z., Zhao, J., Yuille, A., & Cheng, L. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00112 · Cited by 26 · see citing works
Link-RGBD: Cross-Guided Feature Fusion Network for RGBD Semantic Segmentation
Wu, P., Guo, R., Tong, X., Su, S., Zuo, Z., Sun, B., & Wei, J. (2022) · IEEE Sensors Journal · doi:10.1109/jsen.2022.3218601 · Cited by 26 · see citing works
CNN with Pose Segmentation for Suspicious Object Detection in MMW Security Images
Meng, Z., Zhang, M., & Wang, H. (2020) · Sensors · doi:10.3390/s20174974 · Cited by 23 · see citing works
Zhou, W., Zhang, H., Liu, Y., & Luo, T. (2025) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2025.3547613 · Cited by 10 · see citing works
DBNet: A depth-guided and boundary-aware network for amodal instance segmentation
Zhang, S., Yang, H., Zhang, J., & Wang, X. (2025) · Image and Vision Computing · doi:10.1016/j.imavis.2025.105688 · Cited by 8 · see citing works
Zhou, W., Zhang, H., & Qiu, W. (2024) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2024.3505057 · Cited by 8 · see citing works
Exploiting Depth Information for Wildlife MonitoringPreprint
Haucke, T., & Steinhage, V. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.05607 · Cited by 8 · see citing works
A robotic grasping method of box-shaped objects based on Dual-Stream You Only Look Once framework
Deng, H., Shi, J., Cui, J., Bai, S., & Zuo, X. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.111559 · Cited by 6 · see citing works
Self-citationLi, H., Ma, X., & Liu, H. (2024) · Aquaculture International · doi:10.1007/s10499-024-01610-4 · Cited by 2 · see citing works
Weakly-Supervised RGBD Video Object Segmentation
Yang, J., Gao, M., Zheng, F., Zhen, X., Ji, R., Shao, L., & Leonardis, A. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3374130 · Cited by 2 · see citing works
Depth-Guided Cross-Modal Fusion Network for Underwater Salient Instance Segmentation
Zheng, S., Zhou, X., Bao, L., Hu, X., & Zhang, J. (2026) · Symmetry · doi:10.3390/sym18050799 · Cited by 1 · see citing works
2020 Index IEEE Transactions on Image Processing Vol. 29
(2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.3046056
Research on Grasping Detection Method of Manipulator Based on SOLOV2
Yuan, S., Zuo, Z., Sun, B., & Liu, Y. (2023) · Lecture notes in electrical engineering · doi:10.1007/978-981-99-0479-2_51
CIMNet: A Contextual Information Mining Network for Real-time Instance Segmentation
Lyu, X., Yao, J., Zeng, T., Wang, X., Chen, S., Chen, J., Wu, C., & Li, X. (2023) · International Conference on Control, Robotics and Intelligent System (CCRIS) · doi:10.1145/3622896.3622914
DcieNet: dual-branch cross-modal interaction enhanced RGB-D instance segmentation
Lin, Z., Zhong, M., Yuan, B., Tan, J., Fan, K., & Lin, P. (2025) · Engineering Research Express · doi:10.1088/2631-8695/ae25b9
SLSM-Net: Sparse LiDAR Point Clouds Supervised Stereo Matching
Zong, Z., Wu, C., Xie, J., & Zhang, J. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3639911
Li, H., Weng, Y., Li, Z., Cong, R., Li, C., & Zhang, W. (2026) · International Symposium on Machine Learning and Media Computing (MLMC) · doi:10.1109/mlmc69743.2026.11686798
Feng, Y., Pallerla, C. K. R., Davar, A., Sohrabipour, P., Owens, C., Crandall, P. G., & Wang, D. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7675025
- Authors
- Xu, Z., Liu, S., Shi, J., & Lu, C.
- Affiliations
- Shanghai Jiao Tong University (CN)
- Chinese University of Hong Kong (HK)
- Sensetime (China) (CN)
- Venue
- IEEE Transactions on Image Processing
- Published
- 2020
- Volume
- 29
- Pages
- 5301-5309
- ISSN
- 1057-7149 (ISSN-L)
- 1941-0042
- OpenAlex
- W3008978349
- Cited by
- 20
- References
- 39
- Citations by year
- 2026: 3 · 2025: 5 · 2024: 4 · 2023: 3 · 2022: 1 · 2021: 1 · 2020: 3
- FWCI
- 0.78
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Image Processing (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Key R&D Program of China (2017YFA0700800)
- National Natural Science Foundation of China (61772332)
- Shanghai Qi Zhi Institute
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Engineering › Civil and Structural Engineering › Infrastructure Maintenance and Monitoring
- Keywords
- outdoor scene understanding, Mask R-CNN, depth map refinement, RGB-D fusion, cityscape, KITTI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
