Hard pixel mining for depth privileged semantic segmentation
Open accessAbstract
Semantic segmentation has achieved remarkable progress but remains challenging due to the complex scene, object occlusion, and so on. Some research works have attempted to use extra information such as a depth map to help RGB based semantic segmentation because the depth map could provide complementary geometric cues. However, due to the inaccessibility of depth sensors, depth information is usually unavailable for the test images. In this paper, we leverage only the depth of training images as the privileged information to mine the hard pixels in semantic segmentation, in which depth information is only available for training images but not available for test images. Specifically, we propose a novel Loss Weight Module, which outputs a loss weight map by employing two depth-related measurements of hard pixels: Depth Prediction Error and Depth-aware Segmentation Error. The loss weight map is then applied to segmentation loss, with the goal of learning a more robust model by paying more attention to the hard pixels. Besides, we also explore a curriculum learning strategy based on the loss weight map. Meanwhile, to fully mine the hard pixels on different scales, we apply our loss weight module to multi-scale side outputs. Our hard pixels mining method achieves the state-of-the-art results on three benchmark datasets, and even outperforms the methods which need depth input during testing.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LWM, using a ResNet101 backbone, achieves an mIoU of 83.4 on the Cityscapes dataset for 2.5D semantic segmentation.
“| LWM [97] | ResNet101 [16] | 83.4 |”
From Multimodal semantic segmentation in autonomous driving: A review of current approaches and future perspectives (2022) · cites this paper as [97]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · see citing works
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562 · see citing works
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834 · see citing works
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.324 · Cited by 30,581 · see citing works
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392 · see citing works
Zhao, H., Shi, J., Qi, X., Wang, X., & Jia, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.660 · Cited by 16,438 · see citing works
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433 · see citing works
Rethinking Atrous Convolution for Semantic Image Segmentation
Chen, L.-C., Papandreou, G., Schroff, F., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1706.05587 · Cited by 7,517 · see citing works
Indoor segmentation and support inference from rgbd images
Silberman, N., Hoiem, D., Kohli, P., & Fergus, R. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33715-4_54 · Cited by 5,760 · see citing works
Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366 · cited 2× · see citing works
“This training strategy is related to curriculum learning [34] and capable of further improving the segmentation performance.”
BackgroundLearning Deconvolution Network for Semantic Segmentation
Noh, H., Hong, S., & Han, B. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.178 · Cited by 4,422 · see citing works
Multi-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics
Cipolla, R., Gal, Y., & Kendall, A. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00781 · Cited by 3,657 · see citing works
RefineNet: Multi-path Refinement Networks for High-Resolution Semantic Segmentation
Lin, G., Milan, A., Shen, C., & Reid, I. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.549 · Cited by 3,287 · see citing works
CCNet: Criss-Cross Attention for Semantic Segmentation
Huang, Z., Wang, X., Huang, L., Huang, C., Wei, Y., & Liu, W. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00069 · Cited by 3,045 · see citing works
Eigen, D., & Fergus, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.304 · Cited by 2,862 · see citing works
Training Region-Based Object Detectors with Online Hard Example Mining
Shrivastava, A., Gupta, A., & Girshick, R. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.89 · Cited by 2,686 · see citing works
Learning Hierarchical Features for Scene Labeling
Farabet, C., Couprie, C., Najman, L., & LeCun, Y. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.231 · Cited by 2,620 · see citing works
Conditional Random Fields as Recurrent Neural Networks
Zheng, S., Jayasumana, S., Romera-Paredes, B., Vineet, V., Su, Z., Du, D., Huang, C., & Torr, P. H. S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.179 · Cited by 2,386 · cited 2× · see citing works
“Deep learning methods [5], [7], [1], [2], [3], [6], [9], [10], [35] have shown impressive results in RGB semantic segmentation.”
MethodSUN RGB-D: A RGB-D scene understanding benchmark suite
Song, S., Lichtenberg, S. P., & Xiao, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298655 · Cited by 1,943 · see citing works
ERFNet: Efficient Residual Factorized ConvNet for Real-Time Semantic Segmentation
Romera, E., Álvarez, J. M., Bergasa, L. M., & Arroyo, R. (2017) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2017.2750080 · Cited by 1,491 · see citing works
Self-Paced Learning for Latent Variable Models
Kumar, M., Packer, B., & Koller, D. (2010) · neural information processing systems · Cited by 1,204 · see citing works
FuseNet: Incorporating Depth into Semantic Segmentation via Fusion-Based CNN Architecture
Hazirbas, C., Ma, L., Domokos, C., & Cremers, D. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54181-5_14 · Cited by 902 · see citing works
A new learning paradigm: Learning using privileged information
Vapnik, V., & Vashist, A. (2009) · Neural Networks · doi:10.1016/j.neunet.2009.06.042 · Cited by 774 · see citing works
Semantic Image Segmentation via Deep Parsing Network
Liu, Z., Li, X., Luo, P., Loy, C.-C., & Tang, X. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.162 · Cited by 675 · see citing works
Perceptual Organization and Recognition of Indoor Scenes from RGB-D Images
Gupta, S., Arbeláez, P., & Malik, J. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.79 · Cited by 601 · see citing works
Xu, D., Ouyang, W., Wang, X., & Sebe, N. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00077 · Cited by 541 · see citing works
Curriculum Domain Adaptation for Semantic Segmentation of Urban Scenes
Zhang, Y., David, P., & Gong, B. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.223 · Cited by 504 · see citing works
ACNET: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentation
Hu, X., Yang, K., Fei, L., & Wang, K. (2019) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2019.8803025 · Cited by 500 · see citing works
3D Graph Neural Networks for RGBD Semantic Segmentation
Qi, X., Liao, R., Jia, J., Fidler, S., & Urtasun, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.556 · Cited by 482 · see citing works
Towards unified depth and semantic prediction from a single image
Wang, P., Shen, X., Lin, Z., Cohen, S. D., Price, B. L., & Yuille, A. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298897 · Cited by 455 · see citing works
Feedforward semantic segmentation with zoom-out features
Mostajabi, M., Yadollahpour, P., & Shakhnarovich, G. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298959 · Cited by 446 · see citing works
Improving Semantic Segmentation via Video Propagation and Label Relaxation
Zhu, Y., Sapra, K., Reda, F. A., Shih, K. J., Newsam, S. D., Tao, A., & Catanzaro, B. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00906 · Cited by 429 · see citing works
Convolutional feature masking for joint object and stuff segmentation
Dai, J., He, K., & Sun, J. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299025 · Cited by 425 · see citing works
Context Contrasted Feature and Gated Multi-scale Aggregation for Scene Segmentation
Ding, H., Jiang, X., Shuai, B., Liu, A. Q., & Wang, G. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00254 · Cited by 403 · see citing works
Pattern-Affinitive Propagation Across Depth, Surface Normal and Semantic Segmentation
Zhang, Z., Cui, Z., Xu, C., Yan, Y., Sebe, N., & Yang, J. J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00423 · Cited by 339 · see citing works
Depth-Aware CNN for RGB-D Segmentation
Wang, W., & Neumann, U. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01252-6_9 · Cited by 314 · see citing works
RDFNet: RGB-D Multi-level Residual Feature Fusion for Indoor Semantic Segmentation
Lee, S., Park, S., & Hong, K.-S. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.533 · Cited by 294 · see citing works
Not All Pixels Are Equal: Difficulty-Aware Semantic Segmentation via Deep Layer Cascade
Li, X., Liu, Z., Luo, P., Loy, C. C., & Tang, X. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.684 · Cited by 268 · see citing works
Gupta, S., Arbeláez, P., Girshick, R., & Malik, J. (2014) · International Journal of Computer Vision · doi:10.1007/s11263-014-0777-6 · Cited by 248 · see citing works
Locality-Sensitive Deconvolution Networks with Gated Fusion for RGB-D Indoor Semantic Segmentation
Cheng, Y., Cai, R., Li, Z., Zhao, X., & Huang, K. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.161 · Cited by 242 · see citing works
Automated Curriculum Learning for Neural NetworksPreprint
Graves, A., Bellemare, M. G., Menick, J., Munos, R., & Kavukcuoglu, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.03003 · Cited by 237 · see citing works
Chen, Y., Li, W., Chen, X., & Gool, L. V. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00194 · Cited by 236 · see citing works
Joint Task-Recursive Learning for Semantic Segmentation and Depth Estimation
Zhang, Z., Cui, Z., Xu, C., Jie, Z., Li, X., & Yang, J. J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01249-6_15 · Cited by 228 · see citing works
Curriculum learning of multiple tasks
Pentina, A., Sharmanska, V., & Lampert, C. H. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299188 · Cited by 218 · see citing works
Look Deeper into Depth: Monocular Depth Estimation with Semantic Booster and Attention-Driven Loss
Jiao, J., Cao, Y. S., Song, Y., & Lau, R. W. H. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01267-0_4 · Cited by 218 · see citing works
Learning with Side Information through Modality Hallucination
Hoffman, J., Gupta, S., & Darrell, T. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.96 · Cited by 207 · see citing works
Choi, S., Kim, J. T., & Choo, J. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00939 · Cited by 197 · see citing works
Learning to Rank Using Privileged Information
Sharmanska, V., Quadrianto, N., & Lampert, C. H. (2013) · IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.107 · Cited by 196 · see citing works
Learning Sparse High Dimensional Filters: Image Filtering, Dense CRFs and Bilateral Neural Networks
Jampani, V., Kiefel, M., & Gehler, P. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.482 · Cited by 191 · see citing works
DADA: Depth-Aware Domain Adaptation in Semantic Segmentation
Vu, T.-H., Jain, H., Bucher, M., Cord, M., & Pérez, P. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00746 · Cited by 190 · see citing works
Fast, Exact and Multi-scale Inference for Semantic Image Segmentation with Deep Gaussian CRFs
Chandra, S., & Kokkinos, I. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46478-7_25 · Cited by 180 · see citing works
Spatial Information Guided Convolution for Real-Time RGBD Semantic Segmentation
Chen, L.-Z., Lin, Z., Wang, Z., Yang, Y., & Cheng, M.-M. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3049332 · Cited by 177 · see citing works
Learning Common and Specific Features for RGB-D Semantic Segmentation with Deconvolutional Networks
Wang, J., Wang, Z., Tao, D., See, S. C. W., & Wang, G. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46454-1_40 · Cited by 169 · see citing works
A Multi-scale CNN and Curriculum Learning Strategy for Mammogram Classification
Lotter, W. E., Sorensen, G., & Cox, D. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-67558-9_20 · Cited by 155 · see citing works
Cascaded Feature Network for Semantic Segmentation of RGB-D Images
Lin, D., Chen, G., Cohen‐Or, D., Heng, P., & Huang, H. F. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.147 · Cited by 155 · see citing works
Gaussian Conditional Random Field Network for Semantic Segmentation
Vemulapalli, R., Tuzel, O., Liu, M.-Y., & Chellappa, R. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.351 · Cited by 150 · see citing works
Loss Max-Pooling for Semantic Image Segmentation
Bulò, S. R., Neuhold, G., & Kontschieder, P. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.749 · Cited by 145 · see citing works
Real-Time Joint Semantic Segmentation and Depth Estimation Using Asymmetric Annotations
Nekrasov, V., Dharmasiri, T., Spek, A., Drummond, T. W., Shen, C., & Reid, I. D. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8794220 · Cited by 143 · see citing works
Light-Weight RefineNet for Real-Time Semantic SegmentationPreprint
Nekrasov, V., Shen, C., & Reid, I. D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.03272 · Cited by 117 · see citing works
Xu, X., Li, W., & Xu, D. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2405574 · Cited by 104 · see citing works
Deep Learning Under Privileged Information Using Heteroscedastic Dropout
Lambert, J., Sener, O., & Savarese, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00926 · Cited by 93 · cited 1× · see citing works
“Recently, privileged information has also been utilized in deep learning methods [52], [53], [54], which aim to distill knowledge from privileged information or use privileged information to control the training process.”
BackgroundSalient Object Segmentation via Effective Integration of Saliency and Objectness
Ye, L., Liu, Z., Li, L., Shen, L., Bai, C., & Wang, Y. (2017) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2693022 · Cited by 89 · see citing works
SCN: Switchable Context Network for Semantic Segmentation of RGB-D Images
Lin, D., Zhang, R., Ji, Y., Li, P., & Huang, H. F. (2018) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2018.2885062 · Cited by 87 · see citing works
SPIGAN: Privileged Adversarial Learning from SimulationPreprint
Lee, K.-H., Ros, G., Li, J., & Gaidon, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.03756 · Cited by 80 · see citing works
Incorporating Privileged Information Through Metric Learning
Fouad, S., Peter Tiňo, B., Raychaudhury, S., & Schneider, P. (2013) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2013.2251470 · Cited by 78 · see citing works
Privileged information for data clustering
Feyereisl, J., & Aickelin, U. (2011) · Information Sciences · doi:10.1016/j.ins.2011.04.025 · Cited by 75 · see citing works
Yang, H., Zhou, J. T., Cai, J., & Ong, Y.-S. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.635 · Cited by 75 · see citing works
Higher-Order Image Co-segmentation
Wang, W., & Shen, J. (2016) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2016.2545409 · Cited by 73 · see citing works
Depth-Adaptive Deep Neural Network for Semantic Segmentation
Kang, B., Lee, Y., & Nguyen, T. Q. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2798282 · Cited by 71 · see citing works
Collaborative Deconvolutional Neural Networks for Joint Depth Estimation and Semantic Segmentation
Liu, J., Wang, Y., Li, Y., Fu, J., Li, J., & Lu, H. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2017.2787781 · Cited by 63 · see citing works
Hung, S.-W., Lo, S.-Y., & Hang, H. (2019) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2019.8803360 · Cited by 60 · see citing works
Exploiting Privileged Information from Web Data for Image Categorization
Li, W., Niu, L., & Xu, D. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10602-1_29 · Cited by 59 · see citing works
RFBNet: Deep Multimodal Networks with Residual Fusion Blocks for RGB-D Semantic SegmentationPreprint
Deng, L., Yang, M. Z., Li, T., He, Y., & Wang, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.00135 · Cited by 57 · see citing works
Active contour based optical character recognition for automated scene understanding
Olszewska, J. I. (2015) · Neurocomputing · doi:10.1016/j.neucom.2014.12.089 · Cited by 54 · see citing works
Privileged information-based conditional regression forest for facial feature detection
Yang, H., & Patras, I. (2013) · IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2013.6553766 · Cited by 53 · see citing works
Multimodal Neural Networks: RGB-D for Semantic Segmentation and Object Detection
Schneider, L., Jasch, M., Fröhlich, B., Weber, T., Franke, U., Pollefeys, M., & Rätsch, M. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-59126-1_9 · Cited by 42 · see citing works
Learning Deep Conditional Neural Network for Image Segmentation
Wang, Q., Yuan, C., & Liu, Y. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2890360 · Cited by 42 · see citing works
Multiple Instance Curriculum Learning for Weakly Supervised Object Detection
Li, S., Zhu, X., Huang, Q., Xu, H., & Kuo, C. ‐. J. (2017) · British Machine Vision Conference (BMVC) · doi:10.5244/c.31.29 · Cited by 33 · see citing works
Adaptive SVM+: Learning with Privileged Information for Domain Adaptation
Sarafianos, N., Vrigkas, M., & Kakadiaris, I. A. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.313 · Cited by 32 · see citing works
Transfer Hashing with Privileged InformationPreprint
Zhou, J. T., Xu, X., Pan, S. J., Tsang, I. W.-H., Qin, Z., & Goh, R. S. M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.04034 · Cited by 30 · see citing works
Ordinal-based metric learning for learning using privileged information
Fouad, S., & Tiňo, P. (2013) · 2022 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2013.6706799 · Cited by 17 · see citing works
Joint Deep and Depth for Object-Level Segmentation and Stereo Tracking in Crowds
Li, J., Wei, L., Zhang, F., Yang, T., & Lu, Z. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2908350 · Cited by 16 · see citing works
MFDCNN: A Multimodal Fusion DCNN Framework for Object Detection and Segmentation
Zhou, F., Hu, Y., & Shen, X. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00764-5_1 · Cited by 3 · see citing works
Flagged first, then by citations
4 of the 88 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Pareto Refocusing for Drone-View Object Detection
Leng, J., Mo, M., Zhou, Y., Gao, C., Li, W., & Gao, X. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3210207 · Cited by 90 · see citing works
Part-Aware Correlation Networks for Few-Shot Learning
Zhang, R., Tan, J., Cao, Z., Xu, L., Liu, Y., Si, L., & Sun, F. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3394681 · Cited by 76 · see citing works
A brief survey on RGB-D semantic segmentation using deep learning
Wang, C., Wang, C., Li, W., & Wang, H. (2021) · Displays · doi:10.1016/j.displa.2021.102080 · Cited by 71 · see citing works
Rizzoli, G., Barbato, F., & Zanuttigh, P. (2022) · Technologies · doi:10.3390/technologies10040090 · Cited by 58 · see citing works
EH-former: Regional easy-hard-aware transformer for breast lesion segmentation in ultrasound images
Qu, X., Zhou, J., Jiang, J., Wang, W., Wang, H., Wang, S., Tang, W., & Lin, X. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102430 · Cited by 50 · see citing works
Pyramid Fusion Transformer for Semantic Segmentation
Qin, Z., Liu, J., Zhang, X., Tian, M., Zhou, A., Yi, S., & Li, H. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3396281 · Cited by 37 · see citing works
Learning Cross-Channel Representations for Semantic Segmentation
Ma, L., Xie, H., Liu, C., & Zhang, Y. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3151145 · Cited by 31 · see citing works
Rethinking Image Deblurring via CNN-Transformer Multiscale Hybrid Architecture
Zhao, Q., Yang, H., Zhou, D., & Cao, J. (2023) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3230482 · Cited by 30 · see citing works
Learning to see colours: Biologically relevant virtual staining for adipocyte cell images
Wieslander, H., Gupta, A., Bergman, E., Hallström, E., & Harrison, P. J. (2021) · PLoS ONE · doi:10.1371/journal.pone.0258546 · Cited by 22 · see citing works
MutualFormer: Multi-modal Representation Learning via Cross-Diffusion Attention
Wang, X., Wang, X., Jiang, B., Tang, J., & Luo, B. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02067-x · Cited by 20 · see citing works
A Survey on Deep Learning Methods for Semantic Image Segmentation in Real-TimePreprint
Takos, G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.12942 · Cited by 23 · see citing works
Camouflaged Locust Segmentation Based on Pranet
Liu, L., Liu, M., Meng, K., Yang, L., Zhao, M., & Mei, S. (2022) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2022.107061 · Cited by 20 · see citing works
BFANet: Revisiting 3D Semantic Segmentation with Boundary Feature Analysis
Zhao, W., Zhang, R., Wang, Q., Cheng, G., & Huang, K. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02737 · Cited by 15 · see citing works
Real-time Semantic Segmentation with Parallel Multiple Views Feature Augmentation
Qiao, J.-J., Cheng, Z.-Q., Wu, X. Y., Li, W., & Zhang, J. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3547786 · Cited by 15 · see citing works
Chen, S., Xu, X., Yang, N., Chen, X., Du, F., Ding, S., & Gao, W. G. (2022) · Infrared Physics & Technology · doi:10.1016/j.infrared.2022.104164 · Cited by 14 · see citing works
Privileged Modality Learning via Multimodal Hallucination
Wei, S., Luo, C., Luo, Y., & Xu, J. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3282874 · Cited by 13 · see citing works
SRRNet: A Semantic Representation Refinement Network for Image Segmentation
Ding, X., Zeng, T., Tang, J., Che, Z., & Peng, Y. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3198314 · Cited by 12 · see citing works
Boundary-guided multi-scale refinement network for camouflaged object detection
Ye, Q., Li, Q., Huo, G., Liu, Y., & Zhou, Y. (2025) · The Visual Computer · doi:10.1007/s00371-024-03786-5 · Cited by 12 · see citing works
DTS-Net: Depth-to-Space Networks for Fast and Accurate Semantic Object Segmentation
Ibrahem, H., Salem, A., & Kang, H.-S. (2022) · Sensors · doi:10.3390/s22010337 · Cited by 11 · see citing works
Zhou, W., Zhang, H., Liu, Y., & Luo, T. (2025) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2025.3547613 · Cited by 10 · see citing works
Contour-Aware Equipotential Learning for Semantic Segmentation
Yin, X., Min, D., Huo, Y., & Yoon, S. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3205441 · Cited by 10 · see citing works
Self-Supervised Saliency Estimation for Pixel Embedding in Road Detection
Zhou, D., Tian, Y. T., Chen, W., & Huang, G. (2021) · IEEE Signal Processing Letters · doi:10.1109/lsp.2021.3089912 · Cited by 9 · see citing works
Binocular Image Dehazing via a Plain Network Without Disparity Estimation
Nie, J., Pang, Y., Xie, J., Han, J., & Li, X. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3199553 · Cited by 9 · see citing works
Boosting Remote Sensing Change Detection via Hard Region Mining
Li, Z., Tang, C., Hu, X., Li, N., Xiang, S., Li, C., Li, C., & Liu, X. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3598766 · Cited by 8 · see citing works
EfferDeepNet: An Efficient Semantic Segmentation Method for Outdoor Terrain
Wei, Y., Wu, W., & Zhang, Y. (2023) · Machines · doi:10.3390/machines11020256 · Cited by 8 · see citing works
Inharmonious Region Localization
Liang, J., Niu, L., & Zhang, L. (2021) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme51207.2021.9428309 · Cited by 8 · see citing works
Hard Sample Mining: A New Paradigm of Efficient and Robust Model Training
Liu, L., Liang, Y., Yan, X., Huangfu, L., Samtani, S., Yu, Z., Zhang, Y., & Zeng, D. D. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3610948 · Cited by 6 · see citing works
Wieslander, H., Gupta, A., Bergman, E., Hallström, E., & Harrison, P. J. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.01.18.427121 · Cited by 7 · see citing works
Fine-Grained Representation Alignment for Zero-Shot Domain Adaptation
Liu, Y., Wang, J., Zhong, S.-H., Ma, L., & Xu, Y. D. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3347841 · Cited by 6 · see citing works
Yang, T., Zhang, Y., Zhu, M., Wang, Y., An, S., Gu, X., Liu, X., Han, J., He, Y., & Zhu, H. (2022) · IET Image Processing · doi:10.1049/ipr2.12597 · Cited by 6 · see citing works
Accelerating the creation of instance segmentation training sets through bounding box annotation
Sayez, N., & Vleeschouwer, C. D. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956321 · Cited by 6 · see citing works
EFRNet: Efficient Feature Reconstructing Network for Real-Time Scene Parsing
Li, X., Yang, F., Luo, A., Jiao, Z., Cheng, H., & Liu, Z. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3089422 · Cited by 5 · see citing works
GEM: Boost Simple Network for Glass Surface Segmentation via Vision Foundation Models
Hao, J., Liu, M., Yang, J., & Hung, K. F. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3535404 · Cited by 4 · see citing works
Lee, J. Y., Park, R., & Kim, J. (2021) · IEEE Access · doi:10.1109/access.2021.3098819 · Cited by 4 · see citing works
Self-Paced Knowledge Distillation for Real-Time Image Guided Depth Completion
Wang, S., Hu, M. H., Li, B., & Gong, X. (2022) · IEEE Signal Processing Letters · doi:10.1109/lsp.2022.3154243 · Cited by 4 · see citing works
Depth Privileged Object Detection with Depth-Enhanced DCN
Liu, Y., Zhang, Z., & Zhang, L. (2021) · Communications in computer and information science · doi:10.1007/978-3-030-92307-5_51 · Cited by 2 · see citing works
Real-time RGB-D Semantic Segmentation With Scale-invariant Depth Encoding and Noise-robust Fusion
Woo, S., Hyun, J., Lee, S., & Kim, E. (2025) · International Journal of Control Automation and Systems · doi:10.1007/s12555-025-0465-z · Cited by 1 · see citing works
Multimodal Contrastive Learning for Unpaired and Depth-privileged Semantic Segmentation
Yang, X., Ke, D., & Gong, X. (2023) · International Conference on Wireless Communications and Signal Processing (WCSP) · doi:10.1109/wcsp58612.2023.10405240 · Cited by 1 · see citing works
Optimizing against Infeasible Inclusions from Data for Semantic Segmentation through Morphology
Basu, S., Gool, L. V., & Sakaridis, C. (2026) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61042.2026.00726
Topology-guided hard example mining for cell detection
Çakı, O., Ünver, S., Karasayar, A. H. D., Mericoz, C. A., Bulutay, P., Kapucuoğlu, N., Eren, H., Dilbaz, O. F., Osmanli, J., Yetkili, B. S., et al. (2026) · Medical Image Analysis · doi:10.1016/j.media.2026.104155
- Authors
- Gu, Z., Niu, L., Zhao, H., & Zhang, L.
- Affiliations
- Shanghai Jiao Tong University (CN)
- Venue
- IEEE Transactions on Multimedia
- Published
- 2021
- Volume
- 23
- Pages
- 3738-3751
- ISSN
- 1520-9210 (ISSN-L)
- 1941-0077
- OpenAlex
- W3097762837
- Cited by
- 41
- References
- 88
- Citations by year
- 2026: 2 · 2025: 7 · 2024: 5 · 2023: 4 · 2022: 14 · 2021: 8 · 2020: 1
- FWCI
- 2.63
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Multimedia (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Key R&D Program of China (2018AAA0100704)
- Science and Technology Commission of Shanghai Municipality (20511100300)
- National Natural Science Foundation of China (61902247)
- Shanghai Sailing Program (19YF1424400)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Keywords
- curriculum learning, object occlusion
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
