CoChat

Learning Dual Multi-Scale Manifold Ranking for Semantic Segmentation of High-Resolution Images

Open access
Zhang, Mi, Hu, Xiangyun, Zhao, Like, Lv, Ye, Luo, Min, Pang, Shiyan · 2017 · Remote Sensing

Published version of a preprint

Cited by 57

The DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.

Abstract

Semantic image segmentation has recently witnessed considerable progress by training deep convolutional neural networks (CNNs). The core issue of this technique is the limited capacity of CNNs to depict visual objects. Existing approaches tend to utilize approximate inference in a discrete domain or additional aides and do not have a global optimum guarantee. We propose the use of the multi-label manifold ranking (MR) method in solving the linear objective energy function in a continuous domain to delineate visual objects and solve these problems. We present a novel embedded single stream optimization method based on the MR model to avoid approximations without sacrificing expressive power. In addition, we propose a novel network, which we refer to as dual multi-scale manifold ranking (DMSMR) network, that combines the dilated, multi-scale strategies with the single stream MR optimization method in the deep learning architecture to further improve the performance. Experiments on high resolution images, including close-range and remote sensing datasets, demonstrate that the proposed approach can achieve competitive accuracy without additional aides in an end-to-end manner.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3390/rs9050500

Cites

20 of 20 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 71

  1. Ladicky, L., Torr, P., and Zisserman, A. (2013, January 23–28). Human Pose Estimation using a Joint Pixel-wise and Part-wise Formulation. Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA. · DOI
  2. Romera, E., Bergasa, L., and Arroyo, R. (arXiv, 2016). Can we unify monocular detectors for autonomous driving by using the pixel-wise semantic segmentation of CNNs?, arXiv.
  3. Barrnes, D., Maddern, W., and Posner, I. (arXiv, 2016). Find Your Own Way: Weakly-Supervised Segmentation of Path Proposals for Urban Autonomy, arXiv. · DOI · Paper page
  4. Xiao, J., and Quan, L. (October, January 29). Multiple View Semantic Segmentation for Street View Images. Proceedings of the 2009 IEEE 12th International Conference on Computer Vision, Kyoto, Japan.
  5. Floros, G., and Leibe, B. (2012, January 16–21). Joint 2D-3D Temporally Consistent Semantic Segmentation of Street Scenes. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA. · DOI
  6. Chen, C., Seff, A., Kornhauser, A., and Xiao, J. (2015, January 7–13). Deepdriving: Learning affordance for direct perception in autonomous driving. Proceedings of the 2015 IEEE International Conference on Computer Vision, Santiago, Chile. · DOI · Paper page
  7. Toshev, A., and Szegedy, C. (arXiv, 2014). DeepPose: Human Pose Estimation via Deep Neural Networks, arXiv. · DOI · Paper page
  8. Tompson, J.J., Jain, A., LeCun, Y., and Bregler, C. (arXiv, 2014). Joint training of a convolutional network and a graphical model for human pose estimation, arXiv.
  9. Jackson, A., Valstar, M., and Tzimiropoulos, G. (arXiv, 2016). A CNN Cascade for Landmark Guided Semantic Part Segmentation, arXiv. · DOI
  10. Maggiori, E., Tarabalka, Y., Charpiat, G., and Alliez, P. (arXiv, 2016). High-Resolution Semantic Labeling with Convolutional Neural Networks, arXiv. · DOI
  11. Kampffmeyer, M., Salberg, A.B., and Jenssen, R. (July, January 26). Semantic segmentation of small objects and modeling of uncertainty in urban remote sensing images using deep convolutional neural networks. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Las Vegas, NV, USA. · DOI · Paper page
  12. Audebert, N., Saux, B.L., and Lefèvre, S. (arXiv, 2016). Semantic Segmentation of Earth Observation Data Using Multimodal and Multi-scale Deep Networks, arXiv. · DOI · Paper page
  13. Längkvist, M., Kiselev, A., Alirezaie, M., and Loutfi, A. (2016). Classification and segmentation of satellite orthoimagery using convolutional neural networks. Remote Sens., 8. · DOI · Paper page
  14. Muruganandham, S. (2016). Semantic Segmentation of Satellite Images Using Deep Learning. [Master’s Thesis, Department of Computer Science, Electrical and Space Engineering, Luleå University of Technology].
  15. Barron, J.T., and Poole, B. (arXiv, 2016). The fast bilateral solver, arXiv. · DOI · Paper page
  16. Mostajabi, M., Yadollahpour, P., and Shakhnarovich, G. (arXiv, 2015). Feedforward semantic segmentation with zoom-out features, arXiv. · DOI · Paper page
  17. Shelhamer, E., Long, J., and Darrell, T. (arXiv, 2015). Fully Convolutional Networks for Semantic Segmentation, arXiv.
  18. Yu, F., and Koltun, V. (arXiv, 2016). Multi-Scale Context Aggregation by Dilated Convolutions, arXiv.
  19. Chen, L., Papandreou, G., Kokkinos, I., Murphy, K., and Yuille, A. (arXiv, 2015). DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs, arXiv.
  20. Chandra, S., and Kokkinos, I. (arXiv, 2016). Fast, Exact and Multi-Scale Inference for Semantic Image Segmentation with Deep Gaussian CRFs, arXiv. · DOI · Paper page
  21. Badrinarayanan, V., Handa, A., and Cipolla, R. (arXiv, 2015). SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation, arXiv.
  22. Hyeonwoo, N., Hong, S., and Han, B. (arXiv, 2015). Learning deconvolution network for semantic segmentation, arXiv.
  23. Chen, L., Schwing, A., Yuille, A., and Urtasun, R. (arXiv, 2015). Learning Deep Structured Models, arXiv.
  24. Chen, L., Papandreou, G., Kokkinos, I., Murphy, K., and Yuille, A. (arXiv, 2015). Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs, arXiv.
  25. Arnab, A., Jayasumana, S., Zheng, S., and Torr, P. (arXiv, 2016). Higher Order Conditional Random Fields in Deep Neural Networks, arXiv. · DOI · Paper page
  26. Zhou, D., Weston, J., Gretton, A., Bousquent, O., and Scholkopf, B. (2003, January 9–11). Ranking on data manifolds. Proceedings of the 16th International Conference on Neural Information Processing Systems, Whistler, BC, Canada.
  27. Zhou, D., Bousquent, O., Lal, T., Weston, J., and Scholkopf, B. (2003, January 9–11). Learning with Local and Global Consistency. Proceedings of the 16th International Conference on Neural Information Processing Systems, Whistler, BC, Canada.
  28. Bencherif, 2015, Fusion of Extreme Learning Machine and Graph-Based Optimization Methods for Active Classification of Remote Sensing Images, IEEE Geosci. Remote Sens. Lett., 12, 527, 10.1109/LGRS.2014.2349538 · DOI
  29. Krähenbühl, P., and Koltun, V. (2013, January 16–21). Parameter Learning and Convergent Inference for Dense Random Fields. Proceedings of the 30th International Conference on Machine Learning, Atlanta, GA, USA.
  30. Hariharan, B., Arbeláez, P., Girshick, R., and Malik, J. (arXiv, 2014). Simultaneous Detection and Segmentation, arXiv. · DOI · Paper page
  31. Dai, J., He, K., and Sun, J. (arXiv, 2015). Convolutional Feature Masking for Joint Object and Stuff Segmentation, arXiv. · DOI · Paper page
  32. Chen, L., Yang, Y., Wang, J., Xu, W., and Yuille, A. (arXiv, 2016). Attention to Scale: Scale-aware Semantic Image Segmentation, arXiv. · DOI · Paper page
  33. Bearman, A., Russakovsky, O., Ferrari, V., and Li, F.F. (arXiv, 2016). What’s the Point: Semantic Segmentation with Point Supervision, arXiv. · DOI · Paper page
  34. Romero, 2015, Unsupervised Deep Feature Extraction for Remote Sensing Image Classification, IEEE Trans. Geosci. Remote Sens., 54, 1349, 10.1109/TGRS.2015.2478379 · DOI · Paper page
  35. Gatta, 2016, Processing of Extremely High-Resolution LiDAR and RGB Data: Outcome of the 2015 IEEE GRSS Data Fusion Contest–Part A: 2-D Contest, IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., 9, 5547, 10.1109/JSTARS.2016.2569162 · DOI
  36. Tschannen, M., Cavigelli, L., Mentzer, F., Wiatowski, T., and Benini, L. (arXiv, 2016). Deep Structured Features for Semantic Segmentation, arXiv. · DOI
  37. Piramanayagam, S., Schwartzkopf, W., Koehler, F.W., and Saber, E. (2016). Classification of remote sensed images using random forests and deep learning framework. SPIE Remote Sens. Int. Soc. Opt. Photonics. · DOI
  38. Marcu, A., and Leordeanu, M. (arXiv, 2016). Dual Local-Global Contextual Pathways for Recognition in Aerial Imagery, arXiv.
  39. Yuan, 2016, Dual-clustering-based hyperspectral band selection by contextual analysis, IEEE Trans. Geosci. Remote Sens., 54, 1431, 10.1109/TGRS.2015.2480866 · DOI
  40. Simonyan, K., and Zisserman, A. (arXiv, 2015). Very Deep Convolutional Networks for Large-Scale Image Recognition, arXiv.
  41. Hong, S., Noh, H., and Han, B. (arXiv, 2015). Decoupled Deep Neural Network for Semi-supervised Semantic Segmentation, arXiv.
  42. Audebert, N., Saux, B.L., and Lefèvre, S. (2017). Segment-before-Detect: Vehicle Detection and Classification through Semantic Segmentation of Aerial Images. Remote Sens., 9. · DOI · Paper page
  43. Huang, Z., Cheng, G., Wang, H., Li, H., Shi, L., and Pan, C. (2016, January 10–15). Building extraction from multi-source remote sensing images via deep deconvolution neural networks. Proceedings of the IEEE International Geoscience and Remote Sensing Symposium (IGARSS), Beijing, China. · DOI
  44. Audebert, N., Boulch, A., Lagrange, A., Le Saux, B., and Lefevre, S. (2016). Deep Learning for Remote Sensing, ONERA The French Aerospace Lab, DTIM & Univ. Bretagne-Sud & ENSTA ParisTech. Technical Report. · DOI
  45. Paisitkriangkrai, S., Sherrah, J., Janney, P., and Hengel, V.D. (2015, January 7–12). Effective semantic pixel labelling with convolutional networks and conditional random fields. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops, Boston, MA, USA. · DOI · Paper page
  46. Alam, F.I., Zhou, J., Liew, A.W.C., and Jia, X. (2016, January 10–15). CRF learning with CNN features for hyperspectral image segmentation. Proceedings of the IEEE International Geoscience and Remote Sensing Symposium, Beijing, China. · DOI
  47. He, X., Cai, D., and Niyogi, P. (2005, January 5–8). Laplacian Score for Feature Selection. Proceedings of the 18th International Conference on Neural Information Processing Systems, Vancouver, BC, Canada.
  48. Quan, R., Han, J., Zhang, D., and Nie, F. (2016, January 27–30). Object co-segmentation via graph optimized-flexible manifold ranking. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA. · DOI
  49. Wang, 2016, Salient band selection for hyperspectral image classification via manifold ranking, IEEE Trans. Neural Netw. Learn. Syst., 27, 1279, 10.1109/TNNLS.2015.2477537 · DOI · Paper page
  50. Yang, 2013, Graph-regularized saliency detection with convex-hull-based center prior, IEEE Signal Process. Lett., 20, 637, 10.1109/LSP.2013.2260737 · DOI
  51. Xu, B., Bu, J., Chen, C., Cai, D., He, X., Liu, W., and Luo, J. (2011, January 24–28). Efficient Manifold Ranking for Image Retrieval. Proceedings of the 34th international ACM SIGIR Conference on Research and Development in Information Retrieval, Beijing, China. · DOI
  52. Hsieh, C., Han, C., Shih, J., Lee, C., and Fan, K. (2015, January 24–26). 3D Model Retrieval Using Multiple Features and Manifold Ranking. Proceedings of the 2015 8th International Conference on Ubi-Media Computing (UMEDIA), Colombo, Sri Lanka. · doi:10.1109/umedia.2015.7297419
  53. Zhou, 2015, Robust visual tracking via efficient manifold ranking with low-dimensional compressive features, Pattern Recognit., 48, 2459, 10.1016/j.patcog.2015.03.008 · DOI
  54. Brostow, 2009, Semantic object classes in video: A high-definition ground truth database, Pattern Recognit. Lett., 30, 88, 10.1016/j.patrec.2008.04.005 · DOI · Paper page
  55. Everingham, 2010, The Pascal Visual Object Classes (VOC) Challenge, Int. J. Comput. Vis., 88, 303, 10.1007/s11263-009-0275-4 · DOI · Paper page
  56. Rottensteiner, 2012, The ISPRS benchmark on urban object classification and 3D building reconstruction, ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci., I-3, 293, 10.5194/isprsannals-I-3-293-2012 · DOI · Paper page
  57. Glorot, X., Bordes, A., and Bengio, Y. (2011, January 11–13). Deep Sparse Rectifier Neural Networks. Proceedings of the 14th International Conference on Artificial Intelligence and Statistics, Fort Lauderdale, FL, USA.
  58. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and Darrell, T. (2014, January 3–7). Caffe: Convolutional Architecture for Fast Feature Embedding. Proceedings of the 22nd ACM International Conference on Multimedia, Orlando, FL, USA. · DOI · Paper page
  59. Marmanis, 2016, Semantic Segmentation of Aerial Images with an Ensemble of CNSS, ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci., 3, 473, 10.5194/isprs-annals-III-3-473-2016 · DOI · Paper page
  60. Hariharan, B., Arbeláez, P., Bourdev, L., Maji, S., and Malik, J. (2011, January 6–13). Semantic Contours from Inverse Detectors. Proceedings of the 2011 IEEE International Conference on Computer Vision (ICCV), Barcelona, Spain. · DOI · Paper page
  61. Lin, T., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C., and Dollár, P. (arXiv, 2014). Microsoft coco: Common objects in context, arXiv. · DOI · Paper page
  62. Kohli, 2009, Robust higher order potentials for enforcing label consistency, Int. J. Comput. Vis., 82, 302, 10.1007/s11263-008-0202-0 · DOI · Paper page
  63. Quang, N.T., Thuy, N.T., Sang, D.V., and Binh, H.T.T. (2015, January 3–4). An efficient framework for pixel-wise building segmentation from aerial images. Proceedings of the Sixth International Symposium on Information and Communication Technology, Hue City, Vietnam. · DOI
  64. Sherrah, J. (arXiv, 2016). Fully convolutional networks for dense semantic labelling of high-resolution aerial imagery, arXiv.
  65. Immitzer, 2012, Tree species classification with random forest using very high spatial resolution 8-band WorldView-2 satellite data, Remote Sens., 4, 2661, 10.3390/rs4092661 · DOI · Paper page
  66. Dribault, 2012, Monitoring seasonal hydrological dynamics of minerotrophic peatlands using multi-date GeoEye-1 very high resolution imagery and object-based classification, Remote Sens., 4, 1887, 10.3390/rs4071887 · doi:10.3390/rs4071887
  67. Onojeghuo, 2011, Mapping reedbed habitats using texture-based classification of QuickBird imagery, Int. J. Remote Sens., 32, 8121, 10.1080/01431161.2010.532822 · doi:10.1080/01431161.2010.532822
  68. Junwei, 2016, Comparison between GF-1 and Landsat-8 images in land cover classification, Prog. Geogr., 35, 255
  69. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. Advances in Neural Information Processing Systems, The MIT Press.
  70. Mirza, M., and Osindero, S. (arXiv, 2014). Conditional generative adversarial nets, arXiv.
  71. Luc, P., Couprie, C., Chintala, S., and Verbeek, J. (arXiv, 2016). Semantic Segmentation using Adversarial Networks, arXiv.

Cited by

See all 57

Metadata

Authors
Zhang, M., Hu, X., Zhao, L., Lv, Y., Luo, M., & Pang, S.
Venue
Remote Sensing
Publisher
MDPI AG
Published
2017-05-19
Volume
9
Issue
5
Pages
500
ISSN
2072-4292 (ISSN-L)
OpenAlex
W2616727008
Cited by
57
References
20
FWCI
2.13
Type
journal article
Language
en
Open access
gold

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.