Segmentation and semantic labelling of RGBD data with convolutional neural networks and surface fitting
Open accessAbstract
We present an approach for segmentation and semantic labelling of RGBD data exploiting together geometrical cues and deep learning techniques. An initial over‐segmentation is performed using spectral clustering and a set of non‐uniform rational B‐spline surfaces is fitted on the extracted segments. Then a convolutional neural network (CNN) receives in input colour and geometry data together with surface fitting parameters. The network is made of nine convolutional stages followed by a softmax classifier and produces a vector of descriptors for each sample. In the next step, an iterative merging algorithm recombines the output of the over‐segmentation into larger regions matching the various elements of the scene. The couples of adjacent segments with higher similarity according to the CNN features are candidate to be merged and the surface fitting accuracy is used to detect which couples of segments belong to the same surface. Finally, a set of labelled segments is obtained by combining the segmentation output with the descriptors from the CNN. Experimental results show how the proposed approach outperforms state‐of‐the‐art methods and provides an accurate segmentation and labelling.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Cites
Normalized cuts and image segmentation
Shi, J., & Malik, J. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.868688 · Cited by 15,842 · see citing works
Fully convolutional networks for semantic segmentation
Shelhamer, E., Long, J., & Darrell, T. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2572683 · Cited by 11,898 · see citing works
Efficient Graph-Based Image Segmentation
Felzenszwalb, P. F., & Huttenlocher, D. P. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000022288.19776.77 · Cited by 6,205 · see citing works
Selective Search for Object Recognition
Uijlings, J. R. R., van de Sande, K. E. A., Gevers, T., & Smeulders, A. W. M. (2013) · International Journal of Computer Vision · doi:10.1007/s11263-013-0620-5 · Cited by 5,980 · see citing works
Indoor segmentation and support inference from rgbd images
Silberman, N., Hoiem, D., Kohli, P., & Fergus, R. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33715-4_54 · Cited by 5,760 · see citing works
Contour Detection and Hierarchical Image Segmentation
Arbeláez, P., Maire, M., Fowlkes, C. C., & Malik, J. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.161 · Cited by 5,472 · see citing works
Piegl, L. A., & Tiller, W. (1997) · Monographs in visual communication · doi:10.1007/978-3-642-59223-2 · Cited by 2,869 · see citing works
Eigen, D., & Fergus, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.304 · Cited by 2,862 · see citing works
Learning Hierarchical Features for Scene Labeling
Farabet, C., Couprie, C., Najman, L., & LeCun, Y. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.231 · Cited by 2,620 · see citing works
Learning Rich Features from RGB-D Images for Object Detection and Segmentation
Gupta, S., Girshick, R., Arbeláez, P., & Malik, J. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10584-0_23 · Cited by 1,528 · see citing works
Spectral grouping using the nystrom method
Fowlkes, C. C., Belongie, S., Chung, F., & Malik, J. (2004) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2004.1262185 · Cited by 1,310 · see citing works
Perceptual Organization and Recognition of Indoor Scenes from RGB-D Images
Gupta, S., Arbeláez, P., & Malik, J. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.79 · Cited by 601 · see citing works
CPMC: Automatic Object Segmentation Using Constrained Parametric Min-Cuts
Carreira, J., & Sminchisescu, C. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.231 · Cited by 588 · see citing works
Indoor Semantic Segmentation using depth information
Couprie, C., Farabet, C., Najman, L. A., & LeCun, Y. (2013) · HAL · doi:10.48550/arxiv.1301.3572 · Cited by 451 · see citing works
RGB-(D) scene labeling: Features and algorithms
Ren, X., Bo, L., & Fox, D. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247999 · Cited by 410 · see citing works
Dense 3D semantic mapping of indoor scenes from RGB-D images
Hermans, A., Floros, G., & Leibe, B. (2014) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2014.6907236 · Cited by 258 · see citing works
Holistic Scene Understanding for 3D Object Detection with RGBD Cameras
Lin, D., Fidler, S., & Urtasun, R. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.179 · Cited by 253 · see citing works
Gupta, S., Arbeláez, P., Girshick, R., & Malik, J. (2014) · International Journal of Computer Vision · doi:10.1007/s11263-014-0777-6 · Cited by 248 · see citing works
Learning Common and Specific Features for RGB-D Semantic Segmentation with Deconvolutional Networks
Wang, J., Wang, Z., Tao, D., See, S. C. W., & Wang, G. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46454-1_40 · Cited by 169 · see citing works
Differential Geometry of Curves and Surfaces
Tapp, K. (2016) · Undergraduate texts in mathematics · doi:10.1007/978-3-319-39799-3 · Cited by 167 · see citing works
Holzer, S., Rusu, R. B., Dixon, M., et al. (2012) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2012.6385999 · Cited by 158 · see citing works
Time-of-Flight and Structured Light Depth Cameras
Zanuttigh, P., Marin, G., Mutto, C. D., et al. (2016) · Springer eBooks · doi:10.1007/978-3-319-30973-6 · Cited by 120 · see citing works
Self-citationSemantic Segmentation of RGBD Images with Mutex Constraints
Deng, Z., Todorović, S., & Latecki, L. J. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.202 · Cited by 80 · see citing works
Region-Based Semantic Segmentation with End-to-End Training
Caesar, H., Uijlings, J., & Ferrari, V. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46448-0_23 · Cited by 74 · see citing works
Geometry Driven Semantic Labeling of Indoor Scenes
Khan, S. H., Bennamoun, M., Sohel, F. A., & Togneri, R. B. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10590-1_44 · Cited by 59 · see citing works
Fast Semantic Segmentation of RGB-D Scenes with GPU-Accelerated Deep Neural Networks
Höft, N., Schulz, H., & Behnke, S. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-11206-0_9 · Cited by 58 · see citing works
Multi-modal Unsupervised Feature Learning for RGB-D Scene Labeling
Wang, A., Lu, J., Wang, G., Cai, J., & Cham, T.-J. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_30 · Cited by 50 · see citing works
Fusion of Geometry and Color Information for Scene Segmentation
Mutto, C. D., Zanuttigh, P., & Cortelazzo, G. M. (2012) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2012.2194474 · Cited by 49 · see citing works
Self-citationBanica, D., & Sminchisescu, C. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298974 · Cited by 45 · see citing works
Hasnat, A., Alata, O., & Trémeau, A. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2513407 · Cited by 39 · see citing works
Convolutional Nets and Watershed Cuts for Real-Time Semantic Labeling of RGBD Videos
Couprie, C., Najman, L. A., LeCun, Y., Couprie, C., Nets, C., & Paris-est, U. ́. (2014) · Cited by 30 · see citing works
Parsing Indoor Scenes Using RGB-D Imagery
Taylor, C. J., & Cowley, A. (2013) · The MIT Press eBooks · doi:10.7551/mitpress/9816.003.0056 · Cited by 20 · see citing works
Unsupervised RGB-D image segmentation using joint clustering and region merging
Hasnat, M. A., Alata, O., & Trémeau, A. (2014) · British Machine Vision Conference (BMVC) · doi:10.5244/c.28.17 · Cited by 14 · see citing works
Srinivasan, N., & Dellaert, F. (2014) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2014.7026091 · Cited by 7 · see citing works
Scene Segmentation Driven by Deep Learning and Surface Fitting
Minto, L., Pagnutti, G., & Zanuttigh, P. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-49409-8_12 · Cited by 6 · see citing works
Self-citationScene segmentation from depth and color data driven by surface fitting
Pagnutti, G., & Zanuttigh, P. (2014) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2014.7025894 · Cited by 5 · see citing works
Self-citationJoint Color and Depth Segmentation based on Region Merging and Surface Fitting
Pagnutti, G., & Zanuttigh, P. (2016) · doi:10.5220/0005672700930100 · Cited by 4 · see citing works
Self-citationSemantic Instance Labeling Leveraging Hierarchical Segmentation
Hickson, S., Essa, I., & Christensen, H. (2015) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2015.147 · Cited by 3 · see citing works
Unsupervised RGB-D image segmentation by multi-layer clustering
Khan, M. R., Rahman, A. B. M. M., Rahaman, G. M. A., et al. (2016) · International Conference on Informatics, Electronics and Vision · doi:10.1109/iciev.2016.7760095 · Cited by 3 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Rizzoli, G., Barbato, F., & Zanuttigh, P. (2022) · Technologies · doi:10.3390/technologies10040090 · Cited by 58 · see citing works
Self-citation
- Authors
- Pagnutti, G., Minto, L., & Zanuttigh, P.
- Affiliations
- University of Padua (IT)
- Corresponding author
- Zanuttigh, Pietro
- ORCID
- Zanuttigh, Pietro
- Venue
- IET Computer Vision
- Published
- 2017-09-20
- Volume
- 11
- Issue
- 8
- Pages
- 633-642
- ISSN
- 1751-9632 (ISSN-L)
- 1751-9640
- OpenAlex
- W2745846084
- Cited by
- 14
- References
- 39
- Citations by year
- 2023: 1 · 2022: 3 · 2020: 4 · 2019: 4 · 2018: 2
- FWCI
- 2.15
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- Locations
- IET Computer Vision (publishedVersion)
- Padua Research Archive (University of Padova) (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- Nvidia (K40 GPU)
- Topics
- Physical Sciences › Engineering › Computational Mechanics › 3D Shape Modeling and Analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
- Keywords
- semantic segmentation, semantic annotation, surface fitting, NURBS, spectral clustering, over-segmentation, convolutional neural networks, geometric cues, scene understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
