3D Semantic Segmentation with Submanifold Sparse Convolutional Networks

The submanifold sparse convolutional networks outperform all prior state-of-the-art methods on the test set of the ShapeNet part-segmentation competition.

“In particular, our models outperform all prior state-of-the-art on the test set of a recent semantic segmentation competition.”

• Also stated in: § 1 Introduction · plain

  “SSCNs outperform all of these entries.”

• Also stated in: § 2 Related Work · plain

  “Our SSCNs outperform all alternative approaches.”

The submanifold sparse convolutional network achieves an average IoU of 85.98% on the ShapeNet test set.

“Submanifold SparseConvNet (Section 6.5) 85.98%”

SSCNs with downsampling and upsampling, namely FCNs and U-Nets, outperformed SSCNs that operated at a single spatial resolution.

“We observe that SSCNs involving downsampling and upsampling operations (FCNs and U-Nets) outperform SSCNs operating on a single spatial resolution”

At 10^8 FLOPs, SSCNs achieved an average IoU 6-8 percentage points above the baselines.

“For instance, at 10 8 FLOPs, the average IoU of SSCNs is 6-8% higher than that of the baselines.”

• AssumptionThe authors state this as an assumption: At test time, the models are assumed to know the category of the object being segmented.

When depth values were set to zero, the model's FLOPs dropped by 60% because fewer voxels were active.

“a reduction of FLOPs by 60%, as there are fewer active voxels”

The example in Figure 2 shows that applying regular convolutions repeatedly substantially reduces feature sparsity at each layer.

“The example shows that regular convolutions substantially reduce the sparsity of the features with each convolutional layer.”

Setting depth values to zero reduced accuracy from 64.1% to 50.8%.

“a drop in accuracy from 64.1% to 50.8%”

For the voxelized ShapeNet data at scale S = 48, about 99% of voxels were empty.

“At scale S = 48, the voxels are approximately 99% sparse.”

On the NYU Depth dataset (v2), the SSCNs achieved up to 7% higher pixel accuracy than the 2D FCN.

“The results in the table show that SSCNs outperform 2D FCN in terms of pixel accuracy by up to 7%”

• cites [14]Not checked yet

  Not checked in the free run.

SSCN performance was similar across different values of S, particularly when the number of FLOPs was low.

“We observe that the performance of SSCNs is similar for different values of S, particularly for a low number of FLOPs.”

A shape-context segmentation classifier achieves 63.5% IoU on the aligned ShapeNet dataset.

“Aligned 63.5%”

The authors' SSCN models beat every baseline by a wide margin.

“SSCN networks outperform all baseline models by a substantial margin.”

• AssumptionThe authors state this as an assumption: At test time, the models are assumed to know the category of the object being segmented.

In the authors' experiments, removing the axis alignment of the point clouds made the segmentation task harder.

“The results in Table 3 show that removing the alignment, indeed, makes the segmentation task more challenging.”

The three baselines-shape contexts, multi-view 2D ConvNets, and dense 3D ConvNets-had roughly similar accuracy per FLOP.

“The results show that shape context features, multi-view 2D ConvNets, and dense 3D ConvNets perform roughly on par in terms of accuracy per FLOP.”

• AssumptionThe authors state this as an assumption: At test time, the models are assumed to know the category of the object being segmented.

At higher FLOP counts, SSCNs operating at a larger scale performed slightly better.

“At a higher number of FLOPs, the models operating at a larger scale perform slightly better.”

A shape-context segmentation classifier achieves 47.8% IoU on the randomly rotated ShapeNet dataset.

“Random pose 47.8%”

Applying full convolutions to sparse inputs causes rapid growth in the number of active sites, a problem called submanifold dilation.

“This rapid growth of the number of active sites is a poor prospect when implementing modern convolutional network architectures that comprise tens or even hundreds of convolutional layers, such as VGG networks, ResNets, or DenseNets [8,9,20].”

• Also stated in: § 3 Spatial sparsity for ConvNets · plain

  “We refer to this problem as the”

Submanifold sparse convolutions are well suited for point-wise semantic segmentation.

“and that it is well suited for the task of point-wise semantic segmentation.”

Restricting convolution output to active input points can prevent hidden layers from receiving all information needed for classification, because neighboring connected components are treated independently.

“A potential problem of this approach is that hidden layers in the network may not receive all the information they require to classify the input data: in particular, two neighboring connected components are treated completely independently.”

In contrast to OctNets, empty space causes no computational or memory overhead in the implementation of submanifold sparse convolutions.

“However, unlike OctNets, empty space imposes no computational or memory overhead in the implementation of submanifold sparse convolutions.”

In this implementation, the cost of computing an output site scales with the number of active inputs rather than with the receptive field size.

“The above implementation differs from [4] in that the cost of calculating an output site is proportional to the number of active inputs, rather than the size of the receptive field.”

• cites [4]Not checked yet

  Not checked in the free run.

Dense convolutional network implementations are highly inefficient for naturally sparse data such as 3D point clouds.

“Standard”

Earlier sparse convolutional network methods expand the set of active sites with each layer by using full convolutions.

“One of the downsides of prior sparse implementations of convolutional networks is that they”

• Also stated in: § 2 Related Work · plain

  “Prior work on sparse convolutions implements a convolutional operator that increases the number of active sites with each layer [3,4].”

• cites [3]Not checked yet

  Not checked in the free run.

• cites [4]Not checked yet

  Not checked in the free run.

For FCN and U-Net networks, if active sites decrease by a multiplicative factor at each downsampling, building all hash-tables and rule-books costs O(a) independent of network depth.

“For FCN and U-Net networks, assuming the number of active sites reduces by a multiplicative factor with each downsampling operation, the cost of building all the hash-tables and rule-books is also O(a) regardless of the depth of the network.”

• AssumptionThe authors state this as an assumption: In FCN and U-Net networks, the number of active sites decreases by a multiplicative factor at each downsampling operation.

Restricting the convolution output to the set of active input points solves the submanifold dilation problem.

“We explore a simple solution to the submanifold dilation problem that restricts the output of the convolution only to the set of active input points.”

Submanifold convolutions perform no computation in empty regions.

“Conversely, submanifold convolutions require no calculations in empty regions.”

Submanifold sparse convolutions resemble OctNets in preserving the sparsity structure.

“Submanifold sparse convolutions are similar to Oct-Nets [18] in that they preserve the sparsity structure.”

• cites [18]Not checked yet

  Not checked in the free run.

Fixing active site locations makes it feasible to train deep and efficient networks resembling VGG or ResNet architectures.

“We show that this makes it practical to train deep and efficient networks similar to VGG networks [20] or ResNets [7]”

• cites [20]Not checked yet

  Not checked in the free run.

• cites [7]Not checked yet

  Not checked in the free run.

Upsampling feature maps to original resolution instead of using deconvolutions with residual blocks substantially lowers the FCN's parameter count and multiplication-addition operations.

“Our implementation of FCNs upsamples feature maps to their original resolution rather than performing deconvolutions using residual blocks.This substantially reduces the number of parameters and the number of multiplication-addition operations of the FCN.”

The authors' results indicate that limiting information flow to submanifolds in the data did not reduce SSCN performance.

“Importantly, our results show that restricting information to travel along submanifolds in the data does not hamper the performance of SSCNs”

Removing alignment of ShapeNet objects by random 3D rotations makes the part-segmentation problem more challenging.

“The results show that removing the alignment of the ShapeNet objects via random 3D rotations makes the segmentation problem more challenging.”

Restricting information to submanifolds led to considerable savings in computation and memory.

“whilst it does lead to considerable computational and memory savings that can be exploited to train larger models with better accuracies.”

The authors conclude that SSCNs are computationally efficient relative to alternative approaches.

“Moreover, SSCNs are computationally efficient compared to alternative approaches.”

The accuracy drop when depth was removed demonstrates that the SSCNs used 3D structure for segmentation.

“which demonstrates that SSCNs do use 3D structure when performing segmentation.”

The reported ShapeNet results cannot be compared directly with those of Yi et al. [23] because the authors evaluated models in the harder random-pose setting.

“Note that these results are not directly comparable with those in [23] because we are testing the models in the more challenging “random-pose” setting.”

• cites [23]Not checked yet

  Not checked in the free run.