The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks
Open accessAbstract
In many robotics and VR/AR applications, 3D-videos are readily-available input sources (a sequence of depth images, or LIDAR scans). However, in many cases, the 3D-videos are processed frame-by-frame either through 2D convnets or 3D perception algorithms. In this work, we propose 4-dimensional convolutional neural networks for spatio-temporal perception that can directly process such 3D-videos using high-dimensional convolutions. For this, we adopt sparse tensors and propose generalized sparse convolutions that encompass all discrete convolutions. To implement the generalized sparse convolution, we create an open-source auto-differentiation library for sparse tensors that provides extensive functions for high-dimensional convolutional neural networks. We create 4D spatio-temporal convolutional neural networks using the library and validate them on various 3D semantic segmentation benchmarks and proposed 4D datasets for 3D-video perception. To overcome challenges in 4D space, we propose the hybrid kernel, a special case of the generalized sparse convolution, and trilateral-stationary conditional random fields that enforce spatio-temporal consistency in the 7D space-time-chroma space. Experimentally, we show that a convolutional neural network with only generalized 3D sparse convolutions can outperform 2D or 2D-3D hybrid methods by a large margin. Also, we show that on 3D-videos, 4D spatio-temporal convolutional neural networks are robust to noise and outperform the 3D convolutional neural network.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Koller, D., & Friedman, N. (2009) · The MIT Press eBooks · Cited by 7,544
ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes
Dai, A., Chang, A. X., Savva, M., Halber, M., Funkhouser, T., & Niessner, M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.261 · Cited by 4,368
Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
Krähenbühl, P., & Koltun, V. (2012) · arXiv · doi:10.48550/arxiv.1210.5644 · Cited by 2,908
PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
Qi, C. R., Su, H., Mo, K., & Guibas, L. J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.00593 · Cited by 2,817
Conditional Random Fields as Recurrent Neural Networks
Zheng, S., Jayasumana, S., Romera-Paredes, B., Vineet, V., Su, Z., Du, D., Huang, C., & Torr, P. H. S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.179 · Cited by 2,386 · cited 2×
“We use the variational inference and approximate the distribtuion with the meanfield approximation [13] and convert the fixed point update similar to [36].”
BackgroundMethodRos, G., Sellart, L., Materzynska, J., Vazquez, D., & Lopez, A. M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.352 · Cited by 2,338
3D Semantic Parsing of Large-Scale Indoor Spaces
Armeni, I., Sener, O., Zamir, A. R., Jiang, H., Brilakis, I., Fischer, M., & Savarese, S. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.170 · Cited by 1,892
Self-citation3D Semantic Segmentation with Submanifold Sparse Convolutional Networks
Graham, B., Engelcke, M., & Maaten, L. V. D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00961 · Cited by 1,788
Large-Scale Point Cloud Semantic Segmentation with Superpoint Graphs
Landrieu, L., & Simonovsky, M. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00479 · Cited by 1,543
Cited by
Zhao, H., Jiang, L., Jia, J., Torr, P., & Koltun, V. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01595 · Cited by 2,476
Deep Learning for 3D Point Clouds: A Survey
Guo, Y., Wang, H., Hu, Q., Liu, H., Liu, L., & Bennamoun, M. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3005434 · Cited by 2,398
PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection
Shi, S., Guo, C., Jiang, L., Wang, Z., Shi, J., Wang, X., & Li, H. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01054 · Cited by 2,341
RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds
Hu, Q., Yang, B., Xie, L., Rosa, S., Guo, Y., Wang, Z., Trigoni, N., & Markham, A. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01112 · Cited by 2,275
Accurate medium-range global weather forecasting with 3D neural networks
Bi, K., Xie, L., Zhang, H., Chen, X., Gu, X., & Tian, Q. (2023) · Nature · doi:10.1038/s41586-023-06185-3 · Cited by 1,670
ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders
Woo, S., Debnath, S., Hu, R., Chen, X., Liu, Z., Kweon, I. S., & Xie, S. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01548 · Cited by 1,791
Fully Convolutional Geometric Features
Choy, C., Park, J., & Koltun, V. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00905 · Cited by 759
Self-citationSearching Efficient 3D Architectures with Sparse Point-Voxel Convolution
Tang, H., Liu, Z., Zhao, S., Lin, Y., Lin, J., Wang, H., & Han, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58604-1_41 · Cited by 733
PREDATOR: Registration of 3D Point Clouds with Low Overlap
Huang, S., Gojcic, Z., Usvyatsov, M., Wieser, A., & Schindler, K. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00425 · Cited by 721
PointASNL: Robust Point Clouds Processing Using Nonlocal Neural Networks With Adaptive Sampling
Yan, X., Zheng, C., Li, Z., Wang, S., & Cui, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00563 · Cited by 672
Point Transformer V3: Simpler, Faster, Stronger
Wu, X., Jiang, L., Wang, P.-S., Liu, Z., Liu, X., Qiao, Y., Ouyang, W., He, T., & Zhao, H. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.00463 · Cited by 648
PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding
Xie, S., Gu, J., Guo, D., Qi, C. R., Guibas, L., & Litany, O. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58580-8_34 · Cited by 609
Human Action Recognition From Various Data Modalities: A Review
Sun, Z., Ke, Q., Rahmani, H., Bennamoun, M., Wang, G., & Liu, J. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3183112 · Cited by 601
Stratified Transformer for 3D Point Cloud Segmentation
Lai, X., Liu, J., Jiang, L., Wang, L., Zhao, H., Liu, S., Qi, X., & Jia, J. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00831 · Cited by 566
Choy, C., Dong, W., & Koltun, V. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00259 · Cited by 548
Self-citationShi, S., Jiang, L., Deng, J., Wang, Z., Guo, C., Shi, J., Wang, X., & Li, H. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-022-01710-9 · Cited by 546
PAConv: Position Adaptive Convolution with Dynamic Kernel Assembling on Point Clouds
Xu, M., Ding, R., Zhao, H., & Qi, X. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00319 · Cited by 523
D3Feat: Joint Learning of Dense Detection and Description of 3D Local Features
Bai, X., Luo, Z., Zhou, L., Fu, H., Quan, L., & Tai, C.-L. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00639 · Cited by 521
Linking Points With Labels in 3D: A Review of Point Cloud Semantic Segmentation
Xie, Y., Tian, J., & Zhu, X. X. (2020) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2019.2937630 · Cited by 482
PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation
Jiang, L., Zhao, H., Shi, S., Liu, S., Fu, C.-W., & Jia, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00492 · Cited by 460
SpinNet: Learning a General Surface Descriptor for 3D Point Cloud Registration
Ao, S., Hu, Q., Yang, B., Markham, A., & Guo, Y. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01158 · Cited by 397
OpenScene: 3D Scene Understanding with Open Vocabularies
Peng, S., Genova, K., Jiang, C., Tagliasacchi, A., Pollefeys, M., & Funkhouser, T. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00085 · Cited by 395
PointDSC: Robust Point Cloud Registration using Deep Spatial Consistency
Bai, X., Luo, Z., Zhou, L., Chen, H., Li, L., Hu, Z., Fu, H., & Tai, C.-L. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01560 · Cited by 366
RPVNet: A Deep and Efficient Range-Point-Voxel Fusion Network for LiDAR Point Cloud Segmentation
Xu, J., Zhang, R., Dou, J., Zhu, Y., Sun, J., & Pu, S. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01572 · Cited by 336
Focal Sparse Convolutional Networks for 3D Object Detection
Chen, Y., Li, Y., Zhang, X., Sun, J., & Jia, J. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00535 · Cited by 325
ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes
Yeshwanth, C., Liu, Y.-C., Nießner, M., & Dai, A. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00008 · Cited by 316
Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction
Huang, Y., Zheng, W., Zhang, Y., Zhou, J., & Lu, J. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00890 · Cited by 312
Class-balanced Grouping and Sampling for Point Cloud 3D Object Detection
Zhu, B., Jiang, Z., Zhou, X., Li, Z., & Yu, G. (2019) · arXiv · doi:10.48550/arxiv.1908.09492 · Cited by 297
ImVoteNet: Boosting 3D Object Detection in Point Clouds With Image Votes
Qi, C. R., Chen, X., Litany, O., & Guibas, L. J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00446 · Cited by 294
Zhou, C., & Ling, Q. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3336053 · Cited by 284
Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contexts
Hou, J., Graham, B., Niesner, M., & Xie, S. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01533 · Cited by 283
Yan, X., Gao, J., Li, J., Zhang, R., Li, Z., Huang, R., & Cui, S. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i4.16419 · Cited by 282
SoftGroup for 3D Instance Segmentation on Point Clouds
Vu, T., Kim, K., Luu, T. M., Nguyen, T., & Yoo, C. D. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00273 · Cited by 279
Self-Supervised Pretraining of 3D Features on any Point-Cloud
Zhang, Z., Girdhar, R., Joulin, A., & Misra, I. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01009 · Cited by 278
Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusion
Qiu, S., Anwar, S., & Barnes, N. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00180 · Cited by 275
Atlas: End-to-End 3D Scene Reconstruction from Posed Images
Murez, Z., van As, T., Bartolozzi, J., Sinha, A., Badrinarayanan, V., & Rabinovich, A. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58571-6_25 · Cited by 266
Cheng, R., Razani, R., Taghavi, E., Li, E., & Liu, B. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01236 · Cited by 264
Mask3D: Mask Transformer for 3D Semantic Instance Segmentation
Schult, J., Engelmann, F., Hermans, A., Litany, O., Tang, S., & Leibe, B. (2023) · 2023 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48891.2023.10160590 · Cited by 261
DeepGMR: Learning Latent Gaussian Mixture Models for Registration
Yuan, W., Eckart, B., Kim, K., Jampani, V., Fox, D., & Kautz, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58558-7_43 · Cited by 259
MinkLoc3D: Point Cloud Based Large-Scale Place Recognition
Komorowski, J. (2021) · 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv48630.2021.00183 · Cited by 255
xMUDA: Cross-Modal Unsupervised Domain Adaptation for 3D Semantic Segmentation
Jaritz, M., Vu, T.-H., de Charette, R., Wirbel, E., & Perez, P. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01262 · Cited by 224
Perception-Aware Multi-Sensor Fusion for 3D LiDAR Semantic Segmentation
Zhuang, Z., Li, R., Jia, K., Wang, Q., Li, Y., & Tan, M. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01597 · Cited by 216
Interactive Gibson Benchmark: A Benchmark for Interactive Navigation in Cluttered Environments
Xia, F., Shen, W. B., Li, C., Kasimbeg, P., Tchapmi, M. E., Toshev, A., Martin-Martin, R., & Savarese, S. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.2965078 · Cited by 171
Self-citationLatency-Aware Collaborative Perception
Lei, Z., Ren, S., Hu, Y., Zhang, W., & Chen, S. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19824-3_19 · Cited by 147
PLA: Language-Driven Open-Vocabulary 3D Scene Understanding
Ding, R., Yang, J., Xue, C., Zhang, W., Bai, S., & Qi, X. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00677 · Cited by 131
Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation
Werby, A., Huang, C., Büchner, M., Valada, A., & Burgard, W. (2024) · Robotics: Science and Systems XX · doi:10.15607/rss.2024.xx.077 · Cited by 124
“The MinkowskiNet [50] is a privileged method that was trained on the full set of ScanNet [24] scenes to demonstrate the gap between zero-shot and fully-supervised methods.”
MethodCLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIP
Zhang, J., Dong, R., & Ma, K. (2023) · 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) · doi:10.1109/iccvw60793.2023.00219 · Cited by 62
ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations
Mu, T., Ling, Z., Xiang, F., Yang, D., Li, X., Tao, S., Huang, Z., Jia, Z., & Su, H. (2021) · arXiv · doi:10.48550/arxiv.2107.14483 · Cited by 34
Lowis3D: Language-Driven Open-World Instance-Level 3D Scene Understanding
Ding, R., Yang, J., Xue, C., Zhang, W., Bai, S., & Qi, X. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3410324 · Cited by 34
Cheng, D., Zhang, L., Qin, L., Wang, S., Wu, H., & Song, A. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3373019 · Cited by 31
VideoMAC: Video Masked Autoencoders Meet ConvNets
Pei, G., Chen, T., Jiang, X., Liu, H., Sun, Z., & Yao, Y. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02145 · Cited by 23
“ViT Backbone MAE-ST [16] 14 with the MinkowskiEngine sparse convolution library [9].”
Method3D CoMPaT: Composition of Materials on Parts of 3D Things
Li, Y., Upadhyay, U., Slim, H., Abdelreheem, A., Prajapati, A., Pothigara, S., Wonka, P., & Elhoseiny, M. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20074-8_7 · Cited by 13
Metadata
- Authors
- Choy, C., Gwak, J., & Savarese, S.
- Venue
- 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2019-06
- Pages
- 3070-3079
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2939201152
- Cited by
- 2,043
- References
- 27
- FWCI
- 49.76
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.