Voxel Transformer for 3D Object Detection
Open accessAbstract
We present Voxel Transformer (VoTr), a novel and effective voxel-based Transformer backbone for 3D object detection from point clouds. Conventional 3D convolutional backbones in voxel-based 3D detectors cannot efficiently capture large context information, which is crucial for object recognition and localization, owing to the limited receptive fields. In this paper, we resolve the problem by introducing a Transformer-based architecture that enables long-range relationships between voxels by self-attention. Given the fact that non-empty voxels are naturally sparse but numerous, directly applying standard Transformer on voxels is non-trivial. To this end, we propose the sparse voxel module and the submanifold voxel module, which can operate on the empty and non-empty voxel positions effectively. To further enlarge the attention range while maintaining comparable computational overhead to the convolutional counterparts, we propose two attention mechanisms for multi-head attention in those two modules: Local Attention and Dilated Attention, and we further propose Fast Voxel Query to accelerate the querying process in multi-head attention. VoTr contains a series of sparse and submanifold voxel modules, and can be applied in most voxel-based detectors. Our proposed VoTr shows consistent improvement over the convolutional baselines while maintaining computational efficiency on the KITTI dataset and the Waymo Open dataset.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Point-Painting projects LiDAR point clouds onto output of image semantic segmentation network and concatenates channel activation values with LiDAR intensity values, utilizing image semantic information.
“Point-Painting [80] projects LiDAR point clouds onto the output of the image semantic segmentation network and concatenates the channel activation values with the LiDAR point cloud intensity values, effectively utilizing image semantic information.”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [80]
The Related Work section (p. 2) discusses point- and voxel-based detectors, but does not describe Point-Painting. Searches for PointPainting and for LiDAR–image semantic fusion found no passage explaining the claimed projection or concatenation of values. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · see citing works
Vision meets robotics: The KITTI dataset
Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913491297 · Cited by 9,913 · see citing works
PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric SpacePreprint
Qi, C. R., Yi, L., Su, H., & Guibas, L. J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.02413 · Cited by 9,834 · see citing works
VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection
Zhou, Y., & Tuzel, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00472 · Cited by 4,813 · see citing works
PointPillars: Fast Encoders for Object Detection From Point Clouds
Lang, A. H., Vora, S., Caesar, H., Zhou, L., Yang, J., & Beijbom, O. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01298 · Cited by 4,673 · see citing works
Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P. H. S., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.15840 · Cited by 4,004 · see citing works
Self-citationMulti-view 3D Object Detection Network for Autonomous Driving
Chen, X., Ma, H., Wan, J., Li, B., & Xia, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.691 · Cited by 3,450 · see citing works
Scalability in Perception for Autonomous Driving: Waymo Open Dataset
Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaik, V., Tsui, P., Guo, J., Zhou, Y., Chai, Y., Caine, B., et al. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00252 · Cited by 3,437 · see citing works
SECOND: Sparsely Embedded Convolutional Detection
Yan, Y., Mao, Y., & Li, B. (2018) · Sensors · doi:10.3390/s18103337 · Cited by 3,352 · see citing works
PointRCNN: 3D Object Proposal Generation and Detection From Point Cloud
Shi, S., Wang, X., & Li, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00086 · Cited by 3,093 · see citing works
Frustum PointNets for 3D Object Detection from RGB-D Data
Qi, C. R., Liu, W., Wu, C., Su, H., & Guibas, L. J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00102 · Cited by 2,655 · see citing works
PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection
Shi, S., Guo, C., Jiang, L., Wang, Z., Shi, J., Wang, X., & Li, H. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01054 · Cited by 2,341 · see citing works
Joint 3D Proposal Generation and Object Detection from View Aggregation
Ku, J., Mozifian, M., Lee, J., Harakeh, A., & Waslander, S. L. (2018) · 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8594049 · Cited by 1,697 · see citing works
Big Bird: Transformers for Longer Sequences
Zaheer, M., Guruganesh, G., Dubey, A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2007.14062 · Cited by 1,349 · see citing works
3DSSD: Point-Based 3D Single Stage Object Detector
Yang, Z., Sun, Y., Liu, S., & Jia, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01105 · Cited by 1,176 · see citing works
Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud
Shi, W., & Rajkumar, R. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00178 · Cited by 979 · see citing works
Shi, S., Wang, Z., Shi, J., Wang, X., & Li, H. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2977026 · Cited by 951 · see citing works
STD: Sparse-to-Dense 3D Object Detector for Point Cloud
Yang, Z., Sun, Y., Liu, S., Shen, X., & Jia, J. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00204 · Cited by 875 · see citing works
Multi-Task Multi-Sensor Fusion for 3D Object Detection
Liang, M., Yang, B., Chen, Y., Hu, R., & Urtasun, R. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00752 · Cited by 723 · see citing works
Structure Aware Single-Stage 3D Object Detection From Point Cloud
He, C., Zeng, H., Huang, J., Hua, X.-S., & Lei Zhang, A. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01189 · Cited by 657 · see citing works
CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detection
Pang, S., Morris, D., & Radha, H. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9341791 · Cited by 581 · see citing works
Yoo, J. H., Kim, Y., Kim, J., & Choi, J. W. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58583-9_43 · Cited by 520 · see citing works
TANet: Robust 3D Object Detection from Point Clouds with Triple Attention
Liu, Z., Zhao, X., Huang, T., Hu, R., Zhou, Y., & Bai, X. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6837 · Cited by 395 · see citing works
LaserNet: An Efficient Probabilistic 3D Object Detector for Autonomous Driving
Meyer, G. P., Laddha, A., Kee, E., Vallespi-Gonzalez, C., & Wellington, C. K. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01296 · Cited by 374 · see citing works
Interpolated Convolutional Networks for 3D Point Cloud Understanding
Mao, J., Wang, X., & Li, H. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00166 · Cited by 245 · see citing works
Self-citationHVNet: Hybrid Voxel Network for LiDAR Based 3D Object Detection
Ye, M., Xu, S., & Cao, T. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00170 · Cited by 242 · see citing works
Pillar-Based Object Detection for Autonomous Driving
Wang, Y., Fathi, A., Kundu, A., Ross, D. A., Pantofaru, C., Funkhouser, T., & Solomon, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58542-6_2 · Cited by 237 · see citing works
One Million Scenes for Autonomous Driving: ONCE DatasetPreprint
Mao, J., Niu, M., Jiang, C., Liang, H., Chen, J., Liang, X., Li, Y., Ye, C., Zhang, W., Li, Z., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.11037 · Cited by 229 · see citing works
Self-citationStarNet: Targeted Computation for Object Detection in Point CloudsPreprint
Ngiam, J., Caine, B., Han, W., Yang, B., Chai, Y., Sun, P., Zhou, Y., Yi, X., Alsharif, O., Nguyen, P., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.11069 · Cited by 119 · see citing works
AFDet: Anchor Free One Stage 3D Object DetectionPreprint
Ge, R., Ding, Z., Hu, Y., Wang, Y., Chen, S., Huang, L., & Li, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.12671 · Cited by 98 · see citing works
Range Conditioned Dilated Convolutions for Scale Invariant 3D Object DetectionPreprint
Bewley, A., Sun, P., Mensink, T., Anguelov, D., & Sminchisescu, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.09927 · Cited by 80 · see citing works
Center-based 3D Object Detection and TrackingPreprint
Yin, T., Zhou, X., & Krähenbühl, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.11275 · Cited by 67 · see citing works
Patch Refinement -- Localized 3D Object DetectionPreprint
Lehner, J., Mitterecker, A., Adler, T., Hofmarcher, M., Nessler, B., & Hochreiter, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.04093 · Cited by 45 · see citing works
CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object DetectionPreprint
Pang, S., Morris, D., & Radha, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.00784 · Cited by 40 · see citing works
Point TransformerPreprint
Zhao, H., Jiang, L., Jia, J., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.09164 · Cited by 38 · see citing works
MaX-DeepLab: End-to-End Panoptic Segmentation with Mask TransformersPreprint
Wang, H., Zhu, Y., Adam, H., Yuille, A., & Chen, L.-C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.00759 · Cited by 32 · see citing works
Pillar-based Object Detection for Autonomous DrivingPreprint
Wang, Y., Fathi, A., Kundu, A., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.10323 · Cited by 29 · see citing works
Voxel R-CNN: Towards High Performance Voxel-based 3D Object DetectionPreprint
Deng, J., Shi, S., Li, P., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.15712 · Cited by 12 · see citing works
3D Object Detection with PointformerPreprint
Pan, X., Xia, Z., Song, S., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.11409 · Cited by 9 · see citing works
Flagged first, then by citations
Research works printed but not linked: 6
- [2][2] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-toend object detection with transformers. In European Conference on Computer Vision , pages 213-229. Springer, 2020. 2, 3 -
- [3][3] Qi Chen, Lin Sun, Ernest Cheung, and Alan L Yuille. Every view counts: Cross-view consistency in 3d object detection with hybrid-cylindrical-spherical voxelization. Advances in Neural Information Processing Systems , 33, 2020. 7 -
- [14][14] Ming Liang, Bin Yang, Shenlong Wang, and Raquel Urtasun. Deep continuous fusion for multi-sensor 3d object detection. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 641-656, 2018. 7 -
- [23][23] Charles R Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas J Guibas. Frustum pointnets for 3d object detection from rgbd data. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 918-927, 2018. 7 -
- [24][24] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 652-660, 2017. 1 -
- [43][43] Yin Zhou, Pei Sun, Yu Zhang, Dragomir Anguelov, Jiyang Gao, Tom Ouyang, James Guo, Jiquan Ngiam, and Vijay Vasudevan. End-to-end multi-view fusion for 3d object detection in lidar point clouds. In Conference on Robot Learning , pages 923-932. PMLR, 2020. 7 -
Cited by
Sorted by citations
TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers
Bai, X., Hu, Z., Zhu, X., Huang, Q., Chen, Y., Fu, H., & Tai, C.-L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00116 · Cited by 990 · see citing works
VoxelNeXt: Fully Sparse VoxelNet for 3D Object Detection and Tracking
Chen, Y., Liu, J., Zhang, X., Qi, X., & Jia, J. (2023) · arXiv · doi:10.48550/arxiv.2303.11301 · Cited by 486 · see citing works
AdaViT: Adaptive Vision Transformers for Efficient Image Recognition
Meng, L., Li, H., Chen, B.-C., Lan, S., Wu, Z., Jiang, Y.-G., & Lim, S.-N. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01199 · Cited by 262 · see citing works
Object Detection with Transformers: A Review
Shehzadi, T., Hashmi, K. A., Liwicki, M., Stricker, D., & Afzal, M. Z. (2025) · Sensors · doi:10.3390/s25196025 · Cited by 93 · see citing works
Emerging trends in autonomous vehicle perception: Multimodal fusion for 3D object detection
Alaba, S. Y., Gurbuz, A. C., & Ball, J. E. (2024) · World Electric Vehicle Journal · doi:10.3390/wevj15010020 · Cited by 82 · see citing works
Multimodal fusion and vision–language models: A survey for robot vision
Han, X., Chen, S., Fu, Z., Feng, Z., Fan, L., An, D., Wang, C., Guo, L., Meng, W., Zhang, X., et al. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103652 · Cited by 41 · see citing works
“Despite progress, challenges remain, such as the need for large-scale datasets, high computational costs, and limited generalization to real-world conditions.”
- Authors
- Mao, J., Xue, Y., Niu, M., Bai, H., Feng, J., Liang, X., Xu, H., & Xu, C.
- Affiliations
- Chinese University of Hong Kong (HK)
- National University of Singapore (SG)
- Huawei Technologies (China) (CN)
- Hong Kong University of Science and Technology (HK)
- Sun Yat-sen University (CN)
- Beijing University of Posts and Telecommunications (CN)
- Venue
- 2021 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2021-10
- Pages
- 3144-3153
- OpenAlex
- W3204896784
- Cited by
- 520
- References
- 40
- Citations by year
- 2026: 61 · 2025: 114 · 2024: 129 · 2023: 146 · 2022: 68 · 2021: 2
- FWCI
- 28.21
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-sa)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Computational Mechanics › 3D Shape Modeling and Analysis
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- 3D object detection, point clouds, transformer backbone, self-attention, local attention, long-range context modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
