Cross Modal Transformer: Towards Fast and Robust 3D Object Detection
Open accessAbstract
In this paper, we propose a robust 3D detector, named Cross Modal Transformer (CMT), for end-to-end 3D multi-modal detection. Without explicit view transformation, CMT takes the image and point clouds tokens as inputs and directly outputs accurate 3D bounding boxes. The spatial alignment of multi-modal tokens is performed by encoding the 3D points into multi-modal features. The core design of CMT is quite simple while its performance is impressive. It achieves 74.1% NDS (state-of-the-art with single model) on nuScenes test set while maintaining faster inference speed. Moreover, CMT has a strong robustness even if the LiDAR is missing. Code is released at https://github.com/junjie18/CMT.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
CMT enhances object localization by encoding 3D coordinates and using Transformer decoder for multimodal interaction.
“CMT [87] enhances object localization by encoding 3D coordinates and using a Transformer decoder for multimodal interaction.”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [87]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Self-citationEnd-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285 · see citing works
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,989 · see citing works
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522 · see citing works
nuScenes: A Multimodal Dataset for Autonomous Driving
Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., & Beijbom, O. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01164 · Cited by 7,152 · see citing works
VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection
Zhou, Y., & Tuzel, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00472 · Cited by 4,813 · see citing works
PointPillars: Fast Encoders for Object Detection From Point Clouds
Lang, A. H., Vora, S., Caesar, H., Zhou, L., Yang, J., & Beijbom, O. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01298 · Cited by 4,673 · see citing works
Multi-view 3D Object Detection Network for Autonomous Driving
Chen, X., Ma, H., Wan, J., Li, B., & Xia, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.691 · Cited by 3,450 · see citing works
SECOND: Sparsely Embedded Convolutional Detection
Yan, Y., Mao, Y., & Li, B. (2018) · Sensors · doi:10.3390/s18103337 · Cited by 3,352 · see citing works
PointRCNN: 3D Object Proposal Generation and Detection From Point Cloud
Shi, S., Wang, X., & Li, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00086 · Cited by 3,093 · see citing works
Cyclical Learning Rates for Training Neural Networks
Smith, L. N. (2017) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2017.58 · Cited by 2,931 · see citing works
Frustum PointNets for 3D Object Detection from RGB-D Data
Qi, C. R., Liu, W., Wu, C., Su, H., & Guibas, L. J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00102 · Cited by 2,655 · see citing works
Center-based 3D Object Detection and Tracking
Yin, T., Zhou, X., & Krähenbühl, P. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01161 · Cited by 2,197 · see citing works
Deformable DETR: Deformable Transformers for End-to-End Object Detection
Zhu, X., Su, W., Lu, L., Li, B., Wang, X., & Dai, J. (2020) · arXiv · doi:10.48550/arxiv.2010.04159 · Cited by 1,837 · cited 6× · see citing works
“Other improvements in 2D detection mainly focus on modifying the transformer layers [53,57], designing informative object queries [19,25,54], or exploring the label assignment mechanism [5, 16].”
BackgroundInfluentialLi, Z., Wang, W., Li, H., Xie, E., Sima, C., Lu, T., Yu, Q., & Dai, J. (2022) · arXiv · doi:10.48550/arxiv.2203.17270 · Cited by 1,419 · see citing works
Bevfusion: Multi-task multi-sensor fusion with unified bird’s-eye view representation
Liu, Z., Tang, H., Amini, A., Yang, X., Mao, H., Rus, D. L., & Han, S. (2023) · 2023 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48891.2023.10160968 · Cited by 1,307 · see citing works
Lift, Splat, Shoot: Encoding Images from Arbitrary Camera Rigs by Implicitly Unprojecting to 3D
Philion, J., & Fidler, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58568-6_12 · Cited by 1,271 · see citing works
PointPainting: Sequential Fusion for 3D Object Detection
Vora, S., Lang, A. H., Helou, B., & Beijbom, O. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00466 · Cited by 1,236 · see citing works
3DSSD: Point-Based 3D Single Stage Object Detector
Yang, Z., Sun, Y., Liu, S., & Jia, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01105 · Cited by 1,176 · see citing works
DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection
Zhang, H., Li, F., Liu, S., Zhang, L., Su, H., Zhu, J., Ni, L. M., & Shum, H.-Y. (2022) · arXiv · Cited by 1,154 · cited 5× · see citing works
“Based on them, DINO[57] further improves the denoising anchor boxes via a contrastive way.”
BackgroundInfluentialDN-DETR: Accelerate DETR Training by Introducing Query DeNoising
Li, F., Zhang, H., Liu, S., Guo, J., Ni, L. M., & Zhang, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01325 · Cited by 1,102 · see citing works
TrackFormer: Multi-Object Tracking with Transformers
Meinhardt, T., Kirillov, A., Leal-Taixe, L., & Feichtenhofer, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00864 · Cited by 1,081 · see citing works
TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers
Bai, X., Hu, Z., Zhu, X., Huang, Q., Chen, Y., Fu, H., & Tai, C.-L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00116 · Cited by 990 · see citing works
BEVDepth: Acquisition of Reliable Depth for Multi-View 3D Object Detection
Li, Y., Ge, Z., Yu, G., Yang, J., Wang, Z., Shi, Y., Sun, J., & Li, Z. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i2.25233 · Cited by 784 · see citing works
Self-citationCenterMask: Real-Time Anchor-Free Instance Segmentation
Lee, Y., & Park, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01392 · Cited by 767 · see citing works
MOTR: End-to-End Multiple-Object Tracking with Transformer
Zeng, F., Dong, B., Zhang, Y., Wang, T., Zhang, X., & Wei, Y. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19812-0_38 · Cited by 752 · see citing works
Self-citationFCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection
Wang, T., Zhu, X., Pang, J., & Lin, D. (2021) · 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) · doi:10.1109/iccvw54120.2021.00107 · Cited by 683 · see citing works
Self-citationFCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection
Wang, T., Zhu, X., Pang, J., & Lin, D. (2021) · arXiv · doi:10.48550/arxiv.2104.10956 · Cited by 683 · see citing works
Self-citationPETR: Position Embedding Transformation for Multi-view 3D Object Detection
Liu, Y., Wang, T., Zhang, X., & Sun, J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19812-0_31 · Cited by 664 · see citing works
Self-citationDAB-DETR: Dynamic Anchor Boxes are Better Queries for DETRPreprint
Liu, S., Li, F., Zhang, H., Yang, X., Qi, X., Su, H., Zhu, J. J., & Zhang, L. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.12329 · Cited by 631 · see citing works
Anchor DETR: Query Design for Transformer-Based Detector
Wang, Y., Zhang, X., Yang, T., & Sun, J. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20158 · Cited by 511 · see citing works
Self-citationVoxelNeXt: Fully Sparse VoxelNet for 3D Object Detection and Tracking
Chen, Y., Liu, J., Zhang, X., Qi, X., & Jia, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02076 · Cited by 490 · see citing works
Self-citationVoxelNeXt: Fully Sparse VoxelNet for 3D Object Detection and Tracking
Chen, Y., Liu, J., Zhang, X., Qi, X., & Jia, J. (2023) · arXiv · doi:10.48550/arxiv.2303.11301 · Cited by 486 · see citing works
Self-citationPointAugmenting: Cross-Modal Augmentation for 3D Object Detection
Wang, C., Ma, C., Zhu, M. B., & Yang, X. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01162 · Cited by 481 · see citing works
PETRv2: A Unified Framework for 3D Perception from Multi-Camera Images
Liu, Y., Yan, J., Jia, F., Li, S., Gao, A., Wang, T., & Zhang, X. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00302 · Cited by 400 · see citing works
Self-citationFang, Y., Yang, S., Wang, X., Li, Y., Chen, F., Shan, Y., Feng, B., & Liu, W. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00683 · Cited by 371 · see citing works
FUTR3D: A Unified Sensor Fusion Framework for 3D Detection
Chen, X., Zhang, T., Wang, Y., Wang, Y., & Zhao, H. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00022 · Cited by 337 · see citing works
Focal Sparse Convolutional Networks for 3D Object Detection
Chen, Y., Li, Y., Zhang, X., Sun, J. Z., & Jia, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00535 · Cited by 332 · see citing works
Self-citationMegDet: A Large Mini-Batch Object Detector
Peng, C., Xiao, T., Li, Z., Jiang, Y., Zhang, X., Jia, K., Yu, G., & Sun, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00647 · Cited by 311 · see citing works
Self-citationBEVFusion: A Simple and Robust LiDAR-Camera Fusion FrameworkPreprint
Liang, T., Xie, H., Yu, K., Xia, Z., Lin, Z., Wang, Y., Tang, T., Wang, B., & Tang, Z. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.13790 · Cited by 310 · see citing works
Jia, D., Yuan, Y., He, H., Wu, X., Yu, H., Lin, W., Sun, L., Zhang, C., & Hu, H. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01887 · Cited by 298 · see citing works
Class-balanced Grouping and Sampling for Point Cloud 3D Object Detection
Zhu, B., Jiang, Z., Zhou, X., Li, Z., & Yu, G. (2019) · arXiv · doi:10.48550/arxiv.1908.09492 · Cited by 297 · see citing works
BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-ViewPreprint
Huang, J., Huang, G., Zhu, Z. D., Ye, Y., & Du, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.11790 · Cited by 292 · see citing works
RangeDet: In Defense of Range View for LiDAR-based 3D Object Detection
Fan, L., Xiong, X., Feng, W., Wang, N., & Zhang, Z. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00291 · Cited by 260 · see citing works
LiDAR R-CNN: An Efficient and Universal 3D Object Detector
Li, Z., Wang, F., & Wang, N. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00746 · Cited by 246 · see citing works
Pillar-Based Object Detection for Autonomous Driving
Wang, Y., Fathi, A., Kundu, A., Ross, D. A., Pantofaru, C., Funkhouser, T., & Solomon, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58542-6_2 · Cited by 237 · see citing works
Fusionpainting: Multimodal fusion with adaptive attention for 3d object detection
Xu, S., Zhou, D., Fang, J., Yin, J., Bin, Z., & Zhang, L. (2021) · 2021 IEEE International Intelligent Transportation Systems Conference (ITSC) · doi:10.1109/itsc48978.2021.9564951 · Cited by 237 · see citing works
RSN: Range Sparse Net for Efficient, Accurate LiDAR 3D Object Detection
Sun, P., Wang, W., Chai, Y., Elsayed, G. F., Bewley, A., Zhang, X., Sminchisescu, C., & Anguelov, D. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00567 · Cited by 167 · see citing works
Self-citationLargeKernel3D: Scaling up Kernels in 3D Sparse CNNs
Chen, Y., Liu, J., Zhang, X., Qi, X., & Jia, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01296 · Cited by 139 · see citing works
Self-citationAccelerating DETR Convergence via Semantic-Aligned Matching
Zhang, G., Luo, Z., Yu, Y., Cui, K., & Lu, S. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00102 · Cited by 132 · see citing works
Unifying Voxel-based Representation with Transformer for 3D Object Detection
Li, Y., Chen, Y., Qi, X., Li, Z., Sun, J., & Jia, J. (2022) · arXiv · doi:10.48550/arxiv.2206.00630 · Cited by 125 · see citing works
Self-citationDeepInteraction: 3D Object Detection via Modality Interaction
Yang, Z., Chen, J., Miao, Z., Li, W., Zhu, X., & Zhang, L. (2022) · arXiv · doi:10.48550/arxiv.2208.11112 · Cited by 108 · see citing works
SOLQ: Segmenting Objects by Learning Queries
Dong, B., Zeng, F., Wang, T., Zhang, X., & Wei, Y. (2021) · arXiv · doi:10.48550/arxiv.2106.02351 · Cited by 83 · see citing works
Self-citationFully Sparse Fusion for 3D Object Detection
Li, Y., Fan, L., Liu, Y., Huang, Z., Chen, Y., Wang, N., & Zhang, Z. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3392303 · Cited by 53 · see citing works
Self-citationDoll, S., Schulz, R., Schneider, L., Benzin, V., Enzweiler, M., & Lensch, H. P. A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19842-7_14 · Cited by 28 · see citing works
Group DETR v2: Strong Object Detector with Encoder-Decoder Pretraining
Chen, Q., Wang, J., Han, C., Zhang, S., Li, Z., Chen, X., Chen, J., Wang, X., Han, S., Zhang, G., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.03594 · Cited by 19 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Multimodal fusion and vision–language models: A survey for robot vision
Han, X., Chen, S., Fu, Z., Feng, Z., Fan, L., An, D., Wang, C., Guo, L., Meng, W., Zhang, X., et al. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103652 · Cited by 41 · see citing works
“Future research will likely focus on improving scalability, real-time capabilities, and developing unified frameworks that integrate localization with high-level perception and decision-making tasks [145].”
Self-citationMulti-Sem Fusion: Multimodal Semantic Fusion for 3-D Object Detection
Xu, S., Li, F., Song, Z., Fang, J., Wang, S., & Yang, Z.-X. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3387732 · Cited by 26 · see citing works
Transformer-based sensor fusion for autonomous vehicles: A comprehensive review
Abdulmaksoud, A., & Ahmed, R. (2025) · IEEE Access · doi:10.1109/access.2025.3545032 · Cited by 25 · see citing works
- Authors
- Yan, J., Liu, Y., Sun, J., Jia, F., Li, S., Wang, T., & Zhang, X.
- Affiliations
- Megvii (China) (CN)
- Venue
- 2023 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2023-10-01
- Pages
- 18222-18232
- OpenAlex
- W4390874213
- Semantic Scholar
- b23b6117ba6749763bff67c530af1c3b83fa2915
- Cited by
- 182
- References
- 56
- Citations by year
- 2026: 52 · 2025: 76 · 2024: 49 · 2023: 5
- FWCI
- 19.46
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- 3D object detection, multimodal detection, point clouds, image and point cloud fusion, cross-modal transformer, 3D bounding boxes, nuScenes, inference speed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
