SparseBEV: High-Performance Sparse 3D Object Detection from Multi-Camera Videos
Open accessAbstract
Camera-based 3D object detection in BEV (Bird’s Eye View) space has drawn great attention over the past few years. Dense detectors typically follow a two-stage pipeline by first constructing a dense BEV feature and then performing object detection in BEV space, which suffers from complex view transformations and high computation cost. On the other side, sparse detectors follow a query-based paradigm without explicit dense BEV feature construction, but achieve worse performance than the dense counterparts. In this paper, we find that the key to mitigate this performance gap is the adaptability of the detector in both BEV and image space. To achieve this goal, we propose SparseBEV, a fully sparse 3D object detector that outperforms the dense counterparts. SparseBEV contains three key designs, which are (1) scale-adaptive self attention to aggregate features with adaptive receptive field in BEV space, (2) adaptive spatio-temporal sampling to generate sampling locations under the guidance of queries, and (3) adaptive mixing to decode the sampled features with dynamic weights from the queries. On the test split of nuScenes, SparseBEV achieves the state-of-the-art performance of 67.5 NDS. On the val split, SparseBEV achieves 55.8 NDS while maintaining a real-time inference speed of 23.5 FPS. Code is available at https://github.com/MCG-NJU/SparseBEV.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
On the R2R navigation benchmark, early sequence-based models Seq2Seq and SF had poor global path optimization and performed poorly in unseen environments.
“Early sequence-based models like Seq2Seq [265] and SF [266] struggled with global path optimization, leading to poor performance in unseen environments.”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [265]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534 · see citing works
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939 · see citing works
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.324 · Cited by 30,581 · see citing works
He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.322 · Cited by 29,137 · see citing works
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · see citing works
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285 · see citing works
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,759 · see citing works
The Hungarian method for the assignment problem
Kuhn, H. W. (1955) · Naval Research Logistics Quarterly · doi:10.1002/nav.3800020109 · Cited by 13,120 · see citing works
Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., & Xie, S. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01167 · Cited by 8,494 · see citing works
nuScenes: A Multimodal Dataset for Autonomous Driving
Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., & Beijbom, O. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01164 · Cited by 7,152 · cited 4× · see citing works
“By further adopting nuImages [1] pretraining and reducing the number queries to 400, SparseBEV sets a new record of 55.”
BackgroundMethodInfluentialFCOS: Fully Convolutional One-Stage Object Detection
Tian, Z., Shen, C., Chen, H., & He, T. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00972 · Cited by 7,013 · see citing works
International Conference on Learning Representations (ICLR 2013)
松尾, 豊. (2013) · 人工知能学会誌 = Journal of Japanese Society for Artificial Intelligence · Cited by 6,286 · see citing works
PointPillars: Fast Encoders for Object Detection From Point Clouds
Lang, A. H., Vora, S., Caesar, H., Zhou, L., Yang, J., & Beijbom, O. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01298 · Cited by 4,673 · see citing works
SlowFast Networks for Video Recognition
Feichtenhofer, C., Fan, H., Malik, J., & He, K. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00630 · Cited by 4,007 · see citing works
Center-based 3D Object Detection and Tracking
Yin, T., Zhou, X., & Krähenbühl, P. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01161 · Cited by 2,197 · see citing works
Sparse R-CNN: End-to-End Object Detection with Learnable Proposals
Sun, P., Zhang, R., Jiang, Y., Kong, T., Xu, C., Zhan, W., Tomizuka, M., Li, L., Yuan, Z., Wang, C., et al. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01422 · Cited by 1,617 · see citing works
Li, Z., Wang, W., Li, H., Xie, E., Sima, C., Lu, T., Yu, Q., & Dai, J. (2022) · arXiv · doi:10.48550/arxiv.2203.17270 · Cited by 1,419 · see citing works
Self-citationLift, Splat, Shoot: Encoding Images from Arbitrary Camera Rigs by Implicitly Unprojecting to 3D
Philion, J., & Fidler, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58568-6_12 · Cited by 1,271 · see citing works
Wang, Y., Chao, W.-L., Garg, D., Hariharan, B., Campbell, M., & Weinberger, K. Q. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00864 · Cited by 1,154 · see citing works
DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection
Zhang, H., Li, F., Liu, S., Zhang, L., Su, H., Zhu, J., Ni, L. M., & Shum, H.-Y. (2022) · arXiv · Cited by 1,154 · cited 2× · see citing works
“A lot of works [60, 45, 30, 37, 7, 21, 57] were then proposed to accelerate the convergence of DETR by using the sampled features instead of using the global ones.”
BackgroundMethodDN-DETR: Accelerate DETR Training by Introducing Query DeNoising
Li, F., Zhang, H., Liu, S., Guo, J., Ni, L. M., & Zhang, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01325 · Cited by 1,102 · see citing works
InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions
Wang, W., Dai, J., Chen, Z., Huang, Z., Li, Z., Zhu, X., Hu, X., Lu, T., Lu, L., Li, H., et al. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01385 · Cited by 1,080 · see citing works
Self-citationConditional DETR for Fast Training Convergence
Meng, D., Chen, X., Fan, Z., Zeng, G., Li, H., Yuan, Y., Sun, L., & Wang, J. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00363 · Cited by 1,055 · see citing works
BEVDepth: Acquisition of Reliable Depth for Multi-View 3D Object Detection
Li, Y., Ge, Z., Yu, G., Yang, J., Wang, Z., Shi, Y., Sun, J., & Li, Z. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i2.25233 · Cited by 784 · see citing works
CenterMask: Real-Time Anchor-Free Instance Segmentation
Lee, Y., & Park, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01392 · Cited by 767 · see citing works
FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection
Wang, T., Zhu, X., Pang, J., & Lin, D. (2021) · 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) · doi:10.1109/iccvw54120.2021.00107 · Cited by 683 · see citing works
FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection
Wang, T., Zhu, X., Pang, J., & Lin, D. (2021) · arXiv · doi:10.48550/arxiv.2104.10956 · Cited by 683 · see citing works
PETR: Position Embedding Transformation for Multi-view 3D Object Detection
Liu, Y., Wang, T., Zhang, X., & Sun, J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19812-0_31 · Cited by 664 · see citing works
DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETRPreprint
Liu, S., Li, F., Zhang, H., Yang, X., Qi, X., Su, H., Zhu, J. J., & Zhang, L. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.12329 · Cited by 631 · see citing works
Categorical Depth Distribution Network for Monocular 3D Object Detection
Reading, C., Harakeh, A., Chae, J., & Waslander, S. L. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00845 · Cited by 595 · see citing works
BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object DetectionPreprint
Huang, J., & Huang, G. (2022) · arXiv.org · doi:10.48550/arxiv.2203.17054 · Cited by 554 · see citing works
SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentation
Guo, M.-H., Lu, C.-Z., Hou, Q., et al. (2022) · arXiv · doi:10.48550/arxiv.2209.08575 · Cited by 488 · see citing works
PETRv2: A Unified Framework for 3D Perception from Multi-Camera Images
Liu, Y., Yan, J., Jia, F., Li, S., Gao, A., Wang, T., & Zhang, X. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00302 · Cited by 400 · see citing works
Yang, C., Chen, Y., Tian, H., Tao, C., Zhu, X., Zhang, Z., Huang, G. M., Li, H., Qiao, Y., Lu, L., et al. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01710 · Cited by 339 · see citing works
Is Pseudo-Lidar needed for Monocular 3D Object detection?
Park, D., Ambrus, R., Guizilini, V., Li, J., & Gaidon, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00313 · Cited by 334 · see citing works
Jia, D., Yuan, Y., He, H., Wu, X., Yu, H., Lin, W., Sun, L., Zhang, C., & Hu, H. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01887 · Cited by 298 · see citing works
Class-balanced Grouping and Sampling for Point Cloud 3D Object Detection
Zhu, B., Jiang, Z., Zhou, X., Li, Z., & Yu, G. (2019) · arXiv · doi:10.48550/arxiv.1908.09492 · Cited by 297 · see citing works
Cross-View Semantic Segmentation for Sensing Surroundings
Pan, B., Sun, J., Leung, H. Y. T., Andonian, A., & Zhou, B. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.3004325 · Cited by 294 · see citing works
BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-ViewPreprint
Huang, J., Huang, G., Zhu, Z. D., Ye, Y., & Du, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.11790 · Cited by 292 · see citing works
Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection
Wang, S., Liu, Y., Wang, T., Li, Y., & Zhang, X. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00335 · Cited by 292 · see citing works
Sparse4D: Multi-view 3D Object Detection with Sparse Spatial-Temporal FusionPreprint
Lin, X., Lin, T., Pei, Z.-H., et al. (2022) · arXiv.org · doi:10.48550/arxiv.2211.10581 · Cited by 222 · see citing works
BEVStereo: Enhancing Depth Estimation in Multi-View 3D Object Detection with Temporal Stereo
Li, Y., Bao, H., Ge, Z., Yang, J., Sun, J., & Li, Z. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i2.25234 · Cited by 221 · see citing works
PolarFormer: Multi-Camera 3D Object Detection with Polar Transformer
Jiang, Y., Li, Z., Miao, Z., Zhu, X., Gao, J., Hu, W., & Jiang, Y. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i1.25185 · Cited by 196 · see citing works
AdaMixer: A Fast-Converging Query-Based Object Detector
Gao, Z., Wang, L., Han, B., & Guo, S. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00529 · Cited by 134 · see citing works
Self-citationTime Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object DetectionPreprint
Park, J., Xu, C., Yang, S., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.02443 · Cited by 69 · see citing works
LinK: Linear Kernel for LiDAR-based 3D Perception
Lu, T., Ding, X., Liu, H., Wu, G., & Wang, L. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00113 · Cited by 59 · see citing works
Self-citationA Simple Baseline for Multi-Camera 3D Object Detection
Zhang, Y., Zheng, W., Zhu, Z. D., Huang, G., Lu, J., & Zhou, J. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i3.25460 · Cited by 27 · see citing works
BEVPoolv2: A Cutting-edge Implementation of BEVDet Toward DeploymentPreprint
Huang, J., & Huang, G. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.17111 · Cited by 25 · see citing works
Sparse4D v2: Recurrent Temporal Fusion with Sparse ModelPreprint
Lin, X., Lin, T., Pei, Z., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.14018 · Cited by 20 · see citing works
StageInteractor: Query-based Object Detector with Cross-stage Interaction
Teng, Y., Liu, H., Guo, S., & Wang, L. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00605 · Cited by 11 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
RCBEVDet: Radar-Camera Fusion in Bird's Eye View for 3D Object Detection
Lin, Z., Liu, Z., Xia, Z., Wang, X., Wang, Y., Qi, S., Dong, Y., Dong, N., Zhang, L., & Zhu, C. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01414 · Cited by 153 · see citing works
Multimodal fusion and vision–language models: A survey for robot vision
Han, X., Chen, S., Fu, Z., Feng, Z., Fan, L., An, D., Wang, C., Guo, L., Meng, W., Zhang, X., et al. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103652 · Cited by 41 · see citing works
“However, in the real world, the presence of low-quality data seriously affects the performance of multimodal learning, making it difficult for models to work stably in complex environments.”
- Authors
- Liu, H., Teng, Y., Lu, T., Wang, H., & Wang, L.
- Affiliations
- Nanjing University (CN)
- Shanghai Artificial Intelligence Laboratory (CN)
- Venue
- 2023 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2023-10-01
- Pages
- 18534-18544
- OpenAlex
- W4390873371
- Cited by
- 154
- References
- 51
- Citations by year
- 2026: 47 · 2025: 67 · 2024: 40
- FWCI
- 16.50
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
- Keywords
- bird's-eye view, multi-camera video, real-time inference, nuScenes
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
