BAEFormer: Bi-Directional and Early Interaction Transformers for Bird's Eye View Semantic Segmentation
Abstract
Bird's Eye View (BEV) semantic segmentation is a critical task in autonomous driving. However, existing Transformer-based methods confront difficulties in transforming Perspective View (PV) to BEV due to their unidirectional and posterior interaction mechanisms. To address this issue, we propose a novel Bi-directional and Early Interaction Transformers framework named BAEFormer, consisting of (i) an early-interaction PV-BEV pipeline and (ii) a bi-directional cross-attention mechanism. Moreover, we find that the image feature maps' resolution in the cross-attention module has a limited effect on the final performance. Under this critical observation, we propose to enlarge the size of input images and downsample the multi-view image features for cross-interaction, further improving the accuracy while keeping the amount of computation controllable. Our proposed method for BEV semantic segmentation achieves state-of-the-art performance in real-time inference speed on the nuScenes dataset, i.e., 38.9 mIoU at 45 FPS on a single A100 GPU.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Transfer learning through pre-trained networks streamlines training by allowing efficient fine-tuning for specific applications, enhancing adaptability and performance across diverse tasks, particularly when acquiring extensive labeled data is impractical.
“Transfer learning, through the use of pre-trained networks, streamlines the training process by allowing models to be efficiently fine-tuned for specific applications, thereby enhancing adaptability and performance across diverse tasks, and proving particularly beneficial in scenarios where acquiring extensive labeled data is impractical or unfeasible [188], [189].”
From From Google Gemini to OpenAI Q* (Q-Star): A Survey on Reshaping the Generative Artificial Intelligence (AI) Research Landscape (2025) · cites this paper as [189]
Cites
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.324 · Cited by 30,581 · see citing works
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · see citing works
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285 · see citing works
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522 · see citing works
nuScenes: A Multimodal Dataset for Autonomous Driving
Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., & Beijbom, O. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01164 · Cited by 7,152 · see citing works
VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection
Zhou, Y., & Tuzel, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00472 · Cited by 4,813 · see citing works
PointPillars: Fast Encoders for Object Detection From Point Clouds
Lang, A. H., Vora, S., Caesar, H., Zhou, L., Yang, J., & Beijbom, O. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01298 · Cited by 4,673 · see citing works
Gaussian Error Linear Units (GELUs)
Hendrycks, D., & Gimpel, K. (2016) · arXiv · doi:10.48550/arxiv.1606.08415 · Cited by 3,537 · see citing works
Multi-view 3D Object Detection Network for Autonomous Driving
Chen, X., Ma, H., Wan, J., Li, B., & Xia, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.691 · Cited by 3,450 · see citing works
SECOND: Sparsely Embedded Convolutional Detection
Yan, Y., Mao, Y., & Li, B. (2018) · Sensors · doi:10.3390/s18103337 · Cited by 3,352 · see citing works
Li, Z., Wang, W., Li, H., Xie, E., Sima, C., Lu, T., Yu, Q., & Dai, J. (2022) · arXiv · doi:10.48550/arxiv.2203.17270 · Cited by 1,419 · see citing works
Lift, Splat, Shoot: Encoding Images from Arbitrary Camera Rigs by Implicitly Unprojecting to 3D
Philion, J., & Fidler, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58568-6_12 · Cited by 1,271 · see citing works
PointPainting: Sequential Fusion for 3D Object Detection
Vora, S., Lang, A. H., Helou, B., & Beijbom, O. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00466 · Cited by 1,236 · see citing works
Learning Deep Transformer Models for Machine Translation
Wang, Q., Li, B., Xiao, T., Zhu, J., Li, C., Wong, D. F., & Chao, L. S. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1176 · Cited by 673 · see citing works
PETR: Position Embedding Transformation for Multi-view 3D Object Detection
Liu, Y., Wang, T., Zhang, X., & Sun, J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19812-0_31 · Cited by 664 · see citing works
PETRv2: A Unified Framework for 3D Perception from Multi-Camera Images
Liu, Y., Yan, J., Jia, F., Li, S., Gao, A., Wang, T., & Zhang, X. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00302 · Cited by 400 · see citing works
Inverse perspective mapping simplifies optical flow computation and obstacle detection
Mallot, H. A., Bülthoff, H., Little, J. J., & Bohrer, S. (1991) · Biological Cybernetics · doi:10.1007/bf00201978 · Cited by 393 · see citing works
HDMapNet: An Online HD Map Construction and Evaluation Framework
Li, Q., Wang, Y., Wang, Y., & Zhao, H. (2022) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra46639.2022.9812383 · Cited by 386 · see citing works
Cross-view Transformers for real-time Map-view Semantic Segmentation
Zhou, B., & Krähenbühl, P. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01339 · Cited by 317 · see citing works
Cross-View Semantic Segmentation for Sensing Surroundings
Pan, B., Sun, J., Leung, H. Y. T., Andonian, A., & Zhou, B. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.3004325 · Cited by 294 · see citing works
FIERY: Future Instance Prediction in Bird’s-Eye View from Surround Monocular Cameras
Hu, A., Murez, Z., Mohan, N., Dudas, S., Hawke, J., Badrinarayanan, V., Cipolla, R., & Kendall, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01499 · Cited by 277 · see citing works
Predicting Semantic Map Representations From Images Using Pyramid Occupancy Networks
Roddick, T., & Cipolla, R. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01115 · Cited by 263 · see citing works
NEAT: Neural Attention Fields for End-to-End Autonomous Driving
Chitta, K., Prakash, A., & Geiger, A. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01550 · Cited by 229 · see citing works
Monocular Semantic Occupancy Grid Mapping With Convolutional Variational Encoder–Decoder Networks
Lu, C., van de Molengraft, R., & Dubbelman, G. (2019) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2019.2891028 · Cited by 169 · see citing works
BEVSegFormer: Bird's Eye View Semantic Segmentation From Arbitrary Camera Rigs
Peng, L., Chen, Z., Fu, Z., Liang, P., & Cheng, E. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00588 · Cited by 160 · see citing works
Structured Bird’s-Eye-View Traffic Scene Understanding from Onboard Images
Can, Y. B., Liniger, A., Paudel, D. P., & Gool, L. V. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01537 · Cited by 116 · see citing works
Reiher, L., Lampe, B., & Eckstein, L. (2020) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc45102.2020.9294462 · Cited by 108 · see citing works
Automatic dense visual semantic mapping from street-level imagery
Sengupta, S., Sturgess, P., Ladický, Ľ., & Torr, P. H. S. (2012) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2012.6385958 · Cited by 91 · see citing works
Yang, W., Li, Q., Liu, W., Yu, Y., Ma, Y., He, S., & Pan, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01528 · Cited by 89 · see citing works
A Geometric Approach to Obtain a Bird's Eye View From an Image
Abbas, S. A., & Zisserman, A. (2019) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2019.00504 · Cited by 71 · see citing works
Generative Adversarial Frontal View to Bird View Synthesis
Zhu, X., Yin, Z., Shi, J., Li, H., & Lin, D. (2018) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2018.00059 · Cited by 60 · see citing works
Enabling spatio-temporal aggregation in Birds-Eye-View Vehicle Estimation
Saha, A., Méndez, O., Russell, C., & Bowden, R. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9561169 · Cited by 51 · see citing works
FISHING Net: Future Inference of Semantic Heatmaps In GridsPreprint
Hendy, N., Sloan, C. S., Tian, F., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.09917 · Cited by 50 · see citing works
INFER: INtermediate representations for FuturE pRediction
Srikanth, S., Ansari, J. A., Karnik Ram, R., Sharma, S., Jatavallabhula, K. M., & Madhava Krishna, K. (2019) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros40897.2019.8968553 · Cited by 48 · see citing works
BEV-Seg: Bird's Eye View Semantic Segmentation Using Geometry and Semantic Point Cloud
Ng, M. H., Radia, K., Chen, J., Wang, D., Gog, I., & Gonzalez, J. E. (2020) · arXiv · doi:10.48550/arxiv.2006.11436 · Cited by 40 · see citing works
GitNet: Geometric Prior-Based Transformation for Birds-Eye-View Segmentation
Gong, S., Ye, X., Tan, X., Wang, J., Ding, E., Zhou, Y., & Bai, X. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19769-7_23 · Cited by 39 · see citing works
Domain Adaptation via Bidirectional Cross-Attention TransformerPreprint
Wang, X.-Y., Guo, P., & Zhang, Y. (2022) · arXiv.org · doi:10.48550/arxiv.2201.05887 · Cited by 31 · see citing works
A Geometric Approach to Obtain a Bird's Eye View from an ImagePreprint
Abbas, A. N., & Zisserman, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.02231 · Cited by 20 · see citing works
LaRa: Latents and Rays for Multi-Camera Bird's-Eye-View Semantic SegmentationPreprint
Bartoccioni, F., Zablocki, É., Bursuc, A., Pérez, P., Cord, M., & Alahari, K. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.13294 · Cited by 17 · see citing works
HFT: Lifting Perspective Representations via Hybrid Feature TransformationPreprint
Zou, J., Xiao, J., Zhu, Z. D., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.05068 · Cited by 10 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
McIntosh, T. R., Susnjak, T., Liu, T., Watters, P., Xu, D., Liu, D., & Halgamuge, M. N. (2025) · Technologies · doi:10.3390/technologies13020051 · Cited by 102 · see citing works
“These models facilitate accurate and data-efficient analysis by employing multi-view pipelines and cross-attention blocks [158], [159].”
Method
- Authors
- Pan, C., He, Y., Peng, J., Zhang, Q., Sui, W., & Zhang, Z.
- Affiliations
- Chinese Academy of Sciences (CN)
- Horizon Robotics (China) (CN)
- Huawei Technologies (China) (CN)
- Venue
- 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2023-06
- Pages
- 9590-9599
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4386066572
- Cited by
- 33
- References
- 40
- Citations by year
- 2026: 5 · 2025: 13 · 2024: 13 · 2023: 2
- FWCI
- 3.53
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- cross-attention, transformer models, autonomous driving perception, real-time inference, nuScenes
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
