Collaborative Perception in Autonomous Driving: Methods, Datasets, and Challenges
Open accessThe DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
Collaborative perception is essential to address occlusion and sensor failure issues in autonomous driving. In recent years, theoretical and experimental investigations of novel works for collaborative perception have increased tremendously. So far, however, few reviews have focused on systematical collaboration modules and large-scale collaborative perception datasets. This article reviews recent achievements in this field to bridge this gap and motivate future research. We start with a brief overview of collaboration schemes. After that, we systematically summarize the collaborative perception methods for ideal scenarios and real-world issues. The former focuses on collaboration modules and efficiency, and the latter is devoted to addressing the problems in actual application. Furthermore, we present large-scale public datasets and summarize quantitative results on these benchmarks. Finally, we highlight gaps and overlooked challenges between current academic research and real-world applications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Squeeze-and-Excitation Networks
Hu, J., Shen, L., & Sun, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00745 · Cited by 35,667
The pascal visual object classes (VOC) challenge
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010) · International Journal of Computer Vision · doi:10.1007/s11263-009-0275-4 · Cited by 19,599
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · arXiv · doi:10.48550/arxiv.2005.12872 · Cited by 18,253
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897
Object Detection with Discriminatively Trained Part-Based Models
Felzenszwalb, P. F., Girshick, R. B., McAllester, D., & Ramanan, D. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.167 · Cited by 9,853 · cited 2×
“The most common late fusion strategy is leveraging postprocessing methods such as NonMaximum Suppression (NMS) [83] to remove redundant and low-confidence prediction.”
MethodThe graph neural network model
Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M., & Monfardini, G. (2009) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2008.2005605 · Cited by 9,582 · cited 1×
“Graph Neural Networks (GNNs) have the ability to process graphstructured data, which can propagate and aggregate messages from neighbors [58].”
BackgroundEffective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382 · cited 1×
“Since these collaborative datasets do not provide unified evaluation metrics and experimental results for the motion prediction task, we only select three common perception tasks to evaluate, i.e., 3D object detection, 3D object tracking and BEV semantic segmentation.”
BackgroundnuScenes: A Multimodal Dataset for Autonomous Driving
Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., & Beijbom, O. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01164 · Cited by 7,152
PointPillars: Fast Encoders for Object Detection From Point Clouds
Lang, A. H., Vora, S., Caesar, H., Zhou, L., Yang, J., & Beijbom, O. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01298 · Cited by 4,673
Simple online and realtime tracking
Bewley, A., Ge, Z., Ott, L., Ramos, F., & Upcroft, B. (2016) · 2016 IEEE International Conference on Image Processing (ICIP) · doi:10.1109/icip.2016.7533003 · Cited by 4,433
CARLA: An open urban driving simulator
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., & Koltun, V. (2017) · arXiv · doi:10.48550/arxiv.1711.03938 · Cited by 3,718
Attention mechanisms in computer vision: A survey
Guo, M.-H., Xu, T.-X., Liu, J.-J., Liu, Z.-N., Jiang, P.-T., Mu, T.-J., Zhang, S.-H., Martin, R. R., Cheng, M.-M., & Hu, S.-M. (2022) · Computational Visual Media · doi:10.1007/s41095-022-0271-y · Cited by 2,566
PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection
Shi, S., Guo, C., Jiang, L., Wang, Z., Shi, J., Wang, X., & Li, H. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01054 · Cited by 2,341
Self-citationAutonomous vehicle perception: The technology of today and tomorrow
Van Brummelen, J., O’Brien, M., Gruyer, D., & Najjaran, H. (2018) · Transportation Research Part C: Emerging Technologies · doi:10.1016/j.trc.2018.02.012 · Cited by 869
Xu, R., Xiang, H., Xia, X., Han, X., Li, J., & Ma, J. (2022) · 2022 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra46639.2022.9812038 · Cited by 586
V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer
Xu, R., Xiang, H., Tu, Z., Xia, X., Yang, M.-H., & Ma, J. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19842-7_7 · Cited by 584
DAIR-V2X: A Large-Scale Dataset for Vehicle-Infrastructure Cooperative 3D Object Detection
Yu, H., Luo, Y., Shu, M., Huo, Y., Yang, Z., Shi, Y., Guo, Z., Li, H., Hu, X., Yuan, J., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.02067 · Cited by 577 · cited 16×
“TCLF predicts the current infrastructure prediction with the previous adjacent frame.”
BackgroundMethodInfluentialSelf-citationRukhovich, D., Vorontsova, A., & Konushin, A. (2022) · 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv51458.2022.00133 · Cited by 251
Yu, H., Yang, W., Ruan, H., Yang, Z., Tang, Y., Gao, X., Hao, X., Shi, Y., Pan, Y., Sun, N., et al. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00531 · Cited by 182
Learning for Vehicle-to-Vehicle Cooperative Perception Under Lossy Communication
Li, J., Xu, R., Liu, X., Ma, J., Chi, Z., Ma, J., & Yu, H. (2023) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3260040 · Cited by 161 · cited 5×
“To address this issue, Li et al. [33] propose an LC-aware Repair Network (LCRN) to ensure the robustness of collaborative perception under lossy communication.”
BackgroundMethodInfluentialLatency-Aware Collaborative Perception
Lei, Z., Ren, S., Hu, Y., Zhang, W., & Chen, S. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19824-3_19 · Cited by 147
Coopernaut: End-to-End Driving with Cooperative Perception for Networked Vehicles
Cui, J., Qiu, H., Chen, D., Stone, P., & Zhu, Y. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01674 · Cited by 138
Interruption-Aware Cooperative Perception for V2X Communication-Aided Autonomous Driving
Ren, S., Lei, Z., Wang, Z., Dianati, M., Wang, Y., Chen, S., & Zhang, W. (2024) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2024.3371974 · Cited by 75 · cited 4×
“To alleviate the effect, Ren et al. [51] leverage the historical information to recover missing features and propose an interruption-aware robust collaborative perception (V2X-INCOP) framework.”
BackgroundInfluential
Cited by
Mobility-Aware Sensing Data Orchestration for Communication-Efficient Cooperative Perception
Hou, P., Ding, Y., Li, M., & Cheng, Y. (2026) · 2026 International Conference on Computing, Networking and Communications (ICNC) · doi:10.1109/icnc68183.2026.11416959
Metadata
- Authors
- Han, Y., Zhang, H., Li, H., Jin, Y., Lang, C., & Li, Y.
- Venue
- IEEE Intelligent Transportation Systems Magazine
- Published
- 2023-11
- Volume
- 15
- Issue
- 6
- Pages
- 131-151
- ISSN
- 1939-1390 (ISSN-L)
- 1941-1197
- OpenAlex
- W4386634496
- Cited by
- 222
- References
- 26
- FWCI
- 16.13
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.