Complexer-YOLO: Real-Time 3D Object Detection and Tracking on Semantic Point Clouds
Open accessAbstract
Accurate detection of 3D objects is a fundamental problem in computer vision and has an enormous impact on autonomous cars, augmented/virtual reality and many applications in robotics. In this work we present a novel fusion of neural network based state-of-the-art 3D detector and visual semantic segmentation in the context of autonomous driving. Additionally, we introduce Scale-Rotation-Translation score (SRTs), a fast and highly parameterizable evaluation metric for comparison of object detections, which speeds up our inference time up to 20% and halves training time. On top, we apply state-of-the-art online multi target feature tracking on the object measurements to further increase accuracy and robustness utilizing temporal information. Our experiments on KITTI show that we achieve same results as state-of-the-art in all related categories, while maintaining the performance and accuracy trade-off and still run in real-time. Furthermore, our model is the first one that fuses visual semantic with 3D object detection.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Experimental results show that using depth information, optical flow information, and LiDAR point clouds in addition to conventional RGB data can improve the performance of one-stage based detection systems.
“According to experimental results, it has been shown that the use of depth information [183, 184], optical flow information [185], and LiDAR point clouds [186] in addition to conventional RGB data can improve the performance of one-stage based detection systems.”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [186]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · see citing works
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803 · see citing works
(2016) · Cited by 31,590 · see citing works
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.324 · Cited by 30,581 · see citing works
He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.322 · Cited by 29,137 · see citing works
YOLOv3: An Incremental Improvement
Redmon, J., & Farhadi, A. (2018) · arXiv · doi:10.48550/arxiv.1804.02767 · Cited by 17,141 · see citing works
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897 · see citing works
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433 · see citing works
PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
Charles, R., Su, H., Mo, K., & Guibas, L. J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.16 · Cited by 9,767 · see citing works
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593 · see citing works
Multi-scale context aggregation by dilated convolutions
Yu, F., & Koltun, V. (2015) · arXiv · doi:10.48550/arxiv.1511.07122 · Cited by 7,738 · see citing works
SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
Iandola, F. N., Han, S., Moskewicz, M. W., Ashraf, K., Dally, W. J., & Keutzer, K. (2016) · arXiv · doi:10.48550/arxiv.1602.07360 · Cited by 6,201 · see citing works
VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection
Zhou, Y., & Tuzel, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00472 · Cited by 4,813 · see citing works
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1512.03385 · Cited by 4,196 · see citing works
Multi-view 3D Object Detection Network for Autonomous Driving
Chen, X., Ma, H., Wan, J., Li, B., & Xia, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.691 · Cited by 3,450 · see citing works
SECOND: Sparsely Embedded Convolutional Detection
Yan, Y., Mao, Y., & Li, B. (2018) · Sensors · doi:10.3390/s18103337 · Cited by 3,352 · see citing works
RefineNet: Multi-path Refinement Networks for High-Resolution Semantic Segmentation
Lin, G., Milan, A., Shen, C., & Reid, I. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.549 · Cited by 3,287 · see citing works
Frustum PointNets for 3D Object Detection from RGB-D Data
Qi, C. R., Liu, W., Wu, C., Su, H., & Guibas, L. J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00102 · Cited by 2,655 · see citing works
Channel Pruning for Accelerating Very Deep Neural Networks
He, Y., Zhang, X., & Sun, J. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.155 · Cited by 2,647 · see citing works
ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation
Paszke, A., Chaurasia, A., Kim, S., & Culurciello, E. (2016) · arXiv · doi:10.48550/arxiv.1606.02147 · Cited by 1,826 · see citing works
ERFNet: Efficient Residual Factorized ConvNet for Real-Time Semantic Segmentation
Romera, E., Álvarez, J. M., Bergasa, L. M., & Arroyo, R. (2017) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2017.2750080 · Cited by 1,491 · see citing works
PIXOR: Real-time 3D Object Detection from Point Clouds
Yang, B., Luo, W., & Urtasun, R. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00798 · Cited by 1,249 · see citing works
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollár, P. (2017) · arXiv · doi:10.48550/arxiv.1708.02002 · Cited by 1,227 · see citing works
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · arXiv · doi:10.48550/arxiv.1604.01685 · Cited by 1,174 · see citing works
ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation
Mehta, S., Rastegari, M., Caspi, A., Shapiro, L., & Hajishirzi, H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01249-6_34 · Cited by 1,029 · see citing works
The Labeled Multi-Bernoulli Filter
Reuter, S., Vo, B.-T., Vo, B., et al. (2014) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2014.2323064 · Cited by 827 · see citing works
Feature Pyramid Networks for Object DetectionPreprint
Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.03144 · Cited by 645 · see citing works
Luo, W., Yang, B., & Urtasun, R. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00376 · Cited by 644 · see citing works
Vehicle Detection from 3D Lidar Using Fully Convolutional Network
Li, B., Zhang, T., & Xia, T. (2016) · Robotics: Science and Systems XII · doi:10.15607/rss.2016.xii.042 · Cited by 577 · see citing works
Vote3Deep: Fast object detection in 3D point clouds using efficient convolutional neural networks
Engelcke, M., Rao, D., Wang, D. Z., Tong, C. H., & Posner, I. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989161 · Cited by 549 · see citing works
Channel Pruning for Accelerating Very Deep Neural NetworksPreprint
He, Y., Zhang, X., & Sun, J. Z. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.06168 · Cited by 346 · see citing works
VoxelNet: End-to-End Learning for Point Cloud Based 3D Object DetectionPreprint
Zhou, Y., & Tuzel, O. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.06396 · Cited by 318 · see citing works
Vehicle Detection from 3D Lidar Using Fully Convolutional NetworkPreprint
Li, B., Zhang, T., & Xia, T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.07916 · Cited by 275 · see citing works
Beyond Pixels: Leveraging Geometry and Shape Cues for Online Multi-Object Tracking
Sharma, S., Ansari, J. A., Jatavallabhula, K. M., & Krishna, K. M. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8461018 · Cited by 169 · see citing works
Learning Spatiotemporal Features with 3D Convolutional NetworksPreprint
Tran, D., Bourdev, L., Fergus, R., et al. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.0767 · Cited by 155 · see citing works
Joint 3D Proposal Generation and Object Detection from View AggregationPreprint
Ku, J., Mozifian, M., Lee, J., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.02294 · Cited by 134 · see citing works
FollowMe: Efficient Online Min-Cost Flow Tracking with Bounded Memory and Computation
Lenz, P., Geiger, A., & Urtasun, R. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.496 · Cited by 105 · see citing works
A Generalized Labeled Multi-Bernoulli Filter With Object Spawning
Bryant, D. S., Vo, B.-T., Vo, B., et al. (2018) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2018.2872856 · Cited by 70 · see citing works
Multi-View 3D Object Detection Network for Autonomous DrivingPreprint
Chen, X., Ma, H., Wan, J., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.07759 · Cited by 65 · see citing works
A random finite set conjugate prior and application to multi-target tracking
Vo, B.-T., & Vo, B. (2011) · International Conference on Intelligent Sensors, Sensor Networks and Information Processing · doi:10.1109/issnip.2011.6146549 · Cited by 57 · see citing works
EKF/UKF Maneuvering Target Tracking using Coordinated Turn Models with Polar/Cartesian Velocity
Roth, M., Hendeby, G., & Gustafsson, F. (2014) · KTH Publication Database DiVA (KTH Royal Institute of Technology) · Cited by 55 · see citing works
PointFusion: Deep Sensor Fusion for 3D Bounding Box EstimationPreprint
Xu, D., Anguelov, D., & Jain, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.10871 · Cited by 55 · see citing works
3D Object Proposals using Stereo Imagery for Accurate Object Class DetectionPreprint
Chen, X., Kundu, K., Zhu, Y., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.07711 · Cited by 49 · see citing works
Multi-Dimensional Recurrent Neural NetworksPreprint
Graves, A., Fernandez, S., & Schmidhuber, J. (2007) · arXiv (Cornell University) · doi:10.48550/arxiv.0705.2011 · Cited by 48 · see citing works
3D Fully Convolutional Network for Vehicle Detection in Point CloudPreprint
Li, B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.08069 · Cited by 23 · see citing works
Efficient Semantic Segmentation for Visual Bird’s-Eye View Interpretation
Sämann, T., Amende, K., Milz, S., Witt, C., Simón, M., & Petzold, J. (2018) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-01370-7_53 · Cited by 16 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- K. He, G. Gkioxari, P. Dollar, and R. Girshick. Mask R- CNN. Proc. IEEE Int. Conf. Comput. Vis., 2017. 2
Cited by
Sorted by citations
Bayoudh, K., Knani, R., Hamdaoui, F., & Mtibaa, A. (2022) · The Visual Computer · doi:10.1007/s00371-021-02166-7 · Cited by 456 · see citing works
Kumar, S., Singh, S. K., Varshney, S., Singh, S., Kumar, P., Kim, B.-G., & Ra, I.-H. (2023) · Sustainability · doi:10.3390/su152416869 · Cited by 46 · see citing works
- Authors
- Simon, M., Amende, K., Kraus, A., Honer, J., Samann, T., Kaulbersch, H., Milz, S., & Gross, H. M.
- Affiliations
- Valeo (Germany) (DE)
- Technische Universität Ilmenau (DE)
- Venue
- 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
- Publisher
- IEEE
- Published
- 2019-06
- Pages
- 1190-1199
- OpenAlex
- W2940208651
- Cited by
- 304
- References
- 46
- Citations by year
- 2026: 15 · 2025: 28 · 2024: 54 · 2023: 56 · 2022: 54 · 2021: 52 · 2020: 32 · 2019: 12 · 2018: 1
- FWCI
- 12.23
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Physical Sciences › Engineering › Computational Mechanics › 3D Shape Modeling and Analysis
- Keywords
- 3D object detection, semantic point clouds, YOLO, autonomous driving, multi-target tracking, evaluation metrics, real-time detection, KITTI dataset
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
