On the Performance of One-Stage and Two-Stage Object Detectors in Autonomous Vehicles Using Camera Data
Open accessThe DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Abstract
Object detection using remote sensing data is a key task of the perception systems of self-driving vehicles. While many generic deep learning architectures have been proposed for this problem, there is little guidance on their suitability when using them in a particular scenario such as autonomous driving. In this work, we aim to assess the performance of existing 2D detection systems on a multi-class problem (vehicles, pedestrians, and cyclists) with images obtained from the on-board camera sensors of a car. We evaluate several one-stage (RetinaNet, FCOS, and YOLOv3) and two-stage (Faster R-CNN) deep learning meta-architectures under different image resolutions and feature extractors (ResNet, ResNeXt, Res2Net, DarkNet, and MobileNet). These models are trained using transfer learning and compared in terms of both precision and efficiency, with special attention to the real-time requirements of this context. For the experimental study, we use the Waymo Open Dataset, which is the largest existing benchmark. Despite the rising popularity of one-stage detectors, our findings show that two-stage detectors still provide the most robust performance. Faster R-CNN models outperform one-stage detectors in accuracy, being also more reliable in the detection of minority classes. Faster R-CNN Res2Net-101 achieves the best speed/accuracy tradeoff but needs lower resolution images to reach real-time speed. Furthermore, the anchor-free FCOS detector is a slightly faster alternative to RetinaNet, with similar precision and lower memory usage.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×
“The Deep Residual Networks (ResNet) [20] succeeded in addressing complex image problems achieving the best results in competitions such as ILSVRC [39] and COCO [13].”
BackgroundMethodVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 1×
“Faster R-CNN uses a convolutional backbone network, such as VGG [19] or ResNet [20], which outputs global feature maps.”
MethodLeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 2×
“ResNet, ResNeXt, and Res2Net The Deep Residual Networks (ResNet) [20] succeeded in addressing complex image problems achieving the best results in competitions such as ILSVRC [39] and COCO [13].”
BackgroundRich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
(2016) · Cited by 31,590
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939 · cited 1×
“For instance, feature pyramid networks (FPN) were proposed in order to crop RoI features from different levels depending on the scale [21].”
BackgroundMobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755
SSD: Single Shot MultiBox Detector
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., & Berg, A. C. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_2 · Cited by 22,128 · cited 3×
“Many DL detection frameworks were proposed, such as Faster R-CNN [9], Single Shot Detector (SSD) [10] which is also known as RetinaNet, You Only Look Once (YOLO) [11], or the Fully Convolutional One-Stage Detector (FCOS) [12].”
BackgroundMethodYOLO9000: Better, Faster, Stronger
Redmon, J., & Farhadi, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.690 · Cited by 19,892 · cited 4×
“Many DL detection frameworks were proposed, such as Faster R-CNN [9], Single Shot Detector (SSD) [10] which is also known as RetinaNet, You Only Look Once (YOLO) [11], or the Fully Convolutional One-Stage Detector (FCOS) [12].”
BackgroundMethodInfluentialMobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015 · cited 10×
“MobileNet The MobileNets [26] were introduced as an efficient model for vision applications to be used in mobile or embedded devices with constrained resources.”
BackgroundMethodInfluentialYOLOv3: An Incremental Improvement
Redmon, J., & Farhadi, A. (2018) · arXiv · doi:10.48550/arxiv.1804.02767 · Cited by 17,141 · cited 8×
“Furthermore, inspired by ResNet architectures, Darknet-53 included residual connections as shown in Figure 3d.”
BackgroundInfluentialAre we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897 · cited 1×
“For many years, KITTI was the reference benchmark in the field [31].”
BackgroundAggregated Residual Transformations for Deep Neural Networks
Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.634 · Cited by 12,465 · cited 16×
“ResNet, ResNeXt, and Res2Net The Deep Residual Networks (ResNet) [20] succeeded in addressing complex image problems achieving the best results in competitions such as ILSVRC [39] and COCO [13].”
BackgroundInfluentialFully convolutional networks for semantic segmentation
Shelhamer, E., Long, J., & Darrell, T. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2572683 · Cited by 11,898
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2858826 · Cited by 9,556
Cascade R-CNN: Delving Into High Quality Object Detection
Cai, Z., & Vasconcelos, N. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00644 · Cited by 7,370
nuScenes: A Multimodal Dataset for Autonomous Driving
Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., & Beijbom, O. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01164 · Cited by 7,152
FCOS: Fully Convolutional One-Stage Object Detection
Tian, Z., Shen, C., Chen, H., & He, T. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00972 · Cited by 7,013 · cited 2×
“Many DL detection frameworks were proposed, such as Faster R-CNN [9], Single Shot Detector (SSD) [10] which is also known as RetinaNet, You Only Look Once (YOLO) [11], or the Fully Convolutional One-Stage Detector (FCOS) [12].”
BackgroundMethodScalability in Perception for Autonomous Driving: Waymo Open Dataset
Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaik, V., Tsui, P., Guo, J., Zhou, Y., Chai, Y., Caine, B., et al. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00252 · Cited by 3,437 · cited 4×
“For the experiments, we use the Waymo Open Dataset [16], which is the most extensive and diverse benchmark up to date.”
MethodInfluentialRes2Net: A New Multi-Scale Backbone Architecture
Gao, S.-H., Cheng, M.-M., Zhao, K., Zhang, X.-Y., Yang, M.-H., & Torr, P. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2938758 · Cited by 3,385 · cited 2×
“Later, in order to enhance the representation of features at multiple scales and increase the range of receptive field, a novel architecture named Res2Net [23] was introduced.”
BackgroundSpeed/Accuracy Trade-Offs for Modern Convolutional Object Detectors
Huang, J., Rathod, V., Sun, C., Zhu, M., Korattikara, A., Fathi, A., Fischer, I., Wojna, Z., Song, Y., Guadarrama, S., et al. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.351 · Cited by 2,743 · cited 2×
“For instance, in [14], the authors also perform an evaluation with different images sizes.”
BackgroundAccurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., & He, K. (2017) · arXiv · doi:10.48550/arxiv.1706.02677 · Cited by 2,294 · cited 1×
“The initial learning rate is configured depending on the batch size according to the linear scaling rule proposed in [43].”
MethodMMDetection: Open MMLab Detection Toolbox and Benchmark
Chen, K., Wang, J., Pang, J., Cao, Y., Xiong, Y., Li, X., Sun, S., Feng, W., Liu, Z., Xu, J., et al. (2019) · arXiv · doi:10.48550/arxiv.1906.07155 · Cited by 2,227
Deep visual domain adaptation: A survey
Wang, M., & Deng, W. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.05.083 · Cited by 2,209
Wider or Deeper: Revisiting the ResNet Model for Visual Recognition
Wu, Z., Shen, C., & van den Hengel, A. (2019) · Pattern Recognition · doi:10.1016/j.patcog.2019.01.006 · Cited by 1,589
Feng, D., Haase-Schutz, C., Rosenbaum, L., Hertlein, H., Glaser, C., Timm, F., Wiesbeck, W., & Dietmayer, K. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.2972974 · Cited by 1,425 · cited 2×
“Faster R-CNN is the most representative model from the two-stage family, while RetinaNet and YOLOv3 are the most widely used anchor-based one-stage architectures [30].”
BackgroundMethodWang, Y., Chao, W.-L., Garg, D., Hariharan, B., Campbell, M., & Weinberger, K. Q. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00864 · Cited by 1,154 · cited 1×
“However, due to the high cost of the LiDAR hardware, camera-based systems are increasing their popularity [7].”
BackgroundBottom-Up Object Detection by Grouping Extreme and Center Points
Zhou, X., Zhuo, J., & Krahenbuhl, P. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00094 · Cited by 1,088 · cited 2×
“Other methods such as CornerNet [28] or ExtremeNet [29] first locate several keypoints (e.g., top-left corner and bottom-right corner) and then generate bounding boxes to detect objects.”
MethodCornerNet: Detecting Objects as Paired Keypoints
Law, H., & Deng, J. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01204-1 · Cited by 1,054 · cited 2×
“Other methods such as CornerNet [28] or ExtremeNet [29] first locate several keypoints (e.g., top-left corner and bottom-right corner) and then generate bounding boxes to detect objects.”
MethodAutonomous vehicle perception: The technology of today and tomorrow
Van Brummelen, J., O’Brien, M., Gruyer, D., & Najjaran, H. (2018) · Transportation Research Part C: Emerging Technologies · doi:10.1016/j.trc.2018.02.012 · Cited by 869 · cited 1×
“These developments have significantly improved the perception systems used in many applications such as autonomous driving [2,3], security surveillance [4], or land monitoring [5].”
BackgroundObject Detection in 20 Years: A SurveyPreprint
Zou, Z., Chen, K., Shi, Z., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.05055 · Cited by 557 · cited 1×
“AP is the most widely used performance metric in the object-detection literature [44].”
MethodOn Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., & Tang, P. T. P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.04836 · Cited by 525 · cited 1×
“The parameters of a model is an important factor because an excessive number in very deep architectures may lead to overfitting issues [45].”
BackgroundAggregated Residual Transformations for Deep Neural NetworksPreprint
Xie, S., Girshick, R., Dollár, P., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.05431 · Cited by 381 · cited 16×
“ResNet, ResNeXt, and Res2Net The Deep Residual Networks (ResNet) [20] succeeded in addressing complex image problems achieving the best results in competitions such as ILSVRC [39] and COCO [13].”
BackgroundInfluentialCarranza-García, M., García-Gutiérrez, J., & Riquelme, J. C. (2019) · Remote Sensing · doi:10.3390/rs11030274 · Cited by 246 · cited 1×
“These developments have significantly improved the perception systems used in many applications such as autonomous driving [2,3], security surveillance [4], or land monitoring [5].”
BackgroundSelf-citationEvaluation of deep neural networks for traffic sign detection systems
Arcos-García, Á., Álvarez-García, J. A., & Soria-Morillo, L. M. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.08.009 · Cited by 234 · cited 1×
“In [34], several state-of-the-art models for traffic sign detection were evaluated.”
BackgroundTowards Reaching Human Performance in Pedestrian Detection
Zhang, S., Benenson, R., Omran, M., Hosang, J., & Schiele, B. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2700460 · Cited by 191 · cited 1×
“Other studies provided insights on how to adapt convolutional networks to achieve better performance on pedestrian detection [35].”
BackgroundReal-time gun detection in CCTV: An open problem
González, J. L. S., Zaccaro, C., Álvarez-García, J. A., Morillo, L. M. S., & Caparrini, F. S. (2020) · Neural Networks · doi:10.1016/j.neunet.2020.09.013 · Cited by 155
Can autonomous vehicle reduce greenhouse gas emissions? A country-level evaluation
Liu, F., Zhao, F., Liu, Z., & Hao, H. (2019) · Energy Policy · doi:10.1016/j.enpol.2019.06.013 · Cited by 112 · cited 1×
“In recent years, autonomous vehicles are attracting increasing attention given their potential to improve road safety and traffic congestions, while also reducing harmful emissions [6].”
BackgroundEnsemble Methods for Object Detection
Casado-García, Á., & Jónathan, H. (2020) · Frontiers in artificial intelligence and applications · doi:10.3233/faia200407 · Cited by 64 · cited 1×
“Such ensemble approaches are increasingly receiving attention in the object detection literature since they have proved to enhance the accuracy [46].”
BackgroundAsynchronous dual-pipeline deep learning framework for online data stream classification
Lara-Benítez, P., Carranza-García, M., García–Gutiérrez, J., & Riquelme, J. C. (2020) · Integrated Computer-Aided Engineering · doi:10.3233/ica-200617 · Cited by 53 · cited 1×
“The data provided by high-resolution cameras and proximity sensors have helped to develop more powerful machine learning models that have achieved unprecedented results in visual recognition problems [1].”
BackgroundSelf-citationGeng, K., Ge, D., Yin, G., & Hu, J. (2020) · Remote Sensing · doi:10.3390/rs12203274 · Cited by 40
Anchor Generation Optimization and Region of Interest Assignment for Vehicle Detection
Wang, Y., Liu, Z., & Deng, W. (2019) · Sensors · doi:10.3390/s19051089 · Cited by 28 · cited 2×
“An anchor generation optimization procedure to improve vehicle detection was presented in [8].”
MethodThe Analytical Solution of the Clever Eye (CE) Method
Geng, X., Ji, L., & Yang, W. (2020) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2020.2991985 · Cited by 5
Cited by
Ali, M. L., & Zhang, Z. (2024) · Computers · doi:10.3390/computers13120336 · Cited by 383
Sharma, A., Kumar, V., & Longchamps, L. (2024) · Smart Agricultural Technology · doi:10.1016/j.atech.2024.100648 · Cited by 212
Comparing YOLOv8 and Mask R-CNN for instance segmentation in complex orchard environments
Sapkota, R., Ahmed, D., & Karkee, M. (2024) · Artificial Intelligence in Agriculture · doi:10.1016/j.aiia.2024.07.001 · Cited by 200
“…estimation (Lv et al., 2022) 2022 YOLO-V5 Identi fi cation method for the apple growth pattern in the orchard (Su et al., 2022) 2022 YOLO-V5 Tree trunk and obstacle detection in apple orchards (Jia et al., 2022) 2022 Mask R-CNN Ripe and green apple segmentation in orchards (Cong et al., 2022)…”
Ali, R., Kang, D., Suh, G., & Cha, Y.-J. (2021) · Automation in Construction · doi:10.1016/j.autcon.2021.103831 · Cited by 166
Deep learning for visual recognition and detection of aquatic animals: A review
Li, J., Xu, W., Deng, L., Xiao, Y., Han, Z., & Zheng, H. (2022) · Reviews in Aquaculture · doi:10.1111/raq.12726 · Cited by 150
Metadata
- Authors
- Carranza-García, M., Torres-Mateo, J., Lara-Benítez, P., & García-Gutiérrez, J.
- Affiliations
- Universidad de Sevilla (ES)
- Division of Computer Science, University of Sevilla, ES-41012 Seville, Spain
- Corresponding author
- Carranza-García, Manuel
- Venue
- Remote Sensing
- Publisher
- MDPI AG
- Published
- 2020-12-29
- Volume
- 13
- Issue
- 1
- Pages
- 89
- ISSN
- 2072-4292 (ISSN-L)
- OpenAlex
- W3113769242
- Cited by
- 270
- References
- 40
- Citations by year
- 2026: 34 · 2025: 62 · 2024: 75 · 2023: 42 · 2022: 40 · 2021: 17
- FWCI
- 9.63
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Remote Sensing (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Deposito de Investigacion Universidad de Sevilla (University of Seville) (submittedVersion, cc-by-nc-nd)
- Funders
- Junta de Andalucía (US-126334, P18-RT-2778)
- Ministerio de Economía y Competitividad (TIN2017-88209-C2-2-R)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- object detection, autonomous vehicles, one-stage detectors, two-stage detection, Faster R-CNN, RetinaNet, YOLOv3, FCOS, vehicle detection, pedestrian detection, cyclist detection, feature extractor, speed-accuracy trade-off, Waymo Open Dataset, real-time detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.