Real-Time Object Detection Network in UAV-Vision Based on CNN and Transformer
The DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
Unmanned aerial vehicles (UAVs) play an important role in conducting automatic patrol inspections of cities, which can ensure the safety of urban residents’ life and property and the normal operation of cities. However, during the inspection process, problems may arise. For example, numerous small objects in UAV images are difficult to detect, objects in UAV images are severely occluded, and requirements for real-time performances are posed. To address these issues, we first propose a real-time object detection network (RTD-Net) for UAV images. Besides, to deal with the lack of visual features of small objects, we design a feature fusion module (FFM) to interact and fuse features at different levels and improve the feature expression ability of small objects. To achieve real-time detection, we design a lightweight feature extraction module (LEM) to build the backbone network to control the calculation quantity and parameters. To solve the issue of discontinuous features of occluded objects, an efficient convolutional transformer block (ECTB)-based convolutional multihead self-attention (CMHSA) is designed to improve the recognition ability of occluded objects by extracting the context information of objects. Compared with multihead self-attention (MHSA) in the traditional transformer, CMHSA uses convolutional projection to replace the position-linear projection, which can reduce a large amount of calculation without performance loss. Finally, an attention prediction head (APH) is designed based on the attention mechanism to improve the ability of the model to extract attention regions in complex scenarios. The proposed method reaches a detection accuracy of 86.4% mean average precision (mAP) in our UAV image dataset. In addition, it achieves a detection accuracy of 86.0% mAP and a detection speed of 33.4 frames/s in the NVIDIA Jeston TX2 embedded device.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.324 · Cited by 30,581 · cited 1×
“The one-stage detection schemes represented by SSD [14], YOLO [15], [16], [17], and RetinaNet [18] directly predict the exact location and category of objects using a forward network.”
MethodYOLO9000: Better, Faster, Stronger
Redmon, J., & Farhadi, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.690 · Cited by 19,892
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · arXiv · doi:10.48550/arxiv.2005.12872 · Cited by 18,253
YOLOv3: An Incremental Improvement
Redmon, J., & Farhadi, A. (2018) · arXiv · doi:10.48550/arxiv.1804.02767 · Cited by 17,141
Path Aggregation Network for Instance Segmentation
Liu, S., Qi, L., Qin, H., Shi, J., & Jia, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00913 · Cited by 10,011
EfficientDet: Scalable and Efficient Object Detection
Tan, M., Pang, R., & Le, Q. V. (2019) · arXiv · Cited by 9,841 · cited 1×
“In this section, we design a feature fusion module (FFM) based on the idea of bidirectional feature pyramid network (BiFPN) [45], as shown in Fig.”
MethodDeformable DETR: Deformable Transformers for End-to-End Object Detection
Zhu, X., Su, W., Lu, L., Li, B., Wang, X., & Dai, J. (2020) · arXiv · doi:10.48550/arxiv.2010.04159 · Cited by 6,027 · cited 1×
“Based on this paradigm, a series of transformer-based object detection methods [28], [29], [30], [31], [32] have emerged successively, and the detection effect of the transformer method in the field of object detection has been continuously improved.”
BackgroundPyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., & Shao, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00061 · Cited by 5,259 · cited 2×
“…the input image directly, and it is difficult to extract the detailed information in the images and 2) the computation burden and memory of MHSA in transformer grow quadratically with the growth of the spatial dimension, resulting in significant training and inference overhead [26], [34], [37].”
BackgroundMMDetection: Open MMLab Detection Toolbox and Benchmark
Chen, K., Wang, J., Pang, J., Cao, Y., Xiong, Y., Li, X., Sun, S., Feng, W., Liu, Z., Xu, J., et al. (2019) · arXiv · doi:10.48550/arxiv.1906.07155 · Cited by 2,227 · cited 1×
“25.0 [47], and the backbone is ResNet50.”
InfluentialNAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detection
Ghiasi, G., Lin, T.-Y., & Le, Q. V. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00720 · Cited by 2,099
Bottleneck Transformers for Visual Recognition
Srinivas, A., Lin, T.-Y., Parmar, N., Shlens, J., Abbeel, P., & Vaswani, A. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01625 · Cited by 1,385 · cited 2×
“To improve the performance of the transformer in vision tasks, BoTNet [33] introduces MHSA in the transformer into the ResNet [38] [as shown in Fig.”
MethodResultParallel Feature Pyramid Network for Object Detection
Kim, S.-W., Kook, H.-K., Sun, J.-Y., Kang, M.-C., & Ko, S.-J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01228-1_15 · Cited by 326 · cited 1×
“Lin et al. [40], Liu et al. [41], Ghiasi et al. [42], Kim et al. [43], and Zhao et al. [44] proved that the fusion of different level features and the use of different level feature maps to predict different scale objects can significantly improve the detection accuracy of multiscale objects.”
BackgroundSlimYOLOv3: Narrower, Faster and Better for Real-Time UAV Applications
Zhang, P., Zhong, Y., & Li, X. (2019) · 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) · doi:10.1109/iccvw.2019.00011 · Cited by 266 · cited 1×
“To overcome the detection speed problem of UAV images, Zhang et al. [22] developed a UAV image object detection algorithm named SlimYOLOv3 by pruning the YOLOv3, which is twice as fast as the YOLOv3, but its detection accuracy still has much room for improvement.”
MethodDetecting Cars in UAV Images With a Catalog-Based Approach
Moranduzzo, T., & Melgani, F. (2014) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2013.2296351 · Cited by 200
NAM: Normalization-based Attention Module
Liu, Y., Shao, Z., Teng, Y., & Hoffmann, N. (2021) · arXiv · doi:10.48550/arxiv.2111.12419 · Cited by 175 · cited 1×
“To improve the ability of our detection model to resist confusing information, we apply the normalization-based attention module (NAM) [46] attention mechanism to form the APHs before detection heads, which can extract attention regions and enable the model to focus on useful objects.”
Method
Cited by
A survey of the vision transformers and their CNN-transformer based variants
Khan, A., Rauf, Z., Sohail, A., Khan, A. R., Asif, H., Asif, A., & Farooq, U. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10595-0 · Cited by 451
Metadata
- Authors
- Ye, T., Qin, W., Zhao, Z., Gao, X., Deng, X., & Ouyang, Y.
- Venue
- IEEE Transactions on Instrumentation and Measurement
- Published
- 2023
- Volume
- 72
- Pages
- 1-13
- ISSN
- 0018-9456 (ISSN-L)
- 1557-9662
- OpenAlex
- W4319990398
- Cited by
- 160
- References
- 20
- FWCI
- 14.55
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.