CoChat

Deep Learning-Based Real-Time Multiple-Object Detection and Tracking from Aerial Imagery via a Flying Robot with GPU-Based Embedded Devices

Open access
Hossain, Sabir, Lee, Deok-jin · 2019 · Sensors

The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.

Abstract

In recent years, demand has been increasing for target detection and tracking from aerial imagery via drones using onboard powered sensors and devices. We propose a very effective method for this application based on a deep learning framework. A state-of-the-art embedded hardware system empowers small flying robots to carry out the real-time onboard computation necessary for object tracking. Two types of embedded modules were developed: one was designed using a Jetson TX or AGX Xavier, and the other was based on an Intel Neural Compute Stick. These are suitable for real-time onboard computing power on small flying drones with limited space. A comparative analysis of current state-of-the-art deep learning-based multi-object detection algorithms was carried out utilizing the designated GPU-based embedded computing modules to obtain detailed metric data about frame rates, as well as the computation power. We also introduce an effective target tracking approach for moving objects. The algorithm for tracking moving objects is based on the extension of simple online and real-time tracking. It was developed by integrating a deep learning-based association metric approach with simple online and real-time tracking (Deep SORT), which uses a hypothesis tracking methodology with Kalman filtering and a deep learning-based association metric. In addition, a guidance system that tracks the target position using a GPU-based algorithm is introduced. Finally, we demonstrate the effectiveness of the proposed algorithms by real-time experiments with a small multi-rotor drone.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3390/s19153371

Cites

33 of 33 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 3D CAD design of the external structure of an aerial vehicle with a camera mounted on it.
Figure 1 3D CAD design of the external structure of an aerial vehicle with a camera mounted on it.CC BY · PubMed Central
Figure 2 List of developed embedded system types for the aerial vehicle.
Figure 2 List of developed embedded system types for the aerial vehicle.CC BY · PubMed Central
Figure 3 List of the embedded systems implemented inside the common drone structure. (a) On-board embedded GPU system; (b) off-board GPU-based ground station; (c) on-board GPU-constrained system.
Figure 3 List of the embedded systems implemented inside the common drone structure. (a) On-board embedded GPU system; (b) off-board GPU-based ground station; (c) on-board GPU-constrained system.CC BY · PubMed Central
Figure 4 The portrayal of the aerial vehicle looking for a target with a drone taking off from the ground, following a designated waypoint, and looking for objects.
Figure 4 The portrayal of the aerial vehicle looking for a target with a drone taking off from the ground, following a designated waypoint, and looking for objects.CC BY · PubMed Central
Figure 5 (a) On-board target detection system using Jetson TX1, (b) On-board target detection system using Jetson TX2, and (c) On-board target detection system using Jetson AGX Xavier.
Figure 5 (a) On-board target detection system using Jetson TX1, (b) On-board target detection system using Jetson TX2, and (c) On-board target detection system using Jetson AGX Xavier.CC BY · PubMed Central
Figure 6 Target detection using the GPU-based ground station from an aerial vehicle.
Figure 6 Target detection using the GPU-based ground station from an aerial vehicle.CC BY · PubMed Central
Figure 7 On-board target detection drone system using GPU-constrained devices like (a) Raspberry Pi, (b) Latte Panda, and (c) Odroid Xu4.
Figure 7 On-board target detection drone system using GPU-constrained devices like (a) Raspberry Pi, (b) Latte Panda, and (c) Odroid Xu4.CC BY · PubMed Central
Figure 8 List of deep learning algorithms implemented in the aerial vehicle: (a) On-board embedded GPU system; (b) off-board GPU-based ground station; (c) on-board GPU-constrained system.
Figure 8 List of deep learning algorithms implemented in the aerial vehicle: (a) On-board embedded GPU system; (b) off-board GPU-based ground station; (c) on-board GPU-constrained system.CC BY · PubMed Central
Figure 9 Architecture of YOLO.
Figure 9 Architecture of YOLO.CC BY · PubMed Central
Figure 10 YOLOv3 architecture.
Figure 10 YOLOv3 architecture.CC BY · PubMed Central
Figure 11 The architecture of SSD.
Figure 11 The architecture of SSD.CC BY · PubMed Central
Figure 12 After creating a set of region proposals, R-CNN uses a modified version of AlexNet to determine the valid region.
Figure 12 After creating a set of region proposals, R-CNN uses a modified version of AlexNet to determine the valid region.CC BY · PubMed Central
Figure 13 Target detection results of (a) real-time segmentation using DeepLabv-3 using Xavier, (b) faster R-CNN using TX2, (c) YOLOv2 using TX2, (d) faster R-CNN using Xavier, (e) YOLOv3 using TX2, (f) YOLOv3 using Xavier, (g) YOLOv2 using Xavier, (h) SSD-caffe using Xavier, (i) YOLOv3 using Xavier, and (j) SSD-caffe using TX2.
Figure 13 Target detection results of (a) real-time segmentation using DeepLabv-3 using Xavier, (b) faster R-CNN using TX2, (c) YOLOv2 using TX2, (d) faster R-CNN using Xavier, (e) YOLOv3 using TX2, (f) YOLOv3 using Xavier, (g) YOLOv2 using Xavier, (h) SSD-caffe using Xavier, (i) YOLOv3 using Xavier, and (j) SSD-caffe using TX2.CC BY · PubMed Central
Figure 14 Target detection results using the off-board GPU-based ground station: (a) Real-time segmentation using DeepLabv-3, (b) YOLOv3, (c) mask-R-CNN, (d) SSD-Mobilenet, (e,f) YOLOv2, (g) YOLO-9000m, and (h) YOLOv3.
Figure 14 Target detection results using the off-board GPU-based ground station: (a) Real-time segmentation using DeepLabv-3, (b) YOLOv3, (c) mask-R-CNN, (d) SSD-Mobilenet, (e,f) YOLOv2, (g) YOLO-9000m, and (h) YOLOv3.CC BY · PubMed Central
Figure 15 Target tracking results using Deep SORT and YOLOv3 from the drone using (a) Xavier and (b) the off-board GPU-based ground station.
Figure 15 Target tracking results using Deep SORT and YOLOv3 from the drone using (a) Xavier and (b) the off-board GPU-based ground station.CC BY · PubMed Central
Figure 16 Target detection results from the drone of (a) YOLO tiny using Odroid XU4 + NCS and (b) SSD-MobileNet using Odroid XU4 + NCS.
Figure 16 Target detection results from the drone of (a) YOLO tiny using Odroid XU4 + NCS and (b) SSD-MobileNet using Odroid XU4 + NCS.CC BY · PubMed Central
Figure 17 Average GPU usage percentage from (a) Xavier and (b) TX2.
Figure 17 Average GPU usage percentage from (a) Xavier and (b) TX2.CC BY · PubMed Central
Figure 18 (a) Memory usage and (b) average CPU usage percentage in Xavier.
Figure 18 (a) Memory usage and (b) average CPU usage percentage in Xavier.CC BY · PubMed Central
Figure 19 Plot diagram of the error between the image and target center vs. distance.
Figure 19 Plot diagram of the error between the image and target center vs. distance.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 33 references

Research works printed but not linked: 3

  1. Tsang, S.-H. (2019, February 24). Review: YOLOv2 & YOLO9000—You Only Look Once (Object Detection). Available online: towardsdatascience.com/review-YOLOv2-YOLO9000-you-only-look-once-object-detection-7883d2b02a65.
  2. Szegedy, C., Reed, S., Erhan, D., Anguelov, D., and Ioffe, S. (2014). Scalable, high-quality object detection. arXiv.
  3. Qi, H., Zhang, Z., Xiao, B., Hu, H., Cheng, B., Wei, Y., and Dai, J. (2017, January 28). Deformable convolutional networks–coco detection and segmentation challenge 2017 entry. Proceedings of the ICCV COCO Challenge Workshop, Venice, Italy.

Cited by

Metadata

Authors
Hossain, S., & Lee, D.-J.
Affiliations
Kunsan National University (KR)
Corresponding author
Lee, Deok-jin
Venue
Sensors
Publisher
MDPI AG
Published
2019-07-31
Volume
19
Issue
15
Pages
3371
ISSN
1424-8220 (ISSN-L)
OpenAlex
W2957927798
Cited by
184
References
28
Citations by year
2026: 4 · 2025: 18 · 2024: 26 · 2023: 35 · 2022: 38 · 2021: 39 · 2020: 21 · 2019: 3
FWCI
11.29
Type
journal article
Language
en
Open access
Locations
  • Sensors (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Surveillance and Tracking Methods
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
Keywords
multi-target detection, object tracking, aerial imagery, unmanned aerial vehicles, multi-rotor drones, embedded computing, Jetson, DeepSORT, Kalman filter

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.