Application of Deep Reinforcement Learning to UAV Swarming for Ground Surveillance

TL;DR

The paper proposes a hybrid AI system for UAV swarm ground surveillance, combining a centralized rule-based swarm controller with deep reinforcement learning sub-agents for search, tracking, and obstacle avoidance. Simulation results show effective area search, reasonable target acquisition, and consistent tracking, with performance adapting to changing numbers of targets and UAVs.

Key findings

• In a baseline simulation with 8 fixed-wing UAVs tracking 5 targets over a 6 km × 6 km area, run 10 times, most targets were kept in the swarm’s field of view for over 95% of the time since first identification, though some losses occurred mainly due to occlusions behind obstacles.
• Across 10 simulations totaling 50 targets, the swarm searched the operation area effectively, but border cells had higher revisit periods, indicating more difficulty exploring edges.
• With UAVs fixed at 8 and targets varied as 1, 3, 5, and 6, total search time generally increased with more targets, while the percentage of detected targets showed fewer differences because tracking is assigned to a single agent once a target is first detected.
• With targets fixed at 5 and UAVs varied as 8, 9, and 10, increasing the number of UAVs significantly improved revisit time and reduced ordered target acquisition time.

Methods

• The system uses a two-layer centralized architecture: a deterministic rule-based swarm controller assigns search and tracking tasks, while each UAV agent executes tasks through three cooperative DRL sub-agents for search, tracking, and obstacle avoidance.
• The sub-agents were trained independently using proximal policy optimization (PPO) in Unity ML-Agents, with an artificial neural network having an input layer, two fully connected hidden layers of 128 neurons each, and an output layer.
• Evaluation used simulation scenarios with a 6 km square operation area, fixed-wing UAVs at constant altitude, ground targets moving at 20 m/s, static obstacles, and metrics including revisit period, search time, percentage of detected targets, tracking continuity, and target acquisition time.

Limitations

• The paper states that RL-based approaches are expensive in computation, development time, and resources, and that training a swarm control system for an imminent mission with a specific or new scenario could be difficult.
• The paper states that performance strongly depends on the fidelity of training scenarios, that reward function design requires expert knowledge and many design iterations, and that there is no clear method for determining reward functions.
• Evident from the text: the study assumes perfect sensors, negligible navigation error, benign weather, fixed UAV altitude, and simulation-only validation; targets in the tracking training initially move at constant speed in straight lines, and the evaluation uses constant-speed targets.