ROBUST UAV SWARM DECISION-MAKING UNDER UNCERTAINTY VIA COMBINATION OF MULTI-AGENT REINFORCEMENT LEARNING AND HYBRID AUTOMATA

The author is a member of the editorial board of the publication, therefore he did not participate in the review and decision-making regarding the publication of this article.

Abstract

The article is presented in the OnlineFirst format, which involves early posting of materials on the journal website before their final printing in paper form.

The article has passed the review procedure and has been accepted for publication.

The article is available for search and citation by its DOI, which remains unchanged. After the issue is printed in the printing house, a new citation can be made using information about the volume number and page number, using the same DOI reference. 

Background. Unmanned Aerial Vehicle (UAV) swarms are increasingly deployed in complex, uncertain environments such as electronic warfare (EW) scenarios, where traditional rule-based control can struggle. Related works have applied multi-agent reinforcement learning (MARL) to enable UAVs to learn cooperative behaviors in such scenarios, and have explored splitting complex missions (e.g. search & track) into phases to simplify control. However, purely learning-based approaches may lack robustness or require prohibitively long training to handle all edge cases.

Methods. Our approach integrates the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm with a supervisory hybrid automaton that encodes high-level behavior modes. The hybrid automaton defines discrete modes for different mission phases (e.g. exploration vs. target tracking) and emergency conditions (e.g. evasion under jamming), with formal guard conditions triggering mode transitions. Within each mode, UAV agents execute continuous control actions learned via MADDPG’s centralized-training, decentralized-execution paradigm. We develop a simulation environment based on Gym-PyBullet-Drones for centralized training of a UAV swarm, including realistic physics and an EW interference model.

Results. The simulation experiments demonstrated that the proposed hybrid RL-automaton approach outperformed a standard end-to-end MADDPG baseline across all tested conditions. The hybrid swarm achieved consistently higher mission success rates, especially under adversarial jamming, where it maintained over 90% success compared to significant drops for the baseline. Learning curves showed faster convergence and higher final returns for the hybrid method, reflecting the benefit of structured mode-conditioned training. In dynamic target scenarios, the hybrid swarm maintained longer continuous tracking times through coordinated tactics, while the baseline frequently lost contact. These results highlight that integrating a formal automaton with multi-agent RL yields more robust, efficient, and scalable UAV swarm decision-making under uncertainty. These findings also align with recent studies showing that dividing missions into phases and training mode-specialized behaviors can significantly improve performance over monolithic policies.

Conclusions. The proposed hybrid model demonstrates that integrating formal automata-based mode switching with deep RL can yield more reliable and efficient decision-making for UAV swarms in uncertain, adversarial environments. Discrete mode supervision injects domain knowledge (e.g. when to switch tasks or enter a safe mode) that guides learning and ensures safety constraints, while MADDPG optimizes the continuous control within each mode. This hybrid approach offers improved adaptability to evolving threats and environmental changes, as evidenced by robust performance under EW conditions. Future work will extend this framework to more complex swarm scenarios using heterogeneous UAV swarms and explore automatic mode discovery (e.g. via hierarchical RL) to further reduce reliance on expert-defined automata.