Autonomous decision-making of unmanned combat aircraft in air combat driven by hierarchical reinforcement learning
CSTR:
Author:
Affiliation:

(1.School of Equipment Management and Unmanned Aerial Systems Engineering, Air Force Engineering University, Xi’an 710051, China; 2.National Key Laboratory of Unmanned Aerial Vehicle Technology (Northwestern Polytechnical University), Xi’an 710051, China; 3.The Youth Innovation Team of Shaanxi Universities, Air Force Engineering University, Xi’an 710051, China; 4.School of Air Traffic Control and Navigation, Air Force Engineering University, Xi’an 710051, China; 5.School of Aerospace Science and Technology, Beijing Institute of Technology, Beijing 100081, China)

Clc Number:

V249

Fund Project:

undefined

  • Article
  • |
  • Figures
  • |
  • Metrics
  • |
  • Reference
  • |
  • Related
  • |
  • Cited by
  • |
  • Materials
  • |
  • Comments
    Abstract:

    To address the urgent demand for the autonomous decision-making capability of unmanned aerial vehicles in future intelligent air combat and to solve the bottlenecks of low decision-making efficiency and poor strategy robustness of traditional methods in complex and highly dynamic environments, this paper proposed a novel framework for the autonomous decision-making of unmanned aerial vehicles in air combat combining hierarchical reinforcement learning and a two-phase training paradigm. Firstly, this paper designed a hierarchical decision-making architecture to strategically decouple the vast end-to-end decision space. The upper layer utilized a deep Q-network (DQN) to learn high-level tactical decisions, compressing the original continuous action space into a finite set of discrete tactical commands, which effectively reduced the learning complexity; the lower layer mapped the high-level tactical intentions into specific flight control command sequences in real time through multi-mode control and state-driven mechanisms, which ensured the executability and physical safety of the decisions. Secondly, a two-phase training method of “expert imitation and self-game evolution” was proposed and implemented. In the first phase, an expert opponent based on the SECA rule architecture was constructed, and the agent obtained high-quality initial policies and prior knowledge through imitation learning by fighting against it; in the second phase, the agent transitioned into a virtual self-game environment to continuously fight against its historically optimal versions, which drove the policy to continuously break through the limitations of initial expert knowledge. Finally, a high-fidelity air combat simulation environment was built to experimentally verify the performance of the proposed method. The research results indicate that, compared with the traditional DQN and other baseline algorithms, the proposed method effectively improves the reward convergence performance during the agent learning and training and the final confrontation winning rate and effectively enhances the air combat confrontation level of the unmanned aerial vehicle agent.

    Reference
    Related
    Cited by
Get Citation
Related Videos

Article Metrics
  • Abstract:
  • PDF:
  • HTML:
  • Cited by:
History
  • Received:November 29,2025
  • Revised:
  • Adopted:
  • Online: August 29,2026
  • Published:
Article QR Code