分层强化学习驱动无人作战飞机空战自主决策
CSTR:
作者:
作者单位:

(1.空军工程大学 装备管理与无人机工程学院,西安 710051; 2.无人飞行器技术全国重点实验室(西北工业大学),西安 710051; 3.空军工程大学 陕西高校青年创新团队,西安 710051; 4.空军工程大学 空管领航学院,西安 710051; 5.北京理工大学 空天科学与技术学院,北京100081)

作者简介:

王文飞(1998—),男,博士研究生;茹乐(1978—),男,教授,博士生导师;吕茂隆(1991—),男,副教授,博士生导师

通讯作者:

中图分类号:

V249

基金项目:

国家自然科学基金 (62303489, GKJJ2,8, T2121003);中国博士后科学基金面上基金 (2022M723877);中国博士后科学基金(2023T160790);中国博士后科学基金国际交流计划引进项目 (YJ20220347);陕西省青年人才托举计划项目 (20220101);陕西省自然科学基础研究计划项目(2024JC-YBQN-8,5JC-QYCX-052)


Autonomous decision-making of unmanned combat aircraft in air combat driven by hierarchical reinforcement learning
Author:
Affiliation:

(1.School of Equipment Management and Unmanned Aerial Systems Engineering, Air Force Engineering University, Xi’an 710051, China; 2.National Key Laboratory of Unmanned Aerial Vehicle Technology (Northwestern Polytechnical University), Xi’an 710051, China; 3.The Youth Innovation Team of Shaanxi Universities, Air Force Engineering University, Xi’an 710051, China; 4.School of Air Traffic Control and Navigation, Air Force Engineering University, Xi’an 710051, China; 5.School of Aerospace Science and Technology, Beijing Institute of Technology, Beijing 100081, China)

Fund Project:

undefined

  • 摘要
  • |
  • 图/表
  • |
  • 访问统计
  • |
  • 参考文献
  • |
  • 相似文献
  • |
  • 引证文献
  • |
  • 资源附件
  • |
  • 文章评论
    摘要:

    为应对未来智能化空战对无人机自主决策能力提出的迫切需求,解决在复杂高动态环境下传统方法决策效率低、策略鲁棒性差的问题,提出一种结合分层强化学习与两阶段训练范式的无人机空战自主决策新框架。首先,设计了一种分层决策架构,将端到端的庞大决策空间进行策略性解耦:上层利用深度Q网络(DQN)学习高层战术决策,将原始连续动作空间压缩为有限的离散战术指令集,有效降低了学习复杂度;下层通过多模式控制和状态驱动机制,将高层战术意图实时精确映射为具体的飞控指令序列,确保决策的可执行性与物理安全性。其次,提出并实现了一种“专家模仿-自博弈进化”的两阶段训练方法:第1阶段,构建了基于SECA规则架构的专家对手,智能体通过与其对抗进行模仿学习,获得高质量的初始策略和先验知识;第2阶段,智能体转入虚拟自博弈环境,通过与其历史最优版本持续对抗,驱动策略不断突破初始专家知识的局限。最后,搭建高保真空战仿真环境对本文方法性能进行实验验证。研究结果表明,与传统DQN及其他基准算法相比,所提方法可以有效提升智能体学习训练的奖励收敛性能和最终对抗胜率,有效提升无人机智能体空战对抗水平。

    Abstract:

    To address the urgent demand for the autonomous decision-making capability of unmanned aerial vehicles in future intelligent air combat and to solve the bottlenecks of low decision-making efficiency and poor strategy robustness of traditional methods in complex and highly dynamic environments, this paper proposed a novel framework for the autonomous decision-making of unmanned aerial vehicles in air combat combining hierarchical reinforcement learning and a two-phase training paradigm. Firstly, this paper designed a hierarchical decision-making architecture to strategically decouple the vast end-to-end decision space. The upper layer utilized a deep Q-network (DQN) to learn high-level tactical decisions, compressing the original continuous action space into a finite set of discrete tactical commands, which effectively reduced the learning complexity; the lower layer mapped the high-level tactical intentions into specific flight control command sequences in real time through multi-mode control and state-driven mechanisms, which ensured the executability and physical safety of the decisions. Secondly, a two-phase training method of “expert imitation and self-game evolution” was proposed and implemented. In the first phase, an expert opponent based on the SECA rule architecture was constructed, and the agent obtained high-quality initial policies and prior knowledge through imitation learning by fighting against it; in the second phase, the agent transitioned into a virtual self-game environment to continuously fight against its historically optimal versions, which drove the policy to continuously break through the limitations of initial expert knowledge. Finally, a high-fidelity air combat simulation environment was built to experimentally verify the performance of the proposed method. The research results indicate that, compared with the traditional DQN and other baseline algorithms, the proposed method effectively improves the reward convergence performance during the agent learning and training and the final confrontation winning rate and effectively enhances the air combat confrontation level of the unmanned aerial vehicle agent.

    参考文献
    相似文献
    引证文献
引用本文

王文飞,茹乐,吕茂隆,莫雳.分层强化学习驱动无人作战飞机空战自主决策[J].哈尔滨工业大学学报,2026,58(8):125. DOI:10.11918/202511098

复制
分享
相关视频

文章指标
  • 点击次数:
  • 下载次数:
  • HTML阅读次数:
  • 引用次数:
历史
  • 收稿日期:2025-11-29
  • 最后修改日期:
  • 录用日期:
  • 在线发布日期: 2026-08-29
  • 出版日期:
文章二维码