期刊检索

  • 2026年第58卷
  • 2025年第57卷
  • 2024年第56卷
  • 2023年第55卷
  • 2022年第54卷
  • 2021年第53卷
  • 2020年第52卷
  • 2019年第51卷
  • 2018年第50卷
  • 2017年第49卷
  • 2016年第48卷
  • 2015年第47卷
  • 2014年第46卷
  • 2013年第45卷
  • 2012年第44卷
  • 2011年第43卷
  • 2010年第42卷
  • 第1期
  • 第2期

主管单位 中华人民共和国工业和信息化部 主办单位 哈尔滨工业大学 主编 李隆球 国际刊号ISSN 0367-6234 国内刊号CN 23-1235/T

期刊网站二维码
微信公众号二维码
引用本文:王文飞,茹乐,吕茂隆,莫雳.分层强化学习驱动无人作战飞机空战自主决策[J].哈尔滨工业大学学报,2026,58(8):125.DOI:10.11918/202511098
Wang Wenfei,Ru Le,Lü Maolong,Mo Li.Autonomous decision-making of unmanned combat aircraft in air combat driven by hierarchical reinforcement learning[J].Journal of Harbin Institute of Technology,2026,58(8):125.DOI:10.11918/202511098
【打印本页】   【HTML】   【下载PDF全文】   【查看/发表评论】  【下载PDF阅读器】  【关闭】
过刊浏览    高级检索
本文已被:浏览 1404次   下载 112次 本文二维码信息
码上扫一扫!
分享到: 微信 更多
分层强化学习驱动无人作战飞机空战自主决策
王文飞1,2,3,茹乐1,2,3,吕茂隆2,4,莫雳5
(1.空军工程大学 装备管理与无人机工程学院,西安 710051; 2.无人飞行器技术全国重点实验室(西北工业大学),西安 710051; 3.空军工程大学 陕西高校青年创新团队,西安 710051; 4.空军工程大学 空管领航学院,西安 710051; 5.北京理工大学 空天科学与技术学院,北京100081)
摘要:
为应对未来智能化空战对无人机自主决策能力提出的迫切需求,解决在复杂高动态环境下传统方法决策效率低、策略鲁棒性差的问题,提出一种结合分层强化学习与两阶段训练范式的无人机空战自主决策新框架。首先,设计了一种分层决策架构,将端到端的庞大决策空间进行策略性解耦:上层利用深度Q网络(DQN)学习高层战术决策,将原始连续动作空间压缩为有限的离散战术指令集,有效降低了学习复杂度;下层通过多模式控制和状态驱动机制,将高层战术意图实时精确映射为具体的飞控指令序列,确保决策的可执行性与物理安全性。其次,提出并实现了一种“专家模仿-自博弈进化”的两阶段训练方法:第1阶段,构建了基于SECA规则架构的专家对手,智能体通过与其对抗进行模仿学习,获得高质量的初始策略和先验知识;第2阶段,智能体转入虚拟自博弈环境,通过与其历史最优版本持续对抗,驱动策略不断突破初始专家知识的局限。最后,搭建高保真空战仿真环境对本文方法性能进行实验验证。研究结果表明,与传统DQN及其他基准算法相比,所提方法可以有效提升智能体学习训练的奖励收敛性能和最终对抗胜率,有效提升无人机智能体空战对抗水平。
关键词:  深度强化学习  无人机  自主空战决策  专家对抗  自博弈
DOI:10.11918/202511098
分类号:V249
基金项目:国家自然科学基金 (62303489, GKJJ2,8, T2121003);中国博士后科学基金面上基金 (2022M723877);中国博士后科学基金(2023T160790);中国博士后科学基金国际交流计划引进项目 (YJ20220347);陕西省青年人才托举计划项目 (20220101);陕西省自然科学基础研究计划项目(2024JC-YBQN-8,5JC-QYCX-052)
Autonomous decision-making of unmanned combat aircraft in air combat driven by hierarchical reinforcement learning
Wang Wenfei1,2,3,Ru Le1,2,3,Lü Maolong2,4,Mo Li5
(1.School of Equipment Management and Unmanned Aerial Systems Engineering, Air Force Engineering University, Xi’an 710051, China; 2.National Key Laboratory of Unmanned Aerial Vehicle Technology (Northwestern Polytechnical University), Xi’an 710051, China; 3.The Youth Innovation Team of Shaanxi Universities, Air Force Engineering University, Xi’an 710051, China; 4.School of Air Traffic Control and Navigation, Air Force Engineering University, Xi’an 710051, China; 5.School of Aerospace Science and Technology, Beijing Institute of Technology, Beijing 100081, China)
Abstract:
To address the urgent demand for the autonomous decision-making capability of unmanned aerial vehicles in future intelligent air combat and to solve the bottlenecks of low decision-making efficiency and poor strategy robustness of traditional methods in complex and highly dynamic environments, this paper proposed a novel framework for the autonomous decision-making of unmanned aerial vehicles in air combat combining hierarchical reinforcement learning and a two-phase training paradigm. Firstly, this paper designed a hierarchical decision-making architecture to strategically decouple the vast end-to-end decision space. The upper layer utilized a deep Q-network (DQN) to learn high-level tactical decisions, compressing the original continuous action space into a finite set of discrete tactical commands, which effectively reduced the learning complexity; the lower layer mapped the high-level tactical intentions into specific flight control command sequences in real time through multi-mode control and state-driven mechanisms, which ensured the executability and physical safety of the decisions. Secondly, a two-phase training method of “expert imitation and self-game evolution” was proposed and implemented. In the first phase, an expert opponent based on the SECA rule architecture was constructed, and the agent obtained high-quality initial policies and prior knowledge through imitation learning by fighting against it; in the second phase, the agent transitioned into a virtual self-game environment to continuously fight against its historically optimal versions, which drove the policy to continuously break through the limitations of initial expert knowledge. Finally, a high-fidelity air combat simulation environment was built to experimentally verify the performance of the proposed method. The research results indicate that, compared with the traditional DQN and other baseline algorithms, the proposed method effectively improves the reward convergence performance during the agent learning and training and the final confrontation winning rate and effectively enhances the air combat confrontation level of the unmanned aerial vehicle agent.
Key words:  deep reinforcement learning  unmanned aerial vehicle  autonomous decision-making in air combat  expert confrontation  self-game

友情链接LINKS