| 引用本文: | 高志云,唐艺宸,靳舒葳,梁宇玲.CM-MADDPG:面向多无人机博弈对抗的协同决策算法[J].哈尔滨工业大学学报,2026,58(8):173.DOI:10.11918/202510031 |
| Gao Zhiyun,Tang Yichen,Jin Shuwei,Liang Yuling.CM-MADDPG:a cooperative decision-making algorithm for multi-unmanned aerial vehicle game confrontation[J].Journal of Harbin Institute of Technology,2026,58(8):173.DOI:10.11918/202510031 |
|
| 摘要: |
| 为解决多无人机博弈对抗中现有决策方法在复杂场景下奖励设计烦琐,提升算法训练稳定性与样本效率。将多智能体强化学习与元强化学习结合,提出评论家与元策略联合增强的多智能体深度确定性策略梯度(Critic and Meta enhanced multi-agent deep deterministic policy gradient,CM-MADDPG)算法。首先,通过在集中式评论家侧引入适度的结构与学习增强,提升跨机交互的价值估计精度并抑制过估计,从而使训练更稳定,稳态性能更高。其次,在训练期间采用元策略进行自适应调度(如动态调整损失权重与探索强度),提升样本效率与收敛速度,且不增加推理期开销。最后,在相同实验条件下,红方无人机分别采用多智能体深度确定性策略梯度(multi-agent deep deterministic policy gradient,MADDPG)、评论家增强的多智能体深度确定性策略梯度(Critic multi-agent deep deterministic policy gradient,Critic-MADDPG)、元策略增强的多智能体深度确定性策略梯度(Meta enhanced multi-agent deep deterministic policy gradient,Meta-MADDPG),以及CM-MADDPG与随机机动的蓝方无人机进行博弈对抗。仿真结果表明,CM-MADDPG在最终胜率、各无人机奖励,以及收敛速度等指标上均取得显著提升。证明CM-MADDPG在不增加推理计算开销的前提下成功缓解稀疏奖励与训练不稳定问题,提高收敛速度与样本效率。研究成果可为多无人机博弈理论提供一种新思路和途径。 |
| 关键词: 多无人机博弈对抗 多智能体深度确定性策略梯度 元策略增强 集中式评论家 多智能体强化学习 样本效率 |
| DOI:10.11918/202510031 |
| 分类号:TP181;TP273 |
| 基金项目:国家自然科学基金(62403471);中央高校基本科研业务费(3122023QD18);天津市城市空中交通系统技术与装备重点实验室(TJKL-UAM-202402) |
|
| CM-MADDPG:a cooperative decision-making algorithm for multi-unmanned aerial vehicle game confrontation |
|
Gao Zhiyun1,2,Tang Yichen2,Jin Shuwei2,Liang Yuling3
|
|
(1.Key Laboratory of Technology and Equipment of Tianjin Urban Air Transportation System (Civil Aviation University of China), Tianjin 300300, China; 2.College of Transportation Science and Engineering, Civil Aviation University of China, Tianjin 300300, China; 3.School of Artificial Intelligence, Shenyang University of Technology, Shenyang 110870, China)
|
| Abstract: |
| To address the cumbersome reward design of existing decision-making methods in complex scenarios of multi-unmanned aerial vehicle game confrontation and to improve corresponding training stability and sample efficiency, this paper proposed a Critic and Meta enhanced multi-agent deep deterministic policy gradient (CM-MADDPG) algorithm by combining multi-agent reinforcement learning with Meta-reinforcement learning. First, this paper introduced appropriate structural and learning enhancements to the centralized Critic side to improve the value estimation accuracy of cross-unmanned aerial vehicle interaction and suppress overestimation, thereby making training more stable and achieving higher steady-state performance. Second, a Meta-strategy was adopted during the training period for adaptive scheduling (such as dynamically adjusting loss weights and exploration intensity) to improve sample efficiency and convergence speed without increasing inference-time overhead. Finally, under identical experimental conditions, red unmanned aerial vehicles respectively adopting multi-agent deep deterministic policy gradient (MADDPG), Critic multi-agent deep deterministic policy gradient (Critic-MADDPG), Meta enhanced multi-agent deep deterministic policy gradient (Meta-MADDPG), and CM-MADDPG conducted a game confrontation against blue unmanned aerial vehicles with random maneuvering. Simulation results show that CM-MADDPG achieves significant improvements in final win rate, rewards of each unmanned aerial vehicle, and convergence speed. This proves that CM-MADDPG successfully alleviates sparse reward and training instability issues without increasing inference computational overhead and improves convergence speed and sample efficiency. The research results of this paper provide a new idea and approach for theories of multi-unmanned aerial vehicle game confrontation. |
| Key words: multi-unmanned aerial vehicle game confrontation multi-agent deep deterministic policy gradient Meta-strategy enhancement centralized Critic multi-agent reinforcement learning sample efficiency |