基于FPGA的DDPG算法硬件映射解析与机器人运动技能学习

Hardware mapping analysis of DDPG algorithm based on FPGA and robot motion skill learning

摘要:

为研究神经网络和强化学习算法与高等动物进化原理之间的联系,本文结合深度确定性策略梯度（deep deterministic policy gradient,DDPG）算法构建了一套可观测、可解释的轮足机器人自主运动控制系统。首先在FPGA（field-programmable gate arrays）上部署Actor-Critic神经网络,并设计了一套FPGA-ARM机器人控制系统,通过实时导出网络权值激活信号并生成权值热力图,以可视化展示策略演化过程。实验表明,该方案单步计算时延缩减至28 μs,5 000步内完成收敛。同时,权值热力图揭示了策略在初期、中期及后期3个阶段的动态演化,定性分析表明,非关注区域对整体策略影响微弱、资源利用更趋优化。本文提出的硬件算法协同框架为强化学习“黑箱”可观测性研究提供了新范式,展示了FPGA在嵌入式机器人控制中兼具低延迟、高并行和低功耗的独特优势,为多智能体协作与异构平台下的实时技能学习与硬件加速提供了潜在应用前景。

Abstract:

This paper investigates the intrinsic connection between neural networks, reinforcement learning (RL) algorithms, and the evolutionary principles of higher animals by developing an observable and interpretable autonomous control system for a wheel-legged robot. Leveraging the Deep Deterministic Policy Gradient (DDPG) algorithm, an Actor-Critic neural network has been implemented directly on Field-programmable gate arrays (FPGA). An FPGA-ARM robot control system is further designed to export weight activation signals in real time and generate weight heatmaps, thereby visualizing the strategy evolution process. Experimental results demonstrate that the proposed system has the ability of reducing the single-step computation latency to 28 μs and achieves convergence within 5 000 steps. Moreover, the weight heatmaps reveal the dynamic evolution of strategies across three phases——early, middle, and late stages. Qualitative analysis indicates that non-salient regions have minimal impact on the overall strategy, resulting in more efficient resource utilization. The proposed hardware-algorithm co-design framework establishes a novel paradigm for improving the interpretability and reducing the “black-box” nature of RL. It also showcases the unique advantages of FPGA in embedded robot control, namely low latency, high parallelism, and low power consumption. This work lays a robust foundation and presents promising prospects for real-time skill learning and hardware acceleration in scenarios involving multi-agent cooperation and heterogeneous computing platforms.