动态门控扩散去噪与跨层注意力的多模态图像融合网络

Dynamic gating diffusion denoising and cross-layer attention-based multimodal image fusion network

关键词:  多模态图像融合  扩散模型  门控特征选择模块  跨层注意力融合模块  专家卷积核

Key words:  multimodal image fusion  diffusion models  gated feature selection module  cross-layer attention fusion module  expert convolutional kernels

摘要:

针对去噪扩散模型在图像融合任务中难以适应不同噪声水平、普通残差块对特征的筛选能力有限的问题,本文构建了一种动态门控扩散去噪与跨层注意力的多模态图像融合网络。首先,设计并引入4组专家卷积核至动态特征提取器模块,根据输入内容动态组合出最优卷积核,对输入特征实现自适应处理；其次,提出了一种改进的门控特征选择模块来生成门控信号抑制无关信息,提升模型在不同噪声水平下的扩散去噪能力,实现对特征的精准控制；最后,使用R-Transformer块进行特征调整,通过构建的全局局部空间注意力模块实现跨层特征融合,以生成纹理信息丰富、色彩保真度高的融合图像。在MSRS、RoadScene和Harvard三个数据集上的实验结果表明,与近年来图像融合领域中9种具有代表性的重要方法相比,本文方法的7种客观评价指标平均提升了5.11%~15.93%。本文方法在纹理细节保持及解剖结构完整性保留等方面均优于其他方法,符合人眼视觉特性,能够很好地处理各种光照环境场景和医学影像诊断场景下的多模态图像融合任务。

Abstract:

To address the challenges that denoising diffusion models struggle to adapt to varying noise levels and conventional residual blocks have limited feature selection capability in image fusion tasks, this paper constructs a multimodal image fusion network integrating dynamic gating diffusion denoising and cross-layer attention. Firstly, four groups of expert convolution kernels are designed and incorporated into the dynamic feature extractor module. The optimal convolution kernels are dynamically assembled based on input content, enabling adaptive processing of input features. Secondly, an improved gated feature selection module is proposed to generate gating signals that suppress irrelevant information, enhance the model’s diffusion denoising capability under different noise levels, and achieve precise feature control. Finally, R-Transformer blocks are adopted for feature adjustment. A global-local spatial attention module is constructed to realize cross-layer feature fusion, thereby generating fused images with rich texture information and high color fidelity. Experimental results on the MSRS, RoadScene, and Harvard datasets demonstrate that compared with 9 representative state-of-the-art methods in the field of image fusion in recent years, the proposed method achieves an average improvement of 5.11% to 15.93% across 7 objective evaluation metrics. The proposed method outperforms other counterparts in texture detail preservation and anatomical structure integrity maintenance, conforms to human visual perception characteristics, and can effectively handle multimodal image fusion tasks in scenarios such as various lighting environments and medical image diagnosis.