一种Restormer结合细节补偿的红外与可见光图像融合方法

A Restormer-based fusion method with detail compensation for infrared and visible images

关键词:  图像融合  并联结构  细节补偿  可逆神经网络  分解网络

Key words:  image fusion  parallel structure  detail compensation  invertible neural network  decomposition network

摘要:

为提升融合图像的质量和信息完整性,解决红外与可见光图像融合中存在的特征提取能力不足、缺乏纹理细节以及全局上下文信息丢失等问题,提出一种红外与可见光图像的融合与分解网络架构。首先,利用Restormer和Res2Net的并联结构,通过多个深度卷积头转置注意力机制和多尺度残差连接,协同捕获全局上下文信息和局部细节特征；其次,通过带有仿射耦合结构的可逆神经网络,将红外与可见光图像浅层特征分为两部分,利用交替耦合变换实现特征无损保留；然后,重建模块利用拼接及卷积操作生成高质量融合图像;最后,分解网络通过最小化分解损失函数,将融合图像逆向分解为源图像。实验结果表明:在RoadScene数据集上,本文方法的主客观结果均优于多数对比方法,其中标准差、差异相关系数、平均梯度和空间频率较其他对比方法分别平均提升了8.5%、23.1%、49.0%和56.1%；在MSRS数据集上,本文方法较SDCFusion方法在标准差、视觉信息保真度、平均梯度、差异相关系数和空间频率方面分别提升了1.4%、0.4%、0.6%、4.3%和3.4%。所提方法在提升融合图像质量、保留纹理细节和全局信息方面展现出显著优势。

Abstract:

To enhance the quality and information integrity of fused images and tackle issues like inadequate feature extraction, insufficient texture details, and loss of global contextual information in infrared and visible images fusion, a fusion and decomposition network architecture for infrared and visible images is proposed. Firstly, a parallel structure of Restormer and Res2Net is utilized. Multiple deep convolutional heads with transposed attention mechanisms and multi-scale residual connections are employed to collaboratively capture the global contextual information and local detail features. Secondly, an invertible neural network with affine coupling structure is adopted to divide the shallow-level features of infrared and visible images into two parts, using alternating coupling transformations to achieve lossless feature preservation. Then, the reconstruction module generates high-quality fused images through concatenation and convolution operations. Finally, the decomposition network reverses the fusion image into source images by minimizing the decomposition loss function. Experimental results show that on the RoadScene dataset, the objective and subjective results of this method surpass most comparative methods. Specifically, compared to other methods, the standard deviation improves by an average of 8.5%, the difference correlation coefficient by 23.1%, the average gradient by 49.0%, and the spatial frequency by 56.1%. On the MSRS dataset, the proposed method outperforms SDCFusion method by 1.4% in standard deviation, 0.4% in visual information fidelity, 0.6% in average gradient, 4.3% in difference correlation coefficient, and 3.4% in spatial frequency. The proposed method shows significant advantages in improving the quality of fused images, preserving texture details, and retaining global information.