A Restormer-based fusion method with detail compensation for infrared and visible images

Abstract:

To enhance the quality and information integrity of fused images and tackle issues like inadequate feature extraction, insufficient texture details, and loss of global contextual information in infrared and visible images fusion, a fusion and decomposition network architecture for infrared and visible images is proposed. Firstly, a parallel structure of Restormer and Res2Net is utilized. Multiple deep convolutional heads with transposed attention mechanisms and multi-scale residual connections are employed to collaboratively capture the global contextual information and local detail features. Secondly, an invertible neural network with affine coupling structure is adopted to divide the shallow-level features of infrared and visible images into two parts, using alternating coupling transformations to achieve lossless feature preservation. Then, the reconstruction module generates high-quality fused images through concatenation and convolution operations. Finally, the decomposition network reverses the fusion image into source images by minimizing the decomposition loss function. Experimental results show that on the RoadScene dataset, the objective and subjective results of this method surpass most comparative methods. Specifically, compared to other methods, the standard deviation improves by an average of 8.5%, the difference correlation coefficient by 23.1%, the average gradient by 49.0%, and the spatial frequency by 56.1%. On the MSRS dataset, the proposed method outperforms SDCFusion method by 1.4% in standard deviation, 0.4% in visual information fidelity, 0.6% in average gradient, 4.3% in difference correlation coefficient, and 3.4% in spatial frequency. The proposed method shows significant advantages in improving the quality of fused images, preserving texture details, and retaining global information.