论文标题
TGFUSE：基于变压器和生成对抗网络的红外且可见的图像融合方法
TGFuse: An Infrared and Visible Image Fusion Approach Based on Transformer and Generative Adversarial Network
论文作者
论文摘要
端到端图像融合框架已实现了有希望的性能，专用的卷积网络汇总了多模式的本地外观。但是，在现有的CNN融合方法中直接忽略了远程依赖性，阻碍了整个图像级别的感知在复杂场景融合中的平衡。因此，在本文中，我们提出了一种基于轻质变压器模块和对抗性学习的红外且可见的图像融合算法。受到全球互动能力的启发，我们使用变压器技术来学习有效的全球融合关系。特别是，CNN提取的浅特征在提出的变压器融合模块中相互作用，以优化空间范围内和跨通道内的融合关系。此外，对抗性学习是在训练过程中设计的，以通过从输入中施加竞争性一致性来改善输出歧视，这反映了红外和可见图像中的特定特征。实验性能证明了所提出的模块的有效性，并在融合任务中通过变压器和对抗性学习概括了一种新型范式，并具有出色的改进。
The end-to-end image fusion framework has achieved promising performance, with dedicated convolutional networks aggregating the multi-modal local appearance. However, long-range dependencies are directly neglected in existing CNN fusion approaches, impeding balancing the entire image-level perception for complex scenario fusion. In this paper, therefore, we propose an infrared and visible image fusion algorithm based on a lightweight transformer module and adversarial learning. Inspired by the global interaction power, we use the transformer technique to learn the effective global fusion relations. In particular, shallow features extracted by CNN are interacted in the proposed transformer fusion module to refine the fusion relationship within the spatial scope and across channels simultaneously. Besides, adversarial learning is designed in the training process to improve the output discrimination via imposing competitive consistency from the inputs, reflecting the specific characteristics in infrared and visible images. The experimental performance demonstrates the effectiveness of the proposed modules, with superior improvement against the state-of-the-art, generalising a novel paradigm via transformer and adversarial learning in the fusion task.
