论文标题
多模式视频问题回答的方式转移注意力网络
Modality Shifting Attention Network for Multi-modal Video Question Answering
论文作者
论文摘要
本文认为，用于多模式视频答案（MVQA）任务的网络被称为模态转移注意力网络（MSAN）。 MSAN将任务分解为两个子任务：（1）与问题相关的时间矩的定位，以及（2）基于本地化力矩的答案准确预测答案。时间定位所需的方式可能与答案预测不同，而这种移动方式的能力对于执行任务至关重要。为此，MSAN基于（1）试图从每种方式中找到最合适的时间矩的力矩提案网络（MPN），也基于（2）异构推理网络（HRN），该网络（HRN）使用两种方式上的注意机制来预测答案。 MSAN能够使用称为“模态重要性调制”（MIM）的组件对每个子任务的两种模态进行重要的权重。实验结果表明，MSAN在TVQA基准数据集上实现71.13 \％的测试准确性，超过了先前的最先进。进行了广泛的消融研究和定性分析，以验证网络的各种组成部分。
This paper considers a network referred to as Modality Shifting Attention Network (MSAN) for Multimodal Video Question Answering (MVQA) task. MSAN decomposes the task into two sub-tasks: (1) localization of temporal moment relevant to the question, and (2) accurate prediction of the answer based on the localized moment. The modality required for temporal localization may be different from that for answer prediction, and this ability to shift modality is essential for performing the task. To this end, MSAN is based on (1) the moment proposal network (MPN) that attempts to locate the most appropriate temporal moment from each of the modalities, and also on (2) the heterogeneous reasoning network (HRN) that predicts the answer using an attention mechanism on both modalities. MSAN is able to place importance weight on the two modalities for each sub-task using a component referred to as Modality Importance Modulation (MIM). Experimental results show that MSAN outperforms previous state-of-the-art by achieving 71.13\% test accuracy on TVQA benchmark dataset. Extensive ablation studies and qualitative analysis are conducted to validate various components of the network.
