论文标题
在变压器编码器中编码句法知识以进行意图检测和插槽填充
Encoding Syntactic Knowledge in Transformer Encoder for Intent Detection and Slot Filling
论文作者
论文摘要
我们提出了一种基于新颖的变压器编码器的架构，该体系结构用句法知识编码用于意图检测和插槽填充。具体而言，我们通过共同训练它来预测每个代币的句法解析祖先和通过多任务学习的词性来预测句法祖先，并通过多任务学习来编码句法知识。我们的模型基于自我注意事项和前馈层，不需要在推理时提供外部句法信息。实验表明，在两个基准数据集上，我们的模型只有两个变压器编码器层可实现最新的结果。与先前最佳性能的模型相比，我们的模型分别在SNIPS数据集中分别实现了插槽填充和意图检测的绝对F1分数和0.85％的准确性提高。我们的模型还可以在ATIS数据集上分别在先前最佳性能的模型上获得插槽填充和意图检测的绝对F1分数和0.1％和0.34％的准确性提高。此外，自我发挥作用权重的可视化说明了在训练过程中纳入句法信息的好处。
We propose a novel Transformer encoder-based architecture with syntactical knowledge encoded for intent detection and slot filling. Specifically, we encode syntactic knowledge into the Transformer encoder by jointly training it to predict syntactic parse ancestors and part-of-speech of each token via multi-task learning. Our model is based on self-attention and feed-forward layers and does not require external syntactic information to be available at inference time. Experiments show that on two benchmark datasets, our models with only two Transformer encoder layers achieve state-of-the-art results. Compared to the previously best performed model without pre-training, our models achieve absolute F1 score and accuracy improvement of 1.59% and 0.85% for slot filling and intent detection on the SNIPS dataset, respectively. Our models also achieve absolute F1 score and accuracy improvement of 0.1% and 0.34% for slot filling and intent detection on the ATIS dataset, respectively, over the previously best performed model. Furthermore, the visualization of the self-attention weights illustrates the benefits of incorporating syntactic information during training.
