论文标题
使用文本百科全书知识的上下文化表示
Contextualized Representations Using Textual Encyclopedic Knowledge
论文作者
论文摘要
我们提出了一种通过从多个文档中与动态检索的文本百科全书知识共同将输入文本的方法表示来表示输入文本的方法。我们通过编码问题和段落以及有关他们提到的实体的背景句子来读取理解任务的方法。我们表明，从文本中整合背景知识对于专注于事实推理的任务有效，并允许直接重复使用强大的BERT风格编码器。此外，通过通过自我监督的掩盖语言模型目标对背景启动输入文本的单词进行适当的预处理，可以进一步改善知识集成。在Triviaqa上，我们的方法比可比的Roberta模型获得了1.6至3.1 F1的改进，而Roberta模型不会动态地整合背景知识。在MRQA（大量质量检查数据集）中，我们看到了一致的增益，以及在Bioasq（2.1至4.2 F1），TextBookQa（1.6至2.0 F1）和Duorc（1.1至2.0 F1）上的bioASQ外域外的大量改进。
We present a method to represent input texts by contextualizing them jointly with dynamically retrieved textual encyclopedic background knowledge from multiple documents. We apply our method to reading comprehension tasks by encoding questions and passages together with background sentences about the entities they mention. We show that integrating background knowledge from text is effective for tasks focusing on factual reasoning and allows direct reuse of powerful pretrained BERT-style encoders. Moreover, knowledge integration can be further improved with suitable pretraining via a self-supervised masked language model objective over words in background-augmented input text. On TriviaQA, our approach obtains improvements of 1.6 to 3.1 F1 over comparable RoBERTa models which do not integrate background knowledge dynamically. On MRQA, a large collection of diverse QA datasets, we see consistent gains in-domain along with large improvements out-of-domain on BioASQ (2.1 to 4.2 F1), TextbookQA (1.6 to 2.0 F1), and DuoRC (1.1 to 2.0 F1).
