论文标题
KGPLM：知识引导的语言模型通过生成和歧视性学习预训练
KgPLM: Knowledge-guided Language Model Pre-training via Generative and Discriminative Learning
论文作者
论文摘要
关于预训练的语言模型的最新研究表明，他们有能力捕获知识知识的下游任务中的事实知识和应用。在这项工作中，我们提出了一种以事实知识完成和验证为指导的语言模型预训练框架，并使用合作的生成和歧视方法来学习模型。特别是，我们研究了两个命名为两磅的学习方案和管道方案的学习方案，以培训具有共享参数的生成器和歧视器。喇嘛是一组零拍的斗篷式问题，回答任务的实验结果表明，与传统的预训练的语言模型相比，我们的模型包含更丰富的事实知识。此外，当对MRQA进行微调和评估时，该任务由几个机器阅读理解数据集组成，我们的模型可以实现最先进的性能，并在Roberta上对NewsQA（+1.26 F1）和Triviaqa（+1.56 F1）进行了大量改进。
Recent studies on pre-trained language models have demonstrated their ability to capture factual knowledge and applications in knowledge-aware downstream tasks. In this work, we present a language model pre-training framework guided by factual knowledge completion and verification, and use the generative and discriminative approaches cooperatively to learn the model. Particularly, we investigate two learning schemes, named two-tower scheme and pipeline scheme, in training the generator and discriminator with shared parameter. Experimental results on LAMA, a set of zero-shot cloze-style question answering tasks, show that our model contains richer factual knowledge than the conventional pre-trained language models. Furthermore, when fine-tuned and evaluated on the MRQA shared tasks which consists of several machine reading comprehension datasets, our model achieves the state-of-the-art performance, and gains large improvements on NewsQA (+1.26 F1) and TriviaQA (+1.56 F1) over RoBERTa.
