论文标题
用句子连贯目标改善语言产生
Improving Language Generation with Sentence Coherence Objective
论文作者
论文摘要
有条件的故事产生和上下文文本延续已成为NLP社区越来越受欢迎的主题。现有模型通常容易输出逐渐与给定提示的文本段落。尽管生成的文本可能具有合理的困惑和多样性，但人类很容易将其识别为Gibberish。我们项目的目的是提高语言生成模型中句子之间的连贯性和一致性。我们旨在通过首先训练句子对相干分类器与GPT-2预处理的模型来解决此问题，然后使用类似于增强算法的方法，将GPT-2语言模型与此新的连贯物镜共同培训。这个微调的语言模型能够生成以给定主题为条件的冗长段落，而不会分歧太大。该模型的简单性使其适用于各种基础语言模型体系结构，因为它仅修改了预训练模型的最后一层。
Conditional story generation and contextual text continuation have become increasingly popular topics in NLP community. Existing models are often prone to output paragraphs of texts that gradually diverge from the given prompt. Although the generated text may have a reasonable perplexity and diversity, it could easily be identified by human as gibberish. The goal of our project is to improve the coherence and consistency across sentences in a language-generation model. We aim to solve this issue by first training a sentence pair coherence classifier with GPT-2 pretrained model, and then co-train the GPT-2 language model with this new coherence objective using a method analogous to the REINFORCE algorithm. This fine-tuned language model is able to generate lengthy paragraph conditioned on a given topic without diverging too much. The simplicity of this model allows it to be applicable to a variety of underlying language model architecture since it only modifies the final layer of the pre-trained model.
