论文标题
概率掩盖的语言模型，能够以任意单词顺序进行自回归产生
Probabilistically Masked Language Model Capable of Autoregressive Generation in Arbitrary Word Order
论文作者
论文摘要
蒙版语言模型和自回归语言模型是两种类型的语言模型。虽然预验证的蒙版语言模型（例如Bert都压倒了自然语言理解（NLU）任务，但GPT等自回归语言模型在自然语言生成（NLG）中特别有能力。在本文中，我们为掩盖语言模型提出了一种概率掩盖方案，我们称之为概率掩盖语言模型（PMLM）。我们在名为U-PMLM的掩模比上实现了具有统一先前分布的特定PMLM。我们证明U-PMLM等同于自回归排列的语言模型。该模型的主要优点是，它以任意顺序支持文本生成的质量令人惊讶地支持传统的单向生成。此外，验证的U-PMLM在一组下游NLU任务上也表现出BERT的表现。
Masked language model and autoregressive language model are two types of language models. While pretrained masked language models such as BERT overwhelm the line of natural language understanding (NLU) tasks, autoregressive language models such as GPT are especially capable in natural language generation (NLG). In this paper, we propose a probabilistic masking scheme for the masked language model, which we call probabilistically masked language model (PMLM). We implement a specific PMLM with a uniform prior distribution on the masking ratio named u-PMLM. We prove that u-PMLM is equivalent to an autoregressive permutated language model. One main advantage of the model is that it supports text generation in arbitrary order with surprisingly good quality, which could potentially enable new applications over traditional unidirectional generation. Besides, the pretrained u-PMLM also outperforms BERT on a set of downstream NLU tasks.
