论文标题
无监督和几乎没有解析的语言模型
Unsupervised and Few-shot Parsing from Pretrained Language Models
论文作者
论文摘要
通常认为语言模型能够编码语法[Tenney等，2019； Jawahar等，2019； Hewitt和Manning，2019]。在本文中，我们提出了一个无监督的成分解析模型UPOA，该模型仅基于以验证的语言模型为跨度分割的语法距离所学的自我发场权重矩阵计算出OUT关联得分。我们进一步提出了一个增强的版本UPIO，该版本利用了内部关联和外部关联分数来估计跨度的可能性。 UPOA和UPIO的实验揭示了自我注意机制中查询和密钥的线性投影矩阵在解析中起重要作用。因此，我们将无监督的模型扩展到了几个射击模型（FPOA，FPIO），这些模型使用一些注释的树来学习更好的线性投影矩阵进行解析。宾夕法尼亚州立树仓上的实验表明，我们的无监督解析模型UPIO实现了与短句子（长度<= 10）相当的结果。我们的几次解析模型FPIO只有20棵带注释的树木训练的训练，优于以前的几种解析方法，该方法接受了50棵带注释的树木的训练。跨语性解析的实验表明，无监督和少数解析方法都比SPMRL大多数语言的先前方法都更好[Seddah等，2013]。
Pretrained language models are generally acknowledged to be able to encode syntax [Tenney et al., 2019, Jawahar et al., 2019, Hewitt and Manning, 2019]. In this article, we propose UPOA, an Unsupervised constituent Parsing model that calculates an Out Association score solely based on the self-attention weight matrix learned in a pretrained language model as the syntactic distance for span segmentation. We further propose an enhanced version, UPIO, which exploits both inside association and outside association scores for estimating the likelihood of a span. Experiments with UPOA and UPIO disclose that the linear projection matrices for the query and key in the self-attention mechanism play an important role in parsing. We therefore extend the unsupervised models to few-shot parsing models (FPOA, FPIO) that use a few annotated trees to learn better linear projection matrices for parsing. Experiments on the Penn Treebank demonstrate that our unsupervised parsing model UPIO achieves results comparable to the state of the art on short sentences (length <= 10). Our few-shot parsing model FPIO trained with only 20 annotated trees outperforms a previous few-shot parsing method trained with 50 annotated trees. Experiments on cross-lingual parsing show that both unsupervised and few-shot parsing methods are better than previous methods on most languages of SPMRL [Seddah et al., 2013].
