论文标题
要了解预先训练和微调的语言模型中的大规模话语结构
Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Models
论文作者
论文摘要
随着越来越多的伯语工作分析了预训练的语言模型的不同组成部分，我们通过对预先训练和精细的语言模型中的话语信息进行深入分析来扩展这一研究。我们超越了先前的工作三个维度：首先，我们描述了一种从任意长文档中推断出话语结构的新方法。其次，我们提出了一种新型的分析，以探讨BERT和BART模型中在何处以及如何准确地捕获何处。最后，我们评估了生成的结构与各种基线的相似性以及它们在模型之间的分布。
With a growing number of BERTology work analyzing different components of pre-trained language models, we extend this line of research through an in-depth analysis of discourse information in pre-trained and fine-tuned language models. We move beyond prior work along three dimensions: First, we describe a novel approach to infer discourse structures from arbitrarily long documents. Second, we propose a new type of analysis to explore where and how accurately intrinsic discourse is captured in the BERT and BART models. Finally, we assess how similar the generated structures are to a variety of baselines as well as their distribution within and between models.
