论文标题
单词相互依存关系揭示了LSTM的构成表示形式
Word Interdependence Exposes How LSTMs Compose Representations
论文作者
论文摘要
NLP中的最新工作表明，LSTM语言模型在语言数据中捕获了构图结构。为了仔细研究这些表示形式如何层次组成，我们根据内部大门的相互作用，提出了一种新颖的量子含义。为了探讨组成表示如何在训练中出现，我们对合成数据进行了简单的实验，这通过显示高相互依赖性如何损害概括来说明我们的度量。这些综合实验还说明了一个关于如何在培训过程中发现层次结构的特定假设：父母的成分依靠其子女的有效表示，而不是独立学习长期关系。我们通过对英语数据数据进行实验进一步支持这一措施，在句法上，相互依赖性更高。
Recent work in NLP shows that LSTM language models capture compositional structure in language data. For a closer look at how these representations are composed hierarchically, we present a novel measure of interdependence between word meanings in an LSTM, based on their interactions at the internal gates. To explore how compositional representations arise over training, we conduct simple experiments on synthetic data, which illustrate our measure by showing how high interdependence can hurt generalization. These synthetic experiments also illustrate a specific hypothesis about how hierarchical structures are discovered over the course of training: that parent constituents rely on effective representations of their children, rather than on learning long-range relations independently. We further support this measure with experiments on English language data, where interdependence is higher for more closely syntactically linked word pairs.
