论文标题
使用光谱方法从复发性神经网络中蒸馏加权自动机
Distillation of Weighted Automata from Recurrent Neural Networks using a Spectral Approach
论文作者
论文摘要
本文试图弥合深度学习与语法推论之间的差距。确实，它提供了一种算法，可以从培训用于语言建模的任何经常性神经网络中提取（随机）形式的语言。从详细的角度来看，该算法将已经训练的网络用作甲骨文 - 因此不需要访问黑框的内部表示 - 并采用光谱方法来推断加权自动机。 作为加权自动机计算线性函数，它们在计算上比神经网络更有效，因此该方法的性质是知识蒸馏之一。我们详细介绍了62个数据集（合成和来自现实世界应用）的实验，这些数据集允许对所提出算法的能力进行深入研究。结果表明，我们提取的WA是RNN的良好近似值，从而验证了该方法。此外，我们展示了该过程如何为RNN在数据上学习的行为提供有趣的见解，将这项工作的范围扩大到了深度学习模型的解释性之一。
This paper is an attempt to bridge the gap between deep learning and grammatical inference. Indeed, it provides an algorithm to extract a (stochastic) formal language from any recurrent neural network trained for language modelling. In detail, the algorithm uses the already trained network as an oracle -- and thus does not require the access to the inner representation of the black-box -- and applies a spectral approach to infer a weighted automaton. As weighted automata compute linear functions, they are computationally more efficient than neural networks and thus the nature of the approach is the one of knowledge distillation. We detail experiments on 62 data sets (both synthetic and from real-world applications) that allow an in-depth study of the abilities of the proposed algorithm. The results show the WA we extract are good approximations of the RNN, validating the approach. Moreover, we show how the process provides interesting insights toward the behavior of RNN learned on data, enlarging the scope of this work to the one of explainability of deep learning models.
