论文标题
使用层次塔克张分解来压缩复发性神经网络
Compressing Recurrent Neural Networks Using Hierarchical Tucker Tensor Decomposition
论文作者
论文摘要
复发性神经网络（RNN）已被广泛用于序列分析和建模中。但是，在处理高维数据时，RNN通常需要非常大的模型大小，从而带来了一系列的部署挑战。尽管最先进的张量分解方法可以提供良好的模型压缩性能，但这些现有方法仍处于某些固有的局限性，例如限制的表示能力和模型复杂性降低不足。为了克服这些局限性，在本文中，我们建议使用分层塔克（HT）分解开发紧凑的RNN模型。 HT分解为分解的RNN模型带来了强大的分层结构，这对于增强表示能力非常有用且重要。同时，与现有的张量分解方法相比，HT分解可提供更高的存储和计算成本降低。我们的实验结果表明，与最新的压缩RNN模型（例如TT-LSTM，TR-LSTM和BT-LSTM）相比，我们提出的基于HT的LSTM（HT-LSTM）始终同时实现并在不同数据集中同时实现压缩比和测试准确性。
Recurrent Neural Networks (RNNs) have been widely used in sequence analysis and modeling. However, when processing high-dimensional data, RNNs typically require very large model sizes, thereby bringing a series of deployment challenges. Although the state-of-the-art tensor decomposition approaches can provide good model compression performance, these existing methods are still suffering some inherent limitations, such as restricted representation capability and insufficient model complexity reduction. To overcome these limitations, in this paper we propose to develop compact RNN models using Hierarchical Tucker (HT) decomposition. HT decomposition brings strong hierarchical structure to the decomposed RNN models, which is very useful and important for enhancing the representation capability. Meanwhile, HT decomposition provides higher storage and computational cost reduction than the existing tensor decomposition approaches for RNN compression. Our experimental results show that, compared with the state-of-the-art compressed RNN models, such as TT-LSTM, TR-LSTM and BT-LSTM, our proposed HT-based LSTM (HT-LSTM), consistently achieves simultaneous and significant increases in both compression ratio and test accuracy on different datasets.
