论文标题
配置器：朝向基于LSH投影的变压器
ProFormer: Towards On-Device LSH Projection Based Transformers
论文作者
论文摘要
基于文本的神经模型的核心介绍了单词表示形式，这些单词表示功能强大，但占据了许多内存，这使得部署到具有内存约束（例如移动电话，手表和物联网）的设备方面具有挑战性。为了克服这些挑战，我们介绍了基于投影的变压器体系结构，它更快，更轻，使其适合部署到内存约束设备并保留用户隐私。我们使用LSH投影层在不嵌入查找表的情况下动态生成单词表示形式，从而导致内存占地面积从O（v.d）减少到O（t），其中V是词汇大小，D是嵌入尺寸，t是LSH投影表示的维度。 我们还提出了一个局部投影注意（LPA）层，该层使用自我注意力将n LSH单词投影的输入序列转换为N/K表示的序列N/K表示序列，而O（k^2）则四次减少计算。我们在多个文本分类任务上评估了成型，并观察到了对先前最新文本的最新智障方法的改进，以进行短文本分类和长期文本分类任务的可比性。与2层BERT模型相比，配置器将嵌入的存储足迹从92.16 MB降低到1.3 kb，并且需要减少16倍的计算开销，这是非常令人印象深刻的，使其成为最快，最小的机上设备模型。
At the heart of text based neural models lay word representations, which are powerful but occupy a lot of memory making it challenging to deploy to devices with memory constraints such as mobile phones, watches and IoT. To surmount these challenges, we introduce ProFormer -- a projection based transformer architecture that is faster and lighter making it suitable to deploy to memory constraint devices and preserve user privacy. We use LSH projection layer to dynamically generate word representations on-the-fly without embedding lookup tables leading to significant memory footprint reduction from O(V.d) to O(T), where V is the vocabulary size, d is the embedding dimension size and T is the dimension of the LSH projection representation. We also propose a local projection attention (LPA) layer, which uses self-attention to transform the input sequence of N LSH word projections into a sequence of N/K representations reducing the computations quadratically by O(K^2). We evaluate ProFormer on multiple text classification tasks and observed improvements over prior state-of-the-art on-device approaches for short text classification and comparable performance for long text classification tasks. In comparison with a 2-layer BERT model, ProFormer reduced the embedding memory footprint from 92.16 MB to 1.3 KB and requires 16 times less computation overhead, which is very impressive making it the fastest and smallest on-device model.
