论文标题
llm.int8（）：变压器的8位矩阵乘法规模
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
论文作者
论文摘要
大型语言模型已被广泛采用，但需要大量的GPU记忆进行推理。我们为变形金刚中的进料前进和注意力投影层开发了一个INT8矩阵乘法的过程，该过程将推断所需的记忆减少了一半，同时保留了完整的精度性能。使用我们的方法，可以加载175b参数16/32位检查点，转换为INT8，并立即使用而不会降解。通过理解和围绕变压器语言模型中高度系统的新兴特征的属性来理解和工作，这些属性主导着注意力和变形金刚预测性能。为了应对这些功能，我们开发了一个分为两部分的量化程序，llm.int8（）。我们首先将矢量量化与矩阵乘法中的每个内部产物的单独归一化常数一起使用，以量化大多数特征。但是，对于新兴的离群值，我们还包括一种新的混合精液分解方案，该方案将离群特征尺寸分离为16位矩阵乘法，而在8位中仍超过99.9％的值乘以99.9％。使用llm.int8（），我们从经验上显示，可以在LLMS中执行最多175B参数的推断，而不会降解任何性能。这个结果使此类模型更容易访问，例如，可以在带有消费者GPU的单个服务器上使用Opt-175b/Bloom。我们开源我们的软件。
Large language models have been widely adopted but require significant GPU memory for inference. We develop a procedure for Int8 matrix multiplication for feed-forward and attention projection layers in transformers, which cut the memory needed for inference by half while retaining full precision performance. With our method, a 175B parameter 16/32-bit checkpoint can be loaded, converted to Int8, and used immediately without performance degradation. This is made possible by understanding and working around properties of highly systematic emergent features in transformer language models that dominate attention and transformer predictive performance. To cope with these features, we develop a two-part quantization procedure, LLM.int8(). We first use vector-wise quantization with separate normalization constants for each inner product in the matrix multiplication, to quantize most of the features. However, for the emergent outliers, we also include a new mixed-precision decomposition scheme, which isolates the outlier feature dimensions into a 16-bit matrix multiplication while still more than 99.9% of values are multiplied in 8-bit. Using LLM.int8(), we show empirically it is possible to perform inference in LLMs with up to 175B parameters without any performance degradation. This result makes such models much more accessible, for example making it possible to use OPT-175B/BLOOM on a single server with consumer GPUs. We open-source our software.
