论文标题
对结构修剪的神经元的重要性恢复
Receding Neuron Importances for Structured Pruning
论文作者
论文摘要
结构化修剪通过识别和去除不重要的神经元有效地压缩网络。虽然可以通过在batchNorm参数上应用稀疏性诱导正则化来优雅地实现，但L1惩罚将缩小所有缩放因子，而不仅仅是多余的神经元的缩放因素。为了解决这个问题，我们引入了一个简单的批度变化，具有有界缩放参数，基于我们设计一个新颖的正则化项，该术语仅抑制重要性较低的神经元。在我们的方法下，不必要的神经元的权重有效地退缩，产生了两极化的重要性分布。我们表明，以这种方式训练的神经网络可以在更大程度上修剪，并且恶化较小。我们以CIFAR和Imagennet数据集的不同比率为不同比例的resnet体系结构。就VGG风格的网络而言，我们的方法极大地胜过现有方法，尤其是在严重的修剪制度下。
Structured pruning efficiently compresses networks by identifying and removing unimportant neurons. While this can be elegantly achieved by applying sparsity-inducing regularisation on BatchNorm parameters, an L1 penalty would shrink all scaling factors rather than just those of superfluous neurons. To tackle this issue, we introduce a simple BatchNorm variation with bounded scaling parameters, based on which we design a novel regularisation term that suppresses only neurons with low importance. Under our method, the weights of unnecessary neurons effectively recede, producing a polarised bimodal distribution of importances. We show that neural networks trained this way can be pruned to a larger extent and with less deterioration. We one-shot prune VGG and ResNet architectures at different ratios on CIFAR and ImagenNet datasets. In the case of VGG-style networks, our method significantly outperforms existing approaches particularly under a severe pruning regime.
