论文标题
通过敏感性统计的迭代排名修剪
Pruning via Iterative Ranking of Sensitivity Statistics
论文作者
论文摘要
随着SNIP的引入[ARXIV：1810.02340V2]，已经证明现代神经网络可以在训练前有效地修剪。然而，其灵敏度标准此后一直因未正确传播训练信号甚至断开连接层而受到批评。作为一种补救措施，引入了掌握[Arxiv：2002.07376V1]，这损害了简单性。但是，在这项工作中，我们表明，通过以较小的步骤应用灵敏度标准（仍在训练之前），我们可以提高其性能而无需实施。因此，我们介绍了“ Snip-it”。然后，我们演示了如何将其用于结构化和非结构化的修剪，在培训之前和/或在培训期间，可以实现最先进的表现折衷。也就是说，尽管从一开始就已经在培训过程中提供了修剪的计算益处。此外，我们还评估了关于鲁棒性过度拟合，断开和对抗性攻击的方法。
With the introduction of SNIP [arXiv:1810.02340v2], it has been demonstrated that modern neural networks can effectively be pruned before training. Yet, its sensitivity criterion has since been criticized for not propagating training signal properly or even disconnecting layers. As a remedy, GraSP [arXiv:2002.07376v1] was introduced, compromising on simplicity. However, in this work we show that by applying the sensitivity criterion iteratively in smaller steps - still before training - we can improve its performance without difficult implementation. As such, we introduce 'SNIP-it'. We then demonstrate how it can be applied for both structured and unstructured pruning, before and/or during training, therewith achieving state-of-the-art sparsity-performance trade-offs. That is, while already providing the computational benefits of pruning in the training process from the start. Furthermore, we evaluate our methods on robustness to overfitting, disconnection and adversarial attacks as well.
