论文标题
初始化时修剪神经网络：为什么我们缺少标记？
Pruning Neural Networks at Initialization: Why are We Missing the Mark?
论文作者
论文摘要
最近的工作探索了在初始化时修剪神经网络的可能性。我们评估这样做的建议：SNIP（Lee等人，2019年），Grasp（Wang等，2020），Synflow（Tanaka等，2020）和幅度修剪。尽管这些方法超过了随机修剪的微不足道基线，但它们仍然低于训练后的幅度修剪的准确性，我们努力了解原因。我们表明，与训练后修剪不同，将这些方法随机地洗净了每一层中的修剪或取样新的初始值保留或提高准确性。因此，这些方法做出的每次体重修剪决定可以用每层重量的修剪来代替。该属性提出了更广泛的挑战，即基础修剪启发式方法，初始化或两者兼而有之。
Recent work has explored the possibility of pruning neural networks at initialization. We assess proposals for doing so: SNIP (Lee et al., 2019), GraSP (Wang et al., 2020), SynFlow (Tanaka et al., 2020), and magnitude pruning. Although these methods surpass the trivial baseline of random pruning, they remain below the accuracy of magnitude pruning after training, and we endeavor to understand why. We show that, unlike pruning after training, randomly shuffling the weights these methods prune within each layer or sampling new initial values preserves or improves accuracy. As such, the per-weight pruning decisions made by these methods can be replaced by a per-layer choice of the fraction of weights to prune. This property suggests broader challenges with the underlying pruning heuristics, the desire to prune at initialization, or both.
