论文标题
大多数激活功能都可以赢得彩票，而无需过度深度
Most Activation Functions Can Win the Lottery Without Excessive Depth
论文作者
论文摘要
强有力的彩票假说突出了通过修剪训练深神经网络的潜力，这激发了人们对神经网络如何代表功能的有趣实用和理论见解。对于具有RELU激活功能的网络，已证明具有深度$ L $的目标网络可以通过随机初始化的神经网络的子网近似，该网络的子网具有将目标深度$ 2L $两倍，并且通过对数因子更宽。我们表明，深度$ L+1 $网络就足够了。该结果表明，我们可以期望在现实的，常用的深度上找到彩票，而只需要对数过度参数化。我们的新型构造方法适用于大量的激活功能，不仅限于RES。
The strong lottery ticket hypothesis has highlighted the potential for training deep neural networks by pruning, which has inspired interesting practical and theoretical insights into how neural networks can represent functions. For networks with ReLU activation functions, it has been proven that a target network with depth $L$ can be approximated by the subnetwork of a randomly initialized neural network that has double the target's depth $2L$ and is wider by a logarithmic factor. We show that a depth $L+1$ network is sufficient. This result indicates that we can expect to find lottery tickets at realistic, commonly used depths while only requiring logarithmic overparametrization. Our novel construction approach applies to a large class of activation functions and is not limited to ReLUs.
