论文标题
使用深层样条神经网络的Lipschitz函数近似
Approximation of Lipschitz Functions using Deep Spline Neural Networks
论文作者
论文摘要
Lipschitz受限的神经网络在机器学习中有许多应用。由于设计和培训表现力的Lipschitz受限的网络非常具有挑战性，因此需要改进方法和更好的理论理解。不幸的是，事实证明，Relu网络在这种情况下具有可证明的缺点。因此，我们建议使用至少3个线性区域使用可学习的样条激活功能。我们证明，在所有组件的$ 1 $ -LIPSCHITZ激活功能中，这种选择是最佳的，因为没有其他重量约束的体系结构可以近似更大的功能。此外，这种选择至少与最近引入的非分量组的非组成部分激活函数一样表现力。先前发表的数值结果支持我们的理论发现。
Lipschitz-constrained neural networks have many applications in machine learning. Since designing and training expressive Lipschitz-constrained networks is very challenging, there is a need for improved methods and a better theoretical understanding. Unfortunately, it turns out that ReLU networks have provable disadvantages in this setting. Hence, we propose to use learnable spline activation functions with at least 3 linear regions instead. We prove that this choice is optimal among all component-wise $1$-Lipschitz activation functions in the sense that no other weight constrained architecture can approximate a larger class of functions. Additionally, this choice is at least as expressive as the recently introduced non component-wise Groupsort activation function for spectral-norm-constrained weights. Previously published numerical results support our theoretical findings.
