论文标题
分段线性激活基本上塑造了神经网络的损失表面
Piecewise linear activations substantially shape the loss surfaces of neural networks
论文作者
论文摘要
了解神经网络的损失表面对于理解深度学习至关重要。本文介绍了分段线性激活如何基本影响神经网络的损失表面。我们首先证明{\ IT许多神经网络的损失表面都具有无限的伪造局部最小值}，这些杂物被定义为局部最小值，其经验风险高于全球最小值。我们的结果表明，具有分段线性激活的网络与研究良好的线性神经网络具有实质性差异。对于任何具有任意深度和任意分段线性激活函数（不包括线性函数）的神经网络，此结果均可在大多数损失函数下。从本质上讲，基本假设与大多数实际情况一致，即输出层比任何隐藏层都窄。另外，通过非不同的边界将具有分段线性激活的神经网络的损耗表面分为多个平滑和多线性细胞。构建的杂种局部最小值集中在一个山谷中：它们通过连续路径相互连接，经验风险是不变的。进一步对于一个隐藏的网络，我们证明了一个单元格中的所有局部最小值构成等效类别。他们集中在山谷中；它们都是细胞中的全球最小值。
Understanding the loss surface of a neural network is fundamentally important to the understanding of deep learning. This paper presents how piecewise linear activation functions substantially shape the loss surfaces of neural networks. We first prove that {\it the loss surfaces of many neural networks have infinite spurious local minima} which are defined as the local minima with higher empirical risks than the global minima. Our result demonstrates that the networks with piecewise linear activations possess substantial differences to the well-studied linear neural networks. This result holds for any neural network with arbitrary depth and arbitrary piecewise linear activation functions (excluding linear functions) under most loss functions in practice. Essentially, the underlying assumptions are consistent with most practical circumstances where the output layer is narrower than any hidden layer. In addition, the loss surface of a neural network with piecewise linear activations is partitioned into multiple smooth and multilinear cells by nondifferentiable boundaries. The constructed spurious local minima are concentrated in one cell as a valley: they are connected with each other by a continuous path, on which empirical risk is invariant. Further for one-hidden-layer networks, we prove that all local minima in a cell constitute an equivalence class; they are concentrated in a valley; and they are all global minima in the cell.
