论文标题
神经网络宽度和深度的准等效性
Quasi-Equivalence of Width and Depth of Neural Networks
论文作者
论文摘要
尽管经典研究证明了宽网络允许通用近似，但深度学习的最新研究和成功表明了深网的力量。基于对称的考虑，我们研究了人工神经网络的设计是否应具有方向偏好，以及在网络的宽度和深度之间的相互作用机理。受摩根法律的启发，我们通过在两个方面建立relu网络的宽度和深度之间的准等效性来解决这个基本问题。首先，我们制定了两个转换，以分别将任意的Relu网络映射到宽网络和深层网络，以进行回归或分类，以便可以实现原始网络的基本相同功能。然后，我们用二次对应物替换主流人工神经元类型，并利用同一多项式函数的分解和持续分数表示，分别构建宽网络和深层网络。根据我们的发现，深层网络具有广泛的等效性，反之亦然，但遇到了一个任意小的错误。
While classic studies proved that wide networks allow universal approximation, recent research and successes of deep learning demonstrate the power of deep networks. Based on a symmetric consideration, we investigate if the design of artificial neural networks should have a directional preference, and what the mechanism of interaction is between the width and depth of a network. Inspired by the De Morgan law, we address this fundamental question by establishing a quasi-equivalence between the width and depth of ReLU networks in two aspects. First, we formulate two transforms for mapping an arbitrary ReLU network to a wide network and a deep network respectively for either regression or classification so that the essentially same capability of the original network can be implemented. Then, we replace the mainstream artificial neuron type with a quadratic counterpart, and utilize the factorization and continued fraction representations of the same polynomial function to construct a wide network and a deep network, respectively. Based on our findings, a deep network has a wide equivalent, and vice versa, subject to an arbitrarily small error.
