论文标题
达到通用近似的神经网络的最小宽度
Achieve the Minimum Width of Neural Networks for Universal Approximation
论文作者
论文摘要
神经网络的通用近似特性（UAP）对于深度学习至关重要，众所周知，广泛的神经网络是$ l^p $ norm和连续/统一规范中连续功能的通用近似值。但是，确切的最小宽度，$ w _ {\ min} $，尚未对UAP进行彻底研究。最近，使用解码器 - 莫莫尔语编码方案，\ citet {park2021mimine}发现，$ w _ {\ min} = \ max（d_x+1，d_y）$均用于$ l^p $ relu Networks和$ c $ c $ co $ relu+step news $ d_ $ d_ d_ d_ d_ d_ d_ d_ d_ d_ d_ d_ d_ deyy，dyy intut，d yy d_ deyy intut，在本文中，我们考虑具有任意激活功能的神经网络。我们证明，紧凑型域上功能的$ c $ -uap和$ l^p $ -uap共享最小宽度的通用下限；也就是说，$ w^*_ {\ min} = \ max（d_x，d_y）$。特别是，只要输入或输出尺寸大于一个，就可以通过泄漏的relu网络来实现临界宽度，$ w^*_ {\ min} $，可以通过泄漏的relu网络来实现。我们的构建基于神经普通微分方程的近似能力以及通过神经网络近似流量图的能力。还讨论了非单身酮或不连续的激活函数和一维情况。
The universal approximation property (UAP) of neural networks is fundamental for deep learning, and it is well known that wide neural networks are universal approximators of continuous functions within both the $L^p$ norm and the continuous/uniform norm. However, the exact minimum width, $w_{\min}$, for the UAP has not been studied thoroughly. Recently, using a decoder-memorizer-encoder scheme, \citet{Park2021Minimum} found that $w_{\min} = \max(d_x+1,d_y)$ for both the $L^p$-UAP of ReLU networks and the $C$-UAP of ReLU+STEP networks, where $d_x,d_y$ are the input and output dimensions, respectively. In this paper, we consider neural networks with an arbitrary set of activation functions. We prove that both $C$-UAP and $L^p$-UAP for functions on compact domains share a universal lower bound of the minimal width; that is, $w^*_{\min} = \max(d_x,d_y)$. In particular, the critical width, $w^*_{\min}$, for $L^p$-UAP can be achieved by leaky-ReLU networks, provided that the input or output dimension is larger than one. Our construction is based on the approximation power of neural ordinary differential equations and the ability to approximate flow maps by neural networks. The nonmonotone or discontinuous activation functions case and the one-dimensional case are also discussed.
