论文标题
Mehler的公式，分支过程和深神经网络的组成核
Mehler's Formula, Branching Process, and Compositional Kernels of Deep Neural Networks
论文作者
论文摘要
我们通过Mehler的公式使用构图内核和分支过程之间的联系来研究深层神经网络。这种新的概率洞察力为我们提供了关于组成神经网络中激活功能的数学作用的新观点。我们研究组成核的未量化和重新缩放限制，并随着组成深度的增加而探索限制行为的不同阶段。我们通过表征组成深度，样本量，维度和非线性的组成核和神经网络的记忆能力。提供了成分内核特征值上的显式公式，从而量化了相应繁殖核Hilbert空间的复杂性。在方法论方面，我们提出了一种新的随机特征算法，该算法通过设计新的激活函数来压缩组成层。
We utilize a connection between compositional kernels and branching processes via Mehler's formula to study deep neural networks. This new probabilistic insight provides us a novel perspective on the mathematical role of activation functions in compositional neural networks. We study the unscaled and rescaled limits of the compositional kernels and explore the different phases of the limiting behavior, as the compositional depth increases. We investigate the memorization capacity of the compositional kernels and neural networks by characterizing the interplay among compositional depth, sample size, dimensionality, and non-linearity of the activation. Explicit formulas on the eigenvalues of the compositional kernel are provided, which quantify the complexity of the corresponding reproducing kernel Hilbert space. On the methodological front, we propose a new random features algorithm, which compresses the compositional layers by devising a new activation function.
