论文标题
训练期间特征空间饱和
Feature Space Saturation during Training
论文作者
论文摘要
我们提出了层饱和度 - 一种简单的在线计算方法，用于分析神经网络中的信息处理。首先，我们表明，一层的输出可以局限于其方差矩阵的特征空间而不会丢失。我们提出了一种计算轻量级方法，用于近似训练期间的方差矩阵。从其无损特征空间的尺寸，我们得出层饱和度 - 特征空间维度和层宽度之间的比率。我们表明饱和似乎表明哪些层有助于网络性能。我们通过更改网络深度，滤波器大小和输入分辨率来改变神经网络中的层饱和度。此外，我们表明，精心挑选的输入分辨率通过在整个网络中更均匀地分布推理过程来提高网络性能。
We propose layer saturation - a simple, online-computable method for analyzing the information processing in neural networks. First, we show that a layer's output can be restricted to the eigenspace of its variance matrix without performance loss. We propose a computationally lightweight method for approximating the variance matrix during training. From the dimension of its lossless eigenspace we derive layer saturation - the ratio between the eigenspace dimension and layer width. We show that saturation seems to indicate which layers contribute to network performance. We demonstrate how to alter layer saturation in a neural network by changing network depth, filter sizes and input resolution. Furthermore, we show that well-chosen input resolution increases network performance by distributing the inference process more evenly across the network.
