论文标题
在线学习中噪声引起的变性
Noise-induced degeneration in online learning
论文作者
论文摘要
为了阐明因消失梯度引起的高原现象，我们在这里分析了多层感知器中退化的子空间附近的随机梯度下降的稳定性。在Fukumizu-Amari模型的随机梯度下降中，这是显示非平台现象的最小多层感知器，我们表明（1）吸引在多重退化的子空间中存在（2）强大的高原平稳性，它是一种强大的高原势，这是噪音诱导的范围，它是噪音诱导的范围，（3存在以最大程度地减少退化的子空间的逃生时间。预计在此处观察到的噪声引起的变性将在通过神经网络的大量机器学习中找到。
In order to elucidate the plateau phenomena caused by vanishing gradient, we herein analyse stability of stochastic gradient descent near degenerated subspaces in a multi-layer perceptron. In stochastic gradient descent for Fukumizu-Amari model, which is the minimal multi-layer perceptron showing non-trivial plateau phenomena, we show that (1) attracting regions exist in multiply degenerated subspaces, (2) a strong plateau phenomenon emerges as a noise-induced synchronisation, which is not observed in deterministic gradient descent, (3) an optimal fluctuation exists to minimise the escape time from the degenerated subspace. The noise-induced degeneration observed herein is expected to be found in a broad class of machine learning via neural networks.
