论文标题
神经网络无法学习定期功能以及如何修复它
Neural Networks Fail to Learn Periodic Functions and How to Fix It
论文作者
论文摘要
以前的文献提供了有关如何使用现代神经网络学习定期功能的有限线索。我们从研究神经网络的外推特性的研究开始。我们通过实验证明并证明标准激活功能，例如Relu，Tanh，Sigmoid及其变体，都无法学会推断简单的周期性功能。我们假设这是由于它们缺乏“周期性”归纳偏见。为了解决这个问题，我们提出了一种新的激活，即$ x + \ sin^2（x）$，它实现了所需的周期性诱导偏见，以学习定期功能，同时维持基于Relu的激活的优化优化属性。在实验上，我们将提出的方法应用于温度和财务数据预测。
Previous literature offers limited clues on how to learn a periodic function using modern neural networks. We start with a study of the extrapolation properties of neural networks; we prove and demonstrate experimentally that the standard activations functions, such as ReLU, tanh, sigmoid, along with their variants, all fail to learn to extrapolate simple periodic functions. We hypothesize that this is due to their lack of a "periodic" inductive bias. As a fix of this problem, we propose a new activation, namely, $x + \sin^2(x)$, which achieves the desired periodic inductive bias to learn a periodic function while maintaining a favorable optimization property of the ReLU-based activations. Experimentally, we apply the proposed method to temperature and financial data prediction.
