论文标题
在不规则采样的时间序列中学习长期依赖项
Learning Long-Term Dependencies in Irregularly-Sampled Time Series
论文作者
论文摘要
具有连续时间隐藏状态的复发性神经网络（RNN）是自然而然地建模不规则采样时间序列的。但是，当输入数据具有长期依赖性时，这些模型将面临困难。我们证明，与标准RNN类似，此问题的根本原因是训练过程中梯度的消失或爆炸。这种现象是由隐藏状态的普通微分方程（ODE）表示表示的，无论ODE求解器的选择如何。我们通过基于长期短期内存（LSTM）设计新算法来提供解决方案，该算法将其内存与时间连续状态分开。这样，我们在RNN中编码一个连续的时间动态流，使其可以响应到达任意时间段的输入，同时确保通过内存路径进行恒定的误差传播。我们称这些RNN模型ODE-LSTMS。我们通过实验表明，ODE-LSTMS在具有长期依赖性的非均匀采样数据上的表现优于基于RNN的高级同行。所有代码和数据均可在https://github.com/mlech26l/ode-lstms上获得。
Recurrent neural networks (RNNs) with continuous-time hidden states are a natural fit for modeling irregularly-sampled time series. These models, however, face difficulties when the input data possess long-term dependencies. We prove that similar to standard RNNs, the underlying reason for this issue is the vanishing or exploding of the gradient during training. This phenomenon is expressed by the ordinary differential equation (ODE) representation of the hidden state, regardless of the ODE solver's choice. We provide a solution by designing a new algorithm based on the long short-term memory (LSTM) that separates its memory from its time-continuous state. This way, we encode a continuous-time dynamical flow within the RNN, allowing it to respond to inputs arriving at arbitrary time-lags while ensuring a constant error propagation through the memory path. We call these RNN models ODE-LSTMs. We experimentally show that ODE-LSTMs outperform advanced RNN-based counterparts on non-uniformly sampled data with long-term dependencies. All code and data is available at https://github.com/mlech26l/ode-lstms.
