论文标题
Riemannian自适应优化算法及其在自然语言处理中的应用
Riemannian Adaptive Optimization Algorithm and Its Application to Natural Language Processing
论文作者
论文摘要
本文提出了一种Riemannian自适应优化算法，以优化深神经网络的参数。该算法是Euclidean空间中Amsgrad和Riemannian歧管上的Ramsgrad的扩展。该算法有助于解决影响Ramsgrad的两个问题。首先是它可以直接解决Riemannian随机优化问题，而Ramsgrad只能实现低遗憾。另一个是它可以使用恒定的学习率，这在实践中可以实现。此外，我们将所提出的算法应用于Poincar {é}嵌入，该嵌入将Wordnet名词的及其闭合嵌入到多纤维空间的Poincar {é}球模型中。数值实验表明，无论学习率的初始值如何，我们的算法都可以稳定收敛到最佳解决方案，并且收敛速度比RSGD（最基本的Riemannian随机优化算法）更快。
This paper proposes a Riemannian adaptive optimization algorithm to optimize the parameters of deep neural networks. The algorithm is an extension of both AMSGrad in Euclidean space and RAMSGrad on a Riemannian manifold. The algorithm helps to resolve two issues affecting RAMSGrad. The first is that it can solve the Riemannian stochastic optimization problem directly, in contrast to RAMSGrad which only achieves a low regret. The other is that it can use constant learning rates, which makes it implementable in practice. Additionally, we apply the proposed algorithm to Poincar{é} embeddings, which embed the transitive closure of the WordNet nouns into the Poincar{é} ball model of hyperbolic space. Numerical experiments show that regardless of the initial value of the learning rate, our algorithm stably converges to the optimal solution and converges faster than RSGD, the most basic Riemannian stochastic optimization algorithm.
