论文标题
关于线性逆问题的随机梯度下降的饱和现象
On the Saturation Phenomenon of Stochastic Gradient Descent for Linear Inverse Problems
论文作者
论文摘要
随机梯度下降（SGD）是解决大规模逆问题的有前途方法，这是由于其相对于数据大小的出色可伸缩性。正则化理论镜头中的当前数学理论预测，具有多项式腐烂的步骤尺寸时间表的SGD可能会遭受不良饱和现象的障碍，即，当它超出一定范围时，收敛速率不会随着解决方案规则指数而进一步提高。在这项工作中，我们提出了SGD的精制收敛率分析，并证明如果时间表的初始步骤足够小，实际上不会发生饱和度。提供了几个数值实验以补充分析。
Stochastic gradient descent (SGD) is a promising method for solving large-scale inverse problems, due to its excellent scalability with respect to data size. The current mathematical theory in the lens of regularization theory predicts that SGD with a polynomially decaying stepsize schedule may suffer from an undesirable saturation phenomenon, i.e., the convergence rate does not further improve with the solution regularity index when it is beyond a certain range. In this work, we present a refined convergence rate analysis of SGD, and prove that saturation actually does not occur if the initial stepsize of the schedule is sufficiently small. Several numerical experiments are provided to complement the analysis.
