论文标题
L-SVRG和L-katyusha，具有自适应抽样
L-SVRG and L-Katyusha with Adaptive Sampling
论文作者
论文摘要
基于随机梯度的优化方法，例如L-SVRG及其加速变体L-katyusha（Kovalev等，2020），被广泛用于训练机器学习模型。L-SVRG和L-katyusha的理论和经验性能可以通过采样，可以通过采样的分配来提高分布（Qian and a norry-Stian an）（QIAN）（QIAN）。但是，设计所需的采样分布需要对平滑度常数的先验知识，当模型参数的尺寸较高时，在实践中可以在计算上棘手。为了解决这个问题，我们为L-SVRG和L-katyusha提出了一种自适应抽样策略，可以在很少的计算开销中学习采样分布，同时允许其随着迭代方式进行更改，同时不需要任何问题参数的任何先验知识。当采样分布随迭代而变化时，我们证明L-SVRG和L-Katyusha的融合保证。我们的结果表明，即使没有先前的信息，提出的自适应采样策略匹配，在某些情况下，在Qian等人中，采样方案的性能甚至超过了。 （2021）。广泛的模拟支持我们的理论和对真实数据提出的抽样方案的实际实用性。
Stochastic gradient-based optimization methods, such as L-SVRG and its accelerated variant L-Katyusha (Kovalev et al., 2020), are widely used to train machine learning models.The theoretical and empirical performance of L-SVRG and L-Katyusha can be improved by sampling observations from a non-uniform distribution (Qian et al., 2021). However,designing a desired sampling distribution requires prior knowledge of smoothness constants, which can be computationally intractable to obtain in practice when the dimension of the model parameter is high. To address this issue, we propose an adaptive sampling strategy for L-SVRG and L-Katyusha that can learn the sampling distribution with little computational overhead, while allowing it to change with iterates, and at the same time does not require any prior knowledge of the problem parameters. We prove convergence guarantees for L-SVRG and L-Katyusha for convex objectives when the sampling distribution changes with iterates. Our results show that even without prior information, the proposed adaptive sampling strategy matches, and in some cases even surpasses, the performance of the sampling scheme in Qian et al. (2021). Extensive simulations support our theory and the practical utility of the proposed sampling scheme on real data.
