论文标题
基于力矩集中式梯度下降优化器用于卷积神经网络
Moment Centralization based Gradient Descent Optimizers for Convolutional Neural Networks
论文作者
论文摘要
卷积神经网络（CNN）在许多计算机视觉应用中表现出非常吸引人的性能。通常使用基于随机梯度下降（SGD）优化技术进行CNN的训练。基于自适应动量的SGD优化器是最近的趋势。但是，现有的优化器无法在一阶时刻保持零平均值，并且在优化方面挣扎。在本文中，我们提出了针对CNN的基于集中化的SGD优化器。具体而言，我们明确地将零均值约束强加于一阶矩。提出的力矩集中化本质上是通用的，可以与任何基于自适应动量的优化器集成。提出的想法通过三种最先进的优化技术进行了测试，包括基准CIFAR10，CIFAR100和TINYIMAGENET数据集的ADAM，RADAM和ADABELIEF，用于图像分类。与建议的力矩集中化集成时，现有优化器的性能通常会提高。此外，提出的力矩集中化的结果也比现有的梯度集中化更好。使用玩具示例的分析分析表明，所提出的方法导致较短，更平滑的优化轨迹。源代码可在\ url {https://github.com/sumanthsadhu/mc-optimizer}公开可用。
Convolutional neural networks (CNNs) have shown very appealing performance for many computer vision applications. The training of CNNs is generally performed using stochastic gradient descent (SGD) based optimization techniques. The adaptive momentum-based SGD optimizers are the recent trends. However, the existing optimizers are not able to maintain a zero mean in the first-order moment and struggle with optimization. In this paper, we propose a moment centralization-based SGD optimizer for CNNs. Specifically, we impose the zero mean constraints on the first-order moment explicitly. The proposed moment centralization is generic in nature and can be integrated with any of the existing adaptive momentum-based optimizers. The proposed idea is tested with three state-of-the-art optimization techniques, including Adam, Radam, and Adabelief on benchmark CIFAR10, CIFAR100, and TinyImageNet datasets for image classification. The performance of the existing optimizers is generally improved when integrated with the proposed moment centralization. Further, The results of the proposed moment centralization are also better than the existing gradient centralization. The analytical analysis using the toy example shows that the proposed method leads to a shorter and smoother optimization trajectory. The source code is made publicly available at \url{https://github.com/sumanthsadhu/MC-optimizer}.
