论文标题
ZORB：神经网络的无衍生反向传播算法
ZORB: A Derivative-Free Backpropagation Algorithm for Neural Networks
论文作者
论文摘要
梯度下降和反向传播使神经网络能够在许多现实世界应用中取得显着的结果。尽管取得了持续的成功，但培训具有梯度下降的神经网络可能是缓慢而艰苦的事情。我们提出了一种简单而更快的训练算法，称为零级秩序放松反向传播（ZORB）。 Zorb没有计算梯度，而是使用目标的伪源来倒退信息。 Zorb旨在减少训练深神经网络所需的时间而不会惩罚性能。为了说明速度，我们在MNIST上训练了一个带有11层的前馈神经网络，并观察到Zorb收敛的速度比ADAM快300倍，同时达到了可比的错误率，而没有任何超参数调整。我们还将Zorb的范围扩大到卷积神经网络，并将其应用于CIFAR-10数据集的子样本。关于标准分类和回归基准的实验证明了佐尔布在传统的逆向传播中具有梯度下降的优势。
Gradient descent and backpropagation have enabled neural networks to achieve remarkable results in many real-world applications. Despite ongoing success, training a neural network with gradient descent can be a slow and strenuous affair. We present a simple yet faster training algorithm called Zeroth-Order Relaxed Backpropagation (ZORB). Instead of calculating gradients, ZORB uses the pseudoinverse of targets to backpropagate information. ZORB is designed to reduce the time required to train deep neural networks without penalizing performance. To illustrate the speed up, we trained a feed-forward neural network with 11 layers on MNIST and observed that ZORB converged 300 times faster than Adam while achieving a comparable error rate, without any hyperparameter tuning. We also broaden the scope of ZORB to convolutional neural networks, and apply it to subsamples of the CIFAR-10 dataset. Experiments on standard classification and regression benchmarks demonstrate ZORB's advantage over traditional backpropagation with Gradient Descent.
