论文标题
通过平行非线性方程求解加速前馈计算
Accelerating Feedforward Computation via Parallel Nonlinear Equation Solving
论文作者
论文摘要
在机器学习中，馈电计算，例如评估自动回归模型的神经网络或取样的计算。但是，前馈计算的顺序性质需要严格的执行顺序，并且无法通过并行计算轻松加速。为了启用并行化，我们将馈电计算的任务构建为解决非线性方程系统。然后，我们建议使用jacobi或高斯 - 西德尔定点迭代方法以及两者的混合方法找到解决方案。至关重要的是，Jacobi更新在每个方程式上独立运行，并且可以并行执行。我们的方法保证给出与原始的馈电计算完全相同的值，该计算具有减少（或相等）的可行迭代次数，因此给定足够的并行计算能力的时间缩短了时间。在实验上，我们证明了方法在加速（i）RNN的反向传播，（ii）评估登录以及（iii）在各种设置下具有2.1和26之间的加速因子的自动回归取样，以及（iii）自动回归抽样。
Feedforward computation, such as evaluating a neural network or sampling from an autoregressive model, is ubiquitous in machine learning. The sequential nature of feedforward computation, however, requires a strict order of execution and cannot be easily accelerated with parallel computing. To enable parallelization, we frame the task of feedforward computation as solving a system of nonlinear equations. We then propose to find the solution using a Jacobi or Gauss-Seidel fixed-point iteration method, as well as hybrid methods of both. Crucially, Jacobi updates operate independently on each equation and can be executed in parallel. Our method is guaranteed to give exactly the same values as the original feedforward computation with a reduced (or equal) number of parallelizable iterations, and hence reduced time given sufficient parallel computing power. Experimentally, we demonstrate the effectiveness of our approach in accelerating (i) backpropagation of RNNs, (ii) evaluation of DenseNets, and (iii) autoregressive sampling of MADE and PixelCNN++, with speedup factors between 2.1 and 26 under various settings.
