论文标题
发现参数激活功能
Discovering Parametric Activation Functions
论文作者
论文摘要
最近的研究表明，激活功能的选择可以显着影响深度学习网络的性能。但是，新型激活功能的好处是不一致的，并且依赖于任务，因此，整流的线性单元（RELU）仍然是最常用的。本文提出了一种自动自动自定义激活功能的技术，从而可以可靠地提高性能。进化搜索用于发现该功能的一般形式，以及梯度下降，以优化网络不同部分和学习过程的参数。在CIFAR-10和CIFAR-100图像分类数据集上具有四个不同神经网络体系结构的实验表明，此方法是有效的。它既发现一般的激活函数，又发现了不同体系结构的专业功能，从而通过明显的边缘不断提高对relu和其他激活函数的准确性。因此，该方法可以用作将深度学习应用于新任务的自动优化步骤。
Recent studies have shown that the choice of activation function can significantly affect the performance of deep learning networks. However, the benefits of novel activation functions have been inconsistent and task dependent, and therefore the rectified linear unit (ReLU) is still the most commonly used. This paper proposes a technique for customizing activation functions automatically, resulting in reliable improvements in performance. Evolutionary search is used to discover the general form of the function, and gradient descent to optimize its parameters for different parts of the network and over the learning process. Experiments with four different neural network architectures on the CIFAR-10 and CIFAR-100 image classification datasets show that this approach is effective. It discovers both general activation functions and specialized functions for different architectures, consistently improving accuracy over ReLU and other activation functions by significant margins. The approach can therefore be used as an automated optimization step in applying deep learning to new tasks.
