论文标题
EIS - 一个激活功能的家族，结合了指数，ISRU和软质量
EIS -- a family of activation functions combining Exponential, ISRU, and Softplus
论文作者
论文摘要
激活功能在使用神经网络的功能学习中起关键作用。通过重复使用激活函数来实现学习函数中的非线性。多年来，已经提出了许多激活功能来提高多个任务的准确性。诸如Relu，Sigmoid，Tanh或Softplus之类的基本功能由于其简单性而受到深度学习社区的喜爱。近年来，已经提出了这些基本功能引起的几种新颖的激活功能，这些功能在某些具有挑战性的数据集中提高了精度。我们提出了一个五个超参数激活函数家族，即Eis，定义为，\ [\ frac {x（\ ln（1+e^x））^α} {\ sqrt {\ sqrt {β+γx^2}+δe^e^e^{ - θx}}}}}。 \]我们展示了来自EIS家族的激活功能的示例，这些函数的表现超过了一些众所周知的数据集和模型上广泛使用的激活功能。例如，$ \ frac {x \ ln（1+e^x）} {x+1.16e^{ - x}} $ beats relu在densenet-169中以0.89 \％击败0.24 \％，在CIFAR100数据集中inception v3中的inception v3 in Cifar100数据集中为1.13 \％inception v3，0.13，0.13，0.13，0.13 \ inception v3，0.13 \ inception v3，0.13 \ incept。在CIFAR10数据集中的SimpleNet模型中。另外，$ \ frac {x \ ln（1+e^x）} {\ sqrt {1+x^2}} $在169中以1.68 \％击败了1.68 \％，0.30 \％\％inception v3 in Intection v3 in Cifar100数据集中的Inception v3 in Cifar100数据集中为1.0 \％\％inception v3，0.15 in inceent v3，0.15 n in contect inceent v3，0.15 n in inceent v3，0.15 n in conceent v3，0.15 n in incement v3，0.15 n in Chore。 CIFAR10数据集中的SimpleNet模型。
Activation functions play a pivotal role in the function learning using neural networks. The non-linearity in the learned function is achieved by repeated use of the activation function. Over the years, numerous activation functions have been proposed to improve accuracy in several tasks. Basic functions like ReLU, Sigmoid, Tanh, or Softplus have been favorite among the deep learning community because of their simplicity. In recent years, several novel activation functions arising from these basic functions have been proposed, which have improved accuracy in some challenging datasets. We propose a five hyper-parameters family of activation functions, namely EIS, defined as, \[ \frac{x(\ln(1+e^x))^α}{\sqrt{β+γx^2}+δe^{-θx}}. \] We show examples of activation functions from the EIS family which outperform widely used activation functions on some well known datasets and models. For example, $\frac{x\ln(1+e^x)}{x+1.16e^{-x}}$ beats ReLU by 0.89\% in DenseNet-169, 0.24\% in Inception V3 in CIFAR100 dataset while 1.13\% in Inception V3, 0.13\% in DenseNet-169, 0.94\% in SimpleNet model in CIFAR10 dataset. Also, $\frac{x\ln(1+e^x)}{\sqrt{1+x^2}}$ beats ReLU by 1.68\% in DenseNet-169, 0.30\% in Inception V3 in CIFAR100 dataset while 1.0\% in Inception V3, 0.15\% in DenseNet-169, 1.13\% in SimpleNet model in CIFAR10 dataset.
