论文标题
定量突触稀释增强了深网中稀疏编码和辍学的正则化
Quantal synaptic dilution enhances sparse encoding and dropout regularisation in deep networks
论文作者
论文摘要
辍学是一种在随机训练深层网络以减少过度拟合的同时，将单位的活性沉默。在这里，我们介绍了基于神经元突触的量化特性的生物学上合理化的脱落正则化模型（QSD），该模型具有神经元突触的量化特性，该模型将异质性纳入了响应大小和囊泡量的释放概率。 QSD优于Relu多层感知器中的标准辍学，在测试时间稀疏编码增强时，辍学掩码被身份函数代替，而无需训练重量或偏置分布的变化。对于卷积网络，该方法还改善了计算机视觉任务的概括，而有或没有其他形式的正则化。 QSD还胜过反复网络中的标准辍学，用于语言建模和情感分析。 QSD比辍学的许多变化的优点是，它可以在适用标准辍学的所有常规深网中实现。
Dropout is a technique that silences the activity of units stochastically while training deep networks to reduce overfitting. Here we introduce Quantal Synaptic Dilution (QSD), a biologically plausible model of dropout regularisation based on the quantal properties of neuronal synapses, that incorporates heterogeneities in response magnitudes and release probabilities for vesicular quanta. QSD outperforms standard dropout in ReLU multilayer perceptrons, with enhanced sparse encoding at test time when dropout masks are replaced with identity functions, without shifts in trainable weight or bias distributions. For convolutional networks, the method also improves generalisation in computer vision tasks with and without inclusion of additional forms of regularisation. QSD also outperforms standard dropout in recurrent networks for language modelling and sentiment analysis. An advantage of QSD over many variations of dropout is that it can be implemented generally in all conventional deep networks where standard dropout is applicable.
