论文标题
对神经网络中稀疏性和鲁棒性的国家驱动的隐式建模
State-driven Implicit Modeling for Sparsity and Robustness in Neural Networks
论文作者
论文摘要
隐式模型是一种普通学习模型，它放弃了神经网络中典型的层次结构结构，而是基于``平衡''方程来定义内部状态，从而提供竞争性能和减少记忆消耗。但是，培训这些模型通常依赖于昂贵的隐性区分来向后传播。在这项工作中，我们提出了一种新的方法来训练隐式模型，称为州驱动的隐式建模（SIM），在其中，我们限制了内部状态和输出以匹配基线模型的模型，从而规避了昂贵的落后计算。训练问题通过构造变为凸，由于其可分解的结构，可以平行解决。我们演示了如何应用SIM卡方法来显着提高稀疏性（参数降低）和在时尚界和CIFAR-100数据集中训练的基线模型的鲁棒性。
Implicit models are a general class of learning models that forgo the hierarchical layer structure typical in neural networks and instead define the internal states based on an ``equilibrium'' equation, offering competitive performance and reduced memory consumption. However, training such models usually relies on expensive implicit differentiation for backward propagation. In this work, we present a new approach to training implicit models, called State-driven Implicit Modeling (SIM), where we constrain the internal states and outputs to match that of a baseline model, circumventing costly backward computations. The training problem becomes convex by construction and can be solved in a parallel fashion, thanks to its decomposable structure. We demonstrate how the SIM approach can be applied to significantly improve sparsity (parameter reduction) and robustness of baseline models trained on FashionMNIST and CIFAR-100 datasets.
