论文标题
通过受约束优化的控制稀疏性或：我如何学会停止调整处罚和爱情约束
Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraints
论文作者
论文摘要
训练有素的神经网络的性能至关重要。加上深度学习模型的不断增长的规模，这种观察激发了对学习稀疏模型的广泛研究。在这项工作中，我们专注于控制稀疏学习时控制稀疏度的任务。基于稀疏性惩罚的现有方法涉及对罚款因素的昂贵试用和错误调整，因此缺乏直接控制所得模型的稀疏性。作为响应，我们采用了一个约束的公式：使用Louizos等人提出的栅极机制。 （2018年），我们制定了一个受约束的优化问题，其中稀疏以训练目标和所需的稀疏目标以端到端的方式指导。使用WIDERESNET和RESNET {18，50}模型对CIFAR- {10，100}，Tinyimagenet和Imagenet进行实验验证了我们的建议的有效性，并证明我们可以可靠地实现预定的稀疏目标，而不会影响预测性能。
The performance of trained neural networks is robust to harsh levels of pruning. Coupled with the ever-growing size of deep learning models, this observation has motivated extensive research on learning sparse models. In this work, we focus on the task of controlling the level of sparsity when performing sparse learning. Existing methods based on sparsity-inducing penalties involve expensive trial-and-error tuning of the penalty factor, thus lacking direct control of the resulting model sparsity. In response, we adopt a constrained formulation: using the gate mechanism proposed by Louizos et al. (2018), we formulate a constrained optimization problem where sparsification is guided by the training objective and the desired sparsity target in an end-to-end fashion. Experiments on CIFAR-{10, 100}, TinyImageNet, and ImageNet using WideResNet and ResNet{18, 50} models validate the effectiveness of our proposal and demonstrate that we can reliably achieve pre-determined sparsity targets without compromising on predictive performance.
