论文标题
通过数据独立函数表征隐式正则化的限制
Limitation of Characterizing Implicit Regularization by Data-independent Functions
论文作者
论文摘要
近年来，了解神经网络（NNS）的隐式正规化已成为深度学习理论的核心任务。但是，隐式正规化本身并未完全定义和理解。在这项工作中，我们试图数学上定义和研究隐式正则化。重要的是，我们探讨了使用独立于数据的函数来表征隐式正则化的常见方法的局限性。我们提出了两种动态机制，即两点和单点重叠的机制，基于它们，我们提供了两种食谱，用于生产一类隐藏的nnns类，这些食谱不能证明是无法完全表征的，这些食谱不能完全表征或所有独立于数据的功能。在先前的工作之后，我们的结果进一步强调了隐性正则化的深刻数据依赖性，激发了我们详细研究NN隐式正则化的数据依赖性。
In recent years, understanding the implicit regularization of neural networks (NNs) has become a central task in deep learning theory. However, implicit regularization is itself not completely defined and well understood. In this work, we attempt to mathematically define and study implicit regularization. Importantly, we explore the limitations of a common approach to characterizing implicit regularization using data-independent functions. We propose two dynamical mechanisms, i.e., Two-point and One-point Overlapping mechanisms, based on which we provide two recipes for producing classes of one-hidden-neuron NNs that provably cannot be fully characterized by a type of or all data-independent functions. Following the previous works, our results further emphasize the profound data dependency of implicit regularization in general, inspiring us to study in detail the data dependency of NN implicit regularization in the future.
