论文标题
朝着理论分析relu dnns的转化复杂性
Towards Theoretical Analysis of Transformation Complexity of ReLU DNNs
论文作者
论文摘要
本文的目的是从理论上分析具有relu层的分段线性DNN中编码的特征转换的复杂性。我们建议指标根据信息理论衡量转换的三种复杂性。我们进一步发现并证明了转换的复杂性与分离之间的密切相关性。根据提议的指标，我们分析了训练过程中转化复杂性变化的两个典型现象，并探索DNN复杂性的上限。所提出的指标也可以用作学习最小复杂性的DNN的损失，这也控制DNN的过度拟合水平并影响对抗性的鲁棒性，对抗性转移性和知识一致性。全面的比较研究为了解DNN提供了新的观点。
This paper aims to theoretically analyze the complexity of feature transformations encoded in piecewise linear DNNs with ReLU layers. We propose metrics to measure three types of complexities of transformations based on the information theory. We further discover and prove the strong correlation between the complexity and the disentanglement of transformations. Based on the proposed metrics, we analyze two typical phenomena of the change of the transformation complexity during the training process, and explore the ceiling of a DNN's complexity. The proposed metrics can also be used as a loss to learn a DNN with the minimum complexity, which also controls the over-fitting level of the DNN and influences adversarial robustness, adversarial transferability, and knowledge consistency. Comprehensive comparative studies have provided new perspectives to understand the DNN.
