论文标题
班级/跨阶层结构的痕迹遍布深度学习光谱
Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra
论文作者
论文摘要
许多研究人员最近将经验光谱分析应用于现代深度学习分类器的研究。我们识别并讨论了重要的正式阶级/跨阶级结构，并展示了它是如何在DeepNet Spectra中观察到的许多视觉引人注目的特征的起源，其中一些在最近的文章中进行了报道，其他则首次在这里揭幕。这些包括光谱异常值，“尖峰”，以及较小但独特的连续分布，“颠簸”，通常在“主要散装”的边缘超出。 通过三种方式说明了跨阶层结构的意义：（i）在多项式逻辑回归的背景下，我们证明了在Fisher Information Matrix频谱中的离群值与渔民信息谱系中的批量之比； （ii）我们演示了一个网络逐渐能够逐渐能够将类别的信息与班级变异性分开，同时逐拟定类差异信息； （iii）我们建议对KFAC进行校正，KFAC是一种众所周知的二阶优化算法，用于训练深网。
Numerous researchers recently applied empirical spectral analysis to the study of modern deep learning classifiers. We identify and discuss an important formal class/cross-class structure and show how it lies at the origin of the many visually striking features observed in deepnet spectra, some of which were reported in recent articles, others are unveiled here for the first time. These include spectral outliers, "spikes", and small but distinct continuous distributions, "bumps", often seen beyond the edge of a "main bulk". The significance of the cross-class structure is illustrated in three ways: (i) we prove the ratio of outliers to bulk in the spectrum of the Fisher information matrix is predictive of misclassification, in the context of multinomial logistic regression; (ii) we demonstrate how, gradually with depth, a network is able to separate class-distinctive information from class variability, all while orthogonalizing the class-distinctive information; and (iii) we propose a correction to KFAC, a well-known second-order optimization algorithm for training deepnets.
