论文标题
排名神经检查点
Ranking Neural Checkpoints
论文作者
论文摘要
本文涉及将许多预训练的深神经网络（DNN）（称为CheckPoints）对转移到下游任务进行排名。由于广泛使用了DNN，我们可能很容易从各种来源收集数百个检查站。他们中的哪一个将最好的人转移到我们感兴趣的下游任务？为了彻底回答这个问题，我们建立了一个神经检查点排名基准（Neucrab），并研究一些直观的排名措施。这些措施是通用的，适用于不同输出类型的检查点，而不知道检查点是如何对哪个数据集进行预训练的。他们还产生了低计算成本，使它们实际上有意义。我们的结果表明，检查点提取的特征的线性可分离性是可传递性的强烈指标。我们还达到了一种新的排名NLEEP，这在实验中带来了最佳性能。
This paper is concerned with ranking many pre-trained deep neural networks (DNNs), called checkpoints, for the transfer learning to a downstream task. Thanks to the broad use of DNNs, we may easily collect hundreds of checkpoints from various sources. Which of them transfers the best to our downstream task of interest? Striving to answer this question thoroughly, we establish a neural checkpoint ranking benchmark (NeuCRaB) and study some intuitive ranking measures. These measures are generic, applying to the checkpoints of different output types without knowing how the checkpoints are pre-trained on which dataset. They also incur low computation cost, making them practically meaningful. Our results suggest that the linear separability of the features extracted by the checkpoints is a strong indicator of transferability. We also arrive at a new ranking measure, NLEEP, which gives rise to the best performance in the experiments.
