论文标题
有效的有条件预训练用于转移学习
Efficient Conditional Pre-training for Transfer Learning
论文作者
论文摘要
几乎所有用于计算机视觉任务的最新神经网络都通过（1）在大规模数据集中进行预训练和（2）在目标数据集上进行填充的培训。该策略有助于减少对目标数据集的依赖，并提高目标任务的收敛率和概括。尽管大规模数据集的预培训非常有用，但其最重要的缺点是高训练成本。为了解决这个问题，我们提出了有效的过滤方法，以从预训练数据集中选择相关子集。此外，我们发现在训练前的步骤中降低图像分辨率在成本和绩效之间提供了一个很好的权衡。我们通过在无监督和监督的设置中对Imagenet进行预训练以及在各种目标数据集和任务集合中进行填充来验证我们的技术。我们提出的方法大大降低了培训前成本，并提供了强大的绩效提高。最后，我们通过在我们的子集上调整可用模型并在从较大规模的数据集中过滤的数据集中进行预培训，从而将标准图像预训练提高了1-3％。
Almost all the state-of-the-art neural networks for computer vision tasks are trained by (1) pre-training on a large-scale dataset and (2) finetuning on the target dataset. This strategy helps reduce dependence on the target dataset and improves convergence rate and generalization on the target task. Although pre-training on large-scale datasets is very useful, its foremost disadvantage is high training cost. To address this, we propose efficient filtering methods to select relevant subsets from the pre-training dataset. Additionally, we discover that lowering image resolutions in the pre-training step offers a great trade-off between cost and performance. We validate our techniques by pre-training on ImageNet in both the unsupervised and supervised settings and finetuning on a diverse collection of target datasets and tasks. Our proposed methods drastically reduce pre-training cost and provide strong performance boosts. Finally, we improve standard ImageNet pre-training by 1-3% by tuning available models on our subsets and pre-training on a dataset filtered from a larger scale dataset.
