论文标题
如何培训您的超级网络：分析体重分享NAS的训练启发式方法
How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAS
论文作者
论文摘要
重量共享承诺即使在商品硬件上也可以使神经体系结构搜索（NAS）进行处理。该领域的现有方法依靠一组启发式方法来设计和培训共享权重的骨干网络，即超级网络。由于启发式方法和超参数在不同的方法中有很大不同，因此只能通过系统地分析这些因素的影响才能实现它们之间的公平比较。因此，在本文中，我们对经常通过重量共享NAS算法使用的启发式和超参数进行系统评估。我们的分析发现，一些通常用于超级网络训练的启发式方法会对超网络和独立性能之间的相关性产生负面影响，并证明某些超参数和建筑选择的强烈影响。我们的代码和实验树立了强大而可重复的基线，未来的作品可以基础。
Weight sharing promises to make neural architecture search (NAS) tractable even on commodity hardware. Existing methods in this space rely on a diverse set of heuristics to design and train the shared-weight backbone network, a.k.a. the super-net. Since heuristics and hyperparameters substantially vary across different methods, a fair comparison between them can only be achieved by systematically analyzing the influence of these factors. In this paper, we therefore provide a systematic evaluation of the heuristics and hyperparameters that are frequently employed by weight-sharing NAS algorithms. Our analysis uncovers that some commonly-used heuristics for super-net training negatively impact the correlation between super-net and stand-alone performance, and evidences the strong influence of certain hyperparameters and architectural choices. Our code and experiments set a strong and reproducible baseline that future works can build on.
