论文标题
无需搜索而自动化神经建筑设计
Automating Neural Architecture Design without Search
论文作者
论文摘要
神经结构搜索（NAS）是自动化深度神经体系结构设计的主流方法，近年来取得了很大的成功。但是，粘附于NAS的性能估计部分通常是昂贵的，这导致了巨大的计算需求。尽管已大量的努力致力于减轻这个痛点，但尚未达成共识，这是最佳的。在本文中，我们从新的角度研究了自动化体系结构设计，该设计无需顺序评估算法执行过程中生成的每个神经体系结构。具体而言，提出的方法是通过学习设计最先进的体系结构的高级专家的知识来构建的，然后直接根据所学知识生成新的体系结构。我们通过使用图神经网络进行链接预测实现了提出的方法，并从NAS-Bench-101获取了知识。与现有的同伴竞争对手相比，我们发现了一个竞争性的网络，成本最低。此外，我们还利用了从NAS-Bench-101中学的知识来自动化飞镖搜索领域的建筑设计，并在CIFAR10上实现了97.82％的精度，而Imagenet的Imagenet上只有76.51％的Top-1精度，仅消耗$ 2 \ times10^{-4} $ GPU天。这也证明了所提出的方法的高传递性，并可能导致在此研究方向上产生新的，更有效的范式。
Neural structure search (NAS), as the mainstream approach to automate deep neural architecture design, has achieved much success in recent years. However, the performance estimation component adhering to NAS is often prohibitively costly, which leads to the enormous computational demand. Though a large number of efforts have been dedicated to alleviating this pain point, no consensus has been made yet on which is optimal. In this paper, we study the automated architecture design from a new perspective that eliminates the need to sequentially evaluate each neural architecture generated during algorithm execution. Specifically, the proposed approach is built by learning the knowledge of high-level experts in designing state-of-the-art architectures, and then the new architecture is directly generated upon the knowledge learned. We implemented the proposed approach by using a graph neural network for link prediction and acquired the knowledge from NAS-Bench-101. Compared to existing peer competitors, we found a competitive network with minimal cost. In addition, we also utilized the learned knowledge from NAS-Bench-101 to automate architecture design in the DARTS search space, and achieved 97.82% accuracy on CIFAR10, and 76.51% top-1 accuracy on ImageNet consuming only $2\times10^{-4}$ GPU days. This also demonstrates the high transferability of the proposed approach, and can potentially lead to a new, more computationally efficient paradigm in this research direction.
