论文标题
高参数优化：算法和应用的综述
Hyper-Parameter Optimization: A Review of Algorithms and Applications
论文作者
论文摘要
由于开发了深层神经网络，因此他们为日常生活做出了巨大贡献。机器学习提供了比人类在日常生活几乎各个方面都有能力的理性建议。但是，尽管取得了这一成就，但神经网络的设计和培训仍然具有挑战性和不可预测的程序。为了降低普通用户的技术阈值，自动参数优化（HPO）已成为学术和工业领域的流行话题。本文综述了HPO上最重要的主题。第一部分介绍了与模型培训和结构相关的关键超参数，并讨论了它们定义价值范围的重要性和方法。然后，研究重点是主要的优化算法及其适用性，涵盖了它们的效率和准确性，尤其是对于深度学习网络。这项研究接下来回顾了HPO的主要服务和工具包，比较了它们对最先进的搜索算法的支持，与主要深度学习框架的可行性以及用户设计的新模块的可扩展性。本文结论是将HPO应用于深度学习，优化算法之间的比较以及具有有限计算资源的模型评估的突出方法时存在的问题。
Since deep neural networks were developed, they have made huge contributions to everyday lives. Machine learning provides more rational advice than humans are capable of in almost every aspect of daily life. However, despite this achievement, the design and training of neural networks are still challenging and unpredictable procedures. To lower the technical thresholds for common users, automated hyper-parameter optimization (HPO) has become a popular topic in both academic and industrial areas. This paper provides a review of the most essential topics on HPO. The first section introduces the key hyper-parameters related to model training and structure, and discusses their importance and methods to define the value range. Then, the research focuses on major optimization algorithms and their applicability, covering their efficiency and accuracy especially for deep learning networks. This study next reviews major services and toolkits for HPO, comparing their support for state-of-the-art searching algorithms, feasibility with major deep learning frameworks, and extensibility for new modules designed by users. The paper concludes with problems that exist when HPO is applied to deep learning, a comparison between optimization algorithms, and prominent approaches for model evaluation with limited computational resources.
