论文标题
深度学习及其应用中的知识蒸馏
Knowledge Distillation in Deep Learning and its Applications
论文作者
论文摘要
基于深度学习的模型相对较大，并且很难在资源有限的设备（例如手机和嵌入式设备）上部署此类模型。一种可能的解决方案是知识蒸馏，从而通过利用较大模型（教师模型）的信息来训练较小的模型（学生模型）。在本文中，我们介绍了应用于深度学习模型的知识蒸馏技术的调查。为了比较不同技术的性能，我们提出了一种称为蒸馏度量的新指标。蒸馏度量根据大小和准确性得分比较不同的知识蒸馏算法。根据调查，在本文中得出了一些有趣的结论。
Deep learning based models are relatively large, and it is hard to deploy such models on resource-limited devices such as mobile phones and embedded devices. One possible solution is knowledge distillation whereby a smaller model (student model) is trained by utilizing the information from a larger model (teacher model). In this paper, we present a survey of knowledge distillation techniques applied to deep learning models. To compare the performances of different techniques, we propose a new metric called distillation metric. Distillation metric compares different knowledge distillation algorithms based on sizes and accuracy scores. Based on the survey, some interesting conclusions are drawn and presented in this paper.
