论文标题
解释和蒸馏的联合学习
Joint learning of interpretation and distillation
论文作者
论文摘要
模型解释带来的额外信任使其成为机器学习系统必不可少的一部分。但是要解释蒸馏模型的预测，可以与学生模型本身合作，或者转向其教师模型。这导致了一个更基本的问题：是否应该出于与其在同一输入的教师模型相似的原因给出类似的预测？例如，当两个模型具有截然不同的结构时，以GBDT2NN的形式截然不同时，这个问题就变得更加重要。本文对新方法进行了一项实证研究，以解释GBDT2NN的每个预测，以及如何模仿解释可以进一步改善蒸馏过程为辅助学习任务。几个基准的实验表明，所提出的方法在解释和预测方面都能提高性能。
The extra trust brought by the model interpretation has made it an indispensable part of machine learning systems. But to explain a distilled model's prediction, one may either work with the student model itself, or turn to its teacher model. This leads to a more fundamental question: if a distilled model should give a similar prediction for a similar reason as its teacher model on the same input? This question becomes even more crucial when the two models have dramatically different structure, taking GBDT2NN for example. This paper conducts an empirical study on the new approach to explaining each prediction of GBDT2NN, and how imitating the explanation can further improve the distillation process as an auxiliary learning task. Experiments on several benchmarks show that the proposed methods achieve better performance on both explanations and predictions.
