论文标题
通过自适应特征合并进行知识蒸馏的课堂学习学习
Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation
论文作者
论文摘要
我们提出了一种基于深神经网络的新型类增量学习方法，该方法不断地学习新任务，记忆力有限，以存储以前的任务中的示例。我们的算法基于知识蒸馏，并提供了一种原则性的方法来维护旧模型的表示，同时有效地适应新任务。所提出的方法估计表示形式变化和由此产生的损失之间的关系增加了模型更新产生的。它使用表示形式最小化损失的上限增加，这利用了骨干模型中每个特征映射的估计重要性。基于重要性，该模型限制了重要功能的鲁棒性更新，同时允许更改较小的功能以提高灵活性。尽管以前的任务中数据的可访问性有限，但这种优化策略有效地减轻了臭名昭著的灾难性遗忘问题。实验结果表明，与标准数据集上的现有方法相比，所提出的算法的准确性提高了。代码可用。
We present a novel class incremental learning approach based on deep neural networks, which continually learns new tasks with limited memory for storing examples in the previous tasks. Our algorithm is based on knowledge distillation and provides a principled way to maintain the representations of old models while adjusting to new tasks effectively. The proposed method estimates the relationship between the representation changes and the resulting loss increases incurred by model updates. It minimizes the upper bound of the loss increases using the representations, which exploits the estimated importance of each feature map within a backbone model. Based on the importance, the model restricts updates of important features for robustness while allowing changes in less critical features for flexibility. This optimization strategy effectively alleviates the notorious catastrophic forgetting problem despite the limited accessibility of data in the previous tasks. The experimental results show significant accuracy improvement of the proposed algorithm over the existing methods on the standard datasets. Code is available.
