论文标题
受人类学习启发的持续学习效率的研究
A Study on Efficiency in Continual Learning Inspired by Human Learning
论文作者
论文摘要
人类是有效的持续学习系统；我们通过有限的细胞和资源从出生中不断学习新技能。我们的学习在能力和时间方面都高度优化，而没有遭受灾难性遗忘的困扰。在这项工作中，我们研究了连续学习系统的效率，从人类学习中汲取灵感。特别是，受到睡眠机制的启发，我们使用Packnet作为案例研究评估了流行的基于修剪的持续学习算法。首先，我们确定重量冻结在没有生物学理由的情况下用于持续学习中，可能会导致$ 2 \ times $ $ $，因为用于给定性能水平的许多权重。其次，我们注意到人类白天和夜间行为与Packnet的训练和修剪阶段的相似性。我们研究一个设置，使修剪阶段获得时间预算，并确定人类迭代修剪和多个睡眠周期之间的联系。我们表明，存在迭代v.s的最佳选择。时期给出了不同的任务。
Humans are efficient continual learning systems; we continually learn new skills from birth with finite cells and resources. Our learning is highly optimized both in terms of capacity and time while not suffering from catastrophic forgetting. In this work we study the efficiency of continual learning systems, taking inspiration from human learning. In particular, inspired by the mechanisms of sleep, we evaluate popular pruning-based continual learning algorithms, using PackNet as a case study. First, we identify that weight freezing, which is used in continual learning without biological justification, can result in over $2\times$ as many weights being used for a given level of performance. Secondly, we note the similarity in human day and night time behaviors to the training and pruning phases respectively of PackNet. We study a setting where the pruning phase is given a time budget, and identify connections between iterative pruning and multiple sleep cycles in humans. We show there exists an optimal choice of iteration v.s. epochs given different tasks.
