论文标题
冷静地评估在线持续学习
Evaluating Online Continual Learning with CALM
论文作者
论文摘要
在线持续学习（OCL）研究通过连续数据流进行学习，而无需观察到任何一个示例以上，这种设置与必须学习“野外”的人类和系统的体验更接近。然而，通常可用的基准测试远非这些现实世界条件，因为它们明确地向不同的任务发出了信号，缺乏潜在的相似性结构或在不同示例之间假定时间独立性。在这里，我们基于语言建模为OCL提出了一个新的基准测试，其中输入在不同语言和域之间交替而没有任何明确的划界。此外，我们建议在这种情况下研究灾难性遗忘的新指标，并根据专家组成评估多个基线模型。最后，我们引入了一种简单的门控技术，该技术了解不同输入之间的潜在相似性，从而提高了专家模型的性能。
Online Continual Learning (OCL) studies learning over a continuous data stream without observing any single example more than once, a setting that is closer to the experience of humans and systems that must learn "on-the-wild". Yet, commonly available benchmarks are far from these real-world conditions, because they explicitly signal different tasks, lack latent similarity structure or assume temporal independence between different examples. Here, we propose a new benchmark for OCL based on language modelling in which input alternates between different languages and domains without any explicit delimitation. Additionally, we propose new metrics to study catastrophic forgetting in this setting and evaluate multiple baseline models based on compositions of experts. Finally, we introduce a simple gating technique that learns the latent similarities between different inputs, improving the performance of a Products of Experts model.
