论文标题
技术报告：辅助调整及其在有条件的文本生成中的应用
Technical Report: Auxiliary Tuning and its Application to Conditional Text Generation
论文作者
论文摘要
我们引入了一种简单有效的方法，称为辅助调整，用于将预训练的语言模型调整为新任务。我们在有条件文本生成的任务上演示了这种方法。我们的方法用辅助模型为原始的预训练模型提供了根据目标任务移动输出分布的辅助模型。辅助模型是通过将其逻辑添加到预训练的模型逻辑中来训练的，并最大程度地提高目标任务输出的可能性。我们的方法对辅助体系结构没有任何限制。特别是，辅助模型可以独立于预先训练的模型的输入而吸收与目标任务相关的其他输入。此外，在逻辑水平上混合模型提供了对该方法的自然概率解释。我们的方法与从头开始培训的几个不同任务的培训相似，同时使用培训的资源明显更少。我们共享一个以关键字为条件的文本生成的特定示例。
We introduce a simple and efficient method, called Auxiliary Tuning, for adapting a pre-trained Language Model to a novel task; we demonstrate this approach on the task of conditional text generation. Our approach supplements the original pre-trained model with an auxiliary model that shifts the output distribution according to the target task. The auxiliary model is trained by adding its logits to the pre-trained model logits and maximizing the likelihood of the target task output. Our method imposes no constraints on the auxiliary architecture. In particular, the auxiliary model can ingest additional input relevant to the target task, independently from the pre-trained model's input. Furthermore, mixing the models at the logits level provides a natural probabilistic interpretation of the method. Our method achieved similar results to training from scratch for several different tasks, while using significantly fewer resources for training; we share a specific example of text generation conditioned on keywords.
