论文标题
通过对抗训练的语言模型，有效的无监督域适应
Effective Unsupervised Domain Adaptation with Adversarially Trained Language Models
论文作者
论文摘要
最近的工作表明，在目标任务的领域，适应宽覆盖的上下文嵌入模型的重要性。当前的自我监督适应方法很简单，因为训练信号来自\ emph {随机}掩盖令牌的一小部分。在本文中，我们表明，仔细的掩蔽策略可以通过分配需要的自我选择来更有效地弥合掩盖语言模型（MLMS）的知识差距。此外，我们提出了一种有效的培训策略，通过掩盖那些很难通过基础传销重建的代币。对抗性目标导致了代币的\ emph {subsets}的具有挑战性的组合优化问题，我们通过放松将其有效地解决到各种较低和动态编程。在涉及命名实体识别的六个无监督域适应任务上，我们的方法强烈胜过随机掩盖策略，并实现了+1.64 F1得分的改进。
Recent work has shown the importance of adaptation of broad-coverage contextualised embedding models on the domain of the target task of interest. Current self-supervised adaptation methods are simplistic, as the training signal comes from a small percentage of \emph{randomly} masked-out tokens. In this paper, we show that careful masking strategies can bridge the knowledge gap of masked language models (MLMs) about the domains more effectively by allocating self-supervision where it is needed. Furthermore, we propose an effective training strategy by adversarially masking out those tokens which are harder to reconstruct by the underlying MLM. The adversarial objective leads to a challenging combinatorial optimisation problem over \emph{subsets} of tokens, which we tackle efficiently through relaxation to a variational lowerbound and dynamic programming. On six unsupervised domain adaptation tasks involving named entity recognition, our method strongly outperforms the random masking strategy and achieves up to +1.64 F1 score improvements.
