论文标题
证书：对比度的自我监督学习，以了解语言理解
CERT: Contrastive Self-supervised Learning for Language Understanding
论文作者
论文摘要
GPT审计的语言模型（例如Bert）在语言理解中表现出了很大的有效性。在代币上大部分定义了现有预训练方法中的辅助预测任务，因此可能无法很好地捕获句子级的语义。为了解决这个问题，我们提出了证书：来自变形金刚的对比度自我监督的编码器表示，这些编码器表示，该语言表示在句子级别使用对比性的自我监督学习预处理模型。 CERT使用背面翻译创建原始句子的增强。然后，它通过预测两个增强句子是否源于同一句子来列出验证的语言编码器（例如BERT）。 CERT易于使用，可以灵活地插入任何预处理的NLP管道中。我们在胶水基准中评估了11个自然语言理解任务的证书，在胶水基准中，证书在7个任务上的表现优于BERT，在2个任务上取得了与Bert相同的性能，并且在2个任务上的表现要比BERT差。关于11个任务的平均得分，CERT的表现优于Bert。数据和代码可从https://github.com/ucsd-ai4h/cert获得
Pretrained language models such as BERT, GPT have shown great effectiveness in language understanding. The auxiliary predictive tasks in existing pretraining approaches are mostly defined on tokens, thus may not be able to capture sentence-level semantics very well. To address this issue, we propose CERT: Contrastive self-supervised Encoder Representations from Transformers, which pretrains language representation models using contrastive self-supervised learning at the sentence level. CERT creates augmentations of original sentences using back-translation. Then it finetunes a pretrained language encoder (e.g., BERT) by predicting whether two augmented sentences originate from the same sentence. CERT is simple to use and can be flexibly plugged into any pretraining-finetuning NLP pipeline. We evaluate CERT on 11 natural language understanding tasks in the GLUE benchmark where CERT outperforms BERT on 7 tasks, achieves the same performance as BERT on 2 tasks, and performs worse than BERT on 2 tasks. On the averaged score of the 11 tasks, CERT outperforms BERT. The data and code are available at https://github.com/UCSD-AI4H/CERT
