论文标题
游行：一个新数据集，用于识别计算机科学领域知识
PARADE: A New Dataset for Paraphrase Identification Requiring Computer Science Domain Knowledge
论文作者
论文摘要
我们提出了一个新的基准数据集，称为Parade用于释义标识，需要专门的域知识。游行包含在词汇和句法水平上几乎重叠的释义，但基于计算机科学领域知识的语义等同于语义等效，并且非参数在词汇和句法水平上极大地重叠，但基于该领域知识的语义上并不等于语义。实验表明，最新的神经模型和非专业人类注释者在游行中的性能较差。例如，伯特（Bert）在微调后达到0.709的F1分数，这远低于其在其他释义识别数据集上的性能。游行可以作为对结合域知识的测试模型感兴趣的研究人员的资源。我们可以免费提供数据和代码。
We present a new benchmark dataset called PARADE for paraphrase identification that requires specialized domain knowledge. PARADE contains paraphrases that overlap very little at the lexical and syntactic level but are semantically equivalent based on computer science domain knowledge, as well as non-paraphrases that overlap greatly at the lexical and syntactic level but are not semantically equivalent based on this domain knowledge. Experiments show that both state-of-the-art neural models and non-expert human annotators have poor performance on PARADE. For example, BERT after fine-tuning achieves an F1 score of 0.709, which is much lower than its performance on other paraphrase identification datasets. PARADE can serve as a resource for researchers interested in testing models that incorporate domain knowledge. We make our data and code freely available.
