论文标题
关于自动检测在线滥用的跨数据集概括
On Cross-Dataset Generalization in Automatic Detection of Online Abuse
论文作者
论文摘要
NLP研究在滥用语言检测方面取得了很高的表现，作为一项监督分类任务。在研究环境中，通常从类似的数据样本中获得培训和测试数据集，但在实践中，通常将系统应用于与主题和类别分布的培训集不同的数据。同样，此任务中继承的类定义的歧义加剧了源和目标数据集之间的差异。我们探讨了跨数据集概括中的主题偏差和任务配方偏差。我们表明，Wikipedia排毒数据集中的良性示例偏向于平台特定的主题。我们使用无监督的主题建模和手动检查主题的关键字来确定这些示例。消除这些主题会增加交叉概括，而不会降低内域分类性能。对于强大的数据集设计，我们建议使用廉价的无监督方法来检查收集的数据并缩小不可替代内容的尺寸，然后手动注释类标签。
NLP research has attained high performances in abusive language detection as a supervised classification task. While in research settings, training and test datasets are usually obtained from similar data samples, in practice systems are often applied on data that are different from the training set in topic and class distributions. Also, the ambiguity in class definitions inherited in this task aggravates the discrepancies between source and target datasets. We explore the topic bias and the task formulation bias in cross-dataset generalization. We show that the benign examples in the Wikipedia Detox dataset are biased towards platform-specific topics. We identify these examples using unsupervised topic modeling and manual inspection of topics' keywords. Removing these topics increases cross-dataset generalization, without reducing in-domain classification performance. For a robust dataset design, we suggest applying inexpensive unsupervised methods to inspect the collected data and downsize the non-generalizable content before manually annotating for class labels.
