论文标题
人群可以客观地识别错误信息吗？判断量表和评估者背景的影响
Can The Crowd Identify Misinformation Objectively? The Effects of Judgment Scale and Assessor's Background
论文作者
论文摘要
真实性判断是打击错误信息的过程中的基本步骤，因为它们对于训练和评估自动区分真实和错误陈述的分类器至关重要。通常，这些判断是由专家做出的，例如有关政治陈述的记者或医疗陈述的医生。在本文中，我们遵循另一种方法，并依靠（非专家）人群工人。当然，这导致了以下研究问题：可以可靠地使用众包评估信息的真实性并创建大规模标记的信息可信度系统的收藏吗？为了解决这个问题，我们介绍了一项基于众包的广泛研究的结果：我们在两个数据集上收集了数千个真实性评估，并将专家判断与人群判断进行比较，并以各种粒度水平的量表表示。我们还衡量了工人的政治偏见和认知背景，并量化了它们对人群提供的数据可靠性的影响。
Truthfulness judgments are a fundamental step in the process of fighting misinformation, as they are crucial to train and evaluate classifiers that automatically distinguish true and false statements. Usually such judgments are made by experts, like journalists for political statements or medical doctors for medical statements. In this paper, we follow a different approach and rely on (non-expert) crowd workers. This of course leads to the following research question: Can crowdsourcing be reliably used to assess the truthfulness of information and to create large-scale labeled collections for information credibility systems? To address this issue, we present the results of an extensive study based on crowdsourcing: we collect thousands of truthfulness assessments over two datasets, and we compare expert judgments with crowd judgments, expressed on scales with various granularity levels. We also measure the political bias and the cognitive background of the workers, and quantify their effect on the reliability of the data provided by the crowd.
