论文标题
关于命名实体识别和政治推文的情感分析的人群工作者和NLP工具的绩效比较
Performance Comparison of Crowdworkers and NLP Tools on Named-Entity Recognition and Sentiment Analysis of Political Tweets
论文作者
论文摘要
我们报告了在解决两个重要的NLP任务（命名实体识别（NER）和实体级别的情感（ELS）分析）方面比较人群工人和七个自然语言处理（NLP）工具包的结果的结果。我们在这里专注于一个具有挑战性的数据集，2016年2月在美国总统初选期间收集了1,000条政治推文。每条推文都至少是指四个总统候选人中的至少一个，即四个命名实体。由政治传播专家建立的地面确实为推文中提到的每个候选人提供了实体级别的情感信息。我们测试了几种商业和开源工具。我们的实验表明，对于我们的政治推文数据集，最准确的NER系统Google Cloud NL几乎与人群工作者相当，但是最准确的ELS分析系统（浓度）与人群工人的准确性不符合大量超过30％以上。
We report results of a comparison of the accuracy of crowdworkers and seven Natural Language Processing (NLP) toolkits in solving two important NLP tasks, named-entity recognition (NER) and entity-level sentiment (ELS) analysis. We here focus on a challenging dataset, 1,000 political tweets that were collected during the U.S. presidential primary election in February 2016. Each tweet refers to at least one of four presidential candidates, i.e., four named entities. The groundtruth, established by experts in political communication, has entity-level sentiment information for each candidate mentioned in the tweet. We tested several commercial and open-source tools. Our experiments show that, for our dataset of political tweets, the most accurate NER system, Google Cloud NL, performed almost on par with crowdworkers, but the most accurate ELS analysis system, TensiStrength, did not match the accuracy of crowdworkers by a large margin of more than 30 percent points.
