论文标题
一致的稀疏概率
Sparse Probability of Agreement
论文作者
论文摘要
测量通道间的一致性对于注释任务很重要，但是许多指标需要一组完全注销的数据，其中所有注释者都注释了所有样本。我们定义了一致性的稀疏概率，该水疗中心估计并非所有注释者对 - 都可以使用一致的概率。我们表明，在某些条件下，水疗中心是一个公正的估计器，我们提供多种称重方案，用于处理不同程度的注释数据。
Measuring inter-annotator agreement is important for annotation tasks, but many metrics require a fully-annotated set of data, where all annotators annotate all samples. We define Sparse Probability of Agreement, SPA, which estimates the probability of agreement when not all annotator-item-pairs are available. We show that under certain conditions, SPA is an unbiased estimator, and we provide multiple weighing schemes for handling data with various degrees of annotation.
