论文标题
COTK：一种开源工具包，用于快速开发和公平评估文本生成
CoTK: An Open-Source Toolkit for Fast Development and Fair Evaluation of Text Generation
论文作者
论文摘要
在文本生成评估中，许多实际问题，例如不一致的实验环境和指标实施，通常会忽略，但会导致不公平的评估和站不住脚的结论。我们提出了COTK，这是一个开源工具包，旨在支持快速开发和文本生成的公平评估。在模型开发中，COTK有助于处理繁琐的问题，例如数据处理，指标实施和繁殖。它标准化开发步骤并减少人类错误，这可能导致不一致的实验设置。在模型评估中，COTK为不同实验设置的许多常用指标和基准模型提供了实施。作为一个独特的功能，COTK可以表示何时以及哪些度量不能公平地比较。我们证明，将COTK用于模型开发和评估很方便，尤其是在不同的实验环境中。
In text generation evaluation, many practical issues, such as inconsistent experimental settings and metric implementations, are often ignored but lead to unfair evaluation and untenable conclusions. We present CoTK, an open-source toolkit aiming to support fast development and fair evaluation of text generation. In model development, CoTK helps handle the cumbersome issues, such as data processing, metric implementation, and reproduction. It standardizes the development steps and reduces human errors which may lead to inconsistent experimental settings. In model evaluation, CoTK provides implementation for many commonly used metrics and benchmark models across different experimental settings. As a unique feature, CoTK can signify when and which metric cannot be fairly compared. We demonstrate that it is convenient to use CoTK for model development and evaluation, particularly across different experimental settings.
