论文标题
评估自然语言生成多样性的评估
Evaluating the Evaluation of Diversity in Natural Language Generation
论文作者
论文摘要
尽管对产生各种产出的自然语言产生（NLG）模型的兴趣日益增加，但目前尚无评估NLG系统多样性的原则方法。在这项工作中，我们提出了一个评估多样性指标的框架。该框架衡量了提出的多样性度量与多样性参数之间的相关性，该参数控制了生成的文本中多样性的某些方面。例如，多样性参数可能是一种二进制变量，用于指导众包工人生成低内容或高度内容多样性的文本。我们通过以下方式证明了我们的框架的实用性：我们的框架可以促进对不同多样性指标的理解，这是通往更好NLG系统的道路的重要一步。
Despite growing interest in natural language generation (NLG) models that produce diverse outputs, there is currently no principled method for evaluating the diversity of an NLG system. In this work, we propose a framework for evaluating diversity metrics. The framework measures the correlation between a proposed diversity metric and a diversity parameter, a single parameter that controls some aspect of diversity in generated text. For example, a diversity parameter might be a binary variable used to instruct crowdsourcing workers to generate text with either low or high content diversity. We demonstrate the utility of our framework by: (a) establishing best practices for eliciting diversity judgments from humans, (b) showing that humans substantially outperform automatic metrics in estimating content diversity, and (c) demonstrating that existing methods for controlling diversity by tuning a "decoding parameter" mostly affect form but not meaning. Our framework can advance the understanding of different diversity metrics, an essential step on the road towards better NLG systems.
