论文标题
谁的语言算作高质量？在文本数据选择中衡量语言意识形态
Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection
论文作者
论文摘要
语言模型越来越多地依靠大量的Web垃圾箱来用于不同的文本数据。但是，这些来源充满了不良内容。因此，Wikipedia，Books和Newswire等资源通常是自动选择最适合语言建模的Web文本的锚点，该过程通常称为质量过滤。使用来自全国各地的学生撰写的美国高中报纸文章的新数据集 - 我们调查了其语言是由GPT-3使用的质量过滤器所偏爱的。我们发现，来自较富裕，受过教育和城市邮政编码的大型学校的报纸更有可能被归类为高质量。然后，我们证明了过滤器对质量的测量与其他明智的指标，例如事实或文学好评。我们认为，将任何语料库作为高质量的特权都需要语言意识形态，并且需要更多的护理来为语言模型构建培训语料库，并具有更好的透明度和理由，以包含或排除各种文本。
Language models increasingly rely on massive web dumps for diverse text data. However, these sources are rife with undesirable content. As such, resources like Wikipedia, books, and newswire often serve as anchors for automatically selecting web text most suitable for language modeling, a process typically referred to as quality filtering. Using a new dataset of U.S. high school newspaper articles -- written by students from across the country -- we investigate whose language is preferred by the quality filter used for GPT-3. We find that newspapers from larger schools, located in wealthier, educated, and urban ZIP codes are more likely to be classified as high quality. We then demonstrate that the filter's measurement of quality is unaligned with other sensible metrics, such as factuality or literary acclaim. We argue that privileging any corpus as high quality entails a language ideology, and more care is needed to construct training corpora for language models, with better transparency and justification for the inclusion or exclusion of various texts.
