论文标题
人类语言建模
Human Language Modeling
论文作者
论文摘要
自然语言是由人产生的，但是传统语言建模将单词或文档视为独立生成。在这里，我们提出了人类语言建模（HULM），这是对语言建模问题的层次扩展，其中人类水平存在以连接文档序列（例如社交媒体信息）并捕获人类语言通过改变人类状态而调节的概念。我们介绍了HART，这是一种用于HULM任务的大型变压器模型，对大约100,000个社交媒体用户进行了预训练，并在社交媒体的语言建模（Perpolplexity）方面表现出了其有效性，用于社交媒体，以及针对跨越文档和用户级别的4个下游任务进行微调，即：Stance nection tection toction toction toction tection toction toction toction contection，情感分类，年龄估计，年龄估计，和人性评估，并评估。所有任务的结果符合或超过当前的最新技术。
Natural language is generated by people, yet traditional language modeling views words or documents as if generated independently. Here, we propose human language modeling (HuLM), a hierarchical extension to the language modeling problem whereby a human-level exists to connect sequences of documents (e.g. social media messages) and capture the notion that human language is moderated by changing human states. We introduce, HaRT, a large-scale transformer model for the HuLM task, pre-trained on approximately 100,000 social media users, and demonstrate its effectiveness in terms of both language modeling (perplexity) for social media and fine-tuning for 4 downstream tasks spanning document- and user-levels: stance detection, sentiment classification, age estimation, and personality assessment. Results on all tasks meet or surpass the current state-of-the-art.
