论文标题
brain2word：解码语言生成的大脑活动
Brain2Word: Decoding Brain Activity for Language Generation
论文作者
论文摘要
在过去几年中，大脑解码是将大脑活动映射到产生它们的刺激的过程。在语言刺激的情况下，最近的研究表明，可以将fMRI扫描分解成一个受试者正在阅读的单词。但是，这种单词嵌入是为自然语言处理任务而不是大脑解码而设计的。因此，它们限制了我们恢复精确刺激的能力。在这项工作中，我们建议直接对fMRI扫描进行分类，将其映射到固定词汇中的相应单词。与现有工作不同，我们评估了以前看不见的主题的扫描。我们认为这是一个更现实的设置，我们提出了一个模型，可以从看不见的主题中解释fMRI数据。在这项具有挑战性的任务中，我们的模型达到了5.22％的TOP-1和13.59％的前5个准确性，大大优于所有考虑的竞争基线。此外，我们使用解码单词以GPT-2模型指导语言生成。这样，我们将对将大脑活动转化为连贯文本的系统进行了追求。
Brain decoding, understood as the process of mapping brain activities to the stimuli that generated them, has been an active research area in the last years. In the case of language stimuli, recent studies have shown that it is possible to decode fMRI scans into an embedding of the word a subject is reading. However, such word embeddings are designed for natural language processing tasks rather than for brain decoding. Therefore, they limit our ability to recover the precise stimulus. In this work, we propose to directly classify an fMRI scan, mapping it to the corresponding word within a fixed vocabulary. Unlike existing work, we evaluate on scans from previously unseen subjects. We argue that this is a more realistic setup and we present a model that can decode fMRI data from unseen subjects. Our model achieves 5.22% Top-1 and 13.59% Top-5 accuracy in this challenging task, significantly outperforming all the considered competitive baselines. Furthermore, we use the decoded words to guide language generation with the GPT-2 model. This way, we advance the quest for a system that translates brain activities into coherent text.
