论文标题
通过神经语言模型探索软件的自然性
Exploring Software Naturalness through Neural Language Models
论文作者
论文摘要
软件自然假设认为，可以通过自然语言处理中使用的相同技术来理解编程语言。我们通过使用基于预先训练的变压器的语言模型来执行代码分析任务来探讨这一假设。当前的代码分析方法在很大程度上取决于从抽象语法树（AST）得出的特征，而我们的基于变压器的语言模型在原始源代码上工作。这项工作是第一个研究此类语言模型是否可以自动发现AST功能的工作。为了实现这一目标，我们引入了一个序列标记任务，该任务直接探究了对AST的语言模型的理解。我们的结果表明，基于变压器的语言模型在AST标记任务中具有很高的精度。此外，我们在软件漏洞标识任务上评估了模型。重要的是，我们表明我们的方法获得了与基于图形的方法相当的漏洞识别结果，这些方法很大程度上依赖于编译器进行特征提取。
The Software Naturalness hypothesis argues that programming languages can be understood through the same techniques used in natural language processing. We explore this hypothesis through the use of a pre-trained transformer-based language model to perform code analysis tasks. Present approaches to code analysis depend heavily on features derived from the Abstract Syntax Tree (AST) while our transformer-based language models work on raw source code. This work is the first to investigate whether such language models can discover AST features automatically. To achieve this, we introduce a sequence labeling task that directly probes the language models understanding of AST. Our results show that transformer based language models achieve high accuracy in the AST tagging task. Furthermore, we evaluate our model on a software vulnerability identification task. Importantly, we show that our approach obtains vulnerability identification results comparable to graph based approaches that rely heavily on compilers for feature extraction.
