论文标题
评估合成错误
Evaluating Synthetic Bugs
论文作者
论文摘要
自1990年代以来,Fuzz测试已被用来在程序中找到错误,但是尽管进行了数十年的专门研究,但仍未达成共识,其中模糊技术最有效。原因之一是地面真理的匮乏:在具有已知根本原因和触发输入的真实程序中的错误很难以有意义的规模收集。将合成错误添加到实际程序中的错误注入技术似乎提供了解决方案,但是以前尚未大规模探索这些合成错误与有机错误的差异。使用了80多年的CPU时间,我们从Rode0day Finding竞赛和Lava-M Corpus的20个目标中运行了8个模糊器。根据收集的计算资源和指标进行了标准化的实验。这些实验显示了模糊性能的差异以及各种配置选项的影响。例如,很明显,将符号执行与突变模糊化的整合非常有效,并且使用词典可以提高性能。其他结论不太明确。例如,在所有测试中,没有人击败所有其他人。值得注意的是,尽管有50种此类错误可在模糊语料库中发现,但没有fuzzer发现任何有机错误(即在CVE中报告的错误)。对结果的仔细分析表明,可能的解释是:合成和有机虫相对于模糊器发现的“主要路径”的地方之间的巨大差异。我们发现,最近对错误注入系统的更新使合成错误更难发现,但是与我们的目标程序中的有机错误相比,它们仍然更容易找到它们。最后,这项研究确定了错误注入技术中的缺陷,并提出了许多轴,应改善合成错误。
Fuzz testing has been used to find bugs in programs since the 1990s, but despite decades of dedicated research, there is still no consensus on which fuzzing techniques work best. One reason for this is the paucity of ground truth: bugs in real programs with known root causes and triggering inputs are difficult to collect at a meaningful scale. Bug injection technologies that add synthetic bugs into real programs seem to offer a solution, but the differences in finding these synthetic bugs versus organic bugs have not previously been explored at a large scale. Using over 80 years of CPU time, we ran eight fuzzers across 20 targets from the Rode0day bug-finding competition and the LAVA-M corpus. Experiments were standardized with respect to compute resources and metrics gathered. These experiments show differences in fuzzer performance as well as the impact of various configuration options. For instance, it is clear that integrating symbolic execution with mutational fuzzing is very effective and that using dictionaries improves performance. Other conclusions are less clear-cut; for example, no one fuzzer beat all others on all tests. It is noteworthy that no fuzzer found any organic bugs (i.e., one reported in a CVE), despite 50 such bugs being available for discovery in the fuzzing corpus. A close analysis of results revealed a possible explanation: a dramatic difference between where synthetic and organic bugs live with respect to the ''main path'' discovered by fuzzers. We find that recent updates to bug injection systems have made synthetic bugs more difficult to discover, but they are still significantly easier to find than organic bugs in our target programs. Finally, this study identifies flaws in bug injection techniques and suggests a number of axes along which synthetic bugs should be improved.