论文标题
搜索搜索方法：基准测试搜索算法以生成NLP对抗示例
Searching for a Search Method: Benchmarking Search Algorithms for Generating NLP Adversarial Examples
论文作者
论文摘要
我们研究用于生成自然语言处理（NLP）任务的对抗性示例的几种黑框搜索算法的行为。我们对与搜索相关的三个要素进行精细分析：搜索算法，搜索空间和搜索预算。当在过去的工作中提出新的搜索算法时，经常在搜索算法并肩上修改攻击搜索空间。没有消融研究，随着搜索空间的稳定，搜索算法的变化基准了，人们无法确定攻击成功率是否增加是搜索算法改善还是限制性较小的搜索空间的结果。此外，许多以前的研究都无法正确考虑搜索算法的运行时间成本，这对于诸如对抗训练之类的下游任务至关重要。我们的实验为各种搜索空间和查询预算提供了搜索算法的可再现基准，以指导对抗性NLP的未来研究。根据我们的实验，我们建议在时间限制或攻击长输入的情况下，并建议具有单词重要性排名的贪婪攻击，否则，梁搜索或粒子群优化。通过https://github.com/qdata/textattack-search-benchmark共享代码实现
We study the behavior of several black-box search algorithms used for generating adversarial examples for natural language processing (NLP) tasks. We perform a fine-grained analysis of three elements relevant to search: search algorithm, search space, and search budget. When new search algorithms are proposed in past work, the attack search space is often modified alongside the search algorithm. Without ablation studies benchmarking the search algorithm change with the search space held constant, one cannot tell if an increase in attack success rate is a result of an improved search algorithm or a less restrictive search space. Additionally, many previous studies fail to properly consider the search algorithms' run-time cost, which is essential for downstream tasks like adversarial training. Our experiments provide a reproducible benchmark of search algorithms across a variety of search spaces and query budgets to guide future research in adversarial NLP. Based on our experiments, we recommend greedy attacks with word importance ranking when under a time constraint or attacking long inputs, and either beam search or particle swarm optimization otherwise. Code implementation shared via https://github.com/QData/TextAttack-Search-Benchmark
