论文标题
通过环境中毒攻击进行加强学习的政策教学
Policy Teaching in Reinforcement Learning via Environment Poisoning Attacks
论文作者
论文摘要
我们研究了强化学习的安全威胁，攻击者毒害学习环境迫使代理商执行攻击者选择的目标策略。作为受害者，我们认为RL代理人的目的是找到一项最大化无限措施问题的奖励的政策。攻击者可以在训练时间时操纵学习环境中的奖励和过渡动态，并有兴趣以隐秘的方式这样做。我们提出了一个优化框架，以寻找针对不同攻击成本度量的最佳隐身攻击。我们在攻击成本上提供下/上限，并在两个设置中实例化攻击：（i）代理商在中毒环境中进行计划的离线设置，以及（ii）在线环境中，代理商正在学习具有中毒反馈的政策。我们的结果表明，攻击者很容易在轻度条件下向受害者传授任何目标政策，并强调对实践中加强学习者的重大安全威胁。
We study a security threat to reinforcement learning where an attacker poisons the learning environment to force the agent into executing a target policy chosen by the attacker. As a victim, we consider RL agents whose objective is to find a policy that maximizes reward in infinite-horizon problem settings. The attacker can manipulate the rewards and the transition dynamics in the learning environment at training-time, and is interested in doing so in a stealthy manner. We propose an optimization framework for finding an optimal stealthy attack for different measures of attack cost. We provide lower/upper bounds on the attack cost, and instantiate our attacks in two settings: (i) an offline setting where the agent is doing planning in the poisoned environment, and (ii) an online setting where the agent is learning a policy with poisoned feedback. Our results show that the attacker can easily succeed in teaching any target policy to the victim under mild conditions and highlight a significant security threat to reinforcement learning agents in practice.
