论文标题
在顺序决策问题中零射协助
Zero-Shot Assistance in Sequential Decision Problems
论文作者
论文摘要
我们考虑创建助手的问题，可以帮助代理解决新的顺序决策问题，假设代理人无法将奖励功能明确指定给助手。我们没有像目前基于自动化的方法那样代替代理代理代理，而是给助手一个咨询角色，并将代理商作为主要决策者。困难是我们必须考虑代理的潜在偏见，这可能会导致其似乎非理性地拒绝建议。为此，我们介绍了一种新颖的援助形式化，以模拟这些偏见，从而使助手推断和适应它们。然后，我们引入了一种计划助手行动的新方法，该方法可以扩展到大型决策问题。我们通过实验表明，我们的方法适应了这些代理偏见，并且与基于自动化的替代方案相比，对代理的累积奖励更高。最后，我们表明，将建议和自动化结合起来的方法仅优于建议，而损失了一些安全保证。
We consider the problem of creating assistants that can help agents solve new sequential decision problems, assuming the agent is not able to specify the reward function explicitly to the assistant. Instead of acting in place of the agent as in current automation-based approaches, we give the assistant an advisory role and keep the agent in the loop as the main decision maker. The difficulty is that we must account for potential biases of the agent which may cause it to seemingly irrationally reject advice. To do this we introduce a novel formalization of assistance that models these biases, allowing the assistant to infer and adapt to them. We then introduce a new method for planning the assistant's actions which can scale to large decision making problems. We show experimentally that our approach adapts to these agent biases, and results in higher cumulative reward for the agent than automation-based alternatives. Lastly, we show that an approach combining advice and automation outperforms advice alone at the cost of losing some safety guarantees.
