论文标题
将分布式体系结构集成在高度模块化的RL库中
Integrating Distributed Architectures in Highly Modular RL Libraries
论文作者
论文摘要
推进强化学习（RL）需要足够灵活的工具，可以轻松地原型新方法，同时避免了不切实际的实验周转时间。为了符合第一个要求，最受欢迎的RL库倡导高度模块化的代理合并性，这有助于实验和开发。为了在合理的时间范围内解决具有挑战性的环境，将RL扩展到大型抽样和计算资源已证明是成功的策略。但是，到目前为止，这种能力很难与模块化结合。在这项工作中，我们探讨了设计选择，以允许在本地和分布式执行级别的代理合并性。我们提出了一种多功能方法，该方法允许通过独立可重复使用的组件在不同尺度上定义RL代理。我们通过实验证明，我们的设计选择使我们能够重现经典的基准测试，探索多个分布式体系结构，并解决新颖和复杂的环境，同时在代理定义和培训方案定义中为用户充分控制。我们认为，这项工作可以为下一代RL库提供有用的见解。
Advancing reinforcement learning (RL) requires tools that are flexible enough to easily prototype new methods while avoiding impractically slow experimental turnaround times. To match the first requirement, the most popular RL libraries advocate for highly modular agent composability, which facilitates experimentation and development. To solve challenging environments within reasonable time frames, scaling RL to large sampling and computing resources has proved a successful strategy. However, this capability has been so far difficult to combine with modularity. In this work, we explore design choices to allow agent composability both at a local and distributed level of execution. We propose a versatile approach that allows the definition of RL agents at different scales through independent reusable components. We demonstrate experimentally that our design choices allow us to reproduce classical benchmarks, explore multiple distributed architectures, and solve novel and complex environments while giving full control to the user in the agent definition and training scheme definition. We believe this work can provide useful insights to the next generation of RL libraries.
