论文标题
通过重播缓冲区分析随机过程
Analysis of Stochastic Processes through Replay Buffers
论文作者
论文摘要
重播缓冲区是许多强化学习方案中的关键组成部分。然而，他们的理论特性尚未完全理解。在本文中，我们分析了一个系统，将随机过程x推入重型缓冲区，然后随机采样以从重播缓冲区生成随机过程y。我们提供了采样过程的属性分析，例如平稳性，马尔可道和自相关，就原始过程的属性而言。我们的理论分析阐明了为什么重播缓冲液可能是良好的去率。我们的分析提供了理论工具，以证明基于重播缓冲算法的收敛性，这些算法在强化学习方案中很普遍。
Replay buffers are a key component in many reinforcement learning schemes. Yet, their theoretical properties are not fully understood. In this paper we analyze a system where a stochastic process X is pushed into a replay buffer and then randomly sampled to generate a stochastic process Y from the replay buffer. We provide an analysis of the properties of the sampled process such as stationarity, Markovity and autocorrelation in terms of the properties of the original process. Our theoretical analysis sheds light on why replay buffer may be a good de-correlator. Our analysis provides theoretical tools for proving the convergence of replay buffer based algorithms which are prevalent in reinforcement learning schemes.
