论文标题
从历史上学习拜占庭式优化
Learning from History for Byzantine Robust Optimization
论文作者
论文摘要
拜占庭的鲁棒性最近受到了极大的关注，鉴于其对分布式和联合学习的重要性。尽管如此，即使参与者的数据分布相同，我们也确定了现有算法中的严重缺陷。首先，我们展示了现实的例子，即使在没有任何拜占庭攻击者的情况下，当前最新的鲁棒汇总规则也无法融合。其次，我们证明，即使汇总规则可能成功地限制了攻击者在一轮中的影响，攻击者也可以在时间上对攻击进行审对，最终会导致分歧。为了解决这些问题，我们提出了两种令人惊讶的简单策略：一种新的可靠的迭代剪辑程序，并结合了工人动量以克服时间耦合的攻击。这是标准随机优化设置的第一个可证明可靠的方法。我们的代码通过https://github.com/epfml/byzantine-robust-optimizer开放。
Byzantine robustness has received significant attention recently given its importance for distributed and federated learning. In spite of this, we identify severe flaws in existing algorithms even when the data across the participants is identically distributed. First, we show realistic examples where current state of the art robust aggregation rules fail to converge even in the absence of any Byzantine attackers. Secondly, we prove that even if the aggregation rules may succeed in limiting the influence of the attackers in a single round, the attackers can couple their attacks across time eventually leading to divergence. To address these issues, we present two surprisingly simple strategies: a new robust iterative clipping procedure, and incorporating worker momentum to overcome time-coupled attacks. This is the first provably robust method for the standard stochastic optimization setting. Our code is open sourced at https://github.com/epfml/byzantine-robust-optimizer.
