论文标题
私有随机非凸优化：自适应算法和更紧密的概括界限
Private Stochastic Non-Convex Optimization: Adaptive Algorithms and Tighter Generalization Bounds
论文作者
论文摘要
我们研究了随机非凸优化的差异私有算法（DP）算法。在这个问题中，目标是将$ n $ i.i.d. $ p $维空间的人口损失最小化。从分布中得出的样本。我们改进了$ {\ sqrt {p}}/{\ sqrt {n}} $的$ {\ sqrt {p}}/{\ sqrt {n}} $的人口梯度，并从事先工作中获得$ \ sqrt [4] {p}/\ sqrt {n} $的较高速率。我们通过提供基于私有梯度的方法（包括自适应算法DP RMSPROP和DP ADAM）的首次分析来获得此速率。我们的证明技术利用了差异隐私和自适应数据分析与每个迭代的约束梯度估计误差之间的联系，这规避了从标准统一收敛参数限制的较差的概括。最后，我们在两项流行的深度学习任务上评估了提出的算法，并证明了DP自适应梯度方法的经验优势，而不是标准DP SGD。
We study differentially private (DP) algorithms for stochastic non-convex optimization. In this problem, the goal is to minimize the population loss over a $p$-dimensional space given $n$ i.i.d. samples drawn from a distribution. We improve upon the population gradient bound of ${\sqrt{p}}/{\sqrt{n}}$ from prior work and obtain a sharper rate of $\sqrt[4]{p}/\sqrt{n}$. We obtain this rate by providing the first analyses on a collection of private gradient-based methods, including adaptive algorithms DP RMSProp and DP Adam. Our proof technique leverages the connection between differential privacy and adaptive data analysis to bound gradient estimation error at every iterate, which circumvents the worse generalization bound from the standard uniform convergence argument. Finally, we evaluate the proposed algorithms on two popular deep learning tasks and demonstrate the empirical advantages of DP adaptive gradient methods over standard DP SGD.
