论文标题
Rényi差异的变分表示和神经网络估计
Variational Representations and Neural Network Estimation of Rényi Divergences
论文作者
论文摘要
我们为RényiDiverences家族提供了一个新的变分公式，$r_α（q \ | p）$，概率指标$ q $和$ p $。我们的结果概括了Kullback-Leibler Divergence的经典Donsker-Varadhan变异公式。我们进一步表明，该rényi变化公式在一系列函数空间上都有。这导致了在非常弱的假设下的优化器公式，这也是我们开发RényiDivergence估计器一致性理论的关键。通过将该理论应用于神经网络估计器，我们表明，如果神经网络家族满足了通用近似属性的几个增强版本之一，那么相应的rényi散度估计器是一致的。与基于密度估计器的方法相反，我们的估计器仅涉及$ Q $和$ p $的期望，因此在高维系统中更有效。我们通过多达5000维系统中神经网络估计的几个数值示例来说明这一点。
We derive a new variational formula for the Rényi family of divergences, $R_α(Q\|P)$, between probability measures $Q$ and $P$. Our result generalizes the classical Donsker-Varadhan variational formula for the Kullback-Leibler divergence. We further show that this Rényi variational formula holds over a range of function spaces; this leads to a formula for the optimizer under very weak assumptions and is also key in our development of a consistency theory for Rényi divergence estimators. By applying this theory to neural-network estimators, we show that if a neural network family satisfies one of several strengthened versions of the universal approximation property then the corresponding Rényi divergence estimator is consistent. In contrast to density-estimator based methods, our estimators involve only expectations under $Q$ and $P$ and hence are more effective in high dimensional systems. We illustrate this via several numerical examples of neural network estimation in systems of up to 5000 dimensions.
