论文标题
Gazenerf：与神经辐射场的3D感知凝视重定向
GazeNeRF: 3D-Aware Gaze Redirection with Neural Radiance Fields
论文作者
论文摘要
我们提出了Gazenerf，这是一种注视重定向任务的3D感知方法。现有的凝视重定向方法在2D图像上运行，并难以产生3D一致的结果。取而代之的是，我们建立在直觉的基础上，即面部区域和眼球是单独的3D结构，它们以协调但独立的方式移动。我们的方法利用有条件的基于图像的神经辐射场的最新进展，并提出了一个两流体系结构，该体系结构可分别预测面部和眼睛区域的体积特征。通过3D旋转矩阵僵硬地转换眼睛特征，可对所需的凝视角度进行细粒度的控制。然后，通过可区分的音量合成来获得最终的重定向图像。我们的实验表明，就重定向的准确性和身份保存而言，这种体系结构的表现优于天真条件的NERF基线以及先前的最新2D注视重定向方法。
We propose GazeNeRF, a 3D-aware method for the task of gaze redirection. Existing gaze redirection methods operate on 2D images and struggle to generate 3D consistent results. Instead, we build on the intuition that the face region and eyeballs are separate 3D structures that move in a coordinated yet independent fashion. Our method leverages recent advancements in conditional image-based neural radiance fields and proposes a two-stream architecture that predicts volumetric features for the face and eye regions separately. Rigidly transforming the eye features via a 3D rotation matrix provides fine-grained control over the desired gaze angle. The final, redirected image is then attained via differentiable volume compositing. Our experiments show that this architecture outperforms naively conditioned NeRF baselines as well as previous state-of-the-art 2D gaze redirection methods in terms of redirection accuracy and identity preservation.
