论文标题
表达式提供面部额额改善视觉辅助语音处理
Expression-preserving face frontalization improves visually assisted speech processing
论文作者
论文摘要
面部额叶化包括从任意观看的面部综合面孔的脸部。本文的主要贡献是一种额叶化方法，可保留非刚性面部变形，以提高视觉辅助语音交流的性能。该方法在（i）〜刚性变换（刻度，旋转和翻译）的估计和（ii）〜任意观看的面部和面部模型之间的非rigid变形之间交替。该方法具有两个重要的优点：它可以处理数据中的非高斯错误，并结合了动态的面部变形模型。为此，我们将广义的学生T分布与线性动态系统结合使用，以说明僵化的头部运动和由语音产生引起的时变面部变形。我们建议使用零均值的归一化互相关（ZNCC）评分来评估该方法保留面部表情的能力。根据传统的几何模型或深度学习，对该方法进行了彻底评估并与几种最先进的方法进行了比较。此外，我们表明，当将这种方法纳入深度学习管道中时，即唇部阅读和语音增强，可以通过相当大的差距提高单词识别和语音智能分数。可以在https://team.inria.fr/robotlearn/research/facefrontalization/上获取补充材料
Face frontalization consists of synthesizing a frontally-viewed face from an arbitrarily-viewed one. The main contribution of this paper is a frontalization methodology that preserves non-rigid facial deformations in order to boost the performance of visually assisted speech communication. The method alternates between the estimation of (i)~the rigid transformation (scale, rotation, and translation) and (ii)~the non-rigid deformation between an arbitrarily-viewed face and a face model. The method has two important merits: it can deal with non-Gaussian errors in the data and it incorporates a dynamical face deformation model. For that purpose, we use the generalized Student t-distribution in combination with a linear dynamic system in order to account for both rigid head motions and time-varying facial deformations caused by speech production. We propose to use the zero-mean normalized cross-correlation (ZNCC) score to evaluate the ability of the method to preserve facial expressions. The method is thoroughly evaluated and compared with several state of the art methods, either based on traditional geometric models or on deep learning. Moreover, we show that the method, when incorporated into deep learning pipelines, namely lip reading and speech enhancement, improves word recognition and speech intelligibilty scores by a considerable margin. Supplemental material is accessible at https://team.inria.fr/robotlearn/research/facefrontalization/
