论文标题
自由形式的3D场景与双流gan镶嵌
Free-form 3D Scene Inpainting with Dual-stream GAN
论文作者
论文摘要
如今，由于AR和VR技术的发展，在3D场景中对用户编辑的需求迅速增加。但是，现有的3D场景完成任务（和数据集）无法满足需求，因为场景中缺少的区域是由传感器限制或对象遮挡生成的。因此，我们提出了一个名为Free-Form 3D场景介绍的新颖任务。与上一个3D完成数据集中的场景不同，保留了围绕缺失区域的大多数主要结构和详细形状的提示，拟议的indpainting数据集FF-Matterport包含由我们的自由形式3D蒙版生成算法形成的大型且多样的缺失区域，可以模仿3D空间中的3D绘图轨迹。此外，先前的3D完成方法在这项具有挑战性但实用的任务上不能很好地表现，只需插值附近的几何和颜色环境即可。因此，提出了量身定制的双流gan方法。首先，我们的双流生成器融合了几何和颜色信息，会产生独特的语义边界并解决插值问题。为了进一步增强细节，我们轻巧的双流歧视器将预测场景的几何形状和颜色边缘定期为现实和清晰。我们对拟议的FF-Matterport数据集进行了实验。定性和定量结果验证了我们的方法优于现有场景完成方法以及所有提出的组件的功效。
Nowadays, the need for user editing in a 3D scene has rapidly increased due to the development of AR and VR technology. However, the existing 3D scene completion task (and datasets) cannot suit the need because the missing regions in scenes are generated by the sensor limitation or object occlusion. Thus, we present a novel task named free-form 3D scene inpainting. Unlike scenes in previous 3D completion datasets preserving most of the main structures and hints of detailed shapes around missing regions, the proposed inpainting dataset, FF-Matterport, contains large and diverse missing regions formed by our free-form 3D mask generation algorithm that can mimic human drawing trajectories in 3D space. Moreover, prior 3D completion methods cannot perform well on this challenging yet practical task, simply interpolating nearby geometry and color context. Thus, a tailored dual-stream GAN method is proposed. First, our dual-stream generator, fusing both geometry and color information, produces distinct semantic boundaries and solves the interpolation issue. To further enhance the details, our lightweight dual-stream discriminator regularizes the geometry and color edges of the predicted scenes to be realistic and sharp. We conducted experiments with the proposed FF-Matterport dataset. Qualitative and quantitative results validate the superiority of our approach over existing scene completion methods and the efficacy of all proposed components.
