论文标题
遥感的新视图综合与隐式多层表示
Remote Sensing Novel View Synthesis with Implicit Multiplane Representations
论文作者
论文摘要
遥感场景的新型视图综合对于场景可视化，人类计算机相互作用和各种下游应用具有重要意义。尽管计算机图形和摄影测量技术最近取得了进步，但由于其高复杂性，视图稀疏性和有限的视图变化而产生新型视图仍然具有挑战性，特别是对于遥感图像而言。在本文中，我们通过利用隐式神经表示的最新进展提出了一种新型的遥感视图合成方法。考虑到遥感图像的开销和深度成像，我们通过结合隐式多层图像（MPI）表示和深神经网络来表示3D空间。 3D场景通过具有多视图输入约束的可区分的多样式渲染器在自我监督的优化范式下重建。因此，来自任何新颖观点的图像可以根据重建模型自由渲染。作为副产品，可以与渲染输出一起生成与给定观点相对应的深度图。我们将我们的方法称为隐式多层图像（IMMPI）。为了进一步改善稀疏视图输入下的视图综合，我们探讨了基于学习的3D场景的基于学习的初始化，并提出了基于神经网络的先验提取器，以加速优化过程。此外，我们还提出了一个新的数据集，用于使用多视图现实世界的Google Earth Images进行遥感新视图综合。广泛的实验表明，就重建精度，视觉保真度和时间效率而言，IMMPI优于先前最新方法。消融实验还表明了我们方法设计的有效性。可以在https://github.com/wyc-chang/immpi上找到我们的数据集和代码
Novel view synthesis of remote sensing scenes is of great significance for scene visualization, human-computer interaction, and various downstream applications. Despite the recent advances in computer graphics and photogrammetry technology, generating novel views is still challenging particularly for remote sensing images due to its high complexity, view sparsity and limited view-perspective variations. In this paper, we propose a novel remote sensing view synthesis method by leveraging the recent advances in implicit neural representations. Considering the overhead and far depth imaging of remote sensing images, we represent the 3D space by combining implicit multiplane images (MPI) representation and deep neural networks. The 3D scene is reconstructed under a self-supervised optimization paradigm through a differentiable multiplane renderer with multi-view input constraints. Images from any novel views thus can be freely rendered on the basis of the reconstructed model. As a by-product, the depth maps corresponding to the given viewpoint can be generated along with the rendering output. We refer to our method as Implicit Multiplane Images (ImMPI). To further improve the view synthesis under sparse-view inputs, we explore the learning-based initialization of remote sensing 3D scenes and proposed a neural network based Prior extractor to accelerate the optimization process. In addition, we propose a new dataset for remote sensing novel view synthesis with multi-view real-world google earth images. Extensive experiments demonstrate the superiority of the ImMPI over previous state-of-the-art methods in terms of reconstruction accuracy, visual fidelity, and time efficiency. Ablation experiments also suggest the effectiveness of our methodology design. Our dataset and code can be found at https://github.com/wyc-Chang/ImMPI
