论文标题
RT-NERF：实时的设备上神经辐射场朝着沉浸式AR/VR渲染
RT-NeRF: Real-Time On-Device Neural Radiance Fields Towards Immersive AR/VR Rendering
论文作者
论文摘要
由于其最先进的（SOTA）在增强和虚拟现实（AR/VR）中，基于神经辐射场（NERF）的渲染引起了人们的关注。但是，由于AR/VR设备上可实现的吞吐量较低，因此，沉浸式实时（> 30 fps）基于NERF的渲染启用交互仍然受到限制。为此，我们首先在商业设备上介绍了有效的NERF算法，并确定了上述效率低下的两个主要原因：（1）均匀点采样和（2）NERF中所需嵌入的密集访问和计算。此外，我们提出了RT-NERF，据我们所知，这是NERF的第一个算法 - 硬件共同设计加速度。具体而言，在算法级别上，RT-NERF整合了有效的渲染管道，从而通过直接计算预先存在的点的几何形状来大大减轻NERF中通常采用的均匀点采样方法而导致的效率低下。此外，RT-NERF利用粗粒依赖的计算顺序方案消除了不可见点的（不必要的）处理。在硬件级别上，我们提出的RT-NERF加速器（1）采用混合编码方案，在NERF的稀疏嵌入式中适应性地切换基于位图或基于坐标的基于位图或坐标的稀疏编码格式，旨在最大化存储储蓄，从而减少所需的DRAM访问权限，同时支持有效的NERF nerf编码; （2）同时集成了双重用双向加法器和搜索树和高密度稀疏搜索单元，以协调上述两个编码格式。在八个数据集上进行的广泛实验始终验证RT -NERF的有效性，从而实现了较大的吞吐量改进（例如，9.7倍-3,201X），同时与SOTA有效的NERF溶液相比，保持了渲染质量。
Neural Radiance Field (NeRF) based rendering has attracted growing attention thanks to its state-of-the-art (SOTA) rendering quality and wide applications in Augmented and Virtual Reality (AR/VR). However, immersive real-time (> 30 FPS) NeRF based rendering enabled interactions are still limited due to the low achievable throughput on AR/VR devices. To this end, we first profile SOTA efficient NeRF algorithms on commercial devices and identify two primary causes of the aforementioned inefficiency: (1) the uniform point sampling and (2) the dense accesses and computations of the required embeddings in NeRF. Furthermore, we propose RT-NeRF, which to the best of our knowledge is the first algorithm-hardware co-design acceleration of NeRF. Specifically, on the algorithm level, RT-NeRF integrates an efficient rendering pipeline for largely alleviating the inefficiency due to the commonly adopted uniform point sampling method in NeRF by directly computing the geometry of pre-existing points. Additionally, RT-NeRF leverages a coarse-grained view-dependent computing ordering scheme for eliminating the (unnecessary) processing of invisible points. On the hardware level, our proposed RT-NeRF accelerator (1) adopts a hybrid encoding scheme to adaptively switch between a bitmap- or coordinate-based sparsity encoding format for NeRF's sparse embeddings, aiming to maximize the storage savings and thus reduce the required DRAM accesses while supporting efficient NeRF decoding; and (2) integrates both a dual-purpose bi-direction adder & search tree and a high-density sparse search unit to coordinate the two aforementioned encoding formats. Extensive experiments on eight datasets consistently validate the effectiveness of RT-NeRF, achieving a large throughput improvement (e.g., 9.7x - 3,201x) while maintaining the rendering quality as compared with SOTA efficient NeRF solutions.
