论文标题
LIDER：一个有效的高维学到的索引，用于大规模密集通道检索
LIDER: An Efficient High-dimensional Learned Index for Large-scale Dense Passage Retrieval
论文作者
论文摘要
许多最近的通道检索方法都使用了由深神经模型产生的致密嵌入，称为“密集通道检索”。最先进的端到端密集通道检索系统通常部署深度神经模型，然后是大约最近的邻居（ANN）搜索模块。该模型生成语料库和查询的嵌入，然后由高性能ANN模块索引和搜索。随着数据量表的增加，ANN模块不可避免地成为效率的瓶颈。一个替代方案是学习的指数，通过学习数据分布并预测目标数据位置，该指数可实现明显高的搜索效率。但是，大多数现有学习的索引都是为低维数据而设计的，这些索引不适用于具有高维密度嵌入的密集通道检索。在本文中，我们提出了Lider，这是一个有效的高维学到的索引，用于大规模密集的通道检索。 Lider具有基于聚类的分层体系结构，该体系结构由两层核心模型形成。作为索引和搜索数据的基本单位，核心模型包括适应的递归模型索引（RMI）和一个尺寸缩减组件，该组件由扩展的分配keys-lsh（SK-lsh）和键重新缩放模块组成。尺寸还原分量将高维密度嵌入到一维键中，并以特定顺序对其进行分类，然后由RMI将其用于快速预测。实验表明，Lider具有较高的搜索速度，并且与通道检索任务的最先进的ANN指数相比，例如，在大规模数据上，它可以达到1.2倍搜索速度，并且比我们评估中最快的基线要高得多。此外，Lider具有更好的速度质量权衡能力。
Many recent approaches of passage retrieval are using dense embeddings generated from deep neural models, called "dense passage retrieval". The state-of-the-art end-to-end dense passage retrieval systems normally deploy a deep neural model followed by an approximate nearest neighbor (ANN) search module. The model generates embeddings of the corpus and queries, which are then indexed and searched by the high-performance ANN module. With the increasing data scale, the ANN module unavoidably becomes the bottleneck on efficiency. An alternative is the learned index, which achieves significantly high search efficiency by learning the data distribution and predicting the target data location. But most of the existing learned indexes are designed for low dimensional data, which are not suitable for dense passage retrieval with high-dimensional dense embeddings. In this paper, we propose LIDER, an efficient high-dimensional Learned Index for large-scale DEnse passage Retrieval. LIDER has a clustering-based hierarchical architecture formed by two layers of core models. As the basic unit of LIDER to index and search data, a core model includes an adapted recursive model index (RMI) and a dimension reduction component which consists of an extended SortingKeys-LSH (SK-LSH) and a key re-scaling module. The dimension reduction component reduces the high-dimensional dense embeddings into one-dimensional keys and sorts them in a specific order, which are then used by the RMI to make fast prediction. Experiments show that LIDER has a higher search speed with high retrieval quality comparing to the state-of-the-art ANN indexes on passage retrieval tasks, e.g., on large-scale data it achieves 1.2x search speed and significantly higher retrieval quality than the fastest baseline in our evaluation. Furthermore, LIDER has a better capability of speed-quality trade-off.
