Zum Inhalt springen
Jefferson Gonçalves
← Zurück zu den Links

Awesome Multivector Retrieval

#Awesome Multivector Retrieval

Awesome

An extensive and commented list of resources on late-interaction multivector retrieval.

#Contents

#Models

#Foundational Models

  • ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
    Omar Khattab, Matei Zaharia
    SIGIR, 2020
    📄 paper | 🛠️ code

  • COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted List
    Luyu Gao, Zhuyun Dai, Jamie Callan
    NAACL, 2021
    📄 paper

  • ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction
    Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, Matei Zaharia
    NAACL, 2022
    📄 paper | 🛠️ code

#General Models & Training

  • Jina-ColBERT-v2: A General-Purpose Multilingual Late Interaction Retriever
    Rohan Jha, Bo Wang, Michael Günther, Georgios Mastrapas, Saba Sturua, Isabelle Mohr, Andreas Koukounas, Mohammad Kalim Akram, Nan Wang, Han Xiao
    MRL Workshop, 2024
    📄 paper

  • PyLate: Flexible Training and Retrieval for Late Interaction Models
    Antoine Chaffin, Raphaël Sourty
    CIKM, 2025
    📄 paper | 🛠️ code

  • ColBERT-Zero: To Pre-train Or Not To Pre-train ColBERT models
    Antoine Chaffin, Luca Arnaboldi, Amélie Chatelain, Florent Krzakala
    arXiv, 2026
    📄 paper

  • Your Embedding Model is SMARTer Than You Think
    Jianrui Zhang, Hyun Jung Lee, Sukanta Ganguly, Tae-Eui Kam, Donghyun Kim, Yong Jae Lee
    arXiv, 2026
    📄 paper | 🛠️ code

  • Party is over: regularizing ColBERT models to fix efficient ANN methods
    LightOn AI
    Blog, 2026
    📝 blog

  • NumColBERT: Non-Intrusive Numeracy Injection for Late-Interaction Retrieval Models
    Haruki Fujimaki, Makoto P. Kato
    arXiv, 2026
    📄 paper

  • mDenseOn with the mLateOn: Open Multilingual, Long-Context, and Code Retrieval Models
    LightOn AI
    Blog, 2026
    📝 blog

  • GLInt: Geometry-Matched Hard Negatives for Late-Interaction Retrieval
    Aarush Sinha
    Blog, 2026
    📝 blog

  • Multi-Vector (Late Interaction) Embedding Models with Sentence Transformers
    Tom Aarsen, Antoine Chaffin, Raphaël Sourty
    Blog, 2026
    📝 blog

  • Training and Finetuning Multi-Vector Embedding Models with Sentence Transformers
    Tom Aarsen
    Blog, 2026
    📝 blog

  • Exploring Static Embedding Retrieval
    Logan Markewich
    Blog, 2026
    📝 blog

  • KURE-v2: A Korean-English Bilingual Late-Interaction Retrieval Model
    Youngjoon Jang
    Blog, 2026
    📝 blog

#Compression & Token Pruning

  • Compact Token Representations with Contextual Quantization for Efficient Document Re-ranking
    Yingrui Yang, Yifan Qiao, Tao Yang
    ACL, 2022
    📄 paper

  • Learned Token Pruning in Contextualized Late Interaction over BERT (ColBERT)
    Carlos Lassance, Maroua Maachou, Joohee Park, Stephane Clinchant
    SIGIR, 2022
    📄 paper

  • Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reduction
    Sebastian Hofstatter, Omar Khattab, Sophia Althammer, Mete Sertkan, Allan Hanbury
    CIKM, 2022
    📄 paper | 🛠️ code

  • Joint Optimization of Multi-Vector Representation with Product Quantization
    Yufan Fang, Jing Zhan, Yiqun Liu, Jiafeng Mao, Min Zhang, Shaoping Ma
    NLPCC, 2022
    📄 paper

  • Multi-Vector Retrieval as Sparse Alignment
    Yujie Qian, Jinhyuk Lee, Sai Meher Karthik Duddu, Zhuyun Dai, Siddhartha Brahma, Iftekhar Naim, Tao Lei, Vincent Y. Zhao
    arXiv, 2022
    📄 paper

  • CITADEL: Conditional Token Interaction via Dynamic Lexical Routing for Efficient and Effective Multi-Vector Retrieval
    Minghan Li, Sean C. Lin, Barlas Oguz, Arnab Ghoshal, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, Xilun Chen
    ACL, 2023
    📄 paper

  • SLIM: Sparsified Late Interaction for Multi-Vector Retrieval with Inverted Indexes
    Minghan Li, Sheng-Chieh Lin, Xueguang Ma, Jimmy Lin
    SIGIR, 2023
    📄 paper

  • Static Pruning for Multi-Representation Dense Retrieval
    Antonio Acquavia, Craig Macdonald, Nicola Tonellotto
    DocEng, 2023
    📄 paper | 🛠️ code

  • Rethinking the Role of Token Retrieval in Multi-Vector Retrieval
    Jinhyuk Lee, Zhuyun Dai, Sai Meher Karthik Duddu, Tao Lei, Iftekhar Naim, Ming-Wei Chang, Vincent Y. Zhao
    NeurIPS, 2023
    📄 paper

  • SPLATE: Sparse Late Interaction Retrieval
    Thibault Formal, Stephane Clinchant, Herve Dejean, Carlos Lassance
    SIGIR, 2024
    📄 paper

  • Reducing the Footprint of Multi-Vector Retrieval with Minimal Performance Impact via Token Pooling
    Benjamin Clavié, Antoine Chaffin, Griffin Adams
    arXiv, 2024
    📄 paper

  • Muvera: Multi-Vector Retrieval via Fixed Dimensional Encodings
    Laxman Dhulipala, Majid Hadian, Rajesh Jayaram, Jason Lee, Vahab Mirrokni
    NeurIPS, 2024
    📄 paper

  • Enhancing ColBERT: A Method for Reducing Space Complexity and Accelerating Retrieval Speed
    Hai Nguyen T., Huong Le T.
    PACLIC, 2024
    📄 paper

  • Token Pruning Optimization for Efficient Multi-vector Dense Retrieval
    Shanxiu He, Mutasem Al-Darabsah, Suraj Nair, Jonathan May, Tarun Agarwal, Tao Yang, Choon Hui Teo
    ECIR, 2025
    📄 paper

  • CRISP: Clustering Multi-Vector Representations for Denoising and Pruning
    João Veneroso, Rajesh Jayaram, Jinmeng Rao, Gustavo Hernández Ábrego, Majid Hadian, Daniel Cer
    arXiv, 2025
    📄 paper

  • Towards Lossless Token Pruning in Late-Interaction Retrieval Models
    Yuxuan Zong, Benjamin Piwowarski
    SIGIR, 2025
    📄 paper

  • ColPruner: Combining Complementary Pruning Approaches for ColBERT in Web Search
    Wondo Rhee, Chan Lim, Taewon Yoon, Gyuhyeon Choi, Jooyoung Lee
    ReNeuIR Workshop, 2025
    📄 paper

  • Towards Storage-Efficient Visual Document Retrieval: An Empirical Study on Reducing Patch-Level Embeddings
    Yubo Ma, Jinsong Li, Yuhang Zang, Xiaobao Wu, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Haodong Duan, Jiaqi Wang, Yixin Cao, Aixin Sun
    ACL Findings, 2025
    📄 paper

  • Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Framework
    Yibo Yan, Mingdong Ou, Yi Cao, Xin Zou, Jiahao Huo, Shuliang Liu, James Kwok, Xuming Hu
    arXiv, 2026
    📄 paper

  • Coverage Matters: MarginMerge for Compressing Multi-Vector Visual Document Retrievers
    Ailar Mahdizadeh, Aria Salari, Sohail Rajabi, Shahriar Mirabbasi, Panos Nasiopoulos, Alireza Morsali
    arXiv, 2026
    📄 paper

  • Multi-Vector Index Compression in Any Modality
    Hanxiang Qin, Alexander Martin, Rohan Jha, Chunsheng Zuo, Reno Kriz, Benjamin Van Durme
    SIGIR, 2026
    📄 paper | 🛠️ code

  • Learn to Pool: Lightweight Fine-Tuning for Flexible Multi-Vector Compression
    Stefan Josef
    LIR Workshop, 2026
    📄 paper

  • A Voronoi Cell Formulation for Principled Token Pruning in Late-Interaction Retrieval Models
    Yash Kankanampati, Yuxuan Zong, Nadi Tomeh, Benjamin Piwowarski, Joseph Le Roux
    SIGIR, 2026
    📄 paper

  • CrossQ: Task-Aligned Cross-Token Conditional Quantization for Late Interaction Retrieval
    Rohit Kumar Salla, Manoj Saravanan, Ramya Manasa Amancherla
    ICML, 2026
    📄 paper

  • EigenLI: Spectral Approximations to Late Interaction
    Archish S, Sabyasachi Basu, Ankit Garg, Ravishankar Krishnaswamy, Kirankumar Shiragur
    arXiv, 2026
    📄 paper

  • Generative Late-Interaction Embeddings For Visual Document Retrieval
    Mohamed Eltahir, Talal Aloushan, Rose Khairoalsendi, Jana Shata, Mohammed Alhassan, Leen Alrehaili, Tanveer Hussain, Naeemullah Khan
    arXiv, 2026
    📄 paper

#Multimodal & Vision

  • ColPali: Efficient Document Retrieval with Vision Language Models
    Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Celine Hudelot, Pierre Colombo
    ICLR, 2025
    📄 paper

  • Video-ColBERT: Contextualized Late Interaction for Text-to-Video Retrieval
    Arun V. Reddy, Alexander Martin, Eugene Yang, Andrew Yates, Kate Sanders, Kenton Murray, Reno Kriz, Celso M. de Melo, Benjamin Van Durme, Rama Chellappa
    CVPR, 2025
    📄 paper

  • ColMate: Contrastive Late Interaction and Masked Text for Multimodal Document Retrieval
    Ahmed Masry, Megh Thakkar, Patrice Bechard, Sathwik Tejaswi Madhusudhan, Rabiul Awal, Shambhavi Mishra, Akshay Kalkunte Suresh, Srivatsava Daruru, Enamul Hoque, Spandana Gella, Torsten Scholak, Sai Rajeswar
    EMNLP, 2025
    📄 paper

  • MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction
    Zilin Xiao, Qi Ma, Mengting Gu, Chun-cheng Jason Chen, Xintao Chen, Vicente Ordonez, Vijai Mohan
    ICLR, 2026
    📄 paper | 🛠️ code

  • AdaptiveEmbed: Sample-Adaptive Multi-Vector Representation for Multimodal Retrieval
    Xinze Liu, Lei Yang, Dayan Wu, Hengjie Zhu, Zihao Zhang, Hanqi Wu, Tianzhu Hu, Peng Fu, Zheng Lin, Weiping Wang
    arXiv, 2026
    📄 paper

#Theory

  • Multi-Vector Embeddings are Provably More Expressive than Single Vector Embeddings
    Rajesh Jayaram
    arXiv, 2026
    📄 paper

  • Quantifying and Expanding the Theoretical Capacity of Late-Interaction Retrieval Models
    Julian Killingback, Varad Ingale, Hamed Zamani, Cameron Musco
    arXiv, 2026
    📄 paper

  • Retrieval Needs Multivectors: An Exponential Separation
    Mihir Agarwal, Viraj Agrawal, Sabyasachi Basu, Ankit Garg, Kirankumar Shiragur
    arXiv, 2026
    📄 paper

#Retrieval

#Indexing & Search Algorithms

  • Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrieval
    Omar Khattab, Christopher Potts, Matei Zaharia
    NeurIPS, 2021
    📄 paper

  • PLAID: An Efficient Engine for Late Interaction Retrieval
    Keshav Santhanam, Omar Khattab, Christopher Potts, Matei Zaharia
    CIKM, 2022
    📄 paper | 🛠️ code

  • DESSERT: An Efficient Algorithm for Vector Set Search with Vector Set Queries
    Joshua Engels, Benjamin Coleman, Vihan Lakshman, Anshumali Shrivastava
    NeurIPS, 2023
    📄 paper

  • Efficient Multi-Vector Dense Retrieval with Bit Vectors
    Franco Maria Nardini, Cosimo Rulli, Rossano Venturini
    ECIR, 2024
    📄 paper | 🛠️ code

  • Efficient Constant-Space Multi-vector Retrieval
    Sean MacAvaney, Antonio Mallia, Nicola Tonellotto
    ECIR, 2025
    📄 paper

  • IGP: Efficient Multi-Vector Retrieval via Proximity Graph Index
    Ziyang Bian, Man Lung Yiu, Buzhou Tang
    SIGIR, 2025
    📄 paper | 🛠️ code

  • WARP: An Efficient Engine for Multi-Vector Retrieval
    Jan Luca Scheerer, Matei Zaharia, Christopher Potts, Gustavo Alonso, Omar Khattab
    SIGIR, 2025
    📄 paper | 🛠️ code

  • Multivector Reranking in the Era of Strong First-Stage Retrievers
    Silvio Martinico, Franco Maria Nardini, Cosimo Rulli, Rossano Venturini
    ECIR, 2026
    📄 paper | 🛠️ code | 🛠️ code

  • SMVE: Sparse Multi-Vector Retrieval
    Martin Spisak, Marek Galovic
    LIR Workshop, 2026
    📝 blog

  • No More K-means: Single-Stage Sparse Coding for Efficient Multi-Vector Retrieval
    Lixuan Guo, Yifei Wang, Tiansheng Wen, Aosong Feng, Stefanie Jegelka, Chenyu You
    ICML, 2026
    📄 paper

  • LEMUR: Learned Multi-Vector Retrieval
    Elias Jääsaari, Ville Hyvönen, Teemu Roos
    ICML, 2026
    📄 paper | 🛠️ code

  • Efficient Multivector Retrieval with Token-Aware Clustering and Hierarchical Indexing
    Silvio Martinico, Franco Maria Nardini, Cosimo Rulli, Rossano Venturini
    SIGIR, 2026
    📄 paper | 🛠️ code

  • ColBERTSaR: Sparsified ColBERT Index via Product Quantization
    Eugene Yang, Andrew Yates, Dawn Lawrie, James Mayfield, Saron Samuel, Rohan Jha
    SIGIR, 2026
    📄 paper | 🛠️ code

  • PLAID-PRF: Pseudo-Relevance Feedback with Centroid-like Tokens in PLAID
    Xiao Wang, Sean MacAvaney, Craig Macdonald
    SIGIR, 2026
    📄 paper | 🛠️ code

  • Chimera: Efficient Multi-Vector Retrieval via GPU-CPU Co-Processing
    Yanqi Chen, Juelin Liu, Alexandra Meliou, Xiao Yan
    arXiv, 2026
    📄 paper

#Scoring Kernels

  • Col-Bandit: Query-Time Top-K Estimation for Late-Interaction Retrieval
    Roi Pony, Adi Raz Goldfarb, Oshri Naparstek, Idan Friedman, Udi Barzelay, Eli Schwartz
    arXiv, 2026
    📄 paper

  • FLASH-MAXSIM: IO-Aware Fused Kernels for Late-Interaction Scoring
    Roi Pony, Adi Raz Goldfarb, Idan Friedman, Daniel Ezer, Udi Barzelay
    arXiv, 2026
    📄 paper | 🛠️ code

  • TileMaxSim: IO-Aware GPU MaxSim Scoring with Dimension Tiling and Fused Product Quantization
    Ashutosh Sharma
    arXiv, 2026
    📄 paper | 🛠️ code

#Surveys, Analyses & Reproducibility

  • A Reproducibility Study of PLAID
    Sean MacAvaney, Nicola Tonellotto
    SIGIR, 2024
    📄 paper

  • A Replicability Study of XTR
    Rohan Jha, Reno Kriz, Benjamin Van Durme
    arXiv, 2026
    📄 paper

  • Reproduction Beyond Benchmarks: ConstBERT and ColBERT-v2 Across Backends and Query Distributions
    Utshab Kumar Ghosh, Ashish David, Shubham Chatterjee
    SIGIR, 2026
    📄 paper

  • Comparing Token Pruning Approaches for Multi-Vector Retrieval
    Ferdinand Schlatt, Hanno Barschel, Matthias Hagen
    SIGIR, 2026
    📄 paper

  • A Brief Comparison of Training-Free Multi-Vector Sequence Compression Methods
    Rohan Jha, Chunsheng Zuo, Reno Kriz, Benjamin Van Durme
    LIR Workshop, 2026
    📄 paper

  • A Survey of Late-Interaction Neural Retrieval: Paradigms, Systems, and Research Frontiers
    Xiao Wang, Chuting Yu, Minghan Li, Binci Yang, Hang Li, Ben He
    SSRN, 2026
    📄 paper

#Software Libraries

#Training & Inference Frameworks

  • ColBERT Python
    Reference implementation for ColBERT and ColBERTv2, and includes PLAID support for efficient late-interaction retrieval.

  • RAGatouille Python
    Python toolkit to train and serve ColBERT-based late-interaction retrievers.

  • PyLate Python
    Python library for training, fine-tuning, inference, and retrieval with ColBERT-style late-interaction models on single and multi-GPU setups.

  • Sentence Transformers Python
    Framework for dense, sparse, reranker, and (v6.0+) ColBERT-style multi-vector late-interaction models via the MultiVectorEncoder, with a unified training and inference API.

  • PyLate-rs Rust Python
    High-performance Rust inference engine for PyLate models, with Python bindings and optimized integration with FastPlaid for retrieval pipelines.

#Retrieval Engines & Indexes

  • FastPlaid Python
    GPU-optimized engine for ColBERT/PLAID-style late-interaction retrieval.

  • kANNolo Rust Python
    ANN library for dense, sparse, and multivector retrieval.

  • Vectorium Rust
    Rust library for compact storage/access of dense, sparse, and multivector embeddings.

  • Firn Rust
    Rust search engine for single-vector and late-interaction multivector namespaces, backed by LanceDB on object storage with RAM/NVMe result caching.

  • NextPlaid Rust Python
    CPU-oriented local-first multivector retrieval engine with memory-mapped storage.

  • EMVB C++
    Reference implementation for Efficient Multi-Vector Dense Retrieval with Bit Vectors.

  • IGP C++
    Official C++ implementation for IGP: proximity-graph indexing for multi-vector retrieval (with Python scripts for experiments).

  • WARP Python
    Official implementation for WARP, an efficient multi-vector retrieval engine.

  • ColGrep Rust Python
    High-performance code search CLI tool powered by LateOn-Code and NextPlaid, enabling semantic + hybrid (regex + semantic) code retrieval locally with incremental indexing.

  • TACHIOM Rust Python
    Fast and scalable multivector retrieval system with Token-Aware Clustering (TAC) and hierarchical Product Quantization for efficient late-interaction search.

  • TopK Rust Python
    Managed retrieval engine with support for late-interaction search over billions of documents, online index updates, filtering, and more.

#Scoring Kernels

  • Flash-MaxSim Python
    IO-aware Triton kernel for MaxSim scoring in ColBERT/ColPali pipelines: tile-by-tile on-chip computation with zero intermediate memory and INT8 quantization support.

  • maxsim Python
    Ahead-of-time compiled MaxSim kernel with CUDA and Metal backends (NVIDIA + Apple Silicon), distributed as a HuggingFace kernels package.

  • late-interaction-kernels Python
    Fused Triton kernels for MaxSim scoring with CUDA, Metal, and CPU backends, native PyLate/colpali-engine integration, and PLAID-style compressed-index support.

  • maxsim-cpu Rust Python
    CPU-only MaxSim kernel written in Rust (libxsmm on x86, Apple Accelerate on ARM) with Python bindings.

  • TileMaxSim Python
    IO-aware Triton kernel for MaxSim scoring with dimension tiling for embeddings wider than 128 dims and fused product quantization, achieving 80%+ peak HBM bandwidth.

#Model Checkpoints

#General-Purpose

  • colbert-ir/colbertv2.0
    Official ColBERTv2 checkpoint (MS MARCO-trained) from the ColBERT authors, widely used as the canonical baseline model.

  • lightonai/LateOn
    State-of-the-art ColBERT model (149M, ModernBERT-based) achieving 57.22 NDCG@10 on BEIR with fully open training data and strong generalization under decontamination.

  • lightonai/mLateOn
    Multilingual ColBERT model (307M, mmBERT-based) covering 9 languages with SOTA multilingual, long-document, and code retrieval; strong zero-shot generalization to unseen languages (57.56 NDCG@10 on BEIR).

  • jinaai/jina-colbert-v2
    Multilingual late-interaction retriever (0.6B, JinaBERT-based) supporting 89 languages with Matryoshka token embeddings (128/96/64 dims) for flexible efficiency-precision tradeoffs.

  • chungimungi/GLInt
    English ColBERT model (149M, LateOn-based) trained with geometry-matched hard negatives mined via MaxSim geometry, reporting 57.43 NDCG@10 on BEIR.

  • lightonai/LateOn-regularized
    LateOn variant trained with STE-based regularization to fix compatibility with projection-based retrieval methods (MUVERA, SMVE).

  • lightonai/ColBERT-Zero
    Large-scale fully pre-trained ColBERT checkpoint trained on public data and released with the ColBERT-Zero paper.

  • lightonai/GTE-ModernColBERT-v1
    PyLate late-interaction checkpoint based on ModernBERT with 128-dimensional token embeddings and strong long-context retrieval behavior.

  • topk-io/Iso-ModernColBERT
    Isotropically corrected version of GTE-ModernColBERT-v1 built for efficient inference and scalable retrieval.

  • sebastian-hofstaetter/colberter-128-32-msmarco / sebastian-hofstaetter/uni-colberter-128-1-msmarco
    ColBERTer checkpoints trained on MS MARCO (128-dim, with 32 and 1 unique whole-word vectors per document respectively).

#Specialized / Domain

  • lightonai/LateOn-Code
    Specialized ColBERT model (149M parameters) fine-tuned for code retrieval, achieving SOTA on MTEB Code benchmark.

  • lightonai/LateOn-Code-edge
    Lightweight code retrieval model (17M parameters) for edge devices, matching larger models while running efficiently on CPU.

  • lightonai/Reason-ModernColBERT
    Reasoning-focused late-interaction checkpoint fine-tuned on reasonir-hq, with strong BRIGHT benchmark performance for reasoning-intensive retrieval.

  • nlpai-lab/KURE-v2
    Korean-English bilingual late-interaction model (154M, skt/A.X-Encoder-base) with 128-dim token vectors and 8,192-token context, reporting 0.8160 average nDCG@10 on MTEB(kor, v2).

  • nlpai-lab/KURE-v2-unsupervised
    Stage-1 KURE-v2 checkpoint trained with weakly-supervised contrastive learning only (20.7M pairs, no relevance labels), reaching 0.7283 average nDCG@10 on MTEB(kor, v2).

#Datasets and Encodings

#MS MARCO v1

  • Documents: 8,841,823
  • Queries [dev.small]: 6,980
  • Reference Metric: MRR@10
Encoding Link Vector dim Avg vectors per doc Avg vectors per query MRR@10
colbertv2 link 128 67 32 0.397

#LoTTE-pooled

  • Documents: 2,428,854
  • Queries [dev/search]: 2,931
  • Reference Metric: Success@5
Encoding Link Vector dim Avg vectors per doc Avg vectors per query Success@5
colbertv2 link 128 109 32 N/A

#Multimedia Resources

  • Omar Khattab on Late Interaction in 2030. Link
  • Multi-Vector Search with Amélie Chatelain and Antoine Chaffin - Weaviate Podcast #134. Link

Neue Version verfügbar.