Hugging Face trains multi-vector retrieval for token pooling

Hugging Face published a pooling-aware training method on November 11, 2026, that optimizes a multi-vector retrieval model against both full and compressed document representations. In the author’s NanoBEIR@13 evaluation, it retained 97.8% of its own unpooled nDCG@10 score while storing 74.8% fewer document vectors with factor-four pooling. No independent replication was found in the opened evidence.

The method targets the storage burden of multi-vector retrieval. These systems keep contextual vectors for individual document tokens and score them through late interaction, rather than compressing each document into one vector. Token pooling reduces the count by merging similar token vectors. Fewer vectors do not necessarily translate into an equivalent reduction in bytes, latency or cost, which also depend on vector dimensions, quantization, index structure and hardware.

The pooler described in the post uses Ward hierarchical clustering. It repeatedly merges the pair of token-vector groups that produces the smallest increase in within-group squared distance, then replaces each resulting group with a normalized mean vector. With pool_factor=4, documents retain about one quarter of their original vectors by default, while query embeddings remain unpooled.

Pooling-aware training adds the compressed representation to the objective instead of applying pooling only after ordinary training. The demonstrated configuration gives equal weight to retrieval losses from unpooled documents and factor-four pooled documents, with both losses updating the same encoder. Sentence Transformers documentation says hierarchical pooling passes gradients through the cluster means with a straight-through estimator, while the discrete cluster assignments themselves are not differentiated.

For the comparison, the author said both models started from ModernBERT-base and used one epoch over the same 120,000 MS MARCO triplets, seed 42, and matching initialization and optimization settings; only the loss changed. The public training script reflects those settings and creates a separate 1,000-example held-out split. The author also said no hyperparameter search was run.

In the author-run NanoBEIR@13 benchmark, the ordinarily trained model scored 51.84 mean nDCG@10 without pooling and 49.03 with factor-four pooling, or 94.6% retention. The pooling-aware model scored 52.39 unpooled and 51.23 when pooled, or 97.8% retention. Its unpooled score was also 0.55 points higher, so the comparison does not isolate compression robustness from every possible change in overall model quality.

The accompanying evaluation script tests unpooled retrieval, fixed caps of 64, 32, 16, 8 and 4 document vectors, and pool factors from two through six. It reports mean nDCG@10, allowing the same model to be compared across several storage budgets.

The method builds on earlier work rather than introducing token pooling itself. A 2024 paper by Benjamin Clavié, Antoine Chaffin and Griffin Adams described clustering-based token pooling for ColBERT-like indexes without architectural changes or query-time processing. Those findings do not independently validate Hugging Face’s 2026 training results, which remain limited to the disclosed models, training data and NanoBEIR setup.