Multimodal embeddings beyond a single vector

A new family of late-interaction models for text and image retrieval, with a shared embedding space across model sizes.

AuthorsPerplexity Research

Since releasing pplx-embed-v1 in February, we have been working on a new generation of embedding models. Last week, we introduced our new contextual embedding model, pplx-embed-v2-context. Today, we’re expanding the family with pplx-embed-v2-late, a collection of late-interaction models. Alongside stronger retrieval capabilities, the models are the first to combine three core features:

  • Multi-vector representations. Rather than compressing each input into a single vector, these models retain token-level vectors for finer-grained query-document comparisons. This allows different parts of a query to match different parts of a document, preserving detail that is especially valuable for multimodal retrieval.
  • Support for both text and image modalities. In addition to the text-to-text retrieval capabilities of our previous models, these new models handle text-to-image retrieval, allowing them to directly search rendered PDF pages without requiring OCR or parsed text.
  • A shared embedding space. We are releasing two model sizes, 0.6B and 9B, to support different quality, latency, and compute requirements. Each can be used independently, but the shared embedding space also allows the smaller model to query an index created with the larger model, combining higher-quality document embeddings with cheap, fast query-time inference.

These new models achieve state-of-the-art results on a range of vision and text retrieval benchmarks, as well as agentic question-answering tasks. Notably, on ViDoRe (V3), a leading visual document retrieval benchmark, our 0.6B model matches models with five times as many active parameters. Both models are publicly available on Hugging Face. Our technical report provides more details on the methodology and evaluation results.

From single vectors to late interaction

Retrieval systems must find a balance between efficiency and the depth of interaction between queries and documents. Dense embedding models encode each query and document independently, allowing them to be compared with a cheap inner product. This makes approximate nearest-neighbor (ANN) search over billions of documents practical. At the other extreme, cross-encoders process the query and document together, enabling richer interaction but requiring a separate forward pass for every candidate. Applying this computation across a large corpus of documents for every search is prohibitively expensive, so cross-encoders are typically reserved for reranking a small set of retrieved candidates.

Limitations of dense embeddings

Representing an entire document with one fixed-size vector creates an information bottleneck. As documents grow longer and cover more topics, the model must compress more content into the same vector, making it harder to preserve everything relevant to a query. Chunking can reduce this pressure, but it fragments the document and can separate information that only makes sense together, an issue we also tackle with our contextual embedding model. This bottleneck worsens in multimodal settings, where the information density of images requires even more compression.

Single-vector dense retrieval also reduces every comparison to an inner product between two vectors, limiting the expressivity of the scoring function. For a fixed embedding dimension and a large enough corpus, some patterns of scores across query-document pairs are impossible to produce, regardless of training. The LIMIT benchmark turns this theoretical concern into a concrete stress test for single-vector retrieval.

Late-interaction retrieval

Late-interaction models occupy the space between the cheaper dense embedding models and the more expressive cross-encoders. Like dense models, they encode documents independently of the query, allowing document representations to be computed offline. However, rather than reducing each input to a single vector, they retain multiple vectors and use a richer scoring function when comparing a query with a document.

Our pplx-embed-v2-late models use a ColBERT-style architecture, outputting 128-dimensional embeddings for each token in the input. We score query-document pairs using MaxSim; for each query token, the model selects the highest-scoring document token and sums the per-token maximum similarities to produce the final score. This lets different parts of a query match different parts of a document, with stronger token matches contributing more to the score.

An example of MaxSim scoring: Each query token selects its highest-scoring document token, the maximum token-level similarities sum to a query-document level similarity score.

This additional expressiveness comes with higher storage and candidate-scoring costs. Dense retrieval stores one vector per document and scores a candidate with one inner product; late interaction stores vectors for document tokens and evaluates many token-to-token similarities. The cost grows with document length, so late interaction requires a different indexing and serving trade-off than one-vector ANN retrieval. In return, it provides richer first-stage retrieval for quality-sensitive corpora and can also serve as a later stage in a larger web-scale retrieval pipeline.

One model, multiple modalities

This improved depth of interaction has been shown to be very useful for various tasks, including long-context, agentic and out-of-domain retrieval. This difference is particularly evident in multimodal retrieval, as shown by the large lead of ColPali models on the ViDoRe(V3) leaderboard.

Real-world corpora are not composed only of clean text. They arrive as PDFs, slide decks, screenshots, scanned pages, charts, and tables. The conventional pipeline parses these documents, applies OCR or captioning, and searches the resulting text, which is expensive and lossy: it can introduce OCR errors and it discards layout, table structure, figures, and spatial relationships. Visual document search skips that step, projecting the rendered page directly into the representation space where textual queries can search it.

Our new family of models builds on the strong performance of multi-vector models for multi-modal search and advances the Pareto frontier for small models on the most widely used document search benchmark, ViDoRe(V3). While we expect document renderings to be the most common form of visual input, our models also support semantic image search over natural images, such as scenic photographs.

Training a family of interoperable models

Search is fundamentally asymmetric: indexing and query encoding have different operational constraints. Indexing runs offline and can be parallelized, so its compute cost is amortized across many future searches. Query encoding runs online for every request, making it latency-sensitive and sometimes subject to serving or edge-device limits. Our model family makes use of this asymmetry, offering sizes tailored to each side of the retrieval pipeline.

We train an 18B teacher contrastively, then distill it separately into 9B and 0.6B students. Rather than training each student only to reproduce the teacher’s ranking distribution, we use LEAF-style representation distillation: for each input token, we align the student’s output embedding with the teacher’s. While representation-space distillation has previously been used for dense, single-vector models, we extend it to late interaction, where the alignment target is a vector for every retained token.

As a result, both released checkpoints share the same multi-vector embedding space. This enables cross-model retrieval, making additional deployment options possible.

  • Maximum quality: encode documents and queries with the 9B model.
  • Maximum efficiency: encode documents and queries with the 0.6B model for 100% local inference
  • Low-latency retrieval: build the corpus index with the 9B model, then encode live queries with the 0.6B model to leverage index-time computation while keeping querying fast.
  • Local–cloud retrieval: encode private local documents or live queries with the 0.6B model on-device, and compare or merge those representations with results from a 9B index hosted in the cloud.

For latency-sensitive settings, we designed the 0.6B model as a lightweight query encoder that can also run on edge devices. Starting from Qwen3.5-0.8B, we prune its 24-layer text tower to 12 layers, reducing that tower from 498M to 240M non-embedding parameters. The resulting model has 594M parameters in total, but its nominal size overstates the serving cost: 100M are used for image encoding, and 254M sit in the token-embedding table, which has little effect on runtime. Only about 240M parameters are active for text encoding and 340M for image encoding, giving the model the serving cost of a substantially smaller model while retaining compatibility with the 9B index.

We applied the same process to build our 18B teacher from a 27B backbone, which made the teacher considerably cheaper to distill from without weakening the signal it provides.

Multilingual, multimodal, uncontaminated training data

We train both models on a dataset of 186 million query-document pairs drawn from 594 datasets in 46 languages. Of these pairs, 88.3% are text-to-text, 8.3% are text-to-image, and 3.4% are text-to-visual-document pairs. Rather than sampling in proportion to dataset size, we assign each dataset a sampling weight, which yields a more balanced training mixture of 56.5%, 30.9%, and 12.6%, respectively.

When constructing our training data, we exclude any datasets associated with benchmarks that we evaluate on to faithfully represent our results without inflating benchmark scores. This decision comes with the trade-off that some of the datasets we chose to exclude contain high-quality data which could improve the performance of our model on tasks besides the benchmark they came from. Ultimately, we chose to accept that cost: trustworthy numbers are more useful to us during development, and to users when comparing models.

Evaluation

We evaluate both models across text, visual-document, mixed-image, and agentic retrieval. Both models outperform the evaluated baselines on Q2D-Web and BrowseComp+, and achieve strong results on visual-document retrieval and MADQA.

Text retrieval

We evaluate text-to-text retrieval on three benchmarks:

  • Domain-specific benchmarks: A collection of 72 MTEB-style tasks, 55 across five domains (finance, legal, health, conversation, tech), plus 17 miscellaneous multilingual tasks. The collection includes tasks such as FinQA, AILACasedocs and XLSum. None of these datasets appear in our training data, as described above.
  • Q2D-Web: our web-scale retrieval benchmark of about 70,000 agent-reformulated search queries, drawn from production search traffic, over a corpus of 190M web documents. This benchmark measures retrieval performance in a setting much closer to real-world web retrieval than other available retrieval benchmarks, due to its production-like queries and large-scale corpus.
  • ViDoRe(V3) Markdown: ViDoRe(V3) is typically a text-to-visual-document retrieval task, but it can also be evaluated as a text task using a version of the corpus that has been parsed to Markdown, allowing us to evaluate information-rich document retrieval in both image and text modalities.

We measure performance using nDCG@10, except on Q2D-Web, where we report Recall@1000 due to the benchmark’s focus on a first-stage web-retrieval scenario. Throughout our evaluations, we compare against industry-leading embedding models across different size classes. For text retrieval, nemotron-embed-8b, gemini-embedding-2, voyage-large-4, and qwen3-embed-8b represent the strongest and most widely used dense competitors, while voyage-4-nano and qwen3-embed-0.6b serve as sub-billion parameter dense baselines. We additionally compare against TopK's recently released topk-embed-v1 late-interaction models on the benchmarks for which TopK has published results. ViDoRe(V3) scores are taken from their model cards at 128 dimensions, matching the width of our token embeddings; the Q2D-Web score is taken from our leaderboard.

On the domain-specific benchmarks, our 9B and 0.6B models average 81.3% and 78.0%, respectively. Our 9B model achieves the highest score of all models we evaluated, beating the next best model by 1.6pp. Our 0.6B model is competitive with the other top-scoring models, scoring only 0.3pp behind gemini-embedding-2. Importantly, since we did not train on any of the data for these benchmark tasks, the domain-specific benchmarks demonstrate that our embedding models can generalize to new domains.

Domain-specific text retrieval performance, measured by nDCG@10. The average weights the six domains equally, and parentheses show the number of tasks in each group.

The relative strength of our 0.6B model is even greater on the ViDoRe(V3) Markdown task; its score of 61.2% beats all other models besides our 9B model, which scored 64.7%. The next best external model scored 60.6%, and the next best sub-1B model, topk-embed-v1-0.8b, scored 4.3pp behind our 0.6B model.

Text retrieval on the public subset of ViDoRe v3 after converting page images to Markdown with OCR. Our 0.6B model ranks second only to our 9B model on average nDCG@10.

On Q2D-Web, relevance comes from three judgment sets: documents the agent cited (Citation), documents ranked by our production retrieval system (Web), and the union of both extended with LLM judgments of unlabeled documents (Combined). Both pplx-embed-v2-late models lead every baseline on all three judgment sets. We focus on the combined judgments, since the Citation and Web sets are derived from what our own systems already surfaced and so favour models that rank as they do; the LLM judgments added in Combined cover documents neither system retrieved. On that set, both our 9B and 0.6B models beat all competitors, reaching scores of 74.8% and 73.6%, respectively, against the previous best score of 69.3%.

Web-scale text retrieval on Q2D-Web, measured by Recall@1000 under Citation, Web, and Combined relevance judgments. Both models outperform the evaluated baselines across all three judgment sets.

Visual retrieval

Our primary visual retrieval benchmark is the standard public subset of ViDoRe(V3), where the corpus consists of rendered page images rather than the Markdown conversion. We compare against two leading dense vision-language models, qwen3-vl-embed-8b and gemini-embedding-2, and against the topk-embed-v1 late-interaction models. In addition, Tencent's EVIE models and NVIDIA's Nemotron ColEmbed V2 models are “vision-only” multi-vector baselines trained specifically for image retrieval tasks.

Both of our models achieve state of the art performance for vision-language models. Our 9B model scores 65.2% on average, trailing only EVIE. Our 0.6B model achieves a score of 62.3%, ahead of topk-embed-v1-2b and trailing only much larger EVIE and nemotron-colembed-v2-8b among external baselines.

Visual document retrieval on the public subset of ViDoRe v3, measured by nDCG@10. Both Perplexity models lead the evaluated vision-language baselines; vision-only models are shown separately.

Our 0.6B model is particularly competitive in the context of model size. It is ahead of all comparable size models and within 1.2pp of nemotron-colembed-v2-8b while activating only 340M parameters for image inputs. Our models produce 128-dimensional embeddings per token, compared with 2,048 for EVIE-4.5B and 4,096 for EVIE-8B and nemotron-colembed-v2-8b. This difference in storage costs can drastically affect the scenarios in which these models are practical to use.

ViDoRe v3 image-retrieval quality versus active model parameters on a logarithmic scale. Lines connect the Pareto frontier within each panel: models not outperformed by an equal-sized or smaller model.

We also evaluate visual retrieval on MIRACL-Vision, a benchmark whose corpus consists of screenshots of wikipedia articles, and PPLX-Q2I, an internal benchmark covering both visual documents and natural images. PPLX-Q2I is built from Perplexity's production image-search logs and designed to reflect the diversity of image search use cases. We use a 10,000-query sample, stratified by language and domain, against a corpus of 100,000 images; each query's relevant documents are the images served for it. We did not evaluate EVIE-8B or Nemotron ColEmbed on PPLX-Q2I, as indexing a corpus this size at their default output dimensionalities (2,560 for Nemotron ColEmbed 4B and 4,096 for Nemotron ColEmbed 8B and EVIE-8B) was impractical.

Similar to its performance on ViDoRe(V3), our 0.6B model is competitive with much larger models, scoring only 1pp behind qwen3-vl-embed-8b on MIRACL-Vision. Our 9B model outperforms all other models except for gemini-embedding-2.

On PPLX-Q2I, both of our models beat qwen3-vl-embedding-8b by a margin of over 9pp, and our 9B model is behind gemini-embedding-2 by only 2pp. Our results stay competitive on corpora that contain natural images as well as visual documents, so the multi-vector representations that pay off on ViDoRe(V3) do not come at the cost of general image search.

Image retrieval on MIRACL-Vision and PPLX-Q2I, measured by nDCG@10. Nemotron ColEmbed models were not evaluated on PPLX-Q2I because indexing the corpus at their default output dimensionalities was impractical.

Cross-model retrieval

The results in the previous sections evaluate each model against a corpus it embedded itself. The shared embedding space means this is not a requirement: a corpus indexed by the 9B model can be searched with queries embedded by the 0.6B model. This is the configuration we expect most production deployments to use, since indexing is a one-off cost amortized over the lifetime of the corpus while query embedding runs on the latency-critical path of every request.

A deployment that cannot run the 9B model at query time would otherwise index with the 0.6B model as well, so that symmetric 0.6B configuration is the relevant point of comparison. We evaluate against the 72 domain-specific benchmarks used above and on ViDoRe v3 image retrieval. Queries are embedded with pplx-embed-v2-late-0.6b, and we vary only what embedded the corpus.

Indexing with the 9B model is a strict improvement. On the domain-specific benchmarks, the asymmetric configuration averages a gain of 1.6pp. On ViDoRe v3 image retrieval, it scores 63.5%, up from 62.3% with the 0.6B model on both sides, a gain of 1.2 percentage points. Query-encoding cost is unchanged between the symmetric 0.6B and asymmetric configurations, since the same 0.6B model encodes the query in both; the additional document-encoding cost is incurred at indexing time.

Running the 9B model on both sides remains better still, averaging 81.3% on the domain-specific benchmarks. The asymmetric configuration therefore recovers roughly half of the quality difference between the two models on text, for no additional query-encoding cost. Which configuration to choose depends on whether the 9B model is affordable at query time; where it is not, the asymmetric setup captures a substantial part of its benefit.

Embedding the index with the 9B model and queries with the 0.6B model improves retrieval over using the 0.6B model for both. Scores are nDCG@10; the six-domain average excludes the separately reported ViDoRe v3 results.

Agentic retrieval

Agentic retrieval is another setting where multi-vector embeddings are particularly strong. In BrowseComp+, an LLM (GPT-OSS-120B in our case) uses the embedding model to search a corpus for information relevant to a complex user query, and a second LLM judges whether the resulting answer matches the ground truth. Our 9B model achieves 64.0% accuracy, beating the next best ColBERT model by 4.9pp and the next best dense model by 8.7pp, while issuing fewer searches than any other model except our 0.6B. The 0.6B model is also ahead of every model outside our family.

BrowseComp+ answer accuracy and retrieval recall versus the average number of searches for a GPT-OSS-120B agent (high effort). Higher scores and fewer searches indicate better quality and efficiency.

MADQA tests agentic question-answering over 800 heterogeneous real-world PDFs spanning more than 18,000 pages, with 500 human-authored questions that cannot be answered from general knowledge. An agent (Gemini 3.5 Flash in our case) searches the collection with the retriever under test and is scored on answer accuracy and on page F1 against the annotated evidence pages. Our 9B model sets a new state of the art among retrievers at 92.4% accuracy, 3.5pp ahead of the same agent using Mixedbread’s retriever; our 0.6B model reaches 90.1%. We also note that our retriever achieves competitive results against Mixedbread Agentic Search, which uses a search sub-agent in their search system that plans and runs several searches per call from the main agent (93.4%, within our confidence interval).

Answer accuracy and page-level F1 on MADQA. Paired with Gemini 3.5 Flash, our 9B and 0.6B models achieve 92.4% and 90.1% accuracy, respectively, compared with 88.9% for the Mixedbread retriever and 93.4% for Mixedbread Agentic Search.

Getting started

Both our 0.6B and 9B models are available on Hugging Face and are compatible out of the box with sentence-transformers >= 6.0.0 and transformers >= 5.4.0. Further technical details and complete evaluation results will be published in a technical report later this year.

We are currently training the next generation of our dense embedding models, and will progressively roll out support for late-interaction, dense, and contextual embeddings on the Perplexity API Platform.

Citation

BibTeX
@online{perplexity2026MultimodalEmbeddingsBeyondASingleVector,
  author = {{Perplexity} and Markus Krimmel and Louis Milliken and Antoine Chaffin and Sarah Eslami and Ekaterina Glazkova and Maximilian Schall and Bo Wang and Denis Bykov and Jarrod Bryan and Claire Beamer and Jeremy Yang},
  title = {{Multimodal embeddings beyond a single vector}},
  date = {2026-10-07},
  url = {https://perplexity.coolkeypoint.com/hub/blog/multimodal-embeddings-beyond-a-single-vector}
}