Perplexity has recently unveiled pplx-embed, a set of multilingual embedding models tailored for large-scale retrieval tasks. These models are specifically crafted to handle the intricacies and noise of web-scale data, offering a viable alternative to proprietary embedding APIs.
One of the key architectural innovations in these models is the incorporation of bidirectional attention. Unlike most Large Language Models (LLMs) that typically utilize causal, decoder-only architectures, the Perplexity research team implemented bidirectional attention to enable the model to process all tokens in a sequence simultaneously. This results in a more comprehensive hidden state representation, which is crucial for embedding tasks.
Additionally, the models leverage diffusion-based pretraining, a technique commonly used in generative media. By applying diffusion to text embeddings, the models learn to reconstruct clean semantic signals from noisy or fragmented input. This pretraining phase ensures that the models are robust when processing unstructured text commonly found on the open web.
The models are optimized for Retrieval-Augmented Generation (RAG) scenarios, addressing the ‘asymmetry’ between a user’s short search query and a lengthy document chunk. To tackle this challenge, Perplexity offers two specialized model versions:
– pplx-embed-v1: Optimized for independent text embeddings and search queries.
– pplx-embed-context-v1: Specifically tuned for document chunks used in RAG pipelines.
By separating these roles, the models enhance the alignment between user queries and information stored in a database. These models have been successfully validated in real-world search scenarios involving millions of documents.
The models are available in two parameter scales to balance performance and computational cost. The inclusion of native INT8 quantization allows for deployment with a smaller memory footprint and faster inference speeds. This makes the 4B model suitable for production environments that previously required less capable models.
Key takeaways from the release include:
– Bidirectional Architecture via Diffusion: The models utilize bidirectional encoders with diffusion-based pretraining, allowing them to capture the entire context of a sentence at once for more accurate semantic representations.
– Specialized RAG Variants: Two distinct models are provided to optimize Retrieval-Augmented Generation, catering to independent queries and standalone text, as well as document chunks.
– Production-Ready Efficiency: The models support native INT8 and binary quantization, reducing storage and memory requirements without significant loss in accuracy. They also utilize Matryoshka Representation Learning (MRL) for cost-effective vector dimension truncation.
For more information, you can check out the Paper, Model Weights, and Technical details. Don’t forget to follow Perplexity on Twitter and join their ML SubReddit and Newsletter. You can also connect with them on Telegram for the latest updates.
Be the first to comment