Late Interaction Beats Single Vectors For RAG, introduction to PyLate

Hello everyone, my name is Antoine Chapin and I am a research engineer at LightOn. And today I will detail some of the limits of single vector search that have been highlighted by recent usages and evaluations. And then I will introduce multivector models, also known as latent interaction models, and how they can overcome this. And to finish, I will briefly present the PyLatent library that allows you to use as a light interaction model, so you can also benefit from them. So first, really quickly, a few words about me. Before joining LightOn, I did an engineering school followed by a PhD on the subject of multimodal misinformation detection, during which I studied multimodal information retrieval as well as guiding generative models in order to generate better synthetic data. And so at LightOn, I focused more on information retrieval, notably through the scope of encoders model and late interaction, which led me to being a co-creator of the ModernBERT encoder model that you might have heard of, as well as the PyLate library, which I will introduce a bit later today. I'm also still working a lot on multimodality, notably through the scope of creating OCR-free RAG pipeline. For example, with the creation of visual re-rankers such as the MonoQuen model we released earlier this year. And so if you are familiar with those topics, you might have seen me on Twitter where I spend quite a lot of time doing propaganda on this subject and also shitposting a bit. All right, so that's enough about me. Let's dive into today's subject. So I think you are all pretty familiar with the single vector search. Also called nSearch, that allows to search for relevant documents in the knowledge base based on a user query. And so essentially the query and the documents are fed to a transformer model, preferably an encoder such as BERT, or even better its modernized version, right? And the transformer model will create a contextualized vector representation of all of the tokens in the sequence, so in the query and in the document. And once we have all of these embeddings for the query and for the document, we will actually pool all of them into a single vector through a pooling operation such as max pooling, mean pooling, or just like taking the first vector of the sequence. And once we have a unique token for the query and a unique token for the document, we can compute the similarity between the two using a simple vector similarity operations such as the cosine similarity. And so the Transformer models are optimized to create vectors that are highly similar if the document is relevant to the query and lower one if the document is not relevant to the query. And so we can then perform search by looking for the document that has the closest vector to the input query from the user. So I just presented those single vector models, but I'm pretty sure you are all already very familiar with them as they became the go-to type of search for RAG pipeline, and they can be found in all the Ragatom in 2 lines of code boilerplates. And they are very popular because they offer like great performance out of the box. And also there is a lot of different models available with different sizes, languages, and domains. So you really have a large range to choose from. And finally, they are very well integrated now, and they are very easy to deploy with one of the various vectorDB provider and APIs that can serve those embeddings. So you can easily spin up something that works in just a few minutes. And so obviously, as you are all very well aware, evaluation is required to assess the performance of a model. And so the mTab leaderboard is a very nice resource that centralizes various benchmark results for various different models, which allows you to select the model that fits your budget and should perform correctly on your domain quite easily. And so one of the most known benchmarks on MTAB, at least for English retrieval, is the BEIR benchmark. And so it was originally introduced as a heterogeneous benchmark composed of various domains. And so its purpose was to broadly evaluate the performance of a model on various domains in order to assert that the model was correctly working on a domain that it wasn't necessarily trained for. And so it helped a lot in the quest of training better models, and by doing so, it became the main reference to report when evaluating retrieval results in English. However, as stated by the Goodhart's Law, when a measure becomes a target, it ceases to be a good measure. And so by becoming the standard objective to beat in order to get the bold number in a paper or in a blog post, The benchmarks have been gamified and now most of the top models of this leaderboard are actually overfitted on this domain and they can perform pretty poorly in actual use cases. It became more important than ever to run your own evaluation on your own data and use cases in order to make sure that the model actually performs correctly for you. And so besides this high climb league form of beer, other aspects of evaluation have contributed to the lack of realization limitation of some caveats of the existing model. For example, since most models only supported an input length of 512, there were very few evaluations of long context capabilities for retrieval models. And so with the new models supporting longer context length such as 8K, people thought they could just simply use those models to process larger documents. But when you actually evaluate them correctly, it appears pretty clearly that although you can feed these long documents to this model, the model do not actually support them correctly. And the results are really poor. So obviously, having evaluation allows to develop techniques to better train the model to handle such context. For example, most of the existing evaluation benchmarks focus on very brittle notion of similarity, where there is a direct lexical, or at the very least, semantical overlap between the query and the target documents. In modern use cases, especially in agentic workflow, we require more complex type of retrieval in which the definition of the relevance between the query and the target document might require some form of reasoning. For example, when prompted with a mathematical problem, finding examples that aren't the same problem but require the same theorem to be solved might be very useful, yet it is actually very difficult for existing retrieval models. If you are interested in this particular topic, I encourage you to watch Orion's talk from yesterday. He went way deeper into this specific subject and the talk was really good. And so yet again, when evaluating the existing model on this new capability, we observe actually very poor results compared to their usual performance on more traditional benchmarks. And it actually even holds for a very large model that use like billion parameters model as the backbone model. And so one common thing that should be noted is that for both of this evaluation, PM25, which is a very simple lexical method that do not leverage any deep learning, is actually very competitive. And this also holds for out-of-domain retrieval, where BM25 is to this day very much used in addition to the dense search engine in order to enhance the out-of-domain capabilities. And so this might very well be an hint as to why single vector models are actually struggling. Indeed, in the single vector padding, we are putting all the tokens into a single one. And so no matter which type of pooling is used, we are compressing a lot the information originally present in the document and the queries. During the training, the model will learn to keep the information that is helpful to differentiate between the positive documents from the negative documents for the input query, and it will actually ignore the rest because it needs to compress the information, and so it needs to save some space, essentially. So for example, if you train a model on a movie review and like all the queries are about the actors, the model will learn to focus on the information about the actor and it will actually do not encode any information about the plot, the music, etc., etc. And so this is why single vector model actually struggles a lot out of domain, because what if in production now the user asks a question about the plot? Sorry, we don't have the information, and so the model it won't be able to find the relevant documents for this query. And likewise, what the model should encode if it is now used on like cooking recipe? There isn't any information about actors in there, and so the model is lost and it doesn't really know what to encode as an information. And so because we have to learn to compress the information, the model will actually learn one view of the documents, and unfortunately this view is not holistic at all. And so obviously, this is getting worse and worse with larger contexts because the model has to compress the information even more into this single representation. And so it has to make even more drastic choices. And it can even be about just like encoding one view of the document— of a very large document. It is really challenging because the representation is so compressed. And so likewise, my intuition about why single vector model struggles a lot on reasoning-intensive retrieval is because to compress information, the model has to learn one given notion of similarity, and so it cannot be applied to a more abstract matching that isn't present in the training data. And so now you can see why BM25 is actually very competitive on this aspect. Essentially, there is no compression, so we keep all the information, and this is very helpful when applied to new domain, long documents, or reasoning-intensive retrieval. However, one of the main drawbacks of BM25 is that it's not learned. It actually performs a hard matching between the query and the document tokens. And so if there isn't a direct alignment between the two, for example, if they aren't in the same language or if it's a wrong formulation, using BM25 won't work very well. But now, what if I told you that we actually have something that is close to BM25, but that is actually leveraging deep learning. This paradigm is known as a multivector search or a late interaction search. And so the start of the processing is the very same with a single vector model. We will feed the query and the documents to a transformer model, and it will create a contextualized vector representation of all the tokens. But this time, instead of pulling all the vectors into a single one and computing the cosine similarity, we will actually keep all of the vectors, and we will use the MAXIM operator to compute the similarity. That is the sum for all of the query tokens to its maximum similarity with one of the document vectors. And so one fair claim would be that if compression is the only issue, we could just simply use a very large embedding vector instead of like multiple ones. But actually, this is a bit more complicated than that because if you only use one vector, every time you will be making an update during the training, you will actually change all the representation. And so this makes the training very noisy because all the updates will be affecting— for example, the first update will add information about the actors of a movie, And then a subsequent one will add information about the plot of the movie. But by doing so, it will have to remove parts of the existing information about the actors because it needs some space to add this information. And so the Maxim operator, in addition to not requiring to pull the information, it has a built-in mechanism, the maximum operator. And so the update will only update the information of the document related to the input query. So you can see that as for the first query, we will be only updating the token related to the actors. And for the second query, we will only update the token about the plot. And so the intuition is that even if the model is totally out of domain, we can still fall back on a soft lexical matching, which is very close to BM25, but without the hard matching part. So this is the theory, but how does it translate empirically? So one of the earliest recognized perks of ColBERT model, so late interaction model, is that they are very strong out-of-domain compared to usual dense model. And I try to illustrate that with a few examples. So where you can see that ColBERT model dominates out-of-domain evaluation, sometimes even surpassing dense model evaluated in domain. And this can be pushed even further because there have been reports of ColBERT models being able to generalize to other languages that haven't been trained on. And so it is a bit hard to correctly evaluate and illustrate this generalization capabilities because essentially there is an infinite number of possible out-of-domain definition. But really, I cannot emphasize enough how many practitioners reported but ColBERT model generalizing way better to their specific use case. So again, the only thing I can say is I encourage you to just try out this model on your data and see how it performs compared to dense model. If you compare the recent e-Modern ColBERT that leverage latent interaction and you compare it to the best-in-class single vector model on the LongGAN-Bench benchmark, which is a benchmark meant to evaluate the capabilities of long context retrieval. You can see that it is strongly outperforming all of best dense models. And what's more— even more impressive is that it does so by being trained only on MS MARCO with a maximum document length set to 300. And so in addition to be like insanely good at long context retrieval, it does so by only generalizing from short context training. And so training specifically for long-context retrieval should maybe noticeably improve the result further. And so on reasoning-intensive retrieval, the observations are the same, if not even more impressive, because ReasonModN-CORPETS, which is trained on the Reason IR data, it actually outperforms all the existing models up to 7 billion parameter models while being only 150 million parameters. And so this small model is actually beating models that are 45 times bigger than, than him. And it is actually even competitive with the Reasoner model itself, which is an 8-model that is trained on the exact same data. And what's really interesting is that it does so actually just thanks to being a late interaction model. Because when you compare it to a single vector model that is trained in the same setup, so the same backbone, the same data, You can see that the difference in performance is actually pretty huge. And so to conclude on the many advantages of the late interaction model, I want to mention an additional perk that comes for free. Essentially, since the MAXIM operator is doing alignment between the query token and the document token, we can actually see what matched, and thus we have additional explainability allows for exploration and debugging, which is very helpful to show the user when you are doing RAG. Whereas when you are doing a single vector search, you only have one global matching score and you don't have any granularity. Besides this additional explainability, this granular matching also allows to identify subchunks that matched in the bigger document, and thus we can only use these smaller subchunks as the input of the LLM in a RAG setting. In order to augment the precision. If all of these models are so powerful, why isn't like everyone just using them? And to me, there is a few factors explaining this. So first, obviously storing multiple vectors instead of one is— and this part is actively being sold, but right now when using like adapted indexes coupled to quantization and other footprint reduction techniques, the comp becomes actually quite acceptable. Second, because the search is different from the standard cosine search, the vector DB provider did not necessarily support it. But this isn't true anymore as most of the well-known providers support it. And finally, maybe I'm a bit biased, but to me one of the most important reasons is that there was a lack of accessible tools to train, use, and experiment with this model. Because to me, one of the main reasons of the popularity of single dense model is Sentence Transformer. And so to fix this, with Raphaël Surty, we decided to build PyLATE, which is a library that extends Sentence Transformer to late interaction model. Essentially, late interaction model are dense model, but without the pooling operation and the addition of the maximum operator. So you can actually leverage a very large part of Sentence Transformer. And so by doing so, by building upon Sentence Transformer, we actually get a lot of perks, including the support of all the base models as the backbone model. You can use any encoder, any architecture. You can even use large language models if this is what you want to do. And we can also benefit from the efficient training with the support of multi-GPU and multi-node training, FP and BF16 precision, etc., etc. And all of this training is directly monitorable from Weights Biases, which is very helpful to monitor the training, but also to share with your team when you are training the models. Also, Sentence Transformer is very well integrated into the Hugging Face ecosystem. And so the pilot models are easily shareable on the hub alongside model cards that are automatically created and that include information about the model, the training data, the training dynamics, and eval— the evaluation results if you run those during the training. And finally, and maybe most importantly, the code is actually very similar to the one of Sentence Transformer that you are already familiar with, and which in my opinion is very streamlined. And so you can almost swap in PyLATE into existing boilerplates, so you can already leverage the same datasets you are already using. So really you can just reuse your existing setup. For example, here I put a small boilerplate where you can see that we first define a few parameters for the training: the base model name, the batch size, the number of epochs. Then you load the data. So you can see that we actually load directly the Sentence Transformer data. You define the loss, maybe the in-training evaluator if you want to run that. Then you create the Sentence Transformer training arguments based on these arguments. You create the trainer and you just launch the training and voilà. Although this boilerplate is actually very simple, it is actually very similar to the one I used to train the two state-of-the-art models that I presented earlier in this presentation. Those two trainings took like 1 to 3 hours on 8 H100. So really like the training of state-of-the-art models is accessible and you can really just try it out on your data to see how it goes. Okay, so training state-of-the-art model is easy, but if you are here, what you really care about are evaluations, right? And in PyLight, we also got you covered because PyLight has a built-in efficient index based on the BLED index where you can actually store your documents alongside their document IDs, and then you can simply query this index to get the closest document in your database. We also provide helper function to get evaluation metrics using the outputs of this index, and this is using the popular REGEX library, so you can get various metrics easily and quite quickly. And all of this process is using the standard queries document QRL format. So essentially you can use all the MTEB, ER datasets, BEER datasets, and you can even use your own data because this is the most standard format, or at the very least, it's easy to get into this format. We also have an in-training evaluator, as I mentioned earlier. So if you want to keep track of the evaluation during the training to monitor the training, and PyLATE is also integrated in FreshStack that Nandan presented, and it will soon be merged in the mTab library so you can directly run your evaluation in this library. And so I just want to finish this talk by presenting the promising research avenue that should make the multivector models even more awesome. So first, as I said, a big subject is the index footprint for multivector models. And the recently introduced methods allow to compress the representation evenly without losing too many performance, if any. And we actually are starting to get a reasonable index size that can actually match the size of a usual dense model if you consider like reasonably sized document. And so there is more and more research on this subject. In order to try to find like the optimal trade-off between keeping all the tokens and only keeping one. Another promising avenue is applying late interaction to other modalities than just text. You might have heard of the ColPALLI approach, which essentially enabled OCR-free RAG pipeline by applying ColBERT to text-image retrieval. And so Jina recently released and trained and released a model trained to do image-text matching with both a single vector mode and a multivector mode. And the multivector mode consistently outperformed the single one while sharing the same perks as with the text: out-of-domain generalization and the long context handling. And so why stop there? Because some work already applied late interaction to even more modality, including audio. And so finally, the MAXIM operator is pretty naive. And although it has some nice properties that I highlighted earlier, There is definitely room for improvements, notably through the scope of a learned scoring function. All right. So this is the end of this talk, which might have been a bit dense, pun intended. But to summarize the key points really quickly, late interaction models allow us to overcome the intrinsic limitations of single vector search, and this makes them very well suited for your modern real-world use cases. And probably a lot of other benefits to be discovered. It starts to get well supported in the ecosystem. And in order to empower everyone with that model, we built PyLATE, which allows you to train, experiment, and evaluate late interaction models while being as close as possible to the Sentence Transformer experience you are already familiar with. And so really, you do not have any excuse anymore to not at least try the existing model on your data. And even take it a step further by training your own model on your existing setup. So thank you for your attention, and again, if you are interested in this topic, I try to share as much information as I can on Twitter, and I also try to maintain a good information over shitpost ratio. So if you're interested, please do not hesitate to come talk with me. I always like to discuss this subject.

Thank you so much. Do you mind if I ask you some questions?

No, go ahead. Okay, great.

Um, I totally get like why the single dense vector approach compressing it doesn't work. And I agree with you that the vast majority of people, seems like 99% of people are still using dense vector approach, like the one vector. Then you mentioned some trade-offs like the tools may not be as well known, although you did introduce PyLATE as a way to help you train your own models. It seems like you do have PyLATE now if you do want to fine-tune your own. Seems like there is ways to store it without blowing up like crazy. You even talked about the newer research where you compressed things a bit more on top of these adapted indexes and footprint reduction, all that stuff. So like, why do you think it's not despite this? One additional thing beyond this, what I was wondering is, is there any latency hit you get from using late interaction when you're actually retrieving documents? What is the kind of effect of this?

So to answer to the first part of the question, I think it is still early. So like we built PyLATE, but it's still quite new and we are still enhancing it. And most of the vector DB provider, like they support ColBERT model, but for like maybe a few months at top. And so like just changing an embedding model in production might be hard for some teams sometimes. So trying to jump into something this new, especially if your vector DB provider did not support it back then. It might be a bit complex to push into production. I think that right now it is doable to use ColBERT model in production, especially like, like most of the actual users do not scale their databases to like really millions and millions of documents. Because again, like with the Plaid index, we can scale to MS Marco size, which is like 8.8 million documents. I think it's fine. We need people to realize the benefits from them. For example, the reasoning intensive and the long context, it was not that known until a few weeks. So it's really just new. And like in any technology, it takes time to adapt. And I just think it, it will become mainstream because people will start to use the model that has been released. And the more and more models we have to choose from, because for example, all the models I'm training right now are in English. Like this is a big slowdown to push into production because English is not the only language in the world, right? So once like different people will start to train different model of different size, different languages, and like with specific domain, because it's good out of domain, but it's still better to like have something closer to your domain. So I think once we start to have that, there will be more and more people that push that to production. And so it will grow eventually. And about difference in speed, again, it really depends what you are using because like even an index, it doesn't mean, are you using PQEVF? Are you using the article navigable source world? What are you using and stuff? I think there is still a bit, but I think we are getting there. We'll still be a bit slower. But I think as Orion highlighted yesterday, I don't think in today's standard this matters much. Like it will be a few milliseconds different, but I think for deep research stuff and like the AVD RAG thing, it doesn't matter. Matter that much if you really have like insane performance on the other end. I don't want it to be like very slow. And like, for example, you can see Rafael, my awesome co-maintainer, is releasing a lot of, a lot of things in Rust that are actually like really going even faster than what we had before. And honestly, it's really fast. Like even to search in ms-marco, it's really fast. So I think it's acceptable. Maybe not—

That makes sense. You're basically arguing the UX you know, people's expectations usually aren't to where you need it super fast. You have reasoning models and things like that. And if you have a thoughtful UX, hey, like we're thinking whatever, and you communicate that, and you're saying even it's like maybe a few milliseconds.

Like within the full deep research ecosystem, it won't be the bottleneck at all. Obviously people right now are very fine with using 8B model for embeddings. And this is way slower than a 150 million parameter model. Even if you are using late interaction, the inference cost is so much bigger. I still want it to be fast, as fast as possible. But I don't think we are in the same regime than what we were back then, where a few milliseconds was really important. Right now we are used to sending the request and waiting a few, I don't know, seconds, or I didn't make the computation, but we are fine with waiting a bit, and I don't really think it is such of a slowdown, and I think it will get better as we move forward.

Okay, another question I have is comparing to dense vector stuff, um, if a lot of people are fine-tuning their dense vector stuff also on their data, or do you know anything about, okay, if you fine-tune your dense vector stuff and you fine-tune a late interaction model Does the gap still hold? Like the late interaction model still continues to be better similarly as it is? You showed a bunch of benchmarks and it was pretty convincing that, hey, like you get this BM25-like behavior, which is really cool, which doesn't seem like it can happen at all with a single. So I get that.

There isn't a lot of like direct comparison of a single vector model and a multivector model trained on the same data. But this one, for example, is a direct comparison of the two and they are both in domain, right? They are both trained on the same data, which is in domain for the evaluation. And you can see a gap. So this is a very specific use case. This is reasoning-intensive retrieval. So you might argue that it won't hold for everything, but again, Jina, they trained— like, this is the same model. This is the— this is not like two separate models. This is the same model. It is trained to either use the single dense vector and the multivector search. And you can see that like on image-to-text retrieval, which is like way closer to something way more generic, you can see that there is a huge difference. So I cannot guarantee for now because there isn't enough studies that correctly compare both setups in a fair manner. But I will argue that yes, I think in domain it should be better. And I even think that fine-tuning a late interaction model— put aside the cost, which isn't great, but put aside the cost— I think it is easier to fine-tune the late interaction model because you have way less risk to collapse to your new training distribution, right? So you can still keep the original capabilities way easier than if you were collapsing with your single dense representation. So I think it's working nicely out of domain, but I also think that in domain it is stronger than the dense model, especially on some specific use cases, but also in the general case.

How easy is it to fine-tune with PyLATE? Is it easy to like get it going? Any tips, anything to watch out for?

Honestly, we have some boilerplates. So the boilerplates I mentioned to train the two state-of-the-art models, they are openly available alongside the data. On the repo. I don't have like really strange loss curve with late interaction model. What I would advise is maybe to run like in-training evaluation if you can, because this is a very nice signal. And if you are trying to sweep, it is really cool to know where it's going instead of waiting the end to evaluate. But honestly, no, all the boilerplates are exactly the same. You just need to decide the training if you want to do contrastive or distillation. And just use one of the existing boilerplates. We have a nice documentation as well. And again, like, if you have any issue, do not hesitate to open an issue or to reach out to us on Twitter. We try to be as active as possible and we really love to, to help people out.

Okay. Let me ask it at a different angle. Let's say, is there any kind of common mistakes that you see people make or common points of confusion of people coming from single vector to this regime that they misunderstand or get wrong? And anything that comes to your mind, it doesn't have to be, just wanted to ask.

No, I don't think so. I didn't have much this aspect. People really managed to get it working. Essentially, if you are able to train single vector model, you shouldn't be lost within PyLight. There isn't anything different. It converges faster, and it is a bit more stable in my experience.

But I'm not saying it's easier, if anything.

I mean, yes, to me it is easier. Maybe I am a bit biased, but essentially this is all the advice that you can get for like single vector training. For example, tune the temperature if you are using contrastive loss, try to get as big batch size as you can. But I detail everything in the documentation again, so Like the most common pitfalls should be okay. And again, like if you have any question, do not hesitate to reach out and I will update the documentation with this issue to help out the others.

Okay, sounds good. Do you mind going to the last page again? Just so if we can end it so people know how to reach you on all these different links. But yeah, thank you for coming.

Thank you very much for the invitation.

Thank you.