Stop Picking Embedding Models Off The MTEB Leaderboard

Welcome everybody. Really happy to have Radu here today. He's a software engineer at Vespa. He's also a search expert. He's going to be talking about a really important subject today, which is how to choose the right embedding model. A lot of people who use embeddings for search or otherwise, they just kind of, a lot of times, just like either number one, go to a leaderboard and just like pick the best one without thinking too much about it, or just using an embedding model from the provider that they're using, OpenAI or Claude, without giving it too much thought. And if you do this, you're leaving a lot on the table. There's a lot of things you might want to consider, like cost, latency, robustness to, like, changes that you might want to make in your embedding model. And there's a lot of things that you might want to consider that a lot of people don't. So Radu is going to be going over that.

So the agenda, we're going to look at How to choose a base embedder and then how to take that and tune it on your data using a tool called Vespa Embed. And then I'm gonna talk a little bit about the scenario where you may wanna have a great but not greatest embedder, and then, but that would still be cheap to run. And then you can fill that gap with a sort of more expensive re-ranker. So let's kick it off. On choosing the base embedder, We the talk is titled something like "Stop looking at MTAB." We're going to start by looking at MTAB, and then we're going to look at things that MTAB doesn't cover. So, for example, quantization for the model, quantization for the vectors that the model produces, Matryoshka dimensions, some query latency/performance aspects, and then the things that we don't cover, like additional constraints. All right. So let's start with MTAB. So this is the massive text embedding benchmark. We have lots of benchmarks here, and I think the way to go is. To find the benchmark that looks closest to your use case. I'm just going to take one that is known. And then you can pick up things like, okay, maybe I don't want a huge model, so I'm going to restrict the size. I'm— for the purpose of this presentation, I'm going to only look at dense embedders. So those that produce one vector per paragraph or chunk, whatever you're sending. And then you can see the ones that perform best. And also take into account that some of them punch way above their weight. So for example, this one here is much smaller than the rest. So might be faster, which leads me to the next point, quantization. So in largely the performance of an embedder, so like how fast it can produce vectors depends on 2 factors, how big it is and how aggressively it's been quantized. And quantization depends on where you run the model. So if we run it on CPU, I would say int8 is a good quantization to pick because It gets, at least in our tests, most of the precision, but it runs at like a fraction of the cost. It's going to be much faster than the original model. If you're running on GPU, an INT8 model will not fit like its native way to run. So it's actually going to be a whole lot slower, but you can run like an FP16 variant of the same model and you're going to get basically the same result, except that it's going to be significantly cheaper to run. So that was the model itself and how we quantize it. Now, what about the vectors it produced? Normally those are floats as well, like 32-bit floats. But if you go to bfloat16, you're probably not going to lose anything. So that's basically a no-brainer. You can go a lot further. You can go all the way down to bit embeddings, to bit vectors, but then you're going to lose something. Whether that something is worth it or not, that's up to you. And we're going to look at some data to get a feeling of the landscape. The next topic is Matrioshka dimensions. So some models are trained. Such that the earlier dimensions are more important than the later dimensions. You can basically cut the model, cut the vector. I mean, the model can do that. You can just get the first part of the vector and you're going to get, like with quantization, something that might be good enough for you. So this is an example embedding GEMMA, and you can see how these scores of different tests vary with the number of dimensions as the dimensions are Smaller. And yeah, as always, it depends. One particular thing I noticed here is that the 512-dimensional vectors actually perform a little bit better on this particular test than the original model. So it's definitely something to consider and test on your data, I would say. Let's look a bit about query performance. And here I want to focus on the aspects that have to do with the model choice, not as much about the search engine internals. So as far as the model is concerned, there are 2 things that matter. One is how fast it can produce those vectors, because when you send the query, you know, that's gonna be, let's say, a bunch of text. That text needs to be, you know, run by the embedder to produce a vector. How fast it can do that will be reflected directly into the query latency. And then the second thing is, you know, however the search space is structured, search engine chooses to navigate the many, many vectors it may have in there, it's going to have to compute the distance between the query vector and the document vector. And most models are trained with cosine similarity. So that is kind of our baseline. But if vectors, both document and query, are normalized to the same length, then we can use basically a dot product to compute that distance. And dot product is much cheaper than cosine similarity. And then if we have bit vectors, we can use Hamming distance, which is a whole lot faster. It can also be optimized in all those modern CPUs, and a lot of search engines do that. Last but not least, the things that we did not cover here are really constraints of your use case. So if you need the model to be multilingual, you should choose a multilingual model. If you need this larger context, then you should choose a model that supports that kind of context. So with these slides at the end, you're gonna have resources for all the things I'm talking about. This table and this part of the presentation in particular is based on a blog post that my colleague Thomas did, and he tested a lot of those models on a bunch of configurations. And you have this interactive dashboard that you can click around in to get a feel of how these models perform, both in terms of latency as well as quality. So I'm sorting here by query latency, and here I mean not the whole query latency, but the latency added by the embedder. So how fast can it generate a vector from the query? And it's the same with document throughput. It's how fast can we generate a vector from the document itself? And you can see here that if we are running on CPU, then most of the models that are here on the top are int8 quantized. By contrast, if we are looking at a GPU instance, You're gonna see the intake models kind of further down the list. And you will see that largely, but not necessarily, core latency is correlated with document throughput, right? So they're both gonna be faster or slower. Let's talk a bit about vector precision. So if you expand any of those, you're gonna see a comparison of nDCG between the original float embeddings, bfloat16. And these are bit vectors. So it says int8 here, but it's actually because we're using Vespa to test this. Vespa would pack 8 bits into an int8. So this would be actually bit vectors. And you can see there's quite a difference for this particular model. It's quite a difference between the performance of float32 and bfloat16, which are kind of the same, and int8. But if we look at hybrid, which is more likely what you're gonna use in production, then the gap shrinks a little bit. So bit vectors kind of become more viable in this context. Also gonna look a bit at Matrioshka embeddings. So here, this one, Weights Biases Nano supports different dimensions. So the original is 2,000 dimensions, but you can go half that, half of that. And you can see, again, I'm looking at a hybrid because that's what's gonna matter in practice. There's not anything in it between 2,048 and 1,024. We're gonna lose a little bit for 512, a little bit more on 256. So this might work for most, but one thing I'm going to also look at is this score is quite close to this score. So if we go 2,000 dimensions, but with bits, we're gonna get a fourth of the size of the bit float 16s.

Okay.

On 512 dimensions. So again, bit vectors might work here, especially if we can afford to have more dimensions. All right. So let's say we chose our best embedder. Let's look at how we can tune it. So there's this tool, which is again, open source, Hugging Face, links at the end of the presentation. It's called Vespa Embed. You can use sentence transformers on your own, but this is a nice UI on top. That allows you to tune those. So here's an example of a model that's already tuned, but when you do your own, create a new project, and then you provide the data that you want. Typically, you're going to have pairs. You can also have triplets if you have negative examples. So pairs would be typically a query to document, right? Whatever field from your document you want to embed. Then you're going to take the base model from Hugging Face, and then you're going to provide your training data. We're going to have some sample training data in the links at the end as well. It can do auto-split for you with validation, and then you can just go ahead and start training. We have a lot of parameters here. I think these ones are quite interesting because MNR will basically take— you're going to send these— training will be done in batches, right? So in a batch, if you're going to have the pair that is positive, the query to the document pair, it can select other documents from the batch as So that works quite well when I tested it. You can do MNR symmetric, which is gonna do it both ways. So you have query to document. You can also have document to query. And then gist is similar, but it will also take a look at the document it picked from the batch and see based off the model itself whether it's likely to be actually a negative or not, because it might be by coincidence that we're picking something as negative when it actually is something positive. similar to the query just by accident. And then the typical sort of parameters, like how many epochs and batch size and stuff like that. There's more in here, including Matryoshka embeddings. So you can have that sort of model trained here as well.

Do you have any advice for people starting out, like in terms of what parameters to choose, what approach to choose? Should you just go with the default?

Yeah, I would start with default. And if you have negatives, try triplets as well. I tried both in my particular case with the data that is shared there with Amazon dataset. Didn't really make a difference. But then again, I only took negatives from the judgments that I had. So judgments were saying the query and the top-rated products and also some products that don't rank that well. If you want to be sort of more exhaustive about it, you will probably also want to mine for hard negatives. So just to make sure everyone's in sync, a positive means if I search for chocolate, a chocolate bar is a positive. A chocolate-flavored condom is soft negative. It might show up. in the results, but it's a negative. If I have windshield wipers, that's a hard negative, right? So I didn't test with hard negatives, but you should have hard negatives as well. In that case, you will probably want triplets. Again, with the other options, I tried different combinations. In my case, it didn't make a difference. Again, in theory, GIST should be the best option here.

And when it comes to hard negatives, I get this question all the time. How do you like to go about it? Do you iterate? Do you use like a baseline model to figure out like what the nearest sort of wrong answers are and then like use those as hard negatives? Or do you like refresh the hard negatives after every training run? Do you try to make it progressively harder? How do you go about it? How much lift is there to doing that?

I think that would be very different on the use case. Like in my case, I wasn't that much into any of the negatives. Like MNR with just pairs worked just as well as actual triplets and everything, which on paper sounded a whole lot better. So definitely your mileage may vary, but I have seen, especially if you're into e-commerce, there's this small conference called MICEs and the latest, the videos just came up so you can just check them out on YouTube and there were at least 2 or 3 presentations that they mentioned about mining card negatives and how that was important to them. I don't remember how they did it. If you have a search engine with your data in there, it's quite easy to do so. If you search naturally, if the search is decent, then positives will tend to be on top and the negatives on the bottom so you can just kind of Scroll down.

Okay. So when it comes to like this fine-tuning, a lot of people where they get stuck is like having good labeled data. Like some people may not have the best labeled search data of like query to document. They might have many documents that are relevant. They haven't gone to the exercise yet necessarily of like having this like clean dataset for fine-tuning. Do you have any advice how to get started?

Definitely. LLM as a judge. Again, like you're going to have a full sort of ranking course. Linked at the end, and there's some examples there of how to use LLM as a judge. So basically there are a bunch of techniques where you can tell the LLM, hey, rank these query to document pair. You're going to have— want to provide some examples. I think the examples are the most important part. And how do you distinguish between scores, right? So normally you're going to have either binary relevant, irrelevant, or you're going to have something like 1, 2, 3, like relevant, irrelevant, and meh. You don't want to have too many of those. So I think starting with 0, 1, 2 is a good starting place, I would say, but then Then you have to tell the LLM, how do you tell 0 from 1 from 2, right? And give it examples. That's probably going to, if you want to be clear about it, and you should be, pretty big prompt. And ideally you're going to send one document at a time, but that's going to be super slow. So you probably want to have like micro batches of like 8 or 10 documents and have tell the LLM explicitly, like they're not related to each other, things like that. Like, cause you don't want like a list-wise cause you don't have the complete list. But yeah, I think that's a good starting point and you have scripts there to actually And how often do you reach for fine-tuning when you're working on a search project?

How often are you just using out-of-the-box embedding and find that's good enough? How often are you like saying, okay, let me fine-tune? Can you give us intuition for that?

I think it's a matter of effort in general. Like there are a lot of good ideas out there. I think this is one of those ideas that generally work well. Like in this example, I have, you know, with the dataset that I showed, I got a huge jump in nDCG. Okay. It wasn't big to begin with, but it was a huge jump. It was a huge jump. So I think anybody should do it.

Yeah, I agree. The question always comes up, should you fine-tune your LLM? And I say, okay, for that, maybe not. You know, you should try other things first. But the embedding is a much more constrained problem and it's a lot easier to fine-tune the embedding. Yeah, I agree with you. I always find a really good benefit from doing it and it's not that complicated.

Yeah, there's no reason why it should be huge. So it's not a big cost. The model is trained on a bunch of data and your data may be very different. And yeah, with LLMs it's different because you're going to have a lot of context that they can potentially adjust. But if you have a query like chocolate bars, whatever, then—

Yeah, it makes sense.

All right. Let me just quickly go a little bit over the re-rankers part. So this is particular to Vespa, but might apply to some extent to other engines. So in Vespa, you're going to have a stateless layer, which is where your application talks to Vespa. And then you're going to have a data layer, which is typically across multiple nodes. And when you run a query on the data nodes, first we're going to do matching. So which documents match our query at all. And then we're going to do a first phase ranking, right? So all the documents that match your query need to have an initial score. And that first phase ranking needs to be cheap because we can do it potentially on like millions of documents. And then second phase ranking is something that can be more expensive because you only do it on top. And then you can also do another global phase ranking on the final top N, and this can also run on GPU. So in practice, if you're going to do this at scale, you will likely use binary with Hamming distance. That's what most people end up doing. If they care a lot about precision and they're willing to spend the money, then they're going to use the original floats or bfloat16 and stuff like that. Of course, anything cheap qualifies here. Lexical, you can even do like the examples there with arctangent normalization. You can do hybrid search. And then you can use something more expensive to re-rank. So some examples here are you can use the float query with the binary vector that you already have, and you can do the product there. You can do float float. So you can do binary binary on the first phase and float float in second phase. And you can keep the floats on disk. So you can use the binaries with the HNSW and stuff that you have to load in memory, but memory is expensive, but disks are cheaper. So you can do this float float only on the top end so that you can keep it on disk. You can use late interaction. I have an example here of a demo. You can use cross encoders, whatever techniques, and you can define your rank profile as you wish. Here's like a demo. You can go over it. It's just basically a Copilot demo. Here's how this might be defined in Vespa. So you have a schema in Vespa where you define your like numbers and strings and all that. This is the embedding, which is a tensor. Because this is late interaction, we're going to have one vector per patch. So this is for re-ranking. And this is how we can compute Maxim in Vespa. So Vespa, you have like rank profiles, which define how you compute your score. And here I have like the query tensor, which contains the float vector of our query. And this is how you can do Maxim. So it's basically a dot product. This would be just multiplying the vectors. We sum them up. That's our dot product. And we're going to keep only the maximum per patch, right? and then we add them up for the whole document, and that's going to be our maximum. That's literally the definition. And this goes into a function, and this can be a re-rank, right? Like, your first phase could be like your cheap embedding or BM25 or whatever, and then you can have a re-ranker at the second phase or global phase.

And AI can help you write these configurations. You don't need to memorize the syntax.

I just want to kind of explain like how one would do it, right, in sort of more particular fashion.

So you did like high-level overview to give people a taste, all the different things that you can consider, whether it's like there's many different kinds of embeddings. Those embeddings can have lots of parameters. There's different stages of search as well. You have like your initial stage and then a re-ranker after that. And so people, they're asking like, oh, like how do you know whether you need to use a re-ranker? What model do you choose? What dimension do you have? How much data do you need to fine-tune? I know that a lot of this comes down to experimentation. So how would you guide people who are new to this? Like, you're just getting into search. Maybe they're just using off-the-shelf, like a RAG database, and they haven't thought too much about tuning or anything yet. Like, how would you start getting into this so they're not overwhelmed?

Like, I would just pick off MTEB what seems to be good. So that could be our base. And then if I'm running this on CPU, I would use the int8 quantization. If I run it on GPU, I would use FP16. I would just start with what seems better. You can start with your, in your vectors, you can start with bfloat16 because you know it's going to be basically stock precision. You know, do we have a performance problem? Right? Because we may over-optimize, right? Like if you have 2 million documents, then a lot of the performance issues we've been talking about may not apply. So I would start with quote unquote defaults that don't lose too much and then take it from there. Right? Do we need a re-ranker at all? Are we just doing very well? Do we have a lot of data and do we need to quantize further and stuff like that? And with Tuning, I would say, Zach kind of skimmed over that part, but with tuning, I could just quickly go through my dos and don'ts from my own experiments so that others don't go over the same. So it was very tempting for me to say, okay, I have some e-commerce data. I can take the title and description of the product and train off of that. That did not work. You're supposed to train it based on what is actually going to happen. That's going to be your query-document pair. That's going to be your relevance judgments, right?

So beware of proxies. So like people are like, oh, try to be clever. Like, oh, the title is good enough. It's like going to represent queries, but in real life, people's queries might be very different.

Exactly. So it has to be their real queries or, you know, as close to it as possible. And then the other thing that a lot of people run into is depending on how you're measuring it, like most people use nDCG. nDCG has this blind spot. Like every time you change your relevance function, make sure you update all the documents that you compute nDCG off. Because otherwise you change your relevance. function and you're going to bring new results. And those new results, if they're not ranked, they're going to be zero as if they weren't relevant, but you don't know that until you rate them. So that's why a lot of people are like, oh, my NDCG never improves. Well, because you didn't rate all your documents because you're never going to rate all your documents in your corpus for every query.

So can we linger on this NDCG for just a moment? I think people, they also get lost in the metric and it gets really deep really fast. But you have to first think about what problem you're solving. Are you trying to rank? Are you not trying to rank? Um, so on and so forth. How do you know whether your search is good?

Yeah, I think that's important to have some rules of thumb, even though they're not, you know, don't take this as the Bible. I would say for things like e-commerce where you need the ratings and you want the higher, more important results to be on top, you're probably down to nDCG or ERR because those, you know, are effectively looking at are the top results good and the rest is maybe less important. If you're doing something like RAG, then your precision is going to be more important because if you feed the LLM junk, you're going to get hallucination and then you're going to look at recall, but recall is also a factor of like how large of a context can it fit. So I would maybe look at precision more. Yeah, I think those are, those are the ones, but with nDCG, there's this thing that I think with ERR is the same. Just make sure that everything that goes into computing the metric is rated because otherwise you're going to get an invalid metric.

All right. Thank you.

Thanks.