The Hidden Cost of Embeddings in RAG and how to Fix it

Embeddings are one of the most important component of any production ready RAC system, but usually gets the least amount of attention. In general, they are very cheap to compute compared to the LLM calls that you are making as a part of the RAC pipeline, but they are very expensive to store. And that's why it has to be a very important consideration if you are putting any RAC system in production because it posits its own challenging when it comes to scale. So in general, embedding model will receive a text as an input. If you are working with a text embedding model and the output is going to be a fixed size vector, irrespective of the size of the input text, this fixed size vector is going to represent the semantic meanings of your text input. Now, since you are going to be computing embeddings for your input text, there are two different components of the cost associated with it. The first one is the computation cost. So depending on which embedding model or API provider you use, there is a fixed cost that you have to pay in terms of the embedding compute per million or per thousand tokens. And this is largely dependent on your API provider and is usually a one time payment for your data set creation. But there is a storage cost that usually gets ignored but can be substantial. Now the storage cost needs depends on the size of your embedding vector.

In general, most of the bigger or.

State of the art embedding models use a size of 1024. There are some smaller embedding sizes of 3, 84 and 768 as well. And this embedding size determines how much.

Storage you will need.

So to give you an example, let's look at the OpenAI, which generates 1536 dimension embeddings for each vector. Usually they are computed in 32 bit floating point precision. That means each dimension is going to need four bytes of storage. So if you are looking at storing 1 million vectors, you are going to be needing about 5.7 gigabytes. If you are storing 1 billion chunks, that means you will need 5.5 terabytes of vectors storage, which is a substantial amount of storage and it's not unheard of. But there is a cost associated with that storage. So let's assume if you are using the extra instance on AWS, which cost about $4 per gigabytes per month, here are the storage costs that you will be paying. So if you are using the smaller 300 dimensional embedding vector, you are looking at about $5,000 per month. Of storage need if you have 1 billion embeddings. But let's say even if you have about 250 million embeddings or chunks of text, then you are still looking about $1,300. But if you increase the number of dimensions in your embedding vector, which will actually increase the accuracy of retrieval, then, then for a 1 billion embeddings, you are looking about $14,000, which is a substantial amount of money. But if you decide to use the OpenAI embedding, both the smaller and the bigger ones, these are drastically more compared to some of the smaller embedding models. So in general you need to pay close attention to the storage needs. Now, there are ways in which you can reduce the storage needs and that's what we're going to be talking next. Let's look at some of the approaches. So the first one is to reduce the number of dimensions in your embedding vector. A traditional machine learning approach such as principal component analysis can be extremely helpful. And the idea is that instead of using all the dimensions in your embedding vector, you can further reduce it using a dimensionality reduction technique. So, such as you're going to be bringing your six dimension vector into a two dimensional vector. Now, these approaches works great for traditional machine learning problems, but they're not a good fit for text based embeddings because it loses a lot of information. A second approach that has got a lot of attention lately is called metrosha Representation Learning. It's a new embedding model technique which will help you reduce the size of your embedding vector. And the way it works is that you can compute embeddings in the full representation. So for example, you can compute it in 1024, but instead of storing all 1024 dimensions, you just take the first 512 or 256 or 128 dimensions. So depending on your need and your storage requirements, you can store a subset of the dimensions that you have computed. Now, OpenAI uses a similar approach with their newer embedding models. And some of the open weight models like gnomic embeddings also use this representation. But there is still loss of accuracy or loss of performance when you are reducing the number of dimensions. That brings us to this new approach, which is not to touch the number of dimensions in the embedding vector, but actually change the precision that you're using. An extreme example of this is that Instead of using 4 bytes representation, which means 32 bit representation for a single engine, just represent that dimension by a single bit. So you can come up with a threshold. If it's above that threshold, you set the value to 1, otherwise you set it to 0. This will give you substantial increase in speed of computation. So reduce latency, reduce memory requirement. But you're going to be losing some precision. Now, instead of going to the extreme case of using a single bit, you can also use 8 bit or 4 bits. So basically you take the original values, represent them by 256 different levels and pick one of those levels depending on the value of your original input. The approach is very similar to the quantization that we see in large language model. So there we quantize the weights to either 4 bits, 2 bits or 8 bits. Now this will definitely save you on storage cost and compute cost, but what about the performance? So that's why we're going to look at a study done by the Hugging Face team where they showed that for certain embedding models, if you reduce the quantization level from 32 bit to 8 bit, you are still able to preserve up to 99% accuracy. And even if you substantially reduce it to a single bit, you can still preserve up to 96 accuracy of the original model, which is pretty amazing in terms of the storage needs and retrieval speed as well. So next we're going to look at that study and I'll show you a practical example of how you can implement this in your own pipelines. Here's the article that I was referencing. Binary and scalar embedding quantization for significantly faster and cheaper retrieval. It's a must read if you really want to understand which embedding models to choose and which quantization level. It's from the Hugging Face team and they did a study where they compared different quantization level on the NMT benchmark. And the goal was to see what type of storage and performance improvements you will get when you're using a quantized embedding model, very similar to the normal LLMs. Now in the, in their case the baseline is 32 bit quantization. So that's the baseline and we can see from For 250 embeddings we will need about $3600 per month. Now the NDCG score that you get here is about 54. That is the baseline. But if you reduce the quantization to 8 bit, the storage cost reduction is almost 4 times with only 3% reduction in performance compared to the baseline. Now the great thing is if you do binary quantization, the reduction in storage cost are 32 times only around 5% reduction in performance, which is pretty amazing. Now the reduction that you see as well as the performance is really dependent on the embedding model that you choose. So for example, if you use this model which has 708 dimension, if you use 8 bit, you are seeing about 6% reduction. But if you use the binary quantization, then there is a whopping 26% reduction in performance. So you really need to pay attention to which embedding model that you're choosing to start with and then you will need to experiment with your own quantization. I will be putting together another video.

Which will talk more about how to.

Choose an embedding model for a specific application. We'll also discuss what are BI encoder and cross encoders. So if you're interested in that, make sure to subscribe to the channel. But in general, their study shows that the bigger embedding models, which has larger embedding vectors has less impact on the performance if you use lower quantization level. And that really goes in accordance to what we have seen for LLMs as well. Bigger models seems to have less impact on performance if you use a more aggressive quantization. Now the binary quantization that they use has the this additional part called rescoding. So the embedding vector that you get is not going to be the same dimension. And we're going to look at an example. Keep in mind that not all the vector store supports quantize, especially for the open source ones. FIAs, UseArch, OpenSearch, Elasticsearch, malware and quadrant do support some sort of scalar quantization of embedding models. When it comes to proprietary ones. I think Pinecone does have support for it, but something like Chrome, ibdb, I don't think they have support for quantization yet, but hopefully they're going to be implementing this support pretty soon. Now how do you actually implement it? And it's very simple to do if you're using the sentence transformer package. So here's an example of the binary quantization. We load the sentrance transformer package, then you need to pick your embedding model. So in this case we are using one of the state of the art embedding model. Then you pass on the text chunks that you want to embed and when you call the encode function, just use binary precision in there. Another way of doing this is you can compute the embeddings in 32 bits. So using the normal encode function and then use quantized embeddings with the binary precision. Now you can see that in this case we have 1024 is the embedding size vector 32 bit floating point precision. When you are computing directly in 32 bit floating point, that will need about 8,000 bytes of storage for these two embeddings. But if you use the binary encoding, then the storage needs are going to be 256 bytes, which is 32 times reduction. Now, in this case, the ways the sentinel transformer has implemented binary encoding, it uses the rescoring. So that actually will also reduce the number of dimensions in your embedding vector. But instead of using binary encoding, each number is going to be represented by unsigned 8bit integer or sign 8bit depending on how you choose it. Right? So there is a reduction on the embedding vector size if you use binary encoding, but if you use the 8 bit encoding, it actually does not impact the output embedding size. The only thing it impacts is going to be the actual data type. So it does compute the beddings in 8 bit then, and the storage needs, which are four times less than if you use 32 bit precision. So if you are putting an embedding model in production where you're dealing with a large corpus of data, I'll highly recommend to check out quantization for your embedding models. It will definitely save you a lot of money as well as the latency is going to improve substantially in terms of the relative speed for retrieval. Here are the numbers that you can expect. So they did this study. If you keep the 32 bit floating point precision as baseline, then on average for 8 bit quantization you can expect about 3.6 times speedup. And for the binary that is about 25 times speedup. In the max case it was 46 times speed up for binary quantization and almost 5 times speed up for your 8 bit quantization. So these are pretty substantial improvements in terms of latency as well. So I highly recommend to check this out.

In this last section I'm going to show you how you can use an open source vector store such as Quadrant to store these quantized models. Quadrant supports both scalar quantization, so that means eight bit quantization and binary quantization. Now, according to them, here is a reference table that you can use. Again, for scalar quantization, it can preserve to 99% accuracy, gives you 2 times speedup and 4 times compression. For binary quantization, depending on the model, it can give you up to 95% of the baseline accuracy, up to 40 times the speed and 32 times compression. They also have their own mechanism which they call product quantization, but it gives you much lower accuracy at much reduced speed but much higher compression. In order to use it in your own code you will first need to set up a Quadrant client. It can be either running on your local host or you can use the Quadrant cloud client. Once you create that then you need to provide name for your collection or.

A name of your vector store.

You can also provide the size of your embedding vector. That will depend on the embedding model that you choose. Next you need to define the similarity matrix. Again we're going to be using the cosine similarity and after that you need to set your scalar quantization level. So currently it only supports the 8 bit quantization when it comes to scalar quantization level. You also need to set up the quantile so basically it looks at all the different values in your embedding vector when it's convert to bit quantized and only use the up to 99 percentile. Anything above that is simply discarded and you can set whether you want this to be loaded in memory or not. So it's a relatively easy setup. If there's interest. I'm going to create a dedicated video on this. I hope you found this helpful. For production ready systems where you are working with a large corpus of data. This is a very important step because it's not only going to save you money but it also is going to increase the throughput and reduce latency. If you are building production ready systems or looking for consultation, you can reach out to me. Link is going to be in the video description. Also make sure to check out our Discord server and if you are looking for other tips and tricks when it comes to Rack systems, make sure to subscribe to the channel.

I hope you found this video useful.

Thanks for watching and as always see you in the next one.