Understanding Embeddings in RAG and How to use them - Llama-Index

This is the second video in our Llama Index series. We will be looking at different components of the Document Q and A system and see how to improve them. Today we are going to be focusing on embedding models. In this video we will understand what embeddings basically are, how to use them, what type of embeddings are available within Llama Index framework. We will look at how to use OpenAI's as well as other open source embeddings in the Llama Index pipeline. And at the end we will benchmark different embedding models based on their speed. If you are building a system to chat with your documents, you need to consider three different components. The first one is how do you pre process your documents? So mainly the chunking process I have a dedicated video on that link is going to be in the description of the video. The second thing to consider is what type of embedding model that you are using. And the last component is the LLM that you're using to generate the responses. I would personally argue that the document pre processing step as well as the embedding model are the most important components of this pipeline. Now in order to explain why I believe that, let's first explain what exactly embeddings are. In order to understand the concept of embeddings, let's look at this very simple example. So let's say we have four different words, man, woman, boy and girl. We can represent each of these words in this two dimensional space and the features that we are using are age and gender. If you look at this very simple two dimensional representation, you can see that a boy is semantically closer to a man compared to the other two words. Now in this case age and genders are going to be called semantic features and you can represent them by numerical values as shown here. Now you can add more words on this two dimensional semantic space and then you can start seeing a pattern that the words which are semantically closer to each other are going to be closer in this semantic feature space as well. For example, a grandfather is closer to man versus it being closer to a woman. And the beauty is that you can add more features to this feature space. So for example, now we are look looking at the royalty as well. So this is, this becomes a three dimensional space and you can represent each word as a vector of three dimensions. Now the best part is you can actually do some arithmetic and computations on these semantic representations. So for example, if you take the vector that represents a king and a man and subtract man from the king, then the resultant vector is going to be closer to a woman. Now, similarly, if you subtract man from king and then add woman, so you will get a vector which is going to be very closer to queen. Okay, so you might be thinking that, yes, this makes sense. However, these features that we see in here, these are the one that we define. However, we can train neural networks which will come up with different feature representations while preserving the semantic meaning of different words or sentences together. Now, these multidimensional feature vectors are actually called embeddings. So basically that means that you can take word embeddings for a sentence. So, for example, here is a sentence, I want to cancel my shoe order and then compute the word embedding for each word and combine them together to get sentence embeddings. But the main part is that similar sentences are going to have similar embeddings. Okay, so this is great, but how does it work when we try to create vector stores on top of embeddings? Okay, so let me show you a very simple representation of a vector store. So the vector store has three different components. First is the chunk id, as it. Basically, this is the text chunk that was created by splitting the text, then the original chunk text, and then the corresponding embeddings. Now for a retrieval augmented generation system. Now, when a new user query comes in, we need to find the closest text chunk based on the embeddings. So basically, we compute embeddings for the user query and then compare the embeddings of the text chunks present in our vector store and try to figure out which one is the closest text chunk. And as a result, we will get one or more chunks which are very similar to the query that the user has provided. So now, as you can see, this is the most important part, because if the embedding model is not good and it's not able to retrieve the proper chunks, then that will result in bad performance of the LLM. Now why? Because what happens is that we will take the query as well as the chunks that are returned by the embedding model. So this could be one chunk or a number of different chunks based on how many chunks you want to return. Those are fed into the LLM to generate a response. Now, as you can see that the LLM does not have access to the whole document. It's only looking at the user query as well as the chunks returned by the embedding model. That means you really want to pay close attention to what type of embeddings you're going to be using in order to represent the semantic meaning of the chunks that you are providing. Okay, so enough of the theory. Let's look at some of the code. So in this case here again I am just importing different packages that I need. I'm importing the llama index package, OpenAI then transformers, because I'll also show you how to use open source embeddings, including instructor embeddings, BGE embeddings along with OpenAI embeddings. And then we need some other packages, for example, accelerate, instructor embeddings, Sentence transformer and PYPDF reader in order to load PDF files. Okay, so we will first start with OpenAI's embeddings. So for that we need the OpenAI API key. Again, some of the other packages that we need. So we need the OpenAI model, Vector Store, index vector. Then in order to load files from a directory, we're going to be using the simple directory reader and some other packages for formatting. If you have watched my previous video, so I showed you this four lines of code to create a simple document Q and A system. So we load our documents, we create a vector store, then we convert this in into a query engine and then we can ask question. Now in this case we did not define what type of embeddings we were using. So again, this is the part of the pipeline where it's converted into chunks and then embeddings are being computed. Now, by default, if you provide the OpenAI API key, llama index uses the OpenAI's embeddings. Okay, so let me show you how to change defaults in llama index. I have covered this in the previous video link is going to be in the description of the video. So in order to change the defaults, you're going to be using the service context and then we can pass on the different parameters that we want to change. So for example, if you want to change the LLM, you can define the LLM that you want to use and simply pass this to the LLM property. Similarly, you can change the chunk size and chunk overlap, etc. Now these new defaults are going to be used by the vector store index. Now, in order to change the default embedding model, again, just to show you an example, we are loading the service context. Then we are importing OpenAI's embeddings and here we are defining a new variable called embed model, and that's the OpenAI model. We simply pass that to the service context. Now, whenever we use the service context as a part of the vector store index. It's going to be using the OpenAI's embeddings. Now let's look at a practical example of how these embeddings looks like. So what I'm doing in here is I took the OpenAI embeddings object that we just created and then I'm calling this function called Get Text Embedding and I'm passing a simple sentence. So AI is awesome. This is my sentence and this will return the embeddings computed for this sentence. We can actually look at the embeddings. So you can see it's a list of numerical values. These are basically different dimensions across which the vector is being computed. And we can see how many different dimensions this specific embedding vector has. Now, the OpenAI embedding model has 1536 different dimensions. So that means that each paragraph or chunk is going to be represented by a vector of this size. As you probably know that whenever you use OpenAI's embeddings model, you actually have to pay OpenAI, right? But we don't want to pay OpenAI, so we want to look at some alternatives. Now the good news is that there are a bunch of open source embeddings out there which are good for different tasks. So you can actually look at this massive text embedding benchmark which is hosted on Hugging Face, and you can pick embedding model that works best for your use case. For example, if you look here, currently the BGE Large English model is the one that is on top of the leaderboard. Now you can see the number of dimensions that each of these embedding models have. This one has a size of 1024 dimensions. If you look at the smaller model, something like this BGE small English, so it has only 384 dimensions. Now the one that I really personally like is this instructor large embedding model and it has 768 dimensions. So let me show you how to use these embedding models with Llama Index. Okay, so it's as simple as what we did. For the OpenAI's model, there is a class called Hugging Face that you need to import and then you simply need to pass on the name of the embedding model that you want to use. So you can basically go to the leaderboard, select the embedding model that you want to use, simply copy that name of the embedding model and paste it in here. So in this case, I'm actually using this small model, because I'm running this on a T4 Colab GPU. So it downloaded the embedding model. So in order to use this model, we're going to be using the same function, get text embeddings. And this will compute the embeddings for our text. Now, you can see for this small model, the Embedding size is 384Vector. As you can see with this model, we have a Vector Size of 38084 dimensions. Okay, the next model we're going to be looking at is the instructor embedding model. Again, you simply need to provide the model name from Hugging face and will download the model for you. And you can then look at the dimension of the vector itself. So again, it has a Vector Size of 768. Next, we will look at some benchmarks in terms of the speed of computations. So for benchmark computation, we're going to be using this file. It's part of the Llama index documentation. So they provided this example. Now, this specific file has a total of 172 pages. And we want to see which model is the best in terms of time computation. So basically what I have done in here is I downloaded the file. So this is the PDF file that I downloaded, and it has a total size of around 20 megabytes. So once the download is finished, you can actually go and see in here that it downloaded that PDF file. Okay, next we need to load the document. So for that, I'm using the simple directory loader. I am providing the file name so you can directly provide the file name and it will load that file. Right, so this is basically becomes part of the document. Now, in this case, it's converting each different page to a separate document. And that's why you will see a total of 172 different documents. Okay, so next we want to create Vector Store based on the documents that we just loaded. So in this case, we will keep everything to the default except the embedding model. And the goal is to compare the different embedding models that we saw so far in terms of their computational speed. Okay, so now let's see how long it takes for each model to do the embedding computation. So the first One is the OpenAI embedding model. We need to set it as a default model. So for that, again, we're using the service context, change the default model to the OpenAI embedding model. And here what I'm doing is I'm running it two times with the loop of two and Creating a vector index. So basically I want to look at the mean and standard deviation in terms of the compute time. And for this purpose I'm using the time it magic function within Jupyter Collab Jupyter notebook. Right. So it's going to iterate through the vector store or the embedding computation process multiple time and give us the mean and standard deviation of compute time. Now keep in mind for OpenAI's vector embedding, it's actually making call to the OpenAI servers, transferring data and receiving data. So it's going to be longer compared to any local embeddings that you are going to be using. Now, based on our results, it took around 46 seconds on average and the standard deviation is very small. Now a couple of things to notice in here, that for 172 pages it actually created 428 chunks based on the default values of the chunks as well as the overlap. Now let's compare the compute time to the open source embedding models that we were running locally. So for the BGE model it took around nine seconds on average and for the instructor embedding model it took around 19 seconds. That means if you're using a local embedding model, you will get the embeddings computed pretty quickly compared to the OpenAI model because it has to make calls to the OpenAI servers. And the embedding size itself is much larger compared to what we are looking at in here. Another thing to keep in mind is that a larger vector size for the embedding does not mean better embeddings. So in an upcoming video we will do comparison between different embeddings for the information retrieval task. And the goal is going to be look at different embedding types for different types of tasks to figure out which embeddings are best for which type of tasks. Okay, so just a quick recap. We looked at what embeddings are, how do they work and why they are important when it comes to the information retrieval task and what role they play in order to generate responses based on the LLMs. We also looked at examples of how to use them in Llama index. If you want to understand Llama index in more detail, I would recommend you watch this video. Thanks for watching and see you in the next one.