How to Set the Chunk Size in Document Splitter | RAG | LangChain

Welcome to the second video in our series on how to do better rag. In this video we are going to be tackling the topic of how to split your documents. This is the most important step in your rag pipeline and you really need to pay close attention because if you do this part wrong, the RAC pipeline is not going to work properly. For document splitting, you need to consider two things. One is the type of splitter to use, and the second one is the chunk size. I previously created a video on different splitters. In this video we are going to look at how to define the chunk size. If you have used lag chain for rank, you probably have seen this code. So in this case we are using the recursive character text splitter and we're coming up with a arbitrary chunk size of 2,500 characters. You probably have seen different values ranging from a few hundred to a few thousand. And then we are defining our chunk overlap. Now the focus is going to be how to correctly define the chunk size. If you're using the recursive character text player, your chunks are based on number of characters, not number of tokens. Now, the chunk size is highly dependent on the type of documents you're working with. But apart from that, there are two other factors that you need to consider when you are defining your chunk size. The first one is the embedding model that you want to use as a part of your Rack pipeline, and the second one is the context window of your LLM. In this video we're going to be focusing mainly on the embedding model. For this example, we are going to be looking at an open source embedding model and that is going to be BGE Small English version 1.5. Similar concept will apply to OpenAI embedding model and we will cover that in the next video. Now, each embedding model has its max sequence length or max token size. For the BGE small English embedding model, the max sequence length is 512 tokens. And this is a very important number to keep in mind. Okay, so let's look at a few embedding models and try to understand the relationship between the chunk size and the embedding models. If you look at these open source embedding models, there are two things that you need to consider whenever you are selecting one. First is the size of embedding model. So that is the dimension of your final embeddings. And the second one is max tokens. And this is the one that will determine the size of your chunks. Now in this case, there are a variety of different embedding models with different max tokens, which can go up to 32,000 tokens. Let's understand how this embedding model works. So basically, you take your chunks, you pass it to the tokenizer of your embedding model, and that will be converted into tokens for the embedding model to embed. So if your chunk size has more tokens than the max token length of your embedding model, it will simply take the first part up to the max token length and will discard the remaining tokens within the chunk. This is very important to understand. So let's assume our chunk size was 2500 characters. And when we do the tokenization, let's say it becomes 700 tokens. But the embedding model that we chose has a max token length of 512 tokens. That will mean that it will simply take the first 512 tokens from your chunk and discard the rest of the tokens. As you can see, if there is more important information towards the end of the chunk, it's not going to be utilized when we're doing semantic search. So let's go back to our code and let's understand what is the relationship between the chunk size that we define and the number of tokens generated by our embedding model. So in this case, I am defining this function called plot document tokens. It will take chunks from your documents, the embedding model that you want to use, Then it will load the tokenizer for this specific embedding model and create tokens for each of the chunks, and we plot those tokens. Now, I ran the original ARCA paper through this process with a chunk size of 2,500 characters. So in number of tokens that this is what we get. So there are chunks which has more than 700 tokens, but anything beyond 512 tokens is not going to be considered at all. Now, how do we fix this? So the easy solution is to define the chunk size in terms of number of tokens rather than the number of characters. And fortunately, it's very easy to do if you're using LangChain. So instead of using the recursive character text splitter directly, you want to use the from hugging face token as a function within the recursive character text splitter class. Next, we need to provide the model name and it will load the tokenizer. After that, the chunk size is now going to be in terms of the number of tokens, not in terms of the number of characters. And once we have that, then you can define the chunk overlap. Again, the chunk overlap in this case is going to be in terms of number of tokens, not in terms of number of characters. Okay, so once we make this change, here is the distribution of chunks in terms of number of tokens. This is exactly the same data, but now the max tokens that you can have in your chunks are going to be 512 tokens, not 512 characters. Now again, to reiterate, you want to chunk your documents based on the tokenizer of the embedding model, not the number of characters that are in a chunk. Let's look at this example. So this is a space that is running on hugging face. You can copy your own text here and understand the behavior of different splitting mechanism as well as what is going to be the impact of different length functions. So here, for the splitting chunks method, we're going to select first the character text splitter. So this basically takes say 200 characters at a time and simply create chunks based on that. And here we are defining the length function to be characters. So if you use this, this will essentially take 200 characters each and just create a different chunk. And the overlap is 10 characters. So let's say we look at the first chunk, this is the first chunk. Then we're going to look at the second chunk, right? That's the fourth chunk because there is an overlap. So if you click on this, this will highlight the overlaps in this color. Okay, but using character text splitter is not really a good idea. We're going to be using the recursive character text splitter. If you don't know how it works, I have a dedicated video on this. I'll put a link to that in the video description. But basically what it does is it will first try to split your document on paragraphs. Then paragraphs are split based on sentences, then words are split based on spaces, and so on and so forth. And you can change this so you can define what are different splitters that you want to use. For example, if you are working with markdown, so you definitely want to use different mechanism. But by default this uses the character count. And again here we see that if you use 200 characters, so the first chunk is going to contain this part. The second is this third is this. Right now if we change this to our tokens, so we're going to see that it has much better representation. So basically instead of using characters, we're going to now it's looking at individual tokens and looking at 200 tokens at a time to define the chunk boundaries. Now I hope this was helpful and hopefully this will help you in both understanding how the chunk splitting works and how to design better splitters. If you are interested in more detailed coverage of this topic, I'm putting together a more advanced rag resource. Please sign up. Link is going to be in the video description in that we are going to go over the whole RAC pipeline. We are going to look at each component and we'll talk about how to make different design choices and how those are going to impact the behavior of your RAC pipeline. Link is going to be in the video description if you're interested make sure to sign up. Let me know in the comment section below if you if you like more of these in depth tutorials and explanation of concepts. If so, I'll be creating more of these. I hope you found this video useful. Thanks for watching and as always see you in the next one.