LangChain: How to Properly Split your Chunks

Today I'm starting a new series where I will cover and break down concepts related to LLMs, LangChain and generative AI. It's going to be beginner friendly. Every video in this series is going to be focused on a single tool or technique. The goal is to not only teach you how to use these tools, but also explain how things work under the hood. That way you can better utilize these tools and concepts in your own applications. Today we are going to be starting with Recursive Text Splitter in LangChain. So let's look at Recursive Character Text Splitter and understand how exactly this works. If you are trying to extract information from your documents, you probably have seen this. You need to divide your documents or text into smaller chunks in order to process them. Now, how is the size of the chunks determined? So first and foremost, you divide the available text based on a list of characters. So you're dividing it or splitting it by characters, not by tokens. The second thing to remember is that the chunk size is defined by the number of characters, not by the number of tokens. This is a very important distinction to make because a lot of people confuse the chunk size in these text splitters with the number of tokens, which is incorrect. It's actually looking at the number of characters. So in order to understand this, let's define a chunk size of 200 characters. Then the Recursive Character text splitter uses these special tokens or characters to divide our text or documents. Now, the way it works is that the text is first divided into paragraphs. The paragraphs are divided into sentences, those are divided into words, and the words are divided into characters. So initially we start with paragraphs and see if each paragraph exceeds more than the given chunk size or the number of characters in that paragraph. Right. So if a paragraph is smaller than the chunk size, we keep that. If a paragraph is larger than the given chunk size, we further divide it into sentences and see if we can combine multiple sentences within that paragraph to create another chunk. This will become very clear when we look at code examples. Okay, in order to explain everything, let's look at this Google Colab. So I have installed a LangChain, and from LangChain Text Splitters, I am simply importing Recursive Character Text Splitter. Now, keep in mind this Recursive Character Text Splitter plays a very important role when you're extracting or retrieving information from your documents. So for this simple example, we are considering the text here. It's regarding Paris. Now, the title in itself is going to be considered as a single paragraph. Then you have three more paragraphs. So in total, we have four paragraphs here. So here I created a variable called text, and I'm assigning the text that I shown you above. Now, if you check here, here we have split by paragraphs, then this backslash N indicates a new line. Okay, to start, we look at a very simple example. So here I'm defining the chunk size to be 500 characters. There is no overlap between the chunks, and we are looking at each chunks based on the length or the number of characters. Now, we pass our text to this character splitter, and as a result, we are getting a total of three chunks. So if you recall, there were a total of four paragraphs. So how did we get a total of three chunks? Now, in order to explain this, let's look at the chunk size. So the first chunk has a size of 438 characters. The second one has 436 characters, and the last one has 445 characters. And here is how the actual chunks looks like. So what happened was that the first chunk, or the first paragraph, which is the title of the page, it has around 26 characters. And if you combine it with the second paragraph, then the size is still less than the chunk size that we have defined. So that comes out to be 300,438 characters in total. So initially we are defining it based on the paragraphs and then combining the subsequent paragraphs and see if the combined size or the combined number of characters in these two paragraphs exceeds the given chunk or not. So it means that you can combine multiple paragraphs together if the total size does not exceed exceed the chunk size that you have defined. Now, if you look at the second chunk here, this is basically the second paragraph, the third paragraph actually provided in the text, and that is around 430 characters long. Now, you cannot combine it with the last paragraph because for that itself, the size is around 440 characters. If you combine these two together, that will exceed chunk size that we have defined. Okay, I hope this makes it clear how the initial pass works. But now let's look at some more fun examples. So in this case, we reduced the chunk size to 250 characters. Now, as a result, we get a total of 10 chunks. And if you look at each chunk individually, now none of them is actually reaching 250 characters that we have defined. All of them are much less than that limit. So what exactly is happening in here? So for that, we need to look at the individual Chunks. The first chunk is actually the first paragraph, which is the title of the provided text. Now, the second chunk is a sentence from the second paragraph. If you look at here, each one of the chunk ends with a dot. So that means it's a complete sentence. So you might be thinking, what exactly happened here? Now, the way it works is it took the first paragraph and looked at the length of the first paragraph for the number of characters. So it has a total of 26 characters, which is less than the chunk size of 250 characters, right? So it keeps this. Then it tries to combine it with the second paragraph, but the second paragraph itself has more than 400 characters, right? So as a result, it will simply take this as a single chunk. Now, we actually need to further subdivide this paragraph into smaller chunks. And for that it looks at sentences, right? So each of the sentence in itself is less than 250 characters. But let's say if you combine the first sentence with the second sentence, that will exceed the chunk size. So as a result, it has to keep these individual sentences. Now, there could be a case in which you combine two sentences, sentences together, and the size is less than two for 250 characters. So that is going to be fine. But in this specific case, the second division is based on the sentences. Okay, so what happens if we further reduce this chunk size? So let's say we reduce the chunk size to 50 characters. As a result, we are getting 32 different chunks. And this is where things gets very interesting. You see, there is a lot of variation in the chunk sizes. For example, the first one is 26 characters. That's the title. But there is even one with only 11 characters. Again, to understand, we need to look at our chunks. So here is the first chunk, that's the title. Then the second chunk is actually a sub sentence. So basically the third level is it will start dividing sentences based on the characters. And as a result, you see a situation like this where when it tried to divide a sentence, so there is just one word left in one of the chunks, and that's the one which is 11 characters long. Now, the question is why the chunk size is important. And it comes down to information retrieval systems based on embeddings. So, for example, if you're returning only four chunks and you have very small chunk size, then it might be simply returning sub sentences, and you may not be able to extract information from that. So that's why it's very important to understand how this works under the hood and what type of implications it's going to have when you play around with different chunk sizes. Now, a couple of things to keep in mind. You need to select the chunk size based on the data you are playing with, so pay close attention to the type of data you are providing. So you might be actually thinking that having a large chunk size might be the solution, but that's not always the case, and let me explain that with a hypothetical scenario. So if you look at this text, all these paragraphs talk about relatively different things. Now if you define a large chunk size and all of this is put together in a single chunk, then it might confuse the LLM that is looking at this and deriving information on top of it. That is why you need to pay very close attention to what these chunks contain when they are returned by, let's say, a semantic search performed by an embedding model. I hope this was helpful. Now in this introductory video we looked at the default list, but you can actually modify this based on your own applications and needs. Now, if there is interest from the community, I will create a subsequent video on how to do that. If there is interest, I will create these videos. On other topics, I have seen some confusion around the embedding sizes. Well, for example, I like to use instructor embeddings that have a size of 5 12. A lot of people confuses that, whether it's the number of characters or number of tokens. So if there is interest, I can make a video on that. Or if there are any other topics that the community wants me to cover, I would definitely make videos on those as well. If you found this video useful, consider liking the video and subscribe to the channel. Thanks for watching and see you in the next one.