The Truth about GPT-4 128K Context Window

OpenAI released GPT4 Turbo with 128,000 tokens context. But can the model actually use that context effectively? Let's find out. In this video we are going to look at some empirical evidence on the performance of GPT4 Turbo. But before that, let me quickly introduce you to this paper titled Lost in the How Language Models used Long Context. This was an important paper and it specifically looked at the performance of LLM with large context windows. So they wanted to look at the model's ability to extract information from its context. So one of the most important finding of this paper was that if you are looking for information that is present either at the beginning or at the end of the context, the model tends to perform really good. However, the information that you're looking for, if it is right in the middle of a long context, then the performance of these LLMs degrades a lot. And another important finding was that the performance substantially decreased as the input context grow longer, even for explicitly long context windows models. Now they found that you basically get a U shaped performance curve for these LLMs. So for example, if the information is present at the beginning of the context, then the accuracy of retrieval is a lot. So let's say for GPT 3.5 Turbo it's around 75%. Now, towards the end the accuracy of retrieval is also good. However, if the information that you are looking for is present in the middle, then the performance is pretty bad. Now what they found was that these patterns remain consistent across different LLMs. So it's not specifically related to GPT 3.5, but even for other LLMs such as Claude, or even the open source LLMs such as MPT 30B instruct this pattern is there. You can see that in this plot here. So in the beginning almost all the models are performing really good if the information that we are trying to retrieve is in the start of the document. Now as we go towards the middle of the document, the performance degrades in most of the cases and towards the end if the information is present here, then the performance of retrieval is again pretty good. Now, since GPT4 Turbo has a very long context window compared to its predecessor, so does it suffer from the same lost in the middle phenomenon or not? Well, the answer is both. So let me explain. So I'll show you three different takes on this since there are no scientific studies on GPT4 Turbo yet. The first one is from Sean, who is the founder of Small Models. So here is the test setup. So they put 10 different random information in the context of GPT4 Turbo and asked it to retrieve that information. And the goal was to see whether the model is able to find or answer these questions correctly or not. Okay, so here's a simple plot where on the X axis you have the number of tokens that were used for the context and the number of facts that are retrieved correctly. So, so if you see if the information is at the beginning of the context, then the model is able to retrieve that information. So almost close to probably around nine questions are correctly answered by GPT4 Turbo. Now there are two different checkpoints of GPT4, both with 32,000 tokens. So all of them does pretty good when the information is at the beginning of the context. Now as the context window increases or basically the information moves further in the context. So let's say in the middle. So if you're using around 8,000 tokens, then the accuracy of the previous GPT4s drops to only 4 answers out of 10. Now but for this GPT Turbo, it's actually still pretty good because it's right around 8 out of 10 questions. But you will see that there is a decreasing Trend even for GPT4 Turbo, up to 32,000 tokens. Now, since this GPT4 Turbo has a much larger context window, we can see that trend kind of continues. So at 64,000 tokens, it's able to retrieve only 1 out of 10 different facts from the given context. So this kind of shows that large context might not be helpful in this specific case, but there are definitely use cases for much larger context. Now here is a nice summary in a form of a table. So if you look at here, it does perform really good compared to the previous versions of GPT4. However, I would say around 32,000 tokens. That's where it kind of starts dropping off drastically in performance when it comes to information retrieval from its own context. At this point it's only able to retrieve around two and a half questions out of 10 questions that they use. Okay, so now the question is how does GPT4 Turbo performs compared to other proprietary models for information retrieval tasks? So I found this tweet from Alex, who is the author of this paper. Attention Sorting combats recency bias in long context language models. In that paper they showed that the recency bias or lost in the middle phenomena is because of the attention parts that are learned during pre training process. So I'm not going to bother you with the details of the paper, but the main finding is kind of summarized in this plot which shows that the performance of GPT4 Turbo is pretty good, up to around 32,000 tokens. However, it decreases a lot when you go beyond that point. Now, Claude 2, on the other hand, which is shown in this blue line, seems to be able to retrieve information much better from large context windows. Now, one of the best and easiest to understand analysis that I found was from Greg, who actually kind of motivated this whole video. Greg also has an excellent channel on YouTube, so I'll highly recommend everybody to check it out. I'm going to put a link to his channel in the description. I really want to go over the process of how this analysis was done, but let's first look at the final results. Okay, so before looking at the results, let's understand what exactly is going on in this plot. So, effect was placed within the document. GPT4 was then asked to retrieve this and the output was evaluated for accuracy. So basically, he placed a sentence or a certain fact within the document and varied the context window and then asked GPT4 Turbo to retrieve that information. So, as you can see, the test was done at 15 different documents depth. Right, and we're going to understand what this document depth means at 15 different context windows. So what he did was he has performed different experiments with different context windows, starting from just 1,000 tokens up to the total of 128,000 tokens. The Y axis shows where the fact is placed relative to the start of the document. So this is the beginning of the document. And then as you go up, that becomes the top of the document. Basically, he placed that certain fact at different depth that is described by document depth. So 25% depth means that it's in the top 25% of the document or towards the end of the context window. So from this experiment, it seems like up to 64,000 tokens. Irrespective, wherever the effect is placed relative to the start or end of the document, the model is able to retrieve that information correctly. However, when you go beyond 64,000 tokens, it gets really tricky. Now, if the fact is placed in the beginning of the document, you still get a pretty decent performance, even though if you use very large context windows. However, if the fact is placed in the last 50th percentile, then the retrieval for larger context window is pretty bad. Now this kind of aligns from what we saw from Sean's assessment in the beginning of the video. Okay, so here we will quickly summarize the findings of this experiment from Greg. If you go beyond 73,000 tokens, the performance starts to degrade the second finding is low recall performance was correlated when the fact to be recalled was placed between 7 and 50th percentile document depth. That means that if the information that you're trying to retrieve is towards the end of the context window, the model is probably going to have trouble retrieving it. And the last one is if the fact was at the beginning of the document, it was recalled regardless of the context window. Even though if a large language model has a very huge context window, you need to be aware that there are limitations on the accuracy itself when it comes to retrieving certain information from these large contexts. So to summarize, even though if you are working with a model that has a very large context window, it's better to use less context because that will result in more accuracy, even though GPT4 has a very long context window. But I think it's still better to use the conventional wisdom when it comes to Lost in the Middle. I hope you will find this video useful if you're working with GPT4 Turbo models comes to prompt engineering and information retrieval. Now keep in mind that this applies only to extracting information that is present in the context of the model, but if you use something like rag, which is retrieval, augmented generation, you will probably get different results. It's also very important to keep in mind that the analysis that I showed you in this video is based on empirical evidence. There are no scientific studies yet which have thoroughly evaluated whether this is consistent across, like let's say, multiple data sets or not. But I think like it probably is going to hold true based on what we have seen so far. So let me know if you found this video useful. Consider subscribing to the channel for similar content. I make a lot of content on LLMs, not only covering different limbs, but actually how to use them in more practical scenarios. So let me know if there are any interesting papers or projects that you want me to cover in my upcoming videos. Thanks for watching and as always, see you in the next one.