LongNET: WHOLE INTERNET in A SINGLE PROMPT

This is probably the biggest flex that I have seen in a research paper. One of the limiting factors of the current large language models is the context window or the number of tokens that it can work with. You probably have seen different models with different context Windows. For example, GPT4 has supposedly a version with 32,000 tokens. Then there's MPT story writer model with 65,000 tokens. We also covered Claude's hundred thousand context window. These context windows lengths are nothing compared to what I'm about to show you. Today we're going to be looking at Longnet, which is a transformer variant that can scale up to a billion tokens. Now, if this holds true, you can potentially feed in the entire Internet in a single context window. It's simply crazy to even imagine something like this. Now, the good thing is that it's actually coming from a reputable organization. This paper is coming out of Microsoft Research, which already has collaboration with OpenAI. Now, what does it even mean to have a context window of 1 billion tokens? You can basically provide the whole corpus of Internet in a single prompt to the model, and the model will be able to pay close attention to each part. Okay, the question is, how do they achieve it? So essentially it's actually a transformer variant. But in this case, instead of using a simple attention mechanism, they're replacing it with something called dilated attention. We will look at what dilated attention is in a little bit, but according to the author, it is able to expand the attentive field exponentially as the distance grows. So even if you have a longer sequence, it will be able to pay close attention to certain parts of the sequence. Now, there are a few things to consider. First, it has linear computational complexity. That means as the sequence length increases, the computational requirements are not going to be logarithmic or exponential, rather linear. You can train it in a distributed fashion. And the great thing is you can replace standard attention mechanism with this dilated attention mechanism seamlessly. And that's why you will be able to deal with very long sequences, even the entire Internet. Okay, so everybody's talking about this plot and they put it right next to the abstract. Let me explain what is happening in this plot. Generally, when you are comparing smaller and larger quantities in a scientific research paper, you usually put them on logarithmic scale. You will see something like this where very small quantities are compared with relatively large quantities. And you use logarithmic scale, so you'll see the trend. However, in this specific path, they're using a linear Scale on the Y axis just to show you how big of a difference this is relative to all the other models. So for example, if you look at GPT2, it had a context length of or context window of 512. Now there was this RMT model which was released I think a few weeks ago and it had a context window of 1 million token, but they all of them are relatively on flat line. And then you have the long net on top here with a billion tokens. Now this means in theory you will have a single model which will not face any type of hallucination because the model is not going to be limited by the amount of information it can look at at a single point in time. With billion tokens it will have all information available at all times. There are two major contributions of this paper. First is the concept of dilated attention. So with the help of this you can scale it to a billion tokens. And the second one is actually the associated computational complexity. Now if you look at all the previous attention mechanism, the computational complexity is not linear. However, they are able to achieve a linear computation complexity, which is a big deal in terms of the computational requirements that these models are going to have. In the rest of the video, I quickly want to look at how this dilated attention mechanism works. So first let's look at a standard attention mechanism. In a standard attention mechanism, the attention weights are computed over a fixed size window of tokens. This means that the model can only attend to a limited number of tokens at a time. And that's why you see these different LLMs have different context window or token length. So the standard attention mechanism will look something like this. Now, dilated attention mechanism solve this problem by using a sliding window that has gaps between the tokens. So the size of the gap is determined by dilation rate. So you see this parameter called dilation rate, and in this plot it has different values. For example, dilation rate of 2 means that the sliding window will have a gap of two tokens between them. So in this case we are considering every second token. So in this case you essentially double the segment length. If you have a dilation rate of 4, we're going to be considering every fourth token and that will quadruple the sequence length. Now theoretically you can expand the dilation rates and then you will be focusing on different parts of the sequence using these different rates. At the end you simply put them together. Now if you want to look at different parts of the sequence, you can use something called multi headed dilated attention. In that case you successively shift the position and then start paying attention to different parts of the sequence. Here is another plot that I wanted to show. So if you look at vanilla attention mechanism, the runtime requirements increases exponentially as you increase the length of the context window. However, for the dilated attention mechanism it remains pretty linear. That's why the runtime or computational complexity not as high as vanilla transformer implementation. Now the results that they have shown in the paper is actually based on 32,000 tokens context window. Now in terms of perplexity, they have shown that their models outperforms all the other transformer based models. Now this is a pretty big deal because even though if you're increasing the context length, you're not sacrificing the performance, it has been an issue with some of the other open source implementations. For example, there is this technique called superhot, with the help of which you can increase the context and length of llama based model, let's say from 2k up to 8k or 16k. However, you sacrifice a lot of performance, but this does not seem to suffer from that. For more technical coverage of the topic, I would recommend everybody to watch David Shapiro's video One Billion Tokens Singularity Approaches. It's a very interesting video. It goes into his perspective, what he thinks about it. I would recommend everybody to watch it. Link to the video is going to be in the description of this video. Now let's talk about what are the implications. Frankly speaking, I don't know. Now some people think it could lead to AGI, but we don't even know what that is going to look like. Now keep in mind that the actual work that they have done is limited to 32,000 tokens, but according to the authors it can scale up to a billion tokens. And when that happens it will really change the way we think about the capabilities of these large language models. I will end the video with the words of a wise man, what a time to be alive. Thanks for watching and see you in the next one.