Yi-34B - 200K - The BEST & NEW CONTEXT WINDOW KING 🚀 🚀

Welcome back. We have a new model with 200,000 token context window, but I'm not talking about Claude 2.1 from Anthropic. It's a new series of open source large language models called Yi from a new company called 01ai out of China. In my test the models are extremely impressive and it currently sits at the top of the hugging face openllm leaderboard. Early this month the first two version of this model were released. One is the 6 billion version, the other one is the 32 or 34 billion version. In the initial iteration the context window was limited to 4K tokens for training. However at inference time it can be extended to 32k tokens. But later on they released the 200,000 token versions which is extremely impressive. Initial release was limited to the base models. Today they released the instruct version or chat version of these two models. So now they are going to be a lot more useful. And the best part is they also released the quantized version. So you have the models in 4bit and 8bit quantization, which means that you can actually run this on consumer hardware. So for example, the 4 bit version of the 34 billion model, you can run this on either 1390 or 1490 GPU because if you set the batch size to 1, you will need around 19 gigabytes of VRAM. 3090 and 4090 version has 24 gigabytes of VRAM. As you saw earlier on the benchmarks, the results are extremely impressive compared to all the other open source large language models. Before looking at how the model was trained, some comments about the license and testing the model, I want to highlight certain things that are mentioned on the model card, which I think very important whenever it comes to testing any of the open source or proprietary large language models. So here they say While benchmarking open source models, we have observed a disparity between the results generated by our pipeline and those reported in public sources. Then they go on to say, upon conducting a more in depth investigation of this difference, we have discovered that various models may employ different prompts, post processing strategy and sampling techniques, potentially resulting in significant variations in the outcomes. Our prompt and post processing strategy remains consistent with the original benchmark and greedy decoding is employed during evaluation without any post processing for the generated content. I wanted to highlight this for one specific reason and that is different models that we test have actually different prompting techniques. So one prompt that works on one model may not work on another model in the exact same form. So sometime you actually have to modify the prompt for it to work on a different model. And that's why the results that we see on benchmarks may not be a true representation of what the models are capable of. For example, if you simply modify the prompt for a certain model, it might start performing better on these benchmarks as well. So the whole idea is if you prompt a model and it doesn't really work, just try modifying the prompt and you might see a performance boost. Now let's look at how the model is actually trained. So it's a bilingual model, trained only on Chinese and English. So it's not going to work on any other language. And it's trained on approximately 3 trillion tokens. Another thing that I want to highlight before we go into testing is that It's a modified llama 2 architecture. So that means the tool that are developed for llama 2 are probably going to work with the series as well. So that's a good news. Now, in terms of the license, initially they had a very confusing license, but now it's Apache 2.0. That means you can use this for academic purposes as well as for commercial purposes. However, you need to get permission from them for commercial purposes. Now, the application is pretty straightforward. You just need to send them an email and in most of the cases they accept it. You can run the model in two different ways. If you want to use this within your own projects. You can use Transformer package from Hugging face to load and use the model. So here is a quick example. You simply need to import both the auto model for Causal LM as well as the auto tokenizer. You provide the model path or model repo ID from Hugging Face. Now, if you notice they are using the original version, the unquantized version. However, you can use the quantized versions as well. Now, for some reason they have used different techniques for quantization. So for example, for the 8 bit quantized version they are using the GPTQ format and for the four bit quantized version they're using the AWQ format. That means to run the 8 bit version you need an Nvidia GPU for the AWQ format. I think you can run this on Apple Silicon as well, but I haven't tested personally. Another thing is that I'm not sure if the chat version actually supports 200,000 tokens. I have tested in on, on a very long text. It seems to work, but I don't think I'm. I'm reaching that limit. So it might be limited to the 32,000 tokens. We are not sure because that Information is not really clearly mentioned. Now going back to the Python code, you can provide the model repo, then load the tokenizer and in order to use the model, we're going to be using the auto model for causal lm and then you can provide the different rules. So for example here in this example they're just providing the user input or user role, but you can also provide system message and if you use the tokenizer and model on top of it, you will get a response. It's using the Chatlm prompt template, but with the new update of transformer package, you don't really need to explicitly mention this. Now it's part of the model card, so you don't have to worry about it anymore. But if you just want to chat with the model, you have two other options. One is the hugging face spaces, so it's hosted on the Hugging face chat. Or you can run this on replicate as well. We're going to be using the hugging face chat version. So here's the Hugging face chat UI and you can actually ask questions and get responses. And the responses are pretty quick. So I already tested on a few prompts and it seems to be doing pretty good. So the first one was Daniel picks up the football, Daniel drops the newspaper, Daniel picks up the milk, John took the apple. What is Daniel holding? So the model has to do multi step thinking in order to come up with the answer. So here it says based on the information provided, Daniel is holding the football and milk, he dropped the newspaper and John took the apple. So those items are not with Daniel, which is correct. So here is another prompt that I actually like to test. Most of the open source LLMs that I have tested so far are not able to get this right. The prompt is how many days will it take for the pond to be half filled with lilies? If the number of lilies double every day, and initially it takes 48 days for the pond to be completely filled. Okay, so here's the answer and it actually got it right. So it says if the number of lilies double every day and it takes 48 days for the pond to be completely filled, then on the 40th day there would be this many lilies in the pond. Right? So it goes through all the computations and at the end it says therefore it will take 47 days for the pond to be half filled with lilies. Because on the 48th day the pond will be completely filled and on the 47th day it will be half filled. I tried this same prompt a few times. Now, even When I tried it again it gave me the correct answer. But previously I have seen the model to struggle with this question. So I'm not sure if those are just initial temperature conditions that are set, but you could get incorrect answers now one thing I have noticed is it tends to write a lot. So for example here I was asking it for a financial advice. So I said if I had thousand dollars to invest into AI companies on the Nasdaq, what would you invest in and why? So it says if I had thousand dollars to invest in AI companies on the Nasdaq, I would consider the following factors before making an investment decision and the factors are Market leaders Growth potential Diversification Financial health Mergers and acquisition Partnerships and collaborations Technological edge and then it lists these six well known companies, Nvidia, Microsoft, Alphabet, Amazon, Facebook and Apple and also tells us about their business model and AI usage. So for example, for Amazon says for its use of AI in e commerce platform and through Amazon web services. So that's the aws. At the end it states it's important to note that investing in individual stocks carries a higher level of risk than diversified investment like mutual funds or ETFs. Before making any investment decisions, I would also consider consulting with a financial advisor and doing thorough research on the companies and the AI industry as a whole. So the response is pretty nice. Now before looking at whether it can work with very long text, I wanted to see its instruction following abilities. I said write a poem about breaking bad in the style of horatio, limited to 16 lines. Now it came up with a really nice poem. I must say it rhymes pretty good. However, instead of 16 lines it gave me 28 lines so it's not able to keep track of the number of lines. I wanted to see how good this is going to do on large body of text. So we have this ongoing drama at OpenAI regarding the hiring and firing or firing and rehiring of Sam Altman. So I take this article from Guardian and pasted this in Google Doc. So it turned out to be a total of four pages and I wanted to see if the model can summarize it. So I said provide an executive summary of the following text with main points in the form of a list and then I provided the whole text in here and I think it did an absolutely amazing job with the summary. So it says OpenAI's fighting and hiding of Sam Altman has shocked the tech world and raise concerns about the company's leadership and the future of AI development. Then it talks about the power struggle at OpenAI and it refers to that it reflects the immaturity of the AI industry. Now the article actually talks about the divide between accelerationist and decelerationist, so it also captured that. It also talks about how people perceive the action of the board. So it's actually creating a really good summary. It's only four pages, so it's definitely not reaching the limit of the context window. And that brings me to this amazing analysis by Greg. He tested both the GPT4 Turbo 128 context window model as well as the Claude 2.1 200,000 tokens models. Now he put together this interesting looking diagram. So here it's describing the relationship between the context window and the performance of retrieval of certain facts that are present in the document, depending on where the fact is actually located in the document. I have a complete video on this topic where I discuss the performance of GPT4 Turbo in terms of the context length. I would recommend everybody to watch that video, but there are a couple of things that we see when it comes to these large context models. The one thing is that the location of the information that you are trying to retrieve matters, so if it's in the beginning or at the end of the context, you will be able to retrieve those pretty accurately. However, if the information that you want to retrieve is in the middle of the document, then these large context models actually have trouble retrieving that. Another important point is that even if you have a large context window, it's always good to use a smaller portion of it rather than the whole context because that will give you the best performance. Now I'm assuming that the Yi 34B or 6B 200,000 tokens context window models are going to suffer from the same lost in the middle phenomena. Anyways, I hope you found this video useful. This seems like a very interesting model, not only because of its large context window, but its performance and I think it will be very useful for retrieval augmented generation tasks. If you would like to support my work, check out my Patreon or you can join the YouTube membership community as well. We also have a very active Discord community so I would recommend everybody to check that out. If you need help or advice on your own projects, you can reach out to me. There is information in the description of the video. Thanks for watching and as always see you in the next one.