FALCON LLM - NEW KING on OPEN LLM LEADERBOARD!

Today we're going to be looking at yet another open source LLM that was just released. But this one is very special both in terms of its performance compared to other LLMs as well as a very interesting license. Hugging Face has a leaderboard for open LLMs where they compare the performance of open LLMs on four different benchmark data sets. So far Llama based models were on top of the leaderboard, but now they have been taken over by this model called Falcon 40B. This is a new model which is able to beat the rest of all open source models by a large amount and in fact three out of the four benchmark data sets it's able to beat the LLAMA based models. Now in this video we are going to look at it, how this is trained, what's so special about this model and also I will show you how you can run this inside Google Colab. This model was released by Technology Innovation Institute from United Arab Emirates which is one of a new player on the scene when it comes to LLMs. Let's look at some of the details. So Falcon is a foundational large language model with 40 billion parameters trained on 1 trillion tokens. Now they have another version which is smaller which has 7 billion parameters and I believe it's trained on 1.5 trillion tokens. One of the things that are they, they are highlighting is that this model uses a lot less computation compared to the existing models. So for example, and they're saying that it uses only 75% of GPT3's training computer, 45% of chinchilla and 80% of Palm 62 billion parameter model. So there are a few things which makes this model very unique. The first one is custom tooling. So they are not utilizing tools from Nvidia, Microsoft or Hugging Face. Rather they have developed their own data pipeline as well as custom code base to train these models. Now a second contribution or uniqueness is the quality of data that was used for training these models. We will look at the training set in a little bit. It's unique in the sense that it's multilingual as compared to most of the other open source models. They also made some optimization to the compute resources needed and it's one of their selling point. So they're saying that the training time has been reduced substantially and as well as the inference time. So for example, compared to GPT3 it takes only 1/5 of the compute at inference time. These are significant improvements and opens up a lot of possibilities to run these large language models on relatively cheaper hardware. Now we will look at the training data before looking at one of the most controversial parts, which is the license. There are two variants. The 40 billion parameter variant was trained with 1 trillion tokens. And the data set that they are using is called Refine web. So this Refine web data set is essentially a web call of all the Internet where the data is filtered and in some cases it's de duplicated. Now in terms of the distribution of the data, around 75% is coming from English. So that's the websites that were crawled. Right, 7%, which is around 7 billion tokens, it's coming from European languages and then the rest is coming from books, conversations, codes and technical. Now in terms of European languages, that 70 billion tokens consist of 26% German and then 24% Spanish and 23% French. And the rest of the languages have minimal contribution. So I would suspect that this model will work pretty good for English, German, Spanish and French. On the rest of the languages, the performance may not be that great. Now in terms of training, so they used 384A100 GPUs and trained this model for about two months. Now this shows you what type of resources you need if you want to train a foundation model. Now they do have a smaller model which is a 7 billion parameter model. Now this specific model is trained on 1.5 billion trillion tokens. So this is much bigger data set compared to the one that they're using for 40 billion parameter model. And I think it comes down to time as well as resources that you need to train a bigger model with a bigger data set. Now let's look at a controversial aspect of this, which is the license. So it has an open Source License, Apache 2.0, but with some restrictions. So it's not as restrictive as llama, but it has its own limitation. So it says you can freely use our models for research and personal purposes and you are allowed to share build derivatives of these models. But you are required to give attribution and to share like the same license. Now the last part of the license is the one that is the most interesting. So it says for commercial use you are exempt from royalties payment if the attributable revenue are inferior to $1 million per year. Otherwise you should enter in a commercial agreement with the Technical Innovation Institute. So that means that you can use this model for commercial purposes without any royalty payments as long as your revenue is under $1 million per year. Now if you go beyond that, you will need to have a commercial agreement with them. Now let me show you how you can use this in your own workflow. It's a completely new architecture. Therefore tools like Uber Booga Text generation Web UI does not support this yet. However, you can use this with the transformer package from Hugging face. So to show you how to use it, I'm going to walk you through a Google Colab notebook that I put together. Now we cannot run the 40 billion version on Google Colab. That's why we're going to be looking at the 7 billion parameter version. Now this one is a simple pre trained model which should be further fine tuned for most user use cases. So we're going to be specifically looking at the instruct fine tune version. Now in order to run the instruct fine tune version we're going to be utilizing the code that they have provided. I had to make a couple of adjustments and just install the required packages. So let me show you how you can run the Instruct fine tune version for the 7 billion parameter model. Now you can run this code locally and it should work. So first we are simply installing a few packages. So we are mainly using transformer package and then accelerate xformer and Enops. Now these packages are needed in order to run the model. The rest of the code is basically inspired from the code that they have provided in their example. So we are first importing auto tokenizer for tokenization process of the sentences or data that we are going to be providing and then auto model for causal lm. Now the model ID is basically selected from here actually in this case and selected from here. So you just need to copy that model id, pass it on to auto tokenizer from pre trained model so it will download the corresponding tokenizer. So the for each model you need to have a specific tokenizer for it to work. Now if you are familiar with the transform package, the rest is just basically setting up the pipeline. Now in this case we're going to be using the text generation type of models because it's a simple decoder only model. So it's not a combination of encoder decoder. Then we pass on the model ID that we selected the corresponding tokenizer. In this case we are using B float 16. So that's simply a floating precision that we want to use. And then if you have multiple GPUs so you can just set it to auto. In this specific case I'm using T4 GPU from Google Colab. It has a 15 gigabytes of RAM. Next I just want to show you how you can actually run this. So in order to run this, you need to use the pipeline that we just created and then you'll pass on your prompt. So in this case, my prompt is draft an apology email to a customer who experienced a delay in their order and provide reassurances that the issue has been resolved. Right. Then you define the maximum length of the output that you want and just set some other parameters for the pipeline. And here's the output. It's actually not bad at all. It's pretty good. Keep in mind that we are not using the best model in this case because the the model that's on the leaderboard, that is the 40 billion parameter model, whereas the model that we are using is this 7 billion parameter model, which is pretty down below on the leaderboard, but still it's able to generate pretty comprehensive responses. So for example, it says we sincerely apologize for the inconveniences and delay in the delivery of your order. Then it kind of explains that our training team works tirelessly to ensure that all orders are dispatched as soon as possible. But sometimes unforeseen circumstances arises, although it doesn't mention what those circumstances were. Right. And then it simply tries to reassure that actions have been taken to address the issues. So not bad at all for a very small model and you would expect a lot from the much bigger model. Here is just another prompt that I use to create a list of three startup ideas in enterprise B2B spaces and it came up with, I think a good list, but definitely you could do better. Now the goal of this section was just to show you how you can run these models. So if you have the resources and you want to run the 40 billion parameter model, you simply need to replace this model ID at the end of the video I just wanted to put a couple of comments on the leaderboard. It's great to have some kind of quantitative and qualitative measures of the performance of these models, such as benchmarks. However, I won't put too much weightage on these and the reason is that these benchmark data sets, they are very different than what you would use these models in real life. So what I'm trying to say is these benchmarks probably are not going to capture your use cases. That's why these benchmarks can just give you a direction of which model is good. But if you really want to compare the models, my recommendation would be to pick your own use case, pick some of these models and test them on your own data sets. I hope you found this video useful. If you did, consider liking the video and subscribing to the channel. If you haven't, thanks for watching and see you in the next one.