Run Huge GPT Models the on Any Device with Torrent-GPT

Today we're going to be looking at this amazing project called Petals that lets you run large language models at home. The BitTorrent style, similar to BitTorrent. The idea is very simple. You are sharing your resources with other people, so you load a small part of the model and then team up with other people serving the other parts of the model to run inference or fine tuning. As a result, you are able to run huge models like Llama 7B, Guanico 65B, or even Bloom, which is 175 billion parameter model at a reasonable good speed. On the Petals network, people are basically sharing resources. So here for example, you can see different models. That's the llama2.7 billion parameter model. Here is the llama65, the bloom, bloomz and so on and so forth. It's an interesting idea and if you want to learn more about it, I would recommend you to check. Check out this paper, Collaborative Inference and Fine Tuning of Large Language Models, where the author goes into a lot more details about the architecture of the model and how they are running both inference as well as fine tuning. So in this video I'll first show you an online demo that is hosted by the authors. And after that I'll show you how to run this locally on your own machine. And we are talking about huge models such as the Llama 65B which you cannot fit in in a consumer GPU. But thanks to this project, we will not only be able to run it, but we will get pretty decent speed. So let's get started. So first we will look at the demo that is hosted so you can access it by going to chat.peters.dev I will put the link in the description of the video. So here you can notice there are three different families of models hosted here. One is the Llama two, the other one is the original Llama and then the last one is the Bloom models which are the biggest of them all. So for this simple demo, I'm going to select llama 2 and then we will be using the chat model, not the base model. So here is a simple question. What is the capital of Canada? The purpose of this experiment is just to look at the speed of generation. I'm not really looking for accuracy or the length of the generated text. Right. So here actually the speed is pretty good. If you could see, it was almost six tokens per second, which was amazing considering we are running a 70 billion parameter model. Now the great thing about this is that you can actually chat with the model so you can ask subsequent questions. And then it will generate responses for this demo. I have noticed that there seems to be a limit on the maximum tokens that it can generate, right? So it's not going to generate huge responses. But considering that it's actually running on a distributed network of computers or GPUs that people are contributing to, the speed is absolutely great. This is amazing. But the actual benefit of this is that you can run this through an API and integrate it in your own apps. So let me show you how you can install this and run this your own Python codes. So for that we'll go back to the official repo of Petals project where they say run large language models at home, bittorrent style. That's pretty nice. Fine tuning and inference up to 10 times faster than offloading. So if you have an old gpu, this is going to be much faster for you to run rather than run the models on your own local gpu. Now personally I haven't experimented with the fine tuning option, but if you can fine tune the models, that's going to be absolutely amazing. Now in order to run these models you will have to install the official Petals package and then you can use it in pretty much similar way the way you use Transformer models. Now you're going to be using the auto tokenizer from Transformers. However, in order to run the model, we will be using this auto distributed model for Causal LM class from the Petals package. Now it's a community run system, so people are basically sharing their GPUs. In order to run this you will actually need to have a GPU. And if you're not using your GPU you can connect it to the Petals network and contribute to the network capacity. So the more people contributing to the network, the better the capacity and influence speed is going to be. Since I don't have a GPU on my M2, so I'm going to be trying this in Google Colab. But the process should be pretty similar to if you're running this locally on your own gpu. Okay, so they have provided a really detailed Google Colab notebook which walks you through a step by step process. So first and foremost I would recommend you to go and save a copy of this notebook and then make sure that you are running it on a GPU runtime because you need a gpu. Then simply hit connect all and then you will be connected to your Google Colab. I have already created a copy here and it's already connected to a gpu. Once you are connected you need to run this PIP install Petals. This will install the Petals package and then you can start experimenting with the models. And then at the same time, I think it's better to monitor the resources that are available in Google Collab both in terms of the system RAM and GPU RAM and see what type of load is going to put on your system. Okay, so let's look at how you can use the models. So again, you simply need to import the Torch library auto tokenizer from the transformer package then, and then you need to load the auto distributed model for causal LM class from the Petals network. So in this case, according to the authors, this machine will download a small part of the model weights and rely on other computers in the network for the rest of the model. Right. So you're not going to be downloading the whole model, but just a part of it. Right. And since it's a part of the network, then we'll be using resources from other computers and other GPUs to run these models. Then rest of the stuff is very similar to simple transformer package. Right. In this case we're downloading the 65 billion, the original llama. So let's run this and see what happens. Now, if you notice, it literally downloaded a 1 gigabyte file and the rest are actually on the network which it's going to try to access. Now, in this case we are using, as I said, the original 65 billion parameter model, but you can use the LLAMA 2 models if you want. Now, in order to do text generation, you can simply call the generate function on the model object. And in this case we are providing the inputs. So the query or the input text is the capital of France is right. And this is in text completion mode. So here we are expecting the model to generate up to five tokens, right? And the response that it generated is the capital of France is Paris. Now, when you run this query, it's going to actually take a little bit of time because it has to connect to the network. And once it's connected, then the generation speed is going to be pretty fast, as you can see here. And in terms of the resources, you can see that we are using a fraction of both the system as well as the GPU RAM. And in the background we are running a 65 billion parameter model which is simply outstanding and amazing. The great thing about this project is it's not just a demo. You can actually build powerful applications by running these models in distributed fashions. So here they have provided a simple example of how you would create a chatbot using this model. So it accepts an input from the user and generates a response. So let's say I asked the same question before, but I'm going to do that again. So what is your name? And you see that the model is going to be able to generate a response. So it says my name is Hall 900. Funny that it came up with the reference to the Space Odyssey. And then you can keep asking questions here, right? So basically this is a simple example that you can create applications using these models and integrate those. Okay, it says I was created by human. That's great. I will recommend everyone to check out this amazing project and see if you can contribute your resources. This Google Collab Notebooks goes into a lot more detail and if there is interest, I can create subsequent tutorial videos even trying to figure out how to fine tune models. One thing actually I want to highlight is that you can create your own private swarm if you are working on sensitive data. So in that case it's going to be kind of a sub cluster that is private to you and it's going to be run by people that you trust. So that's again a great feature to have to run models in a distributed fashion among people that you trust for your sensitive data. I hope you found this video useful. Check out our Discord server to stay updated on everything that is happening in the generative AI space. Thanks for watching and see you in the next one.