Run Uncensored LLAMA on Cloud GPU for Blazing Fast Inference ⚡️⚡️⚡️

The Cognitive Computation Group is famous for liberating large language models and making them uncensored using the famous Dolphin data set. In this video, I want to show you how you can take one of these models and deploy them on a web server, which enables you to access it from anywhere and you can also make it accessible to other users. For inference, we're going to be using an open source package called Vllm that gives you much higher throughput for LLM inference compared to other packages such as Hugging Face, TGI or Transformers. We also need a cloud instance that will enable multiple users to access our LLM. For that we're going to be using RunPod, who are also the sponsor of this video. RunPod is a cloud platform that you can use to train, deploy and scale AI models. There are a number of different GPU options available, so you can spin up on demand GPUs with GPU cloud and scale ML inferences with serverless. We will use Chainlet to create a UI for our chatbot. The UI is going to be running locally, whereas it's going to be interacting with the model hosted on runpod. The instructions I'm about to show you will work for any model hosted on Hugging Face. For this example, I'm going to be using Dolphin 2.9 Llama 3.8 billion model. I'm not using the latest 3.1 because it has some repetition issues. I'll show you how you can host this model on an on demand GPU as well as how to set this up on a serverless API endpoint. The amount of VRAM we will need is dependent on the size of the model, which is 8 billion model in this case. If you were to Deploy this in 16 bits, we'll probably need about 16 gigabytes of VRAM. If we use a lower quantization, we will need less vram, but it also is going to impact the performance of the model. So in Run Pod, when you sign up, click on Pods, then click on Deploy a Pod. You can see there are a number of different GPU instances that are available. We need at least 16 gigabytes of VRAM. So in order to be on the safe side we you can either select RTX 4090 or RTX 3090 which is the previous generation of the same GPU. So I'm going to select RTX 3090 for this video. Next we need to select VLLM template. If you click on this, you're going to be presented with a number of different templates that are Available on runpod. Some of them are official templates and some of them are built by the community. There is even a template for VLLM llama 3 instruct, so select that, then click on Edit Template. Now, in this case it's using the metalama official version, but in order to use this you will need to provide your hugging Face API key. And this is also a censored model. So we're going to replace the model with the Dolphin version, which is the uncensored version of llama 3. And I'm going to also add another hyper parameter where I'm limiting the max context length to 4096 tokens. Now this is important depending on the GPU that you use. If you use the Llama 3.1 with 128,000 tokens context window, you will need a lot larger GPU than 24 gigabytes. You can also add some environment variables if you want, but I'll keep everything to the default. Just click on Set Overrides. At the moment we are using On Demand, which is about 43 cents per hour. Next click on Deploy On Demand. This will start deploying the POD for us and start creating a GPU instance. This process can take some time. You can actually see the progress of the logs that are being generated. So currently it's downloading the models to this pod. The process will take some time, so I'm going to stop the video and come back. Okay. So I had to wait about five minutes, but after that I came back and looked at the system logs. When you see this message Start container, this indicates that it started the container. You can also see that the model was successfully downloaded. We're going to also look at the container logs here. You want to make sure that Vllm was able to load the model in the memory and we don't really see any other issues or error messages. So we're going to close this. Next, we want to connect to this POD or GPU instance. So for that click on Connect. Here we're going to click on Connect to HTTP service. If you see a message like this, don't worry. This actually shows that the POD was correctly deployed and it's ready for communication. So we're going to copy the URL here. This will become our base URL when we are interacting with this pod using Chainlet app. Now, you can also SSH into this pod if you want, or you can port this map to another port. But before showing you how to use this in the Chainlit app, I also want to show you how you can modify this and some other options. So, for modifying it, click on this icon, then click on Edit Pod. Now, here you can see a few options. One of them is the port at which the API is exposed. So by default it's using port 8000, but you can change it to any other port you want. The container that we are using has a volume disk of 40 gigabytes. Depending on the model that you're downloading, you might want to modify this as well. Okay, so our model is ready. And now we can interact with this model using our Chainlet app. So let me quickly walk you through the Chainlet app. In this case, we need to provide the model name. So this is going to be the same model name as the one that we deployed on our VLLM instance. Next, we need to provide our base URL. So this is the base URL that we copied from runpod. Next we're going to provide a system prompt. This can be anything you want. I'm just using the Dolphin system prompt. We have a function that is going to keep track of the conversation history. So you want the user to be able to interact with the model, and the model remember the previous conversations. So we take the base system prompt that the user have provided and append the chat history to it. Now, for the actual generation, I am using that base prompt plus the header. So we're going to be sending JSON data and the actual data is going to be the model name plus the system prompt, the user message. Then we're setting temperature to 0.2. For dolphin, you also want to make sure that you set the end of sequence token. If you don't set this, it can actually run into a loop where it will keep generating nonsensical data. For the dolphin 3.1, I have noticed that. That's why I'm using a previous version. And we also want to stream the responses. The rest of the code simply keeps track of the text chunks that are coming in. So we set a timeout of 30 seconds. If it takes longer than that, this will timeout and it will not generate a response. We're using chainlit. So chainlit has onmessage function. Basically, whenever it gets the message from the user, it will run this function where we are creating the system prompt, getting the user message or user query, then creating the message and then showing it back on the ui. The code I'm about to show you is based on this example code that was created by Data Centric, who has an excellent YouTube channel. I'll put a link to that. Will highly recommend to check it out. I also updated the base system prompt so now the system prompt is your name is Dolphin and sarcasm is your second language. So you want the model to be sarcastic. You are chatting with someone and every response should be dripping with wit and playful sarcasm. But feel free to roast the user, but keep it friendly. Make sure they know it's all in good fun and we are going to provide it with the conversation history. So it's based on the system prompt that is available in that hugging face repo. Now for this to work we just need a single package. So I created a CONDA virtual environment called Uncensored and you can install chainlet using pip. Install chain.