Deploy Open LLMs with LLAMA-CPP Server

Welcome to the second video of our series on LLM deployment and how to serve multiple users with a single LLM. In the previous video we looked at Nvidia Nim, which is a set of microservices designed by Nvidia for extremely fast and well optimized inference of local LLMs and local AI models. Now in this video we are going to be looking at LLAMA cpp, which is probably one of the most consequential open source projects when it comes to inference of open weights and open source LLMs. If you have used Ollama LLM Studio or John, you have used llama CPP, because these UIs are built on top of llama CPP. And not only that, but virtually any local UI that lets you interact with open source LLMs are pretty much built on top of Llama CPP. And it's not only limited to the UIs, but virtually all the major open source or open weights large language models are supported within LLAMA cpp. We will learn how to install LLAMA CPP on your local machine and then use the Llama CPP server to serve multiple users using a single LLM and a single tpu. So first we will look at the installation, then how to set the server up, and then I'll show you a number of ways in which you can interact with the server and later on we will look at some practical examples. So the easiest way to get started is to use the homebrew package to install LLAMA cpp. This will work natively both on Mac OS as well as a Linux machine. In order to run this on a Windows machine, you will need to use ws. So in order to install LLAMA cpp you are going to use the Brew Install Llama CPP command once you download it. Serving a model is extremely easy. All you have to do is to use the Llama server command. Then you will need to provide the hugging face repo ID for that specific model. And after that you will also need to provide which specific quantization level you want to use. So for quantized models, specifically for GGF quantized models, you will need to provide a specific file name. Now to look at a more concrete example, let's look at a specific model. So in this case I'm serving the Phi 3 model. So here is the repo ID and we're looking at the GGUF format model because we want to use the quantized version. Now then you need to provide the specific quantization level and that's basically the file that you want to use. So here I'm using the 4 bit quantized version again in GGUF format. Once you run this command, it will start the Llama CPP server and it will start listening to incoming requests. Now, you can make requests in multiple ways. I'm going to show you three different ways. So the first one is going to be one of the simplest one. So you can use the curl command to make post requests. So you need to provide the URL by default. Llama CPP is going to start serving the model at port 8080 and it's going to create a chat completion endpoint. Then you also need to provide a header that is going to be application/JSON. Then another field that is necessary is the actual data. So this will contain both the prompt as well as the max number of tokens that you want the model to generate. So if you just want to test your model, you can use the curl command. Now, another great thing about Llama CPP is it uses an API server which is compatible with the OpenAI standard. So that means you can use the OpenAI client to actually make requests to that server. In this case, again, we'll just need to provide the base URL. So this is going to be the local host, 8008 is the port number and v1 is for the chat completion endpoint. Since you are running this locally, you don't need to provide an API key. Now, if you were to host this Llama CPP server on a remote machine, for the example on AWS cloud or run pod, then you need to make sure that this port on the localhost is forwarded to the external port plus the external IP address that you're going to be using. So this is the second way and you can make requests to the served model. Now the third one is just using the request package. So if you're using a Python client, this is probably one of the easiest way we're going to use the request package. We'll need to provide the URL again, in this case port 8080. Then we'll use the chat completion endpoint and the header contents are going to be application JSON. We don't really need any authorization key. And then after that you just need to create a post request. So it'll provide the URL plus the header and then a JSON of the data. So basically the prompt that you want the API endpoint to process. Okay, so let me show you practical examples of what we saw so far. To get started, we will first use The Brew Install Llama CPP command. You will see that I'm running this in Visual Studio Code Terminal. I have already installed it, but if you want to install it, this will take a couple of minutes. Now after that we need to just start our server which will start listening to incoming requests. I'm using the 5.3 model because it's going to be one of the fastest that can run on my local machine. But you can replace this with any other model that is available on Hugging Face. The goal is just to show you how to start the server and how to interact with different models. So when we start the server, you're going to see that we are running the Llama CPP server. I am running this on my Apple M2 Max, but you can run this on a Linux machine as well. Another thing which you want to make sure is that the blast is set to one. So that will mean that means that it is going to be using the GPU available on your system. Now, by default it uses the port 8080, so you want to make sure that no other process is using that specific port. If some other process is using it, you will have to kill that process and then restart this. So now the server is running and it's actively listening to incoming traffic. So before showing you how to make post requests in different possible ways, let me show you some other interesting things. And that is different options that are available. So if you notice, we were just using this Hugging Face repo and the file name option. But apart from that, there are a whole bunch of other things that you can do with this server. It really lets you customize your server. So for example, you can define the max context window of any specific model that you are working with. You can define a batch size if you want to batch your prompts right? You can also change the host address. So by default it's going to be using local host, but you can change it to whatever IP address you want to use. And also you can change the port on which you are serving the model. You can also change the path from which it's actually pulling the model, right? There is also an option to serve embedding models with Llama CPP server. So it can really do anything you want to basically run even a production server. It will be able to track some metrics as well for you, right? So if you decide to use this, I will highly recommend to look at these options and not just stick to the default options that I was showing you. So now let's look at how you will interact with the server. So the first one is going to be just using the curl command. This is basically you are making a request from your terminal. So again we will make a post request and will provide the URL that is the chart completion endpoint which is running on a localhost. Then you need to provide your header and you need to provide a data object that will contain the prompt. And we also are providing how many new tokens we want the model to generate. Now when you run this, this should pretty quickly generate a response because we are just running the Phi 3 model now as a response you will get another dictionary which has content. And here's the response that the model generated. But apart from that, it also gives you some extremely valuable information. So for example, you can look at different parameters that were used to generate this response. So for here is the temperature, then the top p and top k. So these are different hyper parameters that the model was using. But apart from that you can also look at the speed of generation as well. So for example, predicted per second. So how long it took, how long it took to process the prompt, and so on and so forth. So this was how to make post requests from your terminal. Now the same server is running and let me show you another way of making post requests that is going to be using the OpenAI client. Because the llama CPP server adopts the same standard as OpenAI client. So we create the OpenAI client base URL is going to be the URL of our local server. We don't need any API key. Then we will create a chat completion endpoint client. So this is the client that we created. You can provide any model name you want. I'll just keep it to ChatGPT 3.5. You can call it Phi3 if you want. Then in terms of the actual messages, since Phi 3 follows chat LM format for its prompts, so we can use a system prompt. And the system prompt is, you're a helpful assistant. You, your top priority is achieving user fulfillment. We are helping them with their requests. And then after that we're going to just provide a simple prompt that we want it to process. Right? So in order to run this, we're going to go back to our terminal and in this case I'm going to run this file which is LLAMA CPP OpenAI API. When we run it, you will actually see that it had a request, right? And here are the results that we get. So this is the response from the model that is being served here. Now the last way is Going to be just to use vanilla request package. Again, this is based on Python, but here we're going to do something different. We're going to make multiple concurrent requests to the API server and see how it behaves. Okay. In order to use the request package again, we're going to define the URL. Then you need to provide your headers exactly same as before. And we are going to pass multiple different messages. So they are going to be queued because we have a single process that is running on this server. But it will basically process them one at a time. Even though if you are running multiple requests, it's not going to hopefully overwhelm our server and break it down. But let's see how it works. Now to make the API call, we define a function. Here we have the model, the message. So system messages, you're a helpful assistant. And then it will get one of these user messages, right? And then it makes a post request to get a response. Okay, the make the main function that I'm using will just make calls to that specific or that single API endpoint. So basically we're going to be using the concurrent package and it will do multi threading, will make the call to that function simultaneously and let's see what type of results we get. So I'm going to run this file and you can see that it's just queuing those messages one at a time. And for each one of them we are getting responses. And you can also see that it's processing those requests on the server side. So this was a quick video on how to basically install LLAMA CPP and start a server that can listen to incoming traffic. If you are interested in deployment of LLMs for your own applications, I would highly recommend to subscribe to the channel. I'll be creating a lot more content on it. We're going to be covering a lot of different tools in this series, so make sure to subscribe to this channel and also hit that bell notification button so you don't miss any videos. One of the main applications that I'm personally interested is rag, which I think is one of the most practical use case right now. I have a course on it. It's going to be somewhere in the video description, so if you're interested, make sure to check that out as well. And there are a number of ways in which you can support my work if that's something you are interested in. Details are going to be in the video description. Let me know if there are any specific topics that you want me to cover. I'M going to be creating a lot more tech technical content which are more practical. So hope you are enjoying these contents. Let me know in the comment section below. Thanks for watching and as always, see you in the next one.