Self-Host and Deploy Local LLAMA-3 with NIMs

Welcome to another video in our series on how to deploy large language models for inference. Today we are going to look at how to deploy a Llama family of model using Nvidia Nim on your own cloud or on your local machine. Nvidia Nim is a set of microservices developed by Nvidia that really accelerates the deployment of foundation models, which includes both language models as well as other AI models on any cloud or data center. In theory, it's very similar to something like VLLM or Llama cpp, which gives you an inference server that you can deploy in your own environment. However, it can give you much better performance because Nvidia is creating both the hardware as well as the software stack. Now, according to Nvidia, you can expect up to three times improvement when you're using a Nim versus we're not using a Nim. And a Nim is just a Docker container that you can run or on your local environment. First we will look at how to deploy a Nim, then we're going to look at what type of throughput you can get out of it. And Nvidia Nim follows the same standard as OpenAI API server. So I'm going to show you how to use it as a replacement to OpenAI. My friends at Nvidia were kind enough to give me access to an H100 and that's what we're going to be using in this video. But the same steps can be deployed in any cloud environment or on your local servers. As a part of this tutorial we're going to be deploying the llama3.8 billion instruct version with single at 100. You can also deploy the biggest sibling, but to make it much faster we'll just stick to the 8 billion version to access that GPU. I'm going to be using Nvidia Launchpad which is a free program that provides user short term access to large catalog of hands on labs. Now as a part of the Nvidia Launchpad you get access to a code IDE which is VS code. You can also look at all the metrics if you're running or renting a GPU from them and you can also SSH to that specific instance from your local machine. So here is a quick overview of how the metrics dashboard looks like. It's a grafana dashboard which will give you access to different parameters of your gpu. So you can see here how my GPU utilization has been over the last few hours. You can look at the CUDA version, the GPU drivers and all the other information that you will need in order to track how good the utilization has been. You can run all of this on your local machine or on your own cloud servers. I'll just walk you through how to set this up. Now, when you sign up for Nvidia and NIM Access, they also gives you a 90 day Nvidia AI Enterprise evaluation. So you can test it out for 90 days without paying them or without any commitment, which is a pretty good deal to test it out. So in this video I'll show you how to deploy llama3.8 billion instruct model. But the same process going to apply to other models that are available on the Nvidia catalog. So as a first step I'm going to open that IDE here. You can see that I have access to an H100. Now if you were to run this on your local machine, you will need to install a Docker engine as well as Nvidia Container toolkit. I will put links to both of them in the video description. So next we need to set up our key. Now for this you need to actually sign up for an account. I'll put a link to this. So here you can generate both your API key as well as your personal keys. These are going to be required if you are interacting with this remote server through ssh. So for your own local testing, this is the key that you want. And if you are going to deploy a Docker client or through NGSEQ cli, this is the personal key that you want to generate. I already set up my API key. This is going to be set as my environment variable. Next we are going to run a Docker Docker container that is going to deploy that model for us. For the deployment I'm going to just open another terminal. And now we just need to copy this command and paste it there. So first let me walk you to what exactly is happening in this command and, and then we're going to look at the deployment model. So first we're just creating a container name and the container name is metalama3.8 billion instruct. So that's going to be the container name that we want to use with our Docker container. Now in the Docker command we are just telling it to run this specific container, use all the GPUs that are available on the system. And for API key we want it to look at this specific environment variable that we already set. Now it's going to cache the model here at this location and it's going to be using the port 8. And for the model itself, we want to use a NIM that is hosted on Nvidia servers and that NIM is related to llama3.8 billion instructor model. Okay, so we're just going to paste that command in here and this is all you need in order to start an Nvidia nim, which is pretty amazing. This will most download the model if it's already not downloaded. Now, once the model is fully loaded, it's going to tell you how you can interact with the model. So here's a curl command that you can actually use in order to interact with the model. Okay, so I'm going to open another terminal here and we're going to just paste that curl command. Now there are a few things that I'm going to go over, but if we run this curl command, you're going to see that it started generating responses and the responses are looking weird because we enable, I think, streaming here. So that's why it's generating one token at a time. So you can see these tokens being generated and then being shown. And on the server side you can see that there was a request that it generated some tokens for. Okay, next let's look at the structure of the call that we're making. So we'll look at this curl command and later on I'll show you how to do exactly the same thing in Python as well. So we are making a post request. The IP address that we're using is the localhost. But if you were to serve this on, let's say aws, you will need to provide that external IP address and you will also need to enable port forwarding on that specific port. Now it's using OpenAI compatible API server, so you can essentially use the OpenAI client and just use this as base URL. I'll show you an example of how to do that. Next you will need to provide the model number or the model name. So right now we're using the llama3.8 billion model and the structure of messages is similar to what OpenAI expects. You can also set other parameters. So for example, I changed the max number of tokens to 1000. Right. You can also, I think provide something like temperature and stop tokens. So this is how it works. But now let's trust us this API endpoint and see how much traffic we can send here. Now, in order to really stress test the server, let's look at a simple program. So here I'm going to be using a request library to make a lot of requests to that specific API endpoint. So here's my URL. I'm running this on a localhost, but as I said, you can do port forwarding on a remote server and it will still work since we're using the request library. So we also need to provide a header and that's going to be application/JSON. The payload is the model name, the prompt. We are going to be asking the model to generate 50 jokes per prompt. We are also setting the other parameters, but I'm disabling streaming and then there's a simple function which will send all these requests using that URL + header and then dump the payload that we created. Now I'm going to be using multi threading to basically run multiple requests simultaneously and we'll see like what type of responses we get out of it. So here is the API endpoint or like the server that is running, we're going to just run this file. So this is called testserver py actually I need to use Python 3 here, so I'm going to use Python 3 and you will see that we are going to be sending a lot of requests. Now, if you just want to monitor the GPU usage, you can use the Nvidia SMI command, but I want to kind of monitor in real time. So you can use this command watch N and then define the number of seconds that you want it to be updated. So we're going to say updated every second. And you can see that it's right now using about 27 to 30% of the GPU resources. So which is pretty good. Okay. And here you can see the average throughput, which is about 2500 tokens per second for different batches, which is pretty impressive, I would say. Now, the number of input tokens are pretty small, so that throughput is not that high. But the average generation throughput is pretty good. And here we can also look at the utilization on the Grafana dashboard that we have now. I don't think this is real time, you will just need to tell it or like refresh it every time you want it. But you can see the utilization here as well. And here you can see that we are getting some responses from the model. There are a lot of simultaneous requests, so it's generating the text as well. So in this last example I'm going to show you how to use an OpenAI compatible API server with Nvidia name. So we'll need to use the OpenAI API or, sorry, the OpenAI API client and we're using the Python client. Now you need to change the base URL. So we're going to provide the URL of our Nvidia name. So we are running it at Port 8000 and we don't need to provide the completion endpoint because OpenAI internally can use that. Now we don't need to provide the API key either. Next in the model we will say that use the meta llama3.8 billion instruct model and we want it to generate up to 1024 tokens. So with all that settings we can now use the OpenAI API client. So we are going to be using the chat completion endpoint. The model is going to be set to this model. Then the message is tell me 50 different chokes. We set the temperature, we set the max number of tokens and I want to stream the responses. So I wanted to show you how the speed will potentially look like if you are streaming the responses. Now, if you stream the responses, you will get chunks and you just print those chunks individually. So let me show you how the performance looks like. So we're going to go here again and this time we are going to run a new file that I'm calling NIM OpenAI PY So here is a streaming response. This is pretty impressive. Really fast. Now it's running the smaller 8 billion model, but still the response is pretty fast. You can also look at the number of throughput tokens that it's generating. So on average throughput right now is, depending on which state we were in, it went up to 76 tokens per second. So it's definitely an option to consider if you're thinking about deploying LLMs in a production environment. Nvidia NIMS make it a lot easier because it's just a Docker container and it's well optimized for Nvidia hardware, so I would recommend to check it out. You get 90 days free testing with the name that you can run on your environment. So it's a pretty good option to explore. And it's also actually great to see that Nvidia is also adopting the same standard as OpenAI. I think Google is now doing the same thing. I'm not sure about Anthropic, but it's good to see that a lot of these model providers are using the same API standard. So it becomes very easy when you want to switch between models. Anyways, I hope you find this video useful. If you are interested in learning more about deployment options, I'm going to be creating a lot more content around it. Probably we're going to look at Vllm next, so if you are interested in that, make sure to subscribe to the channel and if you are working on your own projects and need advising services I do provide. Those details are going to be in the video description. And if you want to learn about Rack, I recently released my own course. The link is going to be in the video description. Anyways, thanks for watching and as always, see you in the next one.