Install Ollama and run open-source LLMs like Mistral 7B and Llama 2 on your Mac or Linux machine. The tutorial shows how to check token throughput on an M2, switch between verbose and quiet modes, and serve a model through a local API that works with LangChain and LlamaIndex.
Today I'll show you one of the easiest way to run open source large language models locally. The tool that we are going to be using is called Ollama and currently it supports Mac OS and Linux. Windows support is coming soon. It supports a wide range of large language models including Llama 2, Llama 2 uncensored and a whole bunch of other including the newly released Mistral 7B. So I'll walk you through a step by step installation process and then we will chat with the model. Okay, so I'm currently running this on Mac os. For this you can simply come and click on this download link. As I said, it currently supports macOS, Linux and Windows support is coming soon, so we will click on Download for Mac os. Okay, so it downloaded the file, we're going to just double click on it. Now on Mac OS it's asking me it's downloaded from the Internet, do you still want to run it? So click say yes and then it's asking me to move this to Applications. So I'm just going to click yes. We will complete the rest of the installation process. So I'm going to click next. Now, in order to install this on the command line we'll simply click Install and then it's going to ask me for my password. So I'll provide the password. Now, in order to run the Ollama framework, we simply need to run this command. So I'm going to copy this and click Finish. Okay, so by default it provided me the Command to run llama2. So if you come down and look here, this is the command that we're using. So Ollama run and then the model name. But let's say if you want to use another model, for example, something like Mistral 7B. So you can come here and they have a pretty nice model card in here which provides you with different information regarding the model. But let's say if you want to run the Mistral 7B model, we're going to copy this. Okay, so here I opened another terminal. You can simply paste the command that we just copied and hit enter. So this will basically start downloading the model itself. And, and the model that we're using is around 4 gigabytes, so this might take a while. It seems to be a quantized version of the Mistral 7B model. Now you can go to their GitHub page and here they have provided information regarding the different models that are supported, as well as their model size. And they also gives you the rough RAM requirement in order to run different models. So for example, you will need around 8 gigabyte of VRAM to run 3 billion model and around 16 gigabytes of VRAM to run 7 billion model. Okay, so the model is downloaded and now we can start experimenting with the model itself. So let me ask it, what is your name? And let's see what it comes up with. So you can see the response time was pretty good. Okay, so we can play around with the model, but before that let's set the verbose mode. That will give us more information about, let's say, the number of tokens and some of the extra information. Okay, so let's ask another question, how to kill a Linux process. And let's see what it comes up with here. It's giving us a pretty detailed response right now on my computer we can look at the total number of tokens and how many tokens per second we are getting. So for example, if you look at here so prompt evolve rate, it went up to 82 tokens per second. That is, this is pretty great. And then the generation tokens are around 60 tokens per second. There's a pretty good speed on my M2. Okay, so if you type slash question mark it will give you different options that it has. So right now we looked at the verbose mode. You can enable the quiet mode, right? And also show like, let's say parameters, system, some templates that are in there, etc. Okay. Another great feature that this package has is that the model that you installed can be served through an API. For example, we're going to simply copy this. Okay, so we open yet another terminal and here we'll simply paste that. And now all we need to do is to change the model. So we currently have Mistral 7B installed. So I'm going to just change that to Mistral. Basically when we use this, the model is going to be served on this specific port number and it's waiting for the API call. Now if we run this, we should see a streaming response from the model. So basically you can check in here. Our prompt was why is the sky blue? And here's the streaming response from the API. So it says, now here is the response. The sky appears blue due to a phenomenon known as relay scattering. So you can see that this is basically a streaming response from the model. So that means you can serve your model through Ollama. Their platform offers quite a quite a few more features and I would recommend everybody to check out their blog. They also have quite a few integrations with other platforms. So for example, it has integration with LangChain, Llama Index, even Light LLM and a whole bunch of others. It's one of the easiest way to run open large language models on your local machine without a lot of setup. I hope you found this video useful. If you did, consider liking the video and subscribe to the channel. Thanks for watching and as always, see you in the next run.
