Run A Private AI model In Your Computer And On The Command Line with @Ollama

Let's keep going down the rabbit hole of models that you can run locally on your computer. We often think of AI as being something that has to happen out there, right? It takes a whole lot of GPUs. It takes a whole lot of processing. It takes a whole lot of money. We think of these super well-funded companies like, you know, OpenAI or its competitor Anthropic. And the, or Cohere or AI21 or all these others, right? They're raising all this crazy amounts of money. But there's an alternative to this, which is the universe of open source models that are available to us today. And in a previous video, I talked about how we could, you know, access those relatively easy in something called LM Studio. I want to go down that rabbit hole just a little bit more here and talk about Ollama, which is a lower level way of accessing local models using the same fundamental GGUF technology and talk a little bit how you can bring your own custom models to bear in there. Now, the idea behind Ollama is that they are focused just on how to connect your local model with something you can run. And it uses a standard called GGUF, which basically means that they have a library of models they know have been translated into Giga format. That you can then run locally on your computer. So we have, you know, access to Llama 2, access to Mistral, and then like, you know, lots and lots and lots of others. And as we'll see in a couple minutes, you can actually tell Ollama to use any of these open source models once you download them to your local machine. The interface is not as friendly as it is in LM Studio, because it's all really working off of a command line. So let me bring up this command line over here. And we can do, you know, Ollama run llama2, you know. Well, actually, what I can do is just do an ollama list to show which I've actually downloaded. And, you know, ollama, always use --help to find out how a given program wants to be run. This is all available to me, by the way, because the very first thing you can do with Ollama is just right from their homepage, you can just download it. Directly to your computer and install it. And one of the very cool things is once you've installed it, it actually installs a little bit of a server into memory. And that's the reason why you have this Llama up here in your address bar. But right now it's only for macOS. They say they're coming out with Windows relatively soon. So if you're Windows, you want to be looking at LM Studio, which is Mac and Windows. If you are macOS, you can use either of these. And it seems like Ollama also supports Linux on a first-class class basis, whereas that's more beta over on LM Studio. I haven't actually experimented with either of these things on Linux just yet, although I think that gets to be a pretty cool way this thing can proceed. So let's talk a little bit more about what we find here in the terminal. And the, and one of the most straightforward things we can do is just pull one of these models from the registry. And to do that, you just need to know what model you're pulling. And that's where these names come in, or you can reference a local one. We'll talk about that in just a minute, like how you can create your own. And And then we can say, well, let's, you know, Ollama run, Ollama tube. And it will kind of preload the thing into memory so that we can start asking it questions. Like we can ask questions like, you know, make a good description for a YouTube video discussing local large language models, and it will just, you know, calculate and it will run it. This is all happening entirely on my computer. There's nothing that's going up to the cloud, nothing that's going over to OpenAI or to anyone else. And here it's making me a whole bunch of description, which might be, you know, more than we really care for, but it can be relatively useful. Now the question is like, are these model outputs going to be relevant to you? And that's going to be a question of, you know, the usual thing, figuring out which model you want to use. And now, of course, instead of just using OpenAI, you can use any of these. And then the, and then you can say bye and be done. And you can use OpenAI. So instead of having to go send things over to OpenAI, it's all happening locally. You get to choose one of these models or bring your own to the party. And really the only price that you pay for it is going to be speed. That these models are quite large., and this formatting allows them to run on CPUs instead of GPUs, but the things are generally going to run more slowly than they do when they're just running off in the, running off in the cloud. But that's probably fine. There's a lot that we can do with that. Now, I got pretty excited about this technology, and I got excited about it in sort of a couple different directions. The first is the ability to bring whatever model you want to bear. Now, the shopping list and search functionality sitting over in LM Studio is pretty great. It's a bit more limited over here. They just have their known registry of models, but you can also, if we were to click into their GitHub, it's their GitHub page, you can scroll down and you can say you can import your own model. Now, there are a couple ways of doing this. You can import it directly from PyTorch, but the best way to do this, I think, is to have converted the thing into Giga first, and that can be a whole different conversation. But if you have a Giga model that you might have downloaded from Hugging Face, or maybe you copied over from like the cache directory in Elm Studio or whatever, you can say you want to use it in Ollama. And you do that by creating what they call a model file. Now the way you do this is like if I just say ls, you can see here I've got this, you know, this model I previously created and fine-tuned. I use as an example in a previous video as well. And I can, you know, code model file, right, which will then create something called model file. Right. And I can take their instructions here, which I'm just going to move over this to the side and move this over to here. Great. And I'm going to say, well, let's just add from dot slash whatever, right? Now that means it'll only work for this particular directory, but that's okay. Let's copy the path. And we'll say copy that path, right? So from here, it will use this particular model file. Great. I'm going to say save that. And now we can go back over to my terminal. And I can follow again, follow the instructions here and say, oh, llama create yt-sample -f model file, right? And we'll build out, it will basically tell Ollama to take its cues from this particular Giga file. It will simply call it, it'll call it yt-sample. Cool. Cut out a few seconds there, but basically this thing took about 20 seconds to complete. And it creates a layer and writes a layer. And when you see things writing layers, you start to wonder, this server is using some Docker code behind the scenes, but leaving that issue aside, I can do now ollama list. And it will show me not only the Llama 2 that I previously downloaded, but also my YT sample. And now I can do, uh, ollama run yt-sample. Sample. There we go. Keep in mind, you don't get autocomplete for, um, these model names. And I can ask it, why, oh, why is this sky so blue? And we'll see whether it talks about Rayleigh scattering. Yep, right. See, and now it talks about Rayleigh scattering. So this allows me from the command line to have all the chats I'd like to have. So now we can say, let's see, /? to get the list of commands and show to see information about this particular model. Show more specific information about model file, which is just the fact that it came from the from, right? And this is where it downloaded from. And then what the template is, which is not very exciting. Okay, so you can go learn more, explore more with LLAMA all on the command line. Now there's one more thing I want to show off though. Which is as part of this experimentation, I made a little utility that if you have installed Node.js, you can have access to as well. And that is, you know, something I'm calling npx ollama-cli, right? And if you have Node.js installed, you should have access to npx, and you can do npx ollama-cli because npmjs.com, package olama-cli, which actually I created just, just over the course of this weekend, which also demonstrates how you can just implement a little bit of code to make this thing even more useful from a command line point of view. One of the things you probably noticed is that I'm able to do a chat from the command line. But chats from the command line are perhaps less friendly than like what we can do over at LM Studio. So like, how can I make Ollama more useful even than than LM Studio. And that is where things get pretty cool because there is a library that sits on top of Ollama that was made by one of the Ollama maintainers. It's called Ollama Node, which I can go click on here. Ollama Node. And the cool thing about Ollama Node is it provides a library to communicate with Ollama if you've installed it on your machine, i.e., if the little icon is up there in your, in your bar. Because then I can use its API. And I added this to make a little CLI for it. That way, instead of having to say run, I can just say, you know, npx ollama-cli, why is the sky blue. And it actually has a couple other features that are pretty handy. But the first thing, the most important is I can just run something on the command line, and it will just deliver this back to standard output. So from the point of view of creating like a shell script or an automation, this just works, right? As long as Ollama's been installed on the machine, and I'm not gonna say that it's gonna win LAN speed records, I mean, it is running on your local machine. But it is, it's pretty cool that it's able to produce this. And that just went to standard output, which means I can just feed it to the next element of my pipeline, I can put into a file using like redirect or whatever else I want, right? From a shell scripting point of view, you know, sort of lightweight automation, it becomes pretty helpful. But the other thing I can do is, and by the way, if you want the full output, like, you know, finding out, you know, how long it took and what the temperature was, etc., you can, you can get all that through just asking for the JSON output. But there's one more thing that it does, which is most of these models think in terms of Markdown, but they can do things like code generation. So I could do npx, you know, ollama-cli, you know, take the numbers 1, 2, 3 and format them as a JSON, uh, you know, JSON array. Right now ordinarily if I do this, uh, it will do a little throat clearing, explain a bit of the code, but it will create it in the form of, you know, Markdown and this little Markdown indication of like where the code starts and stops. Well, since we know that's going to be Markdown, we know that's where the code's going to start and stop, what we can do is, you know, introduce just a little bit of logic which is which is exactly what I've done in here, to say, add code to it. And instead, it will just output the JSON. And then I could pipe that something like jq, you know, to go find out the first element in there or something. And it will tell me what the, you know, first item is. By using, you know, querying, you know, JSON or whatever it might be, you can feed that into like other inputs or send that off as a payload to web request, or whatever it is you're going to find to be most helpful. But the idea is that in addition, you can spit back the JSON this way, you can spit back, you know, code that gets created, you can turn around and pipe that, say, okay, please run that in Python, or Drought, or Node, or whatever. But what this starts to do is sort of expand our universe of what we can do with this technology. Beyond just making chatbots, we can create things that really automate for us. And rather than saying, hey, we're reaching out into the cloud and allowing the cloud into our machines, we're bringing this piece of code locally, and being able to run it without requiring any network connection. If I were to turn off my Wi-Fi entirely, this thing continues to work and continues to give me pretty good insights, which I find just really, really remarkable. This is all made possible by the efforts of this engineer and the team that he's been putting together and the, and what was originally the GPT-4-All initiative, and now is the— and then for a while there was GGML, and now there's GGUF, which I think is the GG Universal Format, which is the main way that we are formatting these models now, all made possible through a couple projects that this group has put together, one of which is called llama.cpp. And the idea is that it is a set of code that is designed to efficiently run these models using ordinary CPUs. And so what we were doing is we were formatting the models that previously were designed for Hugging Face, PyTorch, and like, you know, designed with Hugging Face Transformers in mind that we're going to take advantage of big heavy GPUs for the purpose of doing all the training, and turns it into a format that is easily portable, just downloadable, just like we did here, that responds to some relatively standard types of inputs. And And, you know, and then we can run it locally on these much lower powered machines. Right now it runs pretty well on both Intel machines as well as Apple Silicon. And I'm really looking forward to what can happen next with it, because the ability to take these models and run them in a way that is under your control and making about your model is really where I think the future is of all this. Okay. Hopefully this has given you a little bit of an intro to Ollama. Hopefully this has given you the idea that you can be bringing what whatever models and whatever use cases you think are going to be most helpful to you, and maybe some ideas of what you can do to extend it just a little bit further, making use of not just Ollama, but maybe the Ollama CLI or other tooling that you might bring to bear to make your own extraordinary applications that are leveraging this very cool AI technology. Certainly AI is usually part of the toughest 5%, and that's the kind of thing that we're always working on over at StateChange for AI, no-code, low-code projects. And for automations. And, you know, if you've got questions about what we've been seeing here, please put them into the comments. We'd love to keep the conversation going. And if you've got particularly challenging things for your, you know, economically meaningful projects, please check us out over at StateChange. I'll see you next