How to Run AI Offline for FREE in 2025

One of the most asked questions I get is, Igor, this AI stuff is fantastic, but how do I do this without sharing my data with some company? Well, in this video we'll be looking exactly at that. You can get to having this private chat interface on your computer in under five minutes. And it doesn't cost anything. You don't even need a credit card, and it all runs locally on your computer. So here's the deal. As you might know by now, there's a lot of LLMs out there that compete with the likes of ChatGPT, Claude as well, etc. But just because they exist, people don't intuitively know how to actually use them. One important thing to know is that all of these open source LLMs are free. And today I'll be showing you a software that can run them. And it's by company you might actually know, especially if you ever did something in a development space. And the app is called Docker. And with their model runner, you can use your computer to run these LLMs locally. I'll show you how to do that and then I'll show you how to set up a simple app that you could then keep on customizing all of it again at no cost. They reached out and told us about the software. And because this is one of the most requested coins questions that we get on the channel, how do I use LLMs locally? I'm excited to now show you how to do this really simply with the Docker model Runner. So first things first, you'll head over to docker.com and you'll download Docker Desktop. As you can see, it's available for all platforms. I concretely am on Apple Silicon here. And the fantastic thing versus some competing products is that when we run the models like this, they use the GPU inside of the Apple silicone max to accelerate the model. So all I'll do is go through the installation and once you log in with your Google account, voila, you can have Docker Desktop installed again. This is both on Windows and Mac, but on a Mac it will always run in the background here on top. And before we proceed, I want to just quickly explain to you what Docker usually does and what we're going to do now. And for that, I'm going to use my trusty little iPad here and I'm going to draw a laptop. Yeah, this is a laptop. Apologies for my drawing skills. So what usually happens if you type a request into your laptop is that that gets sent over to some server in a data center and Once it processes the result, it will send the answer back to you. Now, what we'll be doing with this app that you just downloaded is we'll be cutting out this entire server step because you will be running your model locally. So you type something in, your computer, processes it, and you get your answer right here. And you can do all of this while staying offline. Now, I want to add one more important note, because usually when you have your laptop and you run Docker on it, and some developers might know this, oh, my God, these laptops are getting worse every time I draw them. About a desktop computer. This time, that's better. Think about this as an imac. What Docker usually does is create a second computer inside of your computer. Okay? So as you can see here on screen, when I open up Docker, you can see it has all the controls of a computer. You can pause, you can restart, or you could close the app. And here at the bottom, you can see how much disk space, CPU and RAM it's using up right now. And that's what Docker usually does and is known for. In this case, we're not doing that, though. We're serving up this LLM on your local computer using your cpu, RAM, and especially the gpu. I thought highlighting this is important because the name could easily confuse you to think that you're running inside of Virtual machine. That's the computer inside of a computer. Not in this case, though. So now that we have that out of the way and this installed, we can actually proceed to making this work. So the way it works is very simple. You just need to go to the settings. As I click the settings icon, I can go to Features and Development. And here you need to enable Docker Model Runner and also enable the second check mark. Now, it might be already on by default for you. Just make sure this, this is on, because as soon as we have that, we can move over to the Docker hub, and here we can pull different large language models. If you want a list of all the available ones, you can head on over to this link in the description that lists all of the models here. In our case, you can see the Llama models, you can see some of the Microsoft models and more models that you might know, like deepseek. In our case, we're going to be using this one small LM2. It's just a real winner in terms of what you get for the size, and you can run this on a lot of machines. But what I'll just do in this case, I'll Just look for small LM2. Perfect, that's the one. And once you're here, you can simply click pull and it's going to pull, AKA download the latest version of this model onto your machine. In case that doesn't work for you, you can always go to the terminal and run this simple command Docker model pull. And then you have to say AI slash the model name in this case AI small LM2. And this way you can always pull these models simply by using the model name. I think the most interesting ones here right now, but this will change over time, are 54 for development and small LM2 and Gamma3 for general chat or productivity use cases. Okay, and as I talked it already pulled this little model on my machine. One last tip that you might want to know is that if some models don't work, it might not be the restriction of your computer not being big enough. It might be the fact that you didn't allow Docker to use enough of your computer. Computer that's easily fixable. You can go to the settings here and under Resources you will see that this memory limit is the main thing that constrains what model you can run. In my case you can see from 32 gigabytes. I gave it 28. If you've got the somewhere around 4 or 6 gigabytes, most of these models are not going to work. Okay. And we have the model locally. So now all we need to do is say run and you could already talk to it. Look at that, it gives you an answer. What happens if I turn off my WI fi? In this case I'll just go back to the model, start a new chat, say hi again and I get an instant answer. And look at that speed. It just appears. I don't need Internet, nothing. So that's pretty nice. Already this model just lives on your machine now and you can always run it in here. Now let's take this a step further and this is especially interesting to developers. But I also think that to non technical folks this is something to be aware of because this doesn't just allow you to run the model in here, it also allows you to provide that model to some other application on your computer. Or you could also do it through the web, essentially hosting the model on your computer and you can access it it just like you would a ChatGPT model through the API. Again, remember my terrible drawing with the servers and the computer? Well, in this case our computer is going to be serving as the server here, so we can use our own application and Use our own model. No need to send any data away from your computer. How do we do this? Well, it's quite simple. In this case, I'm going to demo it on a project that Docker provides also completely for free on their GitHub. And so hello, Genai Project. And. And if you work with GitHub, you could simply clone this to your machine. In this case, we're going to do the super low tech version of it. I'm just going to download this to my machine, unzip it, and here it is. All I need to do is the following. Here's a little trick on Mac that you might not know. This is a developer ish thing, but if you hit command shift and dot on a Mac it will reveal hidden files that you're not supposed to see. And there's this env file here that I can now open with a text edit. And this is the only edit I need to make. I need to tell it what model will I be using Again, we need the model name here. So here you can see AI llama 3.2 and then the details of it. In this case, remember we're using AI Smalllm2, so I'll simply change it to that AI Small LM2. I'll click Save, close this out and then I can press it again to hide those files. By the way, that's going to work everywhere in your computer, but usually you should be editing those files if you don't know what you're doing because things might break, they're hidden for a reason. And then what I'll do is I'll right click this, go to Services and open a new terminal, add a folder. Just trust me on this. This is the simplest way to do this. All I'll do is say dot, slash, run sh and as I open the terminal inside of this folder, it will just run this file called run. Nothing is set up to run this entire app for me. Bunch of magic happens in here. I'll give it access to this folder and down here you can see that the apps are already running. All I need to do is copy this URL, this HTTP URL, and in my browser if I open up a chat, it works. And if I click one of these presets, you can see I can talk to it. And there you go, instant email about penguins. And I'm still offline. All of this happened without the Internet. And because I have this model in here and by just simply running this one run file, it will launch different versions of this in different coding languages that you could then build out for yourself. And there you go. Docker is serving up the model and I'm using it in this app. And then if you want to develop a documentation, they have all the details on how to integrate this into any other application, just like you would a OpenAI API. Now, the only difference being that you actually run your own model. And as promised, all of this is completely free and even GPU accelerated if you're on a Mac. So there you go. That's a completely private AI assistant that costs nothing, and you can pick the model depending on the size of your machine. But yeah, you should probably have more than 8 gigabytes of RAM. I hope this was valuable or interesting to you, and with that being said, I sincerely hope you have a wonderful day.