Run AI Locally on Your Computer With These Tools!

Over the course of the last year or so, there's been a divide between the world of open source AI models and those that are more proprietary, especially because large language models are just so big, right? They've got, you know, dozens or hundreds of billions of parameters. And that can require a whole lot of memory in order to run. And the question is, can you run that on mere mortal or consumer-grade machines? The first generation of the stuff would be, you know, things like Anthropic or OpenAI, those were all running, you know, on specialized servers that take advantage of huge infrastructure and run on a classic cloud surface basis. But these many open source models have been coming out, you know, led by our friends over at Facebook, who made the LLaMA model. And then coming out of that, you've got Falcon and Mistral and all sorts of other interesting models. How can we take advantage of them in ways that are different? From the strong points of the production of the, of the proprietary models. And one key thing you can do with open source software is you can download it and run it on whatever machine you see fit. Until relatively recently, this was a big problem because running it on like mere mortal machines like my MacBook here would be quite challenging. But there's this new motion, new movement afoot to try to standardize ways that we can make those models run on local devices much more smoothly. And that has allowed allowed us to have things like LM Studio that I've also made a video about, that we can interact with models now that we download to our own machines. And this works in Windows, this works on Macs. It requires, I mean, not the super old hardware, but, you know, your basic workstation type hardware. If you're running a gaming PC, or even not a gaming PC, like a MacBook, which is not good for gaming, you know, you can make use of the silicon on it in order to interpret. Write the model, and be able to run it locally. Now, the reason it's able to do that is because we have a packaging solution, where we can take these models that were originally written in PyTorch, or using tensors, or what have you, let's say tensors, et cetera, and turn them into a standard format that various client-side software can interpret, and also organize it so it can be run through more consumer-grade hardware. And this is where something called GGML really comes into play. This is a project that started, I think, like, you know, 7 months ago or so, where he created— where this gentleman created both Llama CPP and Whisper CPP, basically, I think, originally working with Whisper, and then creating the Llama model that would basically compress so that you could run the model and fit within the memory limitations of a more, you know, consumer-grade machine like my MacBook. And this becomes a standard, right? So like you have this llama.cpp, which is a way and toolchain for taking a model that you could express through, you know, PyTorch or what have you, and turn it into a new binary standard that is called GGUF, which is their universal format that then packages up their markup language for being able to describe a model in a standardized way that can then run on the consumer hardware. This can run on like Raspberry Pi, like he talks about here. But the other really cool thing about it is it means that if you download a GGUF file, right, a file in that format, it can be run by multiple applications that can all use it locally. And that is the idea behind, say, LM Studio, where you can just, you know, go, it will, it will search Hugging Face for you. And it will find models that are interesting that have been converted to this GGUF format. And then, probably LM Studio is one of the friendliest to use, but it's definitely not the only one. There's one called GPT4All that I think came out before LM Studio, originally supported a separate, and I think kind of proprietary,.it binary format, but now is really standardized on GGUF the same way that LM Studio is. One of the key differences is that GPT4All is also supporting basically a RAG structure where you can be downloading, downloading an embedding model and be able to load documents into it and be able to ask questions of documents using the structure. In general, GPT-4 All is a little bit older than LM Studio, but has this additional feature set that is definitely something to pay attention to. And it's much leaner in terms of its UI. So it's not containing a whole lot of cruft that you might not need. There's also Ollama that I think I've also done a video about before, which is more of a command line interface that is But the really cool thing about it isn't the command line interface itself. It's the fact that it will host itself. And I'm not sure, I think I'm just showing my browser window here, but it's also in my Mac's toolbar because it is creating a little local web server that can then be communicated with by other applications you might have running, right? Now it becomes a service that can be connecting you, connecting other applications with your, with AI models. An example of that actually is ChatD, which is a neat little program that allows you to— it will make use of the Llama 2 model, and you can upload your file into ChatD. It will shred it using fairly naive approaches, and you can just start to ask questions about it, about your documents. It works okay. These things all have room for improvement, but what's really cool about it is that you can start to open it up because this thing is open source, right? The models themselves are open source, and also the applications for building them are open source. And that means we can start digging into them to figure out what are good trends, where can we make improvements, and how can we just do much better with these things. And, and really the key here is packaging. How can we distribute these models that will make it easier for people to download and get value from them, right? So LM Studio is creating this nice little GUI to be able to go select and shop for your model and go try it out, download it, as long as you have enough disk storage, because these things are multiple gigabytes each. GPT-4 All also has a little bit of a shopper, not quite as extensive. The Ollama has support for being able to both pick from its model selection as well as being able to define what they call a model file, which allows you to use any arbitrary giga file, although that requires a little bit more work. And then there's an initiative coming from Mozilla, which is called Llamafile. And their angle on it is they're going to take the The packaging parts allow you to run a model from LLAMA CPP, where I basically take one of these GGOFF models and just jam it right into the executable. And then that way, it becomes easy to distribute one of these models, right? So instead of having, I need to go get a program, and I need to go get a GGOFF file from somewhere, you can just download the one single executable and be able to run that either on a local server or just running that from the command line. I think this is also really neat and just shows how much variety there is out there for being able to try different ways of distributing these interesting models and combining them with applications that allow us to get more value out of them. Now, I don't know where all this is going to go, right? I think there's definitely Llama— LM Studio looks really gorgeous. Ollama looks super composable into other applications. And sort of shows, and ChatD is sort of a good example of that for us to be looking at. The Llama File Initiative is interesting in terms of making an even simpler distribution that instead of having a central server on your application, on your desktop, you just have the individual files you're downloading and just running them each as applications. I don't know which way this is going to go, but the fact that the world is getting really wilder and woolier in this world of local AI leaves me really, really excited for what's going to come next.