Run AI Models Privately On Your Computer

The topic that's very hot nowadays is, of course, AI and large language models. Coming from a data science background myself, I get pretty excited when this topic becomes, you know, more part of the consciousness again. It's one of the reasons why I call StateChange, StateChange.ai. And I think we've seen a lot of press about OpenAI and the services they're offering and the huge models that they have built. And we've also seen there's some competition competition now, right? There's competition in the form of Anthropic, there's competition from Cohere. There's a little company called AI21 based in Israel that is making some pretty cool models. And of course, there are going to be more and more and more. But one of the key changes that we've seen has been the rise of the open source model sort of led by Facebook, which, you know, maybe seeing these other companies creating models they're going to sell, decided that they would kind of go a slightly different way and just open source the large language models they had. And one of the most famous of those is actually called Llama. Llama. And so you're going to find that the term Llama shows up. I think it's called Large Language Analytics by Meta, right? Because Meta is the new corporate name of Facebook. And so Llama as a figure shows up a lot in these systems. So I want to bring your attention to LM Studio, which you call Large Language Studio or Llama Model Studio or whatever. But the core idea is this, that while, you know, if you're using OpenAI, you are sending your sending it back. The whole idea behind open source software is you don't have to send it to somebody else's server. It's just software and software that you can download and that you can run in a place of your choosing. This allows a lot more competition when it comes to hosting. And one of the best places to compete might be your own machine itself. One of the amazing things is that we all sit in front of these machines that, you know, whatever we might think about them, but they are supercomputers compared to what came even just a few years ago. And the, and the, the, these local tools can take advantage of the fact that you have all of this compute power available to you. Compute that's being delivered to you that's being used for nothing more than, you know, making nice animations go on your screen. We can use them for practical things like making models run. LM Studio is a nice GUI. You can download it both for Mac and for Windows. They apparently have some sort of beta for Linux. I haven't experimented with that. You download it, you put it into your applications directory, and you just have access to it. It is really remarkable because the first thing you do is it shows you like all these models that you can just download. You click a button, in here and you download the model. Now, keep in mind, this is downloading several gigabytes of stuff. So it's going to be quite, you know, large. But the— but it starts to teach us about how these models work and gets to think about it in much more comparative ways that I find to be just super, super helpful. So, you know, they certainly have their featured models they put out on top here. And those are actually pretty good. We'll talk about Code Llama just in a minute. But the— you can search for models that you might know their names of or based on the description. And they also have the search functionality in here where you can be looking for models based on, you know, like I use keyword code here, and there are all sorts of code models, most of which are about Code Llama. There's one by Replit that is out there that's pretty cool. And the key thing about these models that allows them to run locally is going to be something called quantization. So you can see over here that many of them, in fact, all of them are going to be quantized. And the idea behind quantization is that Each of these models are based on not just like a bunch of Python or JavaScript or whatever code, but actually a whole bunch of numbers. You can think of it as like a giant pachinko machine, right? That you're just dropping your ball or token or idea in, and it's sort of bouncing all around until it finally comes out with whatever the new version of that token is going to be. And the question of how many pins and like little bouncy points there are in that pachinko machine is going to be the number of parameters that there are in that model. And those parameters are measured in these large language models in the billions. Now, where quantization comes in is to say that each of these little pins that causes the things to move around is what we call the weights of these models, is going to be, you know, it's just a number. It's a number usually between 0 and 1. And there are mathematical reasons for that, but basically you think of them all as being like 0 point something. It's going to be positive, it's going to be between 0 and 1. And the idea is ordinarily, like a normal computer without any further thinking about it, this can store that number using maybe 32 bits of information, using what's called precision and the exponent. And that's, that's basically how floating point numbers work, and that would be the way that you would naively store a number. But in quantization, there's an insight: hey, we're just going between 0 and 1. If we're only going between 0 and 1, what is the difference between being at, say, I don't know, 1/16 versus 1/32? Aren't both of those pretty close to 0? Is there a meaningful difference between those two things? And the answer is there's a difference, but it's actually not a lot. So there's an opportunity opportunity then to instead of storing it using 32 bits of information— by the way, 32 bits of information would allow us to get about 4 billion different possibilities, and usually about half that amount is stuff we can use for the numbers between 0 and 1, so call it, you know, a billion or 2 billion possible values between 0 and 1— and say that we're just going to use, say, 4 bits. 4 bits, which would be based on 2 to the 4th, which is 16. That's 16 different values. And that would say that I don't know the difference between, say, 0 and 1/16, I'm just going to go straight from one to the other, moving up by a quantum, which is— in quantum, it like just means small amount, but it just means it's a stepwise progression. And that means instead of having all these tiny, tiny steps, such as a billion steps going up, I'm only going to have these 16. This is important because then I can store a number just using those 4 bits. And that means instead of using 32 bits, I can use 4 bits. That means I can think about 8 times as many numbers, right? Because 4 bits versus 32 bits are 8 fours and 32, all at the same time. And that quantization is something that these folks take advantage of. They also take advantage of a second thing that I think is really important called GGUF. And GGUF is a format for describing one of these large language models in a way that it can be run on ordinary CPU silicon. And this I find just super exciting because it allows us to package up all sorts of new models, not just the ones that already exist, but new ones that, and we can combine these two ideas of first quantization and then using this format so that we can actually run it on ordinary computers. And that is the insight that the LM Studio and a couple of the others I'm about to talk about take advantage of. And so that means we can use all these wonderful open source models. We can sort of somewhat shrink them in a way, even the really huge ones, shrink them in a way that they can run on our local machines and we can take advantage of them completely privately, very much inexpensively, because we're not paying for hosting costs, and in all sorts of creative new ways. So, you know, when we look up code, I can say, well, let's go find Code Llama. Let's go find, let's go find Replit. Let's go find the Replit Code Instruct model. Great. And here's Replit Code Instruct model. And it's, you know, ordinarily would have 3 billion parameters, but you can see the file is only 1.46 gigs. That's because it has a Q4 on it. You're going to find that Qs usually come in 4, 8, and 16. 4 is 4 bits, 8 is 8 bits, which is 256 gradations, and 16 is, well, 16 bits, which is going to be more like, I think that's 65,000 gradations. And so basically sort of, you know, more stair steps in the middle, and in theory, more precision. So the trade-off is going to be having more quantization, which allows you to shrink it down, versus having more of those little pachinko pins I was talking about, in order for the model to be more thoughtful about giving you back answers. So the more it's quantized and the fewer of those parameters it has, the faster it's going to be. But the more of those parameters it has, the usually the quote unquote smarter it's going to be. And you're going to find there's sort of the give and take to all of that. Now we can, you know, take a Reput model, we can download, I can just click download here and it will immediately download it to my machine. Of course, it's going to be like 1.5 gigs and depending on what your internet speed is, it could take a little bit of a while. And you can examine which of these you're going to use. Note that some of these are bins and some of these are GGUF models. And I actually recommend you use ones that say GGUF instead of the ones that say bin, because the GGUF models are the newer ones. It's the newer standard that is supported as opposed to the bin standard, which is based on GPT4All, which is another interesting model, but kind of deprecated in the face of this new GGUF. So I look for the GGUF. I look, I actually look for a high degree of quantization because I would rather have more parameters and have those parameters to be more quantized, I tend to find that works better, but you should really, you know, download more of them and experiment and see what you like. You'll notice that, you know, sort of get some downloads all going here because all these are relatively large files. So that's how we go shopping, right? That's a shopper's mindset when it comes to models. And we can look for more. I can look for Falcon, right? And go find a whole lot of Falcon models. Note there's a Falcon 180 billion model, which is really quite extraordinary. Note over here that also requires a whole lot of RAM. So, you know, the models, the way they work is they have to load themselves into memory. So a model that requires this much RAM is going to be probably a non-starter for the vast majority of us that have relatively mortal computers. So, you know, just because we do the quantization doesn't mean we can deal with everything at sort of that super high level. And you'll find down here, similarly, it requires a whole lot of RAM in order to make it work. So that's going to be one dimension that drives, that might limit which ones you can really use. My own MacBook, I think, has 16. Yeah, you can see mine has 16 gigabytes of memory, sort of a baseline MacBook Pro, you know, 16-inch from last year. And it is, As such, you know, the ones that have these high requirements, I won't really be able to use. Another way, and so having too many parameters can be something that sort of limits you. You don't want to necessarily use those that you can't then use, that you can't then run. You're going to find there are different types of quantization here. You can click to see what the quality loss is. You have considerable quality loss. That might be a lot. And then, you know, the 8s, usually there's not a whole lot of quantization going on. So I find that like you want to look for 6 or 4. That's going to be usually where the sweet spot is for the majority of these. In fact, that's, you can see at 5 is exactly where they make their recommendations. So we can download this one too. The YI model is actually really cool because it has a very large, what's called context window, which means you can shove a whole lot of information in it when you are asking it questions. And it will consider that as it churns through. This is kind of like how you might have read that like OpenAI now has support for, I think, 120,000 tokens of its history. And that allows you to kind of do some sort of, you know, ERSAT's RAG, which is Retrieval Augmented Generation, which is essentially you can pour a whole lot of reference material into your question and then get that back from your— get that back for your model. Quick note, you're downloading very large files here.. And those large files may eventually cause you to run out of disk space. So pay attention that you're, if you're downloading a whole bunch of them, that you might need to clean it out or need to make other room. I've got a terabyte on my, on my drive. And I do find myself occasionally running out because I'm getting a little too happy downloading these things. Let's talk about how we'd use it. So its main interface looks a lot like a ChatGPT. And, you know, you have your history of chats, whatever. But the most fun part is the fact that I can switch up what model I'm going to look at. So let's first run NewChat. And then let's pick one of these models here. And you can see it hasn't really finished downloading this one.

And I don't know, has it? Was it?

Was it still downloading? It's still downloading the Replit one. And but it did download the CodeInstruct bin one, which is the older. And the, you know, the Code Llama quantization, it has downloaded. And you can see this just sort of, you know, taking a little while here. So let's talk about, you know, using Code Llama, you know, instruct here for a second. Let's have it, you know, write me some code.

Write me code that— write me, I don't know, Python code that will—

well, let's use today's Advent of Code. Advent of Code was asking for, you know, that will manage a light beam going north, south, east, or west hitting a mirror at a 45-degree angle.

The result should be a new direction for the beam, one of north north, south, east, or west.

And just like ChatGPT, this thing will think and it will generate some code for me. It also will generate occasionally the wrong code for you. You can see how this is reflecting light. It says it goes from north to south. It says it goes from east to west.

And the, um, uh, that code reflects the light, but I just want it to redirect by 90 degrees. Light from the north should go to the east, and from the east and from the west go south.

Right, you can have the same kind of interactive experience where it builds in that history because it's sort of aware of how chat works. It's not the smartest, but it does get— it prints east, that's fine, west, south. I need to probably just be more explicit.

Please add to the code for what happens when the light is, uh, the input is east or south.

Ah, see, I did say it was at a 90-degree angle, so this is really quite silly.

I now make the code assume the mirror is at a 45-degree angle.

Go.

All right, east, I guess. Good.

West, south, east, north, southwest. Excellent. All right, well, that's actually what I need for my little Advent of Code project. So I'm going to be very, very happy with that. And I can of course just go copy the code or what have you. Now, a couple things to note about all this. The first is to know there's little presets over here. Presets are basically about metadata that's being applied to how you work with the model. The model underneath is going to be a large language model, which means it's always just doing text completion, right? But the model is, you know, it usually has a certain amount of training to know how to answer questions if they're given in a certain format. So I can do code completion, which just uses baseline. Okay, just complete the text and start and then go based on that. And then instruct, which is the usual, what we think of as a chat model, that sort of question and answer type thing. And then you can find there's some more, you know, chat and what have you. And there's, I think, sort of emerging support now for multimodal vision models. I have not yet been able to get that to work. But I'm very excited about what's coming down the road with that. Okay, so this is sort of a great way to start experiencing how these models work. You can use a chat interface, you can type basic text completion, you can download models to your heart's content. The slightly more advanced use case is you can tell it to run as a local inference server. And the— and make other code, like you could use local Postman, or you could use local code for that matter, to communicate with your server for the purpose of, you know, talking with, you know, having these complicated chats with your systems without actually running up any cloud-based bills. I find this part to be super, super cool. I'm pretty sure that like the little bit of support they have right now for being multimodal is largely expressed through the, you know, through this little server bit, right? IE is more of a high-code thing. But I think that's all, you know, stuff to check out. Also notice that they use OpenAI library, but they are using your local server for doing it. So just because you're using the OpenAI library does not mean you're actually running up an OpenAI bill, you're only using things that are completely local. Okay, cool. One more thing you can do with this, I think is really awesome. Like, obviously, there's a whole lot more depth we could go into with all this. But I guess two things I want to check in on. First is that over on the right-hand side, you can get information about the metadata for the model, usually if you download it from Hugging Face or something. And that can tell you things like what the context length is going to be, which I find to be super, super exciting. Because that then tells you how much history or how much information you can go stuff in there to make the model smarter. Because remember, these are models, they're really good at reasoning. They don't necessarily have all the information in the world. They only have that which these little, little nodes, all these little numbers can actually store. And while we might think billions is a lot, there are billions of pages on the internet, and one of these numbers can't store like a whole page, right? So the being able to give it more information for context can matter. That's the reason why we like to use sort of that interactive assistant or chat-oriented view when we're doing this as humans. And that's and its ability to remember that and work with that is represented in the context length. That's actually one of the reasons I'm quite excited about that Yi model I was alluding to before, because that Yi model is going to have a whole lot more context than others will here. Okay, cool. So let's talk about the next thing that's really cool about this, which is the ability to bring in your own custom models. Now, you can download models from Hugging Face until the cows come home if you are so inclined. But the— but what I find also cool about this is that if you have fine-tuned your own model, you can, you can convert it into this format. And if you do, you can put it into this directory. That is where your other models live. So I can do reveal it in vendor and see my models, right? And you can see all the different vendors for those models and how, you know, for some of them, I've got 3 models or whatever else, right? But so this is sort of structured the same way like your GitHub repository might be, right? It's gonna be the organization followed by what's the actual model or what's the actual, you know, card on Hugging Face. So one thing I can do here is I can say new folder, I can say state change, and I can say, you know, and I have a fine-tuned Mistral model, and I can say, well, let's just make a new, new folder and we'll call it Mistral. Now if I do that, I can now access, you know, from my downloads directory here, I have this, you know, other, other, other model that that we had previously created, and I can put it right into here. Let me just sort of stick it into custom Mistral. Great. And now what I can do is say up in my models, I now have state change custom Mistral and ggml model. Sometimes what I have to do is actually close up LM Studio and start up again in order for it to know about my custom models.

And here it is.

Yeah, see, so I've got my custom Mistral model for state change. It even has a good idea whether it's going to work or not. I'm just going to tell it this is probably just your basic, you know, alpaca model. And I will go back over and say, let's use state change model.

And we'll see how it does. Is the sky blue?

And it will come back with some answers for me. Oh, notice up here how much memory it's using. That's actually kind of a useful thing to keep in mind. The memory sort of tells you how intensive the model is. You should expect that RAM usage to sort of shove up towards the number of parameters. And that the CPU usage can go over 100%. That's going to be because it's using multiple cores that are associated with your model. And, you know, here it told me exactly why, because of Rayleigh scattering. So this is all pretty cool. You get to go shop around with these models. I encourage you to do so. Hopefully this is giving you some idea of how we can be using LM Studio to start to play with our models. There is deeper that we can go in this topic. This GGUF thing really opens up some possibilities in terms of how we can customize models, how we can deliver them wherever we want to, and how we can use them locally. I'll see if I can follow up on that, but right now I hope that you will take the license to go discover LM Studio and some of the ecosystems sort of around this idea of being able to run models locally and, you know, feel empowered to be able to just download, go, and try them out. If you have occasion to, you know, get this and start to, you know, shop for models, you find something that works particularly cool, or you discover something that you think is a little bit weird about the way Elm Studio is working, I hope you'll let me know in the comments because I would love to learn more about this because AI is definitely part of the hardest 5% of almost any project that we'll be working on, and we will all get a lot smarter if we do it together.