LM Studio: The Easiest and Best Way to Run Local LLMs

This is probably one of the best and easiest way to run an open source large language model on your own machine. And this is probably one of the best user interfaces that I have seen for open source large language models. The tool is called LM Studio and it's a potential alternative to something like text generation web ui. The installation is extremely simple and has also advanced features like hosting your open source LLM through an API. The LM Studio supports a wide range of different models including Llama, Falcon, NPT, etc. And is available for Apple, Silicon, Windows and Linux supporters in beta. Now, in order to run LM Studio on your local machine, you will either need an M1 or M2 or a Windows PC with a processor that supports AVX2. LM Studio lets you run quantized models in GGUF format on your local machine and it's using Llama CPP in the background. So in this video I'm going to show you how to download LM Studio on your local machine, how to add different models to it apart from the default models that it comes with, and we will also look at some other more advanced features. In this video I will show you how to run this on an M2 for Windows. I think the instructions are going to be remain pretty similar except for how to install it. So let's first download M Studio by simply clicking on this link. Now in order to run this on a MacBook you simply need to search for LM Studio. Once you open the application, you're going to be presented with a Windows like this. So here you have information regarding what type of models are supported and they have also highlighted some of the models that the community has selected. So for example you have the Mistral 7B instruct code, llama 7B instruct, right and then on the right hand side you will see some more information regarding who created the model and some additional resources. Now with each of the model you have the relevant information as well as highlighted quantization level. So for example if you look at this it says that four bit quantized model, what is going to be the download size and then it says small and fast. So that's actually pretty good to know say for six bit quantization level. So it says less compressed, might be slower, but if you click on this or hover on it, so it says six bit minimal loss of quality. Now if we come to the four bit, so it says medium loss of quality. So it's actually a pretty nice interface where you can get a lot of details about the model that are available in here. Now, in order to download a model, simply click on this download button and select the quantization level you want. Keep in mind, this is not the only way of downloading a model. I'll show you how to download other models later in the video. Okay, so let's look at the other tabs. So here is the search button. If you click on this, this will let you search for different models and also gives you the detail regarding different models. So for example, I Already downloaded this Samantha 1.2 Mistral 7BGU in GGUF format. So on the right hand side it has actually listed the different formats in which it's available with different quantization level. So as I said, mostly it supports the GGUF format because it's based on Llama CPP and Llama CPP currently support only this format. Okay, here's another thing which you can actually do. So you can go to the search button and directly search for models that are available on Hugging Face. So let's say if I'm looking for Vocunia models on the left hand side, it gives me a list of all the different models that are available on Hugging Face. So you don't really have to go to Hugging Face at all. You can simply look at them here and it also basically rank them based on the number of downloads and the number of stars it had. So now I simply typed in uncensored and it basically give me the list of all the uncensored model that are available on Hugging Face. So this is pretty cool. Now if you select one of the models in here on you look on the left hand side. So here you can see different sizes of the models and you can simply click on the download button in here and it will download the model for you. Now yet another interesting thing that I saw was this. So let's say if you select a model it tells you the RAM requirements. So for example here it says requires 30 plus GB of RAM when I selected a smaller model. So it's telling me possibly support it. So probably it's reading my system ram. And based on that it's telling me that you probably will be able to run this model. Okay, so let's look at the other available options. So for example, there is this chat button. When you click on this it takes you to a chat interface. Now again here you have a list of all the models that you have downloaded. So for example, I have a total of three models downloaded. So let's say in order to run a model I'll Simply need to select that model and then it also tells me how much RAM usage is going to be for that specific model. That is pretty neat. In this case I'm using the Samantha 1.1 Llama 7B model. So you can simply start chatting with the model here. So for example, I'm going to say hi and it's actually giving me a pretty quick response. Now the reason is that I'm using the GPU on my M2. Now you can actually create new chats by using this new chat button. So this is pretty awesome. Now at the bottom of the screen you can see the time it took for each token. What was the speed in terms of number of tokens per second? Right, so this is some valuable information that you're getting out of the model. Now let's look at some other options. So on the left hand side you will see this toggle settings sidebar. If you click on this, this will show you more options that you can customize. Okay, so the first thing is presets. These are collection of pre made configuration files depending on different models. Okay, so there are a number of presets available. So for example for alpaca type models, chat, this is basically OpenAI prompt template, right? So for different models you can have different prompt templates and you can simply select them using this preset. I'm going to just keep it to the default. The second tab is the model configuration, so I'm going to keep it to default. Now in terms of the inference parameters, so here you can define the temperature that you want from in the model and also some other things. For example number of words to generate or repetition penalty. Again I'm going to just keep those to the default. Then you can also define the prompt format if you want to change it. So for example it says before message, so what the prefix is going to be, what the suffix is going to be, you can change those if you want. You can also add stop strings as well. And here is a place to define the system prompt if your model supports it. Now a couple of other parameters so you can load your entire model in memory. So you can keep this mlock on. If you have enough ram, you can also change the batch size. And I think one of the most important parameter that you want to set is the context length. So different models are going to have different context lengths. For example, the Mistral 7B has a context length of I think 16,000 tokens or 8,000 tokens. I might be mistaken. Right, so you can change those in here. Now in My case, I want to use the GPU on M2, so I'm going to just disable this and I'm going to go to the hardware settings and here I'm going to select Apple Metal gpu. Now there are some other options that you can explore as well, which are related to the appearance of the chat and stuff, but I'm not going to go in here. Okay, let's look at the available models that you have downloaded before looking at the API. So if you go to this folder icon and click on my models, this basically lists all the model that you have downloaded, right? And if you want to free up some space, you can simply hit on this delete button. You can simply click on this delete button and that will delete the model. Now here you also have the location of where the models are actually stored. So if you want, you can simply change it in here and that will take effect. Now the last thing I want to show you is this inference server. So if you click on this local server option, you will see that you can actually create a localhost and serve your model through an API. So here is how it works. From the models that you have in here you can select a model and then you can define the port number that you want to use. And all you need to do is to simply click on Start server. Now here's the best part. It also support queuing of the request that it's receiving. Also, everything is going to be logged if you toggle this to on, right? So the format that you see here is exactly the same as the OpenAI API. So you have the models endpoint, you have the completions endpoint and then you have the chat completions endpoint for chat model. So in an upcoming video we're going to go into a lot more details on the use of API and I'll also show you how to use this with Autogen in order to replace GPT4 with an open source large language model. So this was a very quick overview of what different features are available within the LM Studio. Do check it out. It's very easy to install if you're running this on a Mac or a Windows machine. And it's very easy to use as well. I hope you found this video useful. Consider liking the video and subscribe to the channel for more content. Thanks for watching and as always, see you in the next one.