AutoGen Studio with 100% Local LLMs (LM Studio)

In the last video I showed you how to install and set up Autogen Studio on your local machine. In that video we were using GPT4 as a primary agent. However, one of the most common request was to replace this model with an open source large language model. And that's exactly what we're going to be doing in this video. This is going to be the easiest way with the minimal effort to use autogen with local LLMs. First we need to serve a local LLM through an API endpoint. For that we are going to be using LLM Studio. LLM Studio is one of the best option if you want to run local LLMs on your machine and it probably has one of the best UI out there. But you can also serve your local LLMs through an API endpoint which makes it very appealing for this to be used with with Autogen Studio. First you need to download LM Studio on your local machine, and now you can install it on all different platforms, including macOS, Windows as well as Linux. Okay, download the instance of LM Studio. I have a dedicated video on LM Studio. I'll put a link to that video. I have an instance of LM Studio running. If you're looking for a specific model, search for the model and then click on Download. This will download the model for you. LM Studio is not open source, but it's probably one of the best UI available that you can use on your local machine. Once you download the model of your choice, all the models are going to be listed in here. So for this example we are going to use the open Hermes 2.5 Mistral 7B model. I have already downloaded the model, so just click on this and this is going to load the model. Now we need to serve this model through an API endpoint and for that we're going to go and click on Local Server. So here you can define your own port number if you want to serve this API endpoint on a certain port. But I'm going to keep it to default of 1234. You want to make sure that the port that you provide is unique enough that no other process on your computer is running that port. Okay, so we are all set to start the server. Click on Start Server. And now we are serving our model through an API endpoint and the API endpoint is this. So localhost 1234/v1 the API endpoint format is exactly the same as the OpenAI API endpoint. That means that this can be used as a replacement for OpenAI without any major changes to the code. Now, once we have this up and running. Now let's go and start an instance of Autogen Studio. In the previous video we looked at how to set up Autogen on your local machine. I'll put a link to that video in the description. In order to start the Autogen Studio we are going to use this command. So autogen studio you UI and then we want to serve it on port 8081. Again you can use any port you wish. Just provide the port number here. We're going to run this. And now the instance of Autogen Studio is running at this location. Next you can create your own agents that will use the open source LLM that we are serving through the LLM Studio. You actually don't need this for simple workflows because you can just create workflows directly and also create the LLMs instances that we want to use. But in case if you have a multi agent framework and you want to use this. So I'll just walk you through the process. So first click on new agent and let's call this sample assistant local. Provide a description of the assistant and in the description we're going to say this is a helpful assistant using OpenHarme's LLM. We are not going to provide any system message, but here you can add the local LLM. So let's remove the GPT4 preview. Click on Add. Now we need to provide a name, so I'm going to call this local. Next we need to provide a base URL for this. We need to go back to LM Studio. Just copy this which is the localhost at port 1234 and then/v1 and we'll just add that. We don't need the API key in here. Click on add model then click ok. So now we have another agent that is using the local LLM, but let's create a completely new workflow. So click on new workflow. Let's call it Local workflow. We're going to set the description to default. This workflow has a sender and a receiver. And here's the main thing where you want to replace or add local LLM. Either you can choose the sample assistant from here, but I will actually just add a local LLM in here. So I'll just call this local. This is going to be a base. We'll add the model. Click ok. So that's our sender agent and then we have the receiver agent. Now for the receiver we again want to modify it. So click on this. By default it's using GPT4 preview, but we want to use a local one. Now you can call it local or whatever you want to name it, but I'll just keep it local. We'll provide the base URL here. Again, now to this you can also add skills. So for example, by default we have three different skills. Generate images, fetch profile, find paper. So let's add this fine paper and also probably add generate images. Now keep in mind, we are probably not going to be able to generate images because by default this was using GPT4 and it was generating images through Dall E. But since we replaced it with an open source model which doesn't have the ability to create images, so let's click ok. But you can actually add a skill here and potentially you can use a stable diffusion model which you serve through automatic 1111 and use that as a new skill. So that would be a way to create images using stable diffusion. Let's check our workflows. Now we're going to create a new session in the playground to test our workflow. So click on new. Here you can select your workflows. So I'm going to select the local workflow that we just created, the create. Okay, so we can start experimenting with it. So the first one I'm going to try is this sine wave, which is write a Python script to plot a sine wave and save it to disk as a PNG file. Right. So here on the terminal you can see all the different messages that are being exchanged. So it actually wrote Python script in here. And here on the API backend you can actually see all the messages that are being exchanged. And this is what the model is actually generating. So here it actually generated a plot with a sine wave and also store that plot. You can see all the messages that are being exchanged between the user proxy agent and the primary assistant agent. And again, keep in mind, both of them are using the open source LLM through the LLM studio that we were serving to the API endpoint. So first messages, write a Python script to plot a sine wave. Here is the response from primary agent or the primary assistant. So it wrote a Python script to execute that. For some reason there are multiple messages that are being exchanged between the proxy user. For some reason there are multiple messages that are being exchanged between the user proxy agent and the primary assistant agent. But overall this seems to work. Now let's try if the open source LLMs can use the skills that we assign to these agents or not. Okay, next I use this prompt, find papers related to flash attention. But here's the response that we got so define academic papers related to flash attention. You can use database like Google Scholar or PubMed. Here is an example of how to search on Google Scholar. Now it walks me through a step by step process of how to do search on the Google Scholar. But the model or the agent itself has a skill to look up papers on archives, but for some reason it's not using that skill. So we might have to look at another open source LLM which has the ability to do function calling and use these skills. This is one of the issues with these smaller open source LLMs that most of them are not created using tools or doing function calling. On the LM Studio side you can also see the total number of tokens both within the prompt as well as in the completion token side. So this was a quick overview of how to run open source large language models with Autogen Studio. I hope you found this video useful. I will be making a lot more content on autonomous agents, both on Auto Agent and similar frameworks like Crewai devchat. I think this is going to be an exciting area that we will see a lot of progress during 2024. If you found this video useful, consider liking the video and subscribe to the channel. Thanks for watching and as always, see you in the next one.