Microsoft's Autogen 2 - Create Custom AI Agents

So Microsoft recently released Autogen version 2 and this is essentially the upgraded version of Microsoft Autogen. And Auto Gen was one of the first Microsoft softwares that actually allowed you to interact and use AI agents on your device. Now essentially this video is going to be a deep dive on how to use Auto Gen Studio, which is the second iteration of Microsoft Autogen. How you you can build multi agent apps using this software. Don't forget to check some of the links down below if you are struggling. And let's take a look at exactly how Microsoft Autogen works. So essentially by this screenshot you can see that this is the Autogen Studio and later on in the video I'll be showing you some of the actual things that you can really, really do. So essentially they say that to rapidly help you prototype multi agent solutions for your tasks, we are introducing autogen Studio, an interface powered by Autogen. It allows you to declaratively define and modify agents and multi agent workflows through point and click and drag and drop interface. And for example, you can select the parameters of two agents that will communicate to solve your task. You can use our user interface to create chat sessions with the specified agents and view the results, for example, view chat history, generated files and the time taken. Then of course you can explicitly add skills to your agent and and accomplish more tasks. And you can also publish your sessions to a local gallery. Now autogen Studio is open source and can be installed via pip. So let's actually take a look quickly at the installation process. It's actually fairly simple and if you do have any issues, I have a one stop solution which I will show you all. So you can see right here the first thing that you need to do is configure a large language model provider. It says here to get started you need to access a language model. You can set this up by following the steps in the Autogen documentation and configuring your environment with either the OpenAI key or an Azure OpenAI. Now essentially it's just an API key. So if you want to get yourself an API key, if you want to get yourself an API key in order to use this, what you need to do is you need to head on over to this page. So essentially all you want to do is you just want to sign in to the OpenAI playground area. And what you want to be able to make sure you do is you then scroll down to here, then you just click API keys. Then you can see here that there are many different API keys that you can use and, and can create. Now, I've created a couple of API keys, and essentially when you create a new key here, it's going to tell you exactly what you need to do. So if I click create a new key here, you just name it as whatever you want. So I could name this as Autogen Studio. And then for example, here I could click Create key. Then of course, I have to verify the puzzle. And I'm not gonna lie to you guys, sometimes these puzzles are actually pretty hard. Like there was one previously that I did have to do that was really, really hard. But, you know, you just do the puzzle or whatever and then you do get your API key. So I just wait for here and then I get my API key. Now, I'm not going to show you guys what the key is, but essentially you just copy it. You copy it and then of course just put that into a text document so that you keep that safe. Because once you do create the key, you're going to be able to have to generate a new one because you're not going to be able to view this key again. So once you do that, just copy that. Done. And then. Yeah, so as you can see, Auto Gen Studio and then last use never. And then of course, I can just revoke the key, just delete it, or I can edit the key just by editing the name of that key right there. So once you have that key, then we go back to this and then you can see you can also specify the model directly in the agents configuration, as shown below. I would just rather not do that. But then of course, in the terminal, this is where you set the OpenAI API key. So now of course we have the installation which is right here. So essentially all you want to do is open up the command prompt and then all you want to do is literally just type in PIP Install Auto Gen Studio. If you actually go onto the Autogen Studio page, you can see that the text is actually right here. So if you just copy this, and then once you copy this, if you actually open up your command prompt and then this is the command prompt, and then you just put that in there. Just put in PIP install. Now, I've already ins. I've already installed it already, so I don't need to do this. But this is something that you do want to do if you do want to install it. And of course this is running the application, so I do have it running. But once you do have it installed and once you do have the application ready, all you need to do is just literally copy the text Like I said, a link will be in the description, so it'll be really easy to copy this and then you just simply copy that. You can put it into your command prompt terminal and then it will essentially give you a link. Now the link is usually here, so once you see the HTTPs a local host, all you need to do is simply do that. So I'm going to do that again just to show you guys how it works. So I've typed in port Y and then of course it's running and then you can see here that it gives me the link. So I just copy this right here, just get that. Then control C and it says waiting for application startup, Application startup complete, running on, yada yada yada. Then all I do is I literally just paste that into a browser and you can see that autogen Studio is loaded up. Now if you are having any trouble with this, because I actually did have a few troubles and a few niggles when I actually started this out, I would just diagnose it with GPT4. So if it says any kind of error, just get the error, paste it into GPT4 and just tell them, look, I'm trying to install this program. So I just copied and pasted the entire page and I was like I'm trying to install this, but I have encountered this error and just copy and paste it in and GPT4 will solve it like it did for me. So that is something that actually does help me copy quite a bit. So then of course we have the actual use cases of what you can actually do with Autogen Studio. So the Auto gen Studio UI is organized into three high level sections which are Build, Playground and Gallery. So I'm going to, you know, firstly walk you through, then I'm going to show you guys how to use everything. So first of all we have the build. So essentially what build is is really simple because this is where you can actually do your general agent workflow and this is where you have your general agents and you can essentially like build them and stuff like that. But like I said, these are just screenshots. I'm going to show you guys in a second. And of course we the skills area. So skills are basically the functions that describe how to solve a task. And in general a good skill has a descriptive name such as for example generate images, extensive doc strings and good defaults, for example writing out files to disk for persistence and reuse. And you can add new skills to Autogen Studio that are via the provided user interface. And at inference time these skills are made available to the assistant agent as they address your tasks. And of course here we have the agent workflows which is just a specification of a set of agents that can work together to accomplish the task. And the simplest version of this is a setup up with two agents. A user proxy agent that represents, for example it compiles code and prints a result. And an assistant that can address tasks requests, for example, generating plans, writing code, evaluating responses and proposing error recovery steps. And a more complex flow could be a group chat where even more agents works towards a solution. So for example, here, like you guys can see, we've got the Build, playground and gallery. And for mine, currently there is a condition. So here you can see we've got Build, we've got playground and we've got gallery. And you can see right here this autogen studio build multi agent apps. And you can see here that these are the default workflows. So we've got the visualization agent workflow. So this workflow is used for visualization tasks. And it says here that visualization assistant to create plans and to write code. And then we can see the system message right here, which we can actually read. It says your task is to ensure you create highly high quality visualization for the user. Your visualizations must follow BREX practices and you must articulate your reasoning for your choices. Visualization must not have a grid, outline or box. The visualization should have an appropriate aspect ratio, for example rectangular for time series data. The title must be boldly yada yada yada. If the chart is a line chart and basically this is how we have our system message, then of course here we have our models and then of course there are skills, so we can have different skills here. So for example, one skill that we have is find papers from ArcSive. I'm not even sure how you say that. But you can see right here it uses the ArcLive API in order to find certain papers. So one thing that you could do is you could get it to research papers. And this is going to be a skill that it does have, of course generate images. I'm presuming that it does use Dall E3 to generate images here. This is something that I'm going to show you guys how to do as well. And you can see right here, response client generate images model DALL E3. So this is another skill, so you can give, you know, your agent different images, which is really cool. I mean not different images, different skills, which is really cool. And of course we've got Fetch profile here which is, you know, you can see right here, you know, fetch a different and so yeah, this is Fetch profile. I'm not actually sure what this is, but it says fetches the text content from a personal website given the URL, yada, yada yada. This function scrapes the page and returns the text found in the body. So yeah, that's actually what it does. And you can actually add different skills. I'm guessing that you can create new skills. I'm guessing that you could probably use GPT4 to create new skills. And then of course you can see here a general agent workflow. And this is a primary assistant default history to generate plans and write code. See helpful assistant user. Then of course we can make new different workflows. Then of course we can have our, we have our agents here. So configure an agent that can be reused in your workflow. And essentially how this works is that you can see that we have different agents and then different workflows. So this workflow right here you can see is the general agent workflow. So we have the sender, and then of course we have the receiver. Then of course we have here we have the sender and then of course we have the receiver and then of course we add a new workflow. Then of course, here we have the skills. And if you want to add the new skills, I'm sure you're going to have to generate the code. Maybe you could ask GPT4 to do it. This is a sample skill replace with your own skill function. I'm not exactly the very best at coding, so I don't know exactly how to do this, but I'm sure there are some other guides that I will leave linked down in the description below. But it does actually work. So now I'm going to show you guys some of the demos of this stuff in action. So this is from a different session where I was loading this on a different device and I will leave instructions on how to actually do this, like a full list in the description, or perhaps it may actually be on the AI Grid website, because the description is actually pretty, pretty extensive. So essentially here you can see that this is Autogen, and essentially I'm going to walk you through exactly what was going on. So here you can see that I'm looking at the skills fetching profile and the different skills that each agent is able to have. You're able to just add different skills. So this is a sample skill that I added. This was generating an ASCII art of a cat, which is Essentially just a drawing of a cat using text. So essentially here you can see that the skill was retrieved successfully and that skill was there. Then of course, once that is done, you can see I'm looking over the agent specification. I'm just making sure that everything is good, just outlining, you know, the models, all the skills. You can see you can find papers on oxave, generate images and all that good stuff. Here was where I had a new session and I said, create an ASCII art of a cat, please. And then you can see here that after some time it was able to generate this image ASCII art of a cat. Now there's some really cool applications that I do want to show you later on in the video, which are really, really cool. And I'm going to show you those right now. So essentially, as this continued and that was the just code there, and of course that's the Python. And then of course there was also this where plot a chart. And this is basically just. This is basically the default one that they do give you. It says plot a chart of Nvidia and Tesla stock price year to date. So save the result of a file named Nvidia Tesla PNG. So right here you can see the agent messages, four messages in 31 seconds. And you can see Nvidia and Tesla PNG and then of course the corresponding code. So you can see exactly, we do have the image and this is of the real stock prices. So all of this stuff is actually really, really accurate. Of course it shows you exactly how it managed to do that. So you can see the agent messages right there. So it's figuring out exactly how to do that using certain libraries, using this library. And yeah, it's really, really cool. So what was actually cool about this was that these agents were interacting with each other and you can see that one of them was actually saying that there is a lack of certain libraries, can you actually load them? And then the second one performs it and then sends a report on the work. So that's why this is pretty fascinating stuff. So I think agents in the future are definitely going to be pretty crazy once we do get a more cohesive software that isn't so not really technically advanced, but just not as easy to use. You can see that there's a primary assistant and then the user proxy here to different agents working together. And of course, here's where I asked for a candlestick chart for Bitcoin and Ethereum stocks. And then here we can see that it is going ahead to complete that. Then after Switching back to tabs, we can see once again that this is completed. Now, there was one really cool thing that I was able to do with this that I think you guys are going to find really, really fascinating. So I asked it, you know, with Dall? E3, to generate, to create a picture of a whale in space. And of course you can see it manages to do that pretty quickly. So picture of a whale in space, but that's not the crazy thing. Okay, so with these agents, what you're able to do is you're able to generate, like long pieces of content. That's really cool. For example, here I asked it to create a PDF file with a presentation of the story history. For example, here I asked it to create a history of London. So create a PDF file with the history of London. So essentially here what I've done is I've said create a PDF file with a presentation of the history of London. So you had to see exactly what it was able to do. So essentially right here, it was able, able to give me three files you can see in one minute and 21 seconds. And of course we can see this is a PDF file with the history of London. Then I said that I wanted more information, because that's not really that much. And then I also said that I want some pictures too. So I've asked to get some additional pictures of historical places in London. And then of course, you're going to see that in around a minute or so seconds. I was able to get, you know, all these kind of pictures. Of course they're generated with dall? E3. It's not just going to like copy pictures from the Internet. But you can see right here that this was really, really fascinating, especially the PDF too. So here we're getting some historical images. I'm guessing that's of Big Ben in London. And of course you can see the PDF is where I ask. I need all this in a PDF file presentation. And then you're about to see here that all of this is able to be generated in a PDF file presentation. So you can see the Tower of London, then you can see Buckingham palace, then you can see Westminster Abbey. So, yeah, this is really, really cool stuff. And you can see that all of this is, of course, really real. So there's many different things and many different ways that you can interact with this agent, multiple agent software. I think it's really cool and I think it's a step in the right direction. But in terms of what is coming next, they do actually have a FAQ on what's actually coming next. So in the future they detail that they're going to have complex agent workflows. So we're working on integrating support for more sophisticated workflows such as group chat, which is allow a richer integration between multiple agents or dynamic topologies, which which is essentially when you have multiple agents in a group chat, you're going to be having multiple agents conversing on how to get a task done better. Also improved user experience, so includes features like streaming intermediate model output for real time feedback, better summarization of agent responses, information costs of each interaction, expansion of agent skills work. So they're going to add multiple different skills, so we don't know what skills is going to be and of course community features. So there's going to be options to share sessions and more easily among users and contributing to a shared repository of skills. Agents and agent workflow testing 121 2.