Qwen-Agent: Build Autonomous Agents with The Best Open Weight Model

Alibaba just released the second version of QAN models. They are calling QIN2 and these are currently the best open weight models available. The focus of this video is going to be on function calling and agents, but let's look at some information regarding the models. This is a family of five different models ranging from half a billion all the way up to 72 billion parameters. You will be able to run the smaller half billion and 1.5 billion models on a modern smartphone. The bigger model supports up to 128,000 tokens compared to 8,000 tokens of Llama 3, which makes it a lot more useful than Llama 3. It has also support for a lot more languages. There is a special focus on Middle Eastern and South Eastern languages which is pretty great because most of the other models that we see are focused on western languages only. Now they have paid special attention to coding and mathematics as well as long context understanding which is pretty useful for these models to be actually usable in real world applications. Now apart from the models, they also updated QuinAgent, which is their own framework for creating agentic workflows and agentic applications. Now this framework has access to its own browser assistant code interpreter and you can create custom assistant. My goal for this video is to show you how to do function calling with Quinn and also how to create custom agents using Quin Agent. As a bonus if you stick to the end, I'll also show you what kind of impact quantization can have on the performance of these large language models, which is pretty critical if you're putting these in production. Let's first understand the difference between agents and function calling before we look at practical examples using Quin. Now these are two concepts that are very important and are often misunderstood. So first, what is function calling or tool usage and why we need it? So imagine that you want the LLM to interact with external world, but it's not able to do that because it doesn't have the real time information. In that case, you will want to use external tools or functions such as APIs to get the current weather. So you will take an external API or it could be like a database call, wrap it around a function, and then tell the LLM that it has access to these different functions or tools. Now the LLM based on the user input will determine which function to use and for the function that it picks to use, it will generate a list of inputs. But the LLM is not able to actually make a function call or execute that function, so it will simply generate the list of inputs for the function that it picked, then you as a user will have to make that function call or call that specific API endpoint. You will get the results, you return those results back to the LLM and the LLM will generate the final response based on the output of that function. Now on the other hand, agents are a lot more complicated. So agents are essentially instances of these LLMs, but it has access to a number of tools just like we described in the previous section. But apart from that, it can also do planning, it has memory and it can perform actions as well. So let's say the user want the LLM to do a certain thing. The LLM in this case is going to be the agent. So the agent will first create a very detailed plan, then it will decompose it into sub goals and then it can use these tools to perform specific parts of the plan which are going to be done through the actions that the LLM can actually do. So essentially it's going to have access to a code interpreter and it can keep track of both the operation that it has performed so far as well as what operation it's supposed to do. So agents have access to both short term as well as long term memory, and agents are very critical for these elements to be actually useful. Now with this difference in mind, let's see how we can use both function calling as well as age agentic calls with Quin. To get started with Quin agents, we will be using the 72 billion version of Quin 2 and we will be running that locally using Ollama. Now you can also use their external API that is available, but to use QUIN Agent we, we have two options. We can install this using PIP as an independent Python package, or if you want to get the latest version, which is the development version, you can clone the repo and then run it run the installation locally. As I said, I'm going to be running this locally using Ollama. So if you don't have Ollama, just go to the link in the video description, then download the Ollama server. Depending on your operating system you will need different command, then go to models and here you're going to select Quinto and I'm going to be running the 72 billion version. So for this I need at least 41 gigabytes of VRAM. I'm running the bigger model because this is going to be a lot more accurate. Later on we'll look at what happens if you use different quantization levels. That is a question that comes often and you want to be very careful with which quantization level that you want to use in production environment. So make sure to stick to the end of the video to find that out. Okay, to run this 72 billion version, I need a lot of VRAM. So I am using a virtual machine from Mast Computer. It's a virtual machine running in the cloud. If you want to use the same machine, link is going to be in the video description for discounted price. So first we need to start an Ollama server and we'll use the Ollama run quin2.72 billion command. If you run this for the first time, this will download Ollama and then download the model. So depending on your Internet speed, this stuff can take some time. Okay, so the model is loaded and we can start interacting with the model. So if I say hi, then you can instantly see that it started generating responses because it's running on a relatively powerful gpu. Okay, so we'll keep this running. Next we need to download the package. So I created a virtual environment which I'm calling Quinn using conda. So here's the command that I used, CONDA Create n Queen Python 3.10 these commands are going to be in the video description, so you don't have to stop and type them in. And then we activated the virtual environment using conda Activate Queen command and you can see that the virtual environment is actually active right now. Now I have two different Python scripts. One is for function calling and the second one is going to be example of agent usage. So we'll first start with function calling, but before that we actually need to install our package. So we'll just use the PIP install quin agent command. Alright, so here's the command. If I run this, I have already installed the package, but in your case it will just download the package and install it and we are ready to go. Okay, we'll start with the function calling example. If you have seen my previous videos on function calling, you probably have seen this. So essentially the model needs to pick what function to use for a specific task, then figure out what are the inputs to that function. I will pass out that on to a Python interpreter. The Python interpreter or the user will execute that function, get the response and then feed those back into the LLM. So just like other large language models, you actually need to do all these steps manually. Only Gemini Flash actually has a way to do wrap everything in one module and do the function execution automatically as well. I have a detailed video on that if you are interested. Link is going to be in the video description. So let's say in this case the function that we want to call is getweather current weather. So basically we created a dummy data set. In real life you'll probably have an API which you want to call. So whenever the user asks for current weather information, it's going to call this function. And then there are different locations. Depending on the location, there is different temperature as well as the units for temperature are different. Okay, now to do function calling, we first need the LLM itself. So since we're using Ollama, we're going to just provide the base API to the model server. It's compatible with OpenAI standard and the Ollama server by default runs on this specific port number. Now, if you use their own API, you can actually provide the API here that's running on Dash Scope. But we are running this locally, so you just need to provide the model name exactly. Same as what the OLAMA is running. And then the IP address with this specific port on which you are running and V1 is since it's compatible with OpenAI standard. Now after that we're going to have the user message. So this is basically the user query and it's talking about what's the current weather like in Paris right? Now for function calling, it uses a very similar structure to something like OpenAI and Claude. So you need to provide what function is going to be calling what that function is supposed to do. So you need to provide detailed description and then you need to tell it what are the input of that function. So in this case the main input that is needed is just the location. You can also auxiliary provide the type of unit that you want to use for temperature. Right. So basically here is the list of functions. It can have multiple functions. Then you are going to pass that initial user query or user input along with the function or list of functions. It will look at the user query and determine whether I want to use the any of the available functions or not. Right? If it decides to use a function, then there is the reason of ending. The prompt is going to be basically function call. So whenever we have the first pass through the LLM, we then check what was the reason. If it's a function call, then we looked at all the available functions, get the function that the model decided to use, then we also get the inputs or arguments based on the model call. Right? We we actually execute that function. So we'll have to basically ourselves execute the function or make that function call. We get the results appended back to the Messages and give it back to the LLM to generate our final response. So if you look here, there are two independent LLM calls that are happening. Okay, so here are the results. After the first call to the LLM, it decided to use a function and based on the user query, which is what was the temperature like in Paris? The input is going to be Paris. Then the output contains both the temperature as well as the units that are being used. And then we pass this output of the function call to the second call to the LLM and the LLM will generate the final response. Now it can do sequential function calls where the input of the second function call is dependent on the output of the first function call. I have a detailed video on how to do that with Gemini Flash if you're interested. You can modify that code link is going to be in the video description. So this is going to be a fun one. We will use the Quinn agent to actually do some interesting tasks. Now remember, the agents have the ability to plan a certain task and then they have the ability to execute those tasks using available tools in a can also keep track of what has happened in between. So it has memory to actually remember what operation it has taken so far and what operations it's supposed to do in future. Okay, so again we will be using the Onama API server. We are using the same model. Now in this case we want the agent to to be able to actually generate images. So, so this is going to be a text to image API that the agent is going to be using. And then we want the agent to download the generated images to a specific folder. In order to do this we will need to first create an agent. We will use the QUIN agent library and there is an Assistant class which is basically an agent that you can use. Now for image generation, we will need to create our custom tool. So this is basically a custom tool class which we are calling My Image Generation. The description is that it will be able to generate images based on the user input and it's using an external API. The API that we're using is polynations AI. Now this is a custom agent, so we can call this to generate the image for us. But in order to actually call it, we need Code interpreter. So Code Interpreter is another tool that is available within QUIN Agent so it can actually execute the code for you. And once we generate the image using this API endpoint, we'll have to download that. That is again going to be done through Code Interpreter. And if the agent run into any issues, it will be able to update its code. Now keep in mind that the code interpreter is not running in a sandbox. So you need to be very careful with the commands that you pass on to this agent, because the commands are going to be executed on your local machine, which is the virtual machine that I'm using. So basically we, we wait for the user input. Based on the user input, we will pass that on to our agent, and then the agent will just run those and come up with a plan, execute that plan. And if it needs to modify that plan, it will be able to do that. So I'm going to disable streaming. We'll just see a couple of messages. Now, in order to run this, we're going to go back to a terminal, we will run that file, and let's ask it to create. Create an image of Llama with sunglasses. Okay. So the quality of the generated image is going to be dependent on how good that API endpoint is. Now, just to show you, I think I ran this before, so I'm going to just delete everything from the workspace, right? So everything is completely clean. We will run this and we will wait for the results. Okay, so the execution is complete. Let's see what exactly happened. So first it decided to use my image generation tool. And the arguments are Llama wearing sunglasses. So this is basically the input to the tool. Now it actually updates, updated the API endpoint. So let's actually look at what exactly it generated on the server. All right, so we see image of a Llama wearing sunglasses. So this is pretty great. And the way this happens is since the agent has the ability to run the code using the code interpreter. So not only it generates the initial base URL, but then it says, here is the image you requested and basically will make a call to that API endpoint and we get the image generated on the remote server. Now the next part was to download that image. So basically it says, okay, define the URL of interest. So it came up with the URL. Then it says send a get request to download the image contents. Right? And in this case it is using the code interpreter. So the second tool that we asked it, now it generated a valid request. But the problem is it ran into an issue because that images folder did not exist. So then it had to create that image folder for us and store or download that file to that specific image folder. Right? So here actually it came up with a path of that where it's going to download, Right. I think before that it here it actually created that folder for us, right? And then it ran a request, a get request to download that file for us. So if you go here, it created a workspace folder within the, there is a tool then code interpreter and it created the images folder. And within the images folder we see, we see the exactly same image that we saw on the server. So this is pretty neat because using this Quinn agent you are able to create your custom agents and those agents have the ability to execute code, which is pretty awesome. In this last section we will look at a very important question and that is whether quantization has a significant impact on the overall performance of the models. And this is a question that I get asked a lot. And the good thing is that the Quinn team actually addressed this question with the updated models. So here we have Quinn 2,72 billion and also starting from the half a billion parameter model. Now what they have done is they ran evaluations for MMLU, CEVOL and IF evolve for floating point 16, 8 bit quantization, 4 bit quantization and AWQ which is a specific quantization format. Now if you look at this based on the average numbers, you will kind of see a trend when it comes to the bigger models. For example 16 bit, you get around 81%. For even this 4 or 8 bit quantization, the differences are not that critical. But for the 8.0 version of quantization, that difference in performance is definitely significant. Now if you come to smaller models then you will see drastic differences. So we started with 66.9% or that's the average score for four bit. We see a difference of five points for the same model. So which is actually pretty significant. Now that difference could be a lot more when you are looking at smaller models, depending on how much data was used to train it. So in general I would say if you were to put any of these models in production, I would highly go against 4 bit quantized version. You want to go with at least 8 bit or 16 bit if that that's possible. And again quantization is going to have a lot more impact on the smaller models compared to the bigger models. So if you have the option, you, you always want to run the bigger models. I hope this was helpful. I'm going to be creating some more content on QIN2 because I think it's a very good alternative to LLAMA3, especially because of its much longer context window and also its support for a lot more languages compared to llama 3. I hope you found this video useful. Thanks for watching and as always see you in the next one.