Huggingface Agents: Multimodal Transformers Agents Are Here & Its Open Source

This is just like R2GPT, but now we have the ability to use free open source models. If you recall, a few weeks ago, Microsoft released a paper called Hugging Chat where they proposed to use ChatGPT as a brain, which has access to all the Hugging Face models. Hugging Face hosts tens of thousands of open source models related to tasks like computer vision, natural language processing, speech processing and everything in between. They call the system Jarvis. Given a task, it can decide which set of models to choose to execute that task. A few days ago, Hugging Face announced their own version of that system and they're calling it Transformers Agents. It's now part of the Hugging Face Transformers library and it has some really cool abilities. Now it's a fully multimodal agent, so it has access to models related to text, images, videos, audios, document and a whole bunch of other things. We are still waiting for OpenAI to release multimodal version of GPT4. So in this video we are going to be looking at Transformers agents in action and I'll show you how you can run this. The best part is you can actually create your own tools that the agents can use. So it's like creating plugins for GPT4. As I said, it's part of the new version of Transformer package and they are introducing the concept of tools and agents. Now these agents are also very similar to what LangChain offers. Hugging Face actually provided a Google Colab that you can play with. But first, let's understand how this whole thing works. So the agents basically provide a natural language API on top of the transformers. So you can simply provide instructions in natural language. And then using the agents and tools, the natural language commands are going to be executed by these different models. To process the natural language requests, we need a large language model. Here is an overview of the whole system. So the user provides instructions in natural language. So for example, read out loud the content of an image. Then the large language model creates a detailed prompt based on your instructions. And a part of this prompt creation, you define different tools that the large language model has access to. So for example, you can have a tool which generate images or create captions, or convert text to speech. Now, based on the initial prompt, the large language model will create a set of steps that it's supposed to follow to execute your initial instructions. So again, the large language model will act as the brain of this whole system. It will create a set of instructions. So for example, for a given task, it selects the type of tools that it will need from all available tools and then Feed that into a pipeline. Now, the pipeline is executed in Python. The user gets a response or result. So before looking at the code, let's look at a couple of examples of how the system works. So you basically create an agent and then call a run function or run command on it. So for example, you pass an image to your agent and say caption the following image. And then the output is going to be this, the caption that the agent came up with. So the agent will call a few different tools and using those tools, it came up with a caption. Now, the best part is you can use the output from the previous stage and use that as an input to the next stage. So, for example, we again run a different command that is read the following text cloud. So you give it, give the text to the agent again and then it actually will use a text to speech model to create the output. So here's the output.

A beaver is swimming in the water.

Hugging Face actually provided this notebook. So in order to execute the code, first go and make sure that you create a copy of the notebook. Then you want to make sure that you are running it on a gpu and then you simply connect it. Now a couple of things to note that it will run only on transferware version 4.4.29. Right? So here we are simply checking the transferable version. Other than that, it's also downloading some other packages. For example, it's downloading Diffusers, Accelerate Dataset Storage, Sound file, right, and some other libraries. Now, it's not going to download all the models that are available in the tools, but whenever we need a certain tool or certain model based on the input that the user provides, the I agent will automatically download that specific tool from Hugging Face. So if I run this, it will actually ask me to provide the Hugging Face token. Okay, it's actually asking me to provide the token. So just go to your Hugging Face accounts, then go to Settings and Settings, then click on Access Tokens, Just copy that and we'll simply paste that and click login. And now we are successfully logged in. Okay, next we need to set up the agent. So there are three types of agents depending on which LLM you use. So you have three different options. Either you can use OpenAI's LLM or Open Assistant or Starcoder. Now, they Recommend to use OpenAI because that is the most powerful agent. Now you can use Open Assistant or Starcoder as well, but you just need to be very careful with the prompt that you provide. I'm going to also show you Some tips and tricks that they have in order to get the most out of your agent. So if you're interested in the code, here is the code that it simply select. The default one is OpenAI, right? But then there are these other options as well. Starcode and Open Assistant. For the first two, it simply creates a hugging face agent object. And for the last one it's getting OpenAI Agent object. All right, so in this case we're going to be selecting OpenAI. So I'm going to run this, you know, it's asking me for my OpenAI API key. So I'm going to just provide that hit enter and it simply downloads the corresponding agent and now it's initiated. Okay, a side trick, if you want to hide the code, so go to these three dots, then click on form hide code. Or you can hide the form that you are seeing. So it will simply show you the code. But I'm going to simply click on hide code. Okay. Now how do you use the agent? So currently it supports two commands. One is run and the other one is chat. We're going to look at the chat in a little bit now just to show you the capabilities of this agent. I'm going to be using the examples provided by HungerFest, but we're going to go into a little bit more details. So first we are going to be running, looking at this run command. All you need to do is simply provide a prompt. So in this case the prompt is generate an image of a boat in the water. Now depending on all the available tools and models that this agent has access to, it will decide which models or agents to use, which actually tools or models to use to execute this task. So in this case it's just one simple task. So it will use a single tool, but it can be a multi layered prompt as well. And then in that case it's going to be using multiple tools. We are going to look at that example in a little bit. So here is you actually get an explanation of what's going on. So this is coming from the agent. The agent says I will use the following tool. So since it's related to image generation, so it's going to use image generator. And then the code that is actually executing is image is equal to image generator. And the prompt that it's passing on to the image generator is a boat in the water. Now it's smart enough to actually ignore the first part of the prompt, right? Because based on the first prompt, it decided which tool or model to use. And the actual thing that we Want is in the second part of the mountain. That's why it used the second part. So it will simply download the corresponding model that it needs. And here's the image that it generated. So there is a boat and it's right in the middle of the water. Now you can actually do operations on the output that it was generated. So for example, next we want to create a caption for this image. So we took the output of the last execution. So that's the port that is going to be the image. Right. We feed that in as an input and then we ask the agent to run another prompt or query. And this, this time it is can you caption the image that we just generated? Right. And let's see what it comes up with. Now in this case it says that I'm going to use image captioner to generate caption for the image. So basically it will download either. I think it's going to be using a blip model that takes in the image and generate text description. Right. And the code that is running is image captioner boat. That is the caption. We're going to look at what different tools tools that are currently available. The new tools are being added, but we're going to look at the current available tools. And here is the output. So a boat floating in the ocean. Now in this case, and the only input that it received was this image, not the initial text prompt that we gave. Okay, so far we simply look at single stage prompt execution. But let's look at this. Can you create an image of a boat? Please read out the contents of the image afterwards. So it's supposed to first create an image, then it's supposed to create the description of the image. So we need a second stage or model and then we need a third stage model that will take the text input and convert it to do into an audio and then we will play that audio. So let's execute it and let's see what different steps it comes up with. As I said before, the agent itself decides which tools to use and which models to use. Right. So it simply looks at this natural language description and based on this it will decide all the steps. So here are the steps. I'll be using the following tools. So image generator to generate an image of the board, then text reader to read out loud the contents of the image. Right. Then here is how it's going to be done. So first generate the image, then generate the caption. So you get the text description and then you convert it into speech. This is amazing to Think of how this is being done. Right now we're using OpenAI, but you can literally integrate any other large language model with this. So let's see what it came up with.

A boat is floating in the water.

That's pretty cool. And the best part is that the caption it generated this text. The caption is a boat is floating in the water. But to text to speech, convert text to speech model, it only pass this. The boat is floating in the water. So it's pretty neat. It can actually understand which is the most important bit so far in the task that we asked it to execute. We actually described them step by step in our prompt. However, this works great when the query implies the use of tools which haven't been described directly. So like as an example, the read out loud the summary of hugging face, right? So now it has to deduce by itself what type of tools to use. And the agent automatically selects the tool appropriate for the task you want to perform. The agent is saying, I will use the following tools. So text downloader. So first it needs to actually access this website and download the text, then summarize it to create a summary of the text and then text reader to read it out loud, right? And the actual tools or models that it use. So it's using text downloader, then summarization and then an audio text to speech. Right?

So Hugging face is an AI community building the future. More than a organizations are using hugging faces ichat models. The hub is open to all mistle models and has support from libraries like Flare, Asteroid, espnet and Pinot.

Okay, so you saw that the text to speech didn't really do a good job with let's say 5000 or AI, but you can actually replace whatever that model is with another model. I'm going to show you how these tools are defined later in the video. Run is one command that works on the agent and it doesn't keep memory across runs, but performs better for multiple operations at once. So for example, if your prompt is using two or more tools or models, then it, it runs very well, right? There's another command that you can execute and this is called chat. So it keeps memory across across runs, but performs better at a single instruction. Now in order to do chat, all you need to do is just call dot chat on your agent. So let's run it like here my prompt is show me an image of a car. Actually I have an extra ant, but let's see how good this can work. Here it's, it's passing a prompt of A car to the image generator. So I think it will work. Now in the process it will download the model that it needs. So here it generated an image of a car. Now you can continue the chat. So it will assume that you are working with the same system. So for example, if you can ask it transform the image so that it can snow like the initial prompt was showing this animal and now it seems like it's snowing, right, and you can even execute other operations on top of that. So for example, this was show me a mask Snowy kp. Now the chat is great if you want to keep the history of the prompts that you have used so far in order to start a new chat, you will simply execute this function, prepare for new chat, and it will clean up the history of the previous chat and you can start over with a new chat. Let's look at which tools are available. So there is a tool for document question answering such as PDF files and the model that's used is called Donut. Then for text question answering it's using flat T5. For unconditional image captioning, it's using the blip. And then there are other models related to, let's say image segmentation, text to speech, speech to text, and so on and so forth. There are some community based tools as well. For example the text downloader from a website, that's actually a community tool. So this list of tools will keep expanding. It was the initial release, so expect this is going to grow a lot. This is very similar to the plugins for GPT4. They also provided a section on what if you want to add a new tool yourself. Right? So here is an example of image reading, right? So there's an API actually, which I wasn't aware, where you can make a call and you will get a random image of a cat. So let's assume you wanted to create a tool around it which whenever it's used will call that API and create image of a cat. Right? So here is how it's going to look like. You are going to be using the tool object from the transformer and you define a class. And there are two very important things that you need to consider when creating your tool. So first is the name of the tool and then a concise description of what this tool actually does. Now this description is going to be used by the agent to decide whether to use this tool for a specific task. Then you simply define the input and outputs. And when this tool is called, what operations are going to be performed. So here is an Example like if you simply use an object of that function of that class. So you will get an image now whatever agent you're using. So you can pass a list of tools and let's say in this specific case we are passing that one specific tool, right? So based on the prompt, the agent will decide which tool to use. So it says fit an image of a cat online and caption it for me. So it will look at the description of all the available tools. So for example, in this case it picks this tool to be the most appropriate because it says fetch an actual image of a cat online Here it used that specific tool to get the image and then use the image captioner tool to create a caption for that too caption for that image. This also bring us to the importance of the prompt that you are providing. So you want to make sure that the prompt prompt is very clear so that the agent can actually decide on which tools to use for to execute your prompt. So that is why they actually have a whole section on custom tools and prompts guidelines on their website. For example, a shorter version of the run prompt that is being used. So it says, I will ask you to perform a task, right? Your job is to come up with a series of simple command sign and then it's given a list of tools, then a task and example answer, right? So that's how the master prompt is defined. Now they have given a couple of good examples of how to get the most out of your prompt. So for example, if you simply tell it show me a tree, right? And it's the hugging face agent. So the steps that came up with is I'm going to use an image segmenter. Now the reason is this prompt is not very clear based on the tools available and their description. So in this case they looked at image generator its description. So this, the description says this is a tool that creates an image according to a prompt, which is a text description, right? So it takes in input name prompt which contains the description of the image and outputs an image, right? Now you want your prompt to have some words used within the descriptor description of your tool. So let's say if we update our prompt to create an image of a tree, right? So now it has the word image in there, right? And it's a lot more clear to the agent. And as a result you see that it's actually using image generator tool. Now. Now you can also customize the description of your tools, even the ones that are already available. So this gives you the ability to customize them to your own prompts, which is, I think a really great feature to have. Now you can also customize your prompts and using custom tools. We already covered this. It's an extremely powerful model or agent that you can use it in your open operations. Like you can build something like AutoGBT on top of this. Now a couple of channel related announcements. So now we have a Discord server. So come and join us over there. Quite a few people are already here. Let's learn from each other. Link is in the description of the video. As always, if you have any questions or comments, put them in the comment section below. If you are new here, please consider subscribing to the channel. And if you are a subscriber, make sure that you press that Bell notification button. Hope you learned something new today. Thanks for watching. See you in the next one.