Agents with Vision 👀 - Powered by Claude 3 Haiku & Opus

You probably have noticed that AI agents are everywhere. There are dedicated frameworks like Crewai and Autogen that will help you orchestrate AI agents. However, in most of the cases to build LLM based agent applications, you don't need these frameworks. In this video I'll show you how you can use a cloud to build your own agent AI framework. And we're going to use its advanced reasoning as well as vision analysis to create a financial analyst. So we will explore how we can use a bigger model like Opus as super Agent, which will provide subtasks to sub agents to smaller models like Haiku. That way we will be able to save money by outsourcing most of the smaller tasks to these relatively weaker models. The example I'm about to show you is based on the amazing cookbook from Anthropic, which I'll highly recommend you to check it out. Okay, so let's try to understand what we are trying to achieve here. So we are trying to create a financial analyst and in this case we are going to provide Apple's 2023 financial earning reports for each quarter. And then we're going to use the Claude 3 haiku based sub agents which will analyze financial report from each quarter. And the results are going to be sent to a bigger OPUS model. The Opus model will generate the final report based on the user query. Now first we need to install all the required packages. So in this case I'm installing the Anthropic Python client, then a library to read PDF files, and then we're going to be using matplotlib to generate plots which will help us explain the results to the user. Next we are importing all the required packages. You will also need to set your Anthropic API key. In this case, it's part of my Google Colab notebook secrets. I'm just activating this and using it directly from my notebook. Next we need to define our subagents. So in this case I'm using the Anthropic client and my sub agents are based on the Claude 3 haiku, which are the smallest model. Now this is pretty helpful because the subtasks are going to be relatively simpler. So that's why we're going to assign Those subtasks to Claude 3 Haiku and it will become clearer as we go through the notebook. Now, to create our financial analyst we need data. So for this we are downloading the financial earnings for different quarters. So this is Q1, Q2, Q3 and Q4. Let me show you how they look like. So basically they are tables which has all the financial earning information for all three quarters. Now, it's really hard to actually parse tables from PDF files, but fortunately for us, Haiku has vision capabilities. And those capabilities are going to be extremely helpful when we are parsing these documents. Okay, next we are going to write a couple of functions to help us download the files. So this function will receive a URL folder path where we want to store the files and it will download the PDF files from the URL. Now, since we want to use the vision capabilities of Haiku, we are going to write another function called PDF to base64png. So this will basically take the PDF file, extract each page individually, convert that page into an image, will send us back a list of images rather than the original text. Okay, next we need to provide a path of the folder we want to download our files. So if you're using Google Colab Notebook, just make sure you create this structure. There is an image folder and within that there is using subagents. And the way you create a new folder is just right click click on new folder and provide a name to that folder. Okay, so the original code was actually using multi threading to download PDF files concurrently. But I was running into some issues, so I simply wrote this code which will download each file individually. And then we also want to check if there are any issues with the files that are being downloaded. So you just ignore the files with issues and here when I printed it, so I got all my four statements successfully downloaded. Let's say we have a user question which is how did Apple's net sales change quarter to quarter in 2023 financial year? And what were the key contributors to the changes. So basically we want to make sure to keep track of what changed from one quarter to another quarter and for each quarter what are the key contributors. So now we can use sub agents which can extract this information from each quarter. And then we can use a super agent based on OPUS which will combine all these insights together and give us our final response. So in this case, we first need to provide a prompt to each of our subagents and for that we are going to use our bigger model which is opus, to create a prompt for these subagents. So the function is generate Haiku prompt which will receive the original question from the user and then it gets this prompt. So this is basically controlling the super agent behavior. The prompt is based on the following question. Please generate a specific prompt for for an LLM sub agent to extract relevant information from earning reports PDF files Now, each sub agent only has access to a single quarterly earning, right? And output only the prompt and nothing else. We just wanted to generate a prompt for us that we can feed into these smaller sub agents. So here we are using Cloud 3 Opus. Okay, so once we run this with our initial question from the user, here is the prompt that it generated that we need to run through each of the sub agents. And the prompt is extract the following information from Apple's earning report for the quarter you have access to. So as I said, each agent is going to have access to only one quarterly report, and we want this information from each one of them. Okay, but how do we run this? Right, so for that we are going to use this function, extract information which will receive a single file as an input. Then we use the PDF to base 64 PNG converter to convert it into a list of images. And then we use the smaller model which are going to be a Haiku as sub agent. But here we provide the input image as a source, right? And then we also provide the prompt that OPUS generated for this sub agent, right? So again we have the bigger model which we use to generate a prompt for these sub agents. And each sub agent is just extracting information from a single document. Okay, so basically this will help us extract the information from one earning report. And, and then we'll have to run this four different times so that we use these sub agents to get information from each different quarterly report. So we put everything together in this process PDF file. We were going to just provide the file path and that will create a sub agent automatically. Then what we do is we run this in parallel. So we execute this in parallel. We have four different calls to Haiku API Endpoint, and then we combine everything together. But in this case, we actually use these XML tags to separate the financial information from each agent. And this is actually an extremely powerful feature that Cloud uses. So within the prompt you can use these XML tags which will separate different portions of the prompt. And in this way the model will be able to actually pay attention to these different portions. So if you look at the output, we have financial information for Q3, then we have a separate section with XML tag for Q2. Similarly, we have the same thing for Q4, and there is one for Q1 as well. And for each one of them we have the product segment, sales, how it increased from one quarter to the other. So all the information that we need is extracted by each of these individual agents. Now we need to collect everything and send it back to a bigger model. Which is going to be again our super agent. So in this case, the prompt for that is based on the following extracted information from Apple's earning releases. Please provide a response to the question. So we pass on the original question from the user. Then we say also please generate Python code using matplotlib library. And we wanted to enclose the code within the code tags. So again, this is a perfect example of using these XML tags which will really help you separate different sections of not only the input prompt, but also the response that Claude 3 generates, which is extremely helpful. And we provide our whole response that was collectively generated by these four sub agents to this super agent. Okay, and this super agent is based on Cloud 3 Opus, right? So now it will look at the earning reports from different quarters and it has this super prompt that based on which is supposed to extract information from these earning reports. Okay, so then we set up our API endpoint client, we run that through the model and here is what we see. So there are two sections. The first section is actually talking about what happened. So it's basically the financial information that is extracted based on the financial reports. And the second section is a code that will let you generate a plot. Okay, so then what we can do is we can actually use these tags to separate the code that we can use to generate plots from the analysis that Cloud 3 has provided. Okay, so there is a code here. Basically it will extract code in response. So it will look at these tags and based on the tags it separates the code segments from the response segment. So here you can see this is the actual response. And now using that code and we have a pretty nice looking plot. So it shows that the net sales actually starts decreasing from Q1 up to Q3 and there is an upward take in Q4 2023. This is probably related to the holidays season or maybe even related to the release of new products. So this was a quick example of how you can create your own agents and sub agents with the help of Claude 3. And you can also use the vision capabilities. So in certain cases, if you have more complex PDF files, I think definitely something worth exploring. If you are interested in extraction of information from PDF files or other documents, RAG is another approach that will really help you. If you are interested in learning more about rag, I have an upcoming course you can sign up for. That link is going to be in the description. And if you want to use CLAUDE for your own business applications, you can reach out to me. I do offer advising and consulting services. Details are in the description of the video. I hope you found this video useful. Thanks for watching, and as always, see you in the next one.