Creating an AI Agent for Financial Report Analysis

Hi there, Data Scamps and Data Champs, this is Richie. Now before we get started, I just want to say it's pretty amazing. The comments here are started by someone called Starter Joe. That felt very impressive to me. So hi Joe. Hi to everyone else who's joining. I'm very excited for this session and the reason for that is that AI agents are the hottest of hot topics right now. And the idea is very simple. Every job has boring bits that you don't want to do. So getting software to do those bits just saves you a lot of tedium. And the idea of automating stuff with software has been around for decades but now that we've got decent generative AI it means there are suddenly a lot more tasks that are more easy to automate. And in particular I feel like report writing is no one's idea of a good time. So that's what we're looking at building agents for today. Maybe that's a little bit harsh. I know like a lot of people who business analysts, they have like reporting is a fairly large component of their job but actually having some AI assistance there is going to take out a lot of the, the grunt work and mean you can focus on more of the fun stuff. So that's what we're doing today. We're going to build some agents for financial reporting. One warning though. So the agents we're going to build, they're a little bit more sophisticated than the things that we usually do in these code longs. So I'm suggest today is probably going to be more of a watch along for most people rather than actively typing code. So as you're going through this I would say focus on like understand, understanding like what's possible with AI agents and thinking about how you can maybe use them in your own job, particularly if you write reports. And then also yeah I would think about like trying to understand the architecture like the code patterns used here. You can always get into the details and figure out how the, the individual lines of code works later, later on when you're playing around yourself. So uh, with that said let's introduce our guest. Uh, so today's guest is J. Putatunda. Uh, she's an award winning lead data scientist and director at the financial information company Fitch Group and she uses generative AI and uh, NLP to create products for credit ratings. So uh, Jita was listed as one of the 100 most influential AI leaders in the USA in 2024 by AIM Research. She's also the New York City chapter lead for women in AI and an ambassador for Stanford's Women in Data Science Group. So highly credentialed guest today. Please take it away. Jaita.

Thank you very much, Richie and thank you very much Reese. Thank you for giving me the opportunity here and for the very kind introduction. I think like you very correctly said and welcome. Welcome everybody, whoever joined, thank you for being here. I think AI agents are a topic that everybody wants to learn about but we want to start in a way that is absorbable or that's implementable and it's not like too complicated infrastructure because of which we feel like oh, it's too hard and maybe not for me. So that's why I'm here to build it step by step using open source tech. I don't want to give an introduction again for myself. Richie did a great job. So let's move along. We have one hour and we have a lot to cover. Okay, next slide please. So, building financial agents, right, we're gonna talk through couple of slides here just to give a little bit of background because I feel if we start coding without like you know, understanding why we are doing a few things the certain way or what the workflow looks like or what the problem we are trying to solve, it sometimes leads to unexpected expectations because we all know with tech, especially with generative AI, expectation is through the roof. That is just the magic pill that we need to swallow and everything is going to solve by itself. So that's not right. Let's build the solution step by step in the right way and then try to solve for it. So we're going to talk a little bit about AI agents versus agentic AI. I'm sure you're being hearing this comparative terms back and forth so we'll understand a little bit about that and then we'll look at a couple of examples from the financial services domain that we feel where agents can definitely add value. And then we. I go through the code Colab notebook to actually show you like four different kinds of agents that I have. Like Rich said, I think for this one I would really like you to take the time to listen in and without like worrying about the coding side of it because you are going to get the collab notebook, it's completely open source tech so you can run it on your time and you can always feel free to reach out to me if you have any questions. But for today take the time to understand each of the step and then we're also going to talk about evaluation frameworks because without that there's no way we're going to put any agent or any services into production. Before I start. I think for every tech there's some kind of hype cycle that we go through with generative AI it was peaking, then we inflated and now we know exactly which areas it can add value for us. So similar for agent Aki, Everybody wants to build agents, but let's identify use cases that actually adds value to the business rather than just trying to fit like an agent. Even when like where a traditional ML model can fix your solution. So let's not over complicate and over engineer. Let's go problem first approach. So AI agent components, right? What are these are the three baseline or basic components that we're going to talk about today. You as a task or a user have a request that you want to send to the agent and what the agent is doing doing is just helping you automate some of the processes or like you know, sub layer of work that you were doing maybe sequentially or like parallelly, but you were orchestrating it yourself or I can say like manually orchestrating it. Now it is possible to kind of chain together some of these components so that the entire workflow runs a little bit in an autonomous way. But sorry, but I would use the word autonomous in a very careful way. It's not that we are automating everything end to end and there's no human supervision. That's completely not it. Because at each layer and at every evaluation process, step and framework, there needs to be human oversight. And we're going to talk about that in the framework part of it. So we're going to talk and look at what tools can help us do. A couple of examples being you can do external tool calls. Say you are reading a newspaper and you don't understand a particular word. What do we do? Look up the word in Google instead of manually doing it. For organizational workflow for your personal use case, please feel free to Google. But for the organization workflow, we are going to use a tool calling function that would help us look up some of these concepts, write a summary, identify some links, all within the code without you being creating a separate workflow for that memory is again the database, the knowledge base, the context that the large language model has about your problem, about your instructions or whatever you have given it beforehand, or the chat history that you have been conversing on. All that comes from the knowledge base and the memory base. And then planning. Planning helps the agent break down complicated instructions. So if you are saying that hey, book a ticket, that's just one goal, right? So how to go about 0 to 1 to actually go book a ticket. That's the planning component of it breaking down step by step of the workflow and, and then kind of going one at a time asking or like utilizing tools or memory in whichever flow it's required. And then the task is complete. It's as simple as that. There are a couple of things this kind of shows like all the components you see on the right here, interaction with the environment. So the agent's ability to respond to external functions is nothing but like the tool calling that we just discussed. There's of course data collection because you need to gather information from various sources. It can be your knowledge database that you have at hand. It can also be Internet search like we talked about, identify the latest 10 papers that were released like yesterday. And this is actually a personal agent that I have because in the space every day there's like thousands of papers that coming up and I want to only read about a couple of niche content genres. And I have set up a very small agent which kind of pings me every day by giving me that These are the 10 top papers that were released in the last week. And then there's of course task performance. We want to achieve a particular specific goals and the performance is measured by, you know, the evaluation framework that we're going to talk about. On the left, what you see is basically the, the characteristics of the agent is that we have a little bit of autonomy. Like I said, we will independently operate at each step, but there will be some kind of human intervention at each layer to make sure the results are right or we are going the right. And you'll also see that in the code like we talked about planning as well as reasoning. Reasoning has become a big part of solving complex problems because we have seen that sometimes when you ask a very big question or there's a very big prompt with a lot of instructions, there tends to be situations where the LLM might miss some nuanced components. So we need to make sure that when we are doing this planning part, we are breaking down components in a way where the LLM would be able to pick up reasoning and say that hey, my step one planning was to, for the ticket booking example go to a website. I went there. And after that my process is going to like look for the keywords. So if you want to go to like, I don't like, say maybe you travel to Canada, you're going to look up like airline tickets for Canada. And that's how the reasoning goes step by step in the in the planning process to set us up for success. One last point to highlight here, AI agents versus agentic AI. What we're going to build here today is the AI agents for very specific tasks. So we are task oriented, we have limited autonomy here. And whatever the agent is learning is the baseline learning that it's getting from the instructions as well as the knowledge base that it has access to. It is not automating or it is not auto optimizing at this point for the at the AI agent level, its output or its performance. That happens at the agentic AI where I'm sure you've come across, you know the OpenAI that released Swarm, that is kind of a more goal oriented rather than a task oriented framework. It's higher in the autonomy side and it's also advanced in terms of like how it kind of relearns and then applies automatically and optimize its performance on the go. That is not it. So I've seen people like talking about AI agents and agent AI as if they're the same concept. So I wanted to make sure that you understand this out the bat. So how do you choose? How do you choose which is your best approach? And what we're going to do today is, is we're going to take a problem. So we go problem first, not fancy tech first. I know our innate human nature is to jump and get into shinier tech. I do that all the time. But I think when you try to solve a specific problem from the business side, you're very cognizant about how it's actually helping the business solve some issues rather than what tech you're using. So AI agents, we're going to start looking at how we define. This is a very generic framework of how you start thinking about it. Let's not go into it, but talk about the applications in the financial services. Right. If you see the chart here, it talks about some use cases that needs to have very high real time response like customer service and those kind of automation agents if we build. But there are some. It might not need that split second responses, maybe compliance monitoring. It can take a while to identify all the resources, all the knowledge and then kind of come up with the solution. So mapping your use case and identifying processes to kind of see which works when really is helpful here. So let's talk about couple of these examples. So streamlining operations, it can help you with automated repetitive tasks for customer engagements. Like if you want to have a very tailored financial advice specific to the customer and you have all the data that's sitting about the customer, about their preferences, likings in your database. You can map that to create that adaptive, a robo advisor kind of a agent that caters to particular customers process. Yep, that's about it I think. Feel free to look at this when you have some time or go through it and let me know if you have any questions. Let's talk about today's use case. Right. This is our use case that we're going to try to solve for today. What we're going to do is that we are enhancing some of these financial analysis that we would run on various PDFs, complicated reports and as you know, in finance, each company releases thousands of documents. It can be an annual report, a 10K ESG report. So how do you make sure that you are creating a pipeline that can handle all these various document types? And we're going to utilize ESG report today because it's very dense in content, in text and we're going to see how we can work through it. This is the agent workflow. Let's not worry about that too much. It basically talks about the four sections or the four agents you're going to see in the code you have an input layer where you're inputting any of the financial documents that you need. And then there's the user's query or the request that's coming in as a prompt. This area of a rag agent that you see is what you're going to start with first because we want to optimize some, some database that holds all the chunked and embedded information from the PDF and build this relevance engine to understand or process the user's query. Then we have a research agent where we will work through how do we search, do a web search for a particular query, how do we create some context around it and then aggregate and create a report. This report generation part also helps us kind of get at some level of understanding of what we found from our initial searches and then sets up for a process of getting started with the process. All the solution name that you see here is what we're going to use for our code base. They are completely open source. We were saying in the beginning of the process there's no paid services, so you don't have to worry about getting paid paid APIs. You need to create some APIs and we'll go through that. But there's nothing that's made here. So we'll talk about. So let's start coding. Right. We've talked enough so I'm just gonna shift to my next screen. Okay. Okay. Can we make this bigger? I think we can make this bigger. Okay, that's about it. I think I should have. Okay, this is better. Okay, so let's start. So the code base talks about the goal. What we are going to do today, we're going to build a couple of AI agents that can handle various financial tasks. We are creating four agents here. One for RAG based query, which is RAG is retrieval, augmented generation where you have a vector database. You have a question. The vector database holds all the information about the documents that you have passed and you can query that through to get your answer. And we will see how. We are also going to build another one like tool calling for web search. The other one is for a deep research. I think this deep research we are going to do for a couple of stock processes and then we also have evaluation where we are going to do LLM as a judge agent just to see how another LLM agent kind of scores a particular agent's answer. And that is very crucial because if we want to scale, sometimes manually reviewing, there's always a human intervention. But manually reviewing code or output can get tedious. So we need like one layer of automated outputs that we can then review with our subject matter experts. So very quickly I want to show you because we need two API keys. One for Groq, which is the fast OS open source, like LLM Inference. It's absolutely free. What you have to do is just go to groq.com you can create, you can create, what do you call like you can create an account. Once you create an account and log in, it will look like something like this. You will see that there's a get started overview with a lot of documentation, a lot of, you know, cookbooks and a lot of. You can also join the developer community. They're very helpful. If you have any questions or you get stuck, you can talk about, you can look through that. What you need to do is create your API key here. I already created my API key here called Finance Agent. You've seen, I have a couple of calls there. If you want to create a new one, just say create new one. Enter your details and it's going to be here. We'll use this secret key in our notebook so that it can call all the models that we would like to use. If you see models here, these are all the production open source models that are hosted in Grok that you can directly utilize without having to set them up on your own system or own machine, which you can still do. It but this just gets you started with an easy access, right. It reduces the stress a little bit when you're just starting to experiment. We're going to use a couple of models here. We're going to use the Llama 370B as well as the Llama 370B as well as the instant model for our evaluation agent. But you can choose and test with whichever model you want to use. There's also Quinn Deep Seq all mentioned here. I think they're still preview. They may not be like in production yet, but start with something simple like the Llama 7B, 8B or 70B, whichever you're more comfortable with and then we are good to go. The second framework where we need an API is Agno. They are a fantastic open source framework for creating or like you know, chaining the entire agentic workflow. If you have not heard about them or look them up, please check them out today. I think they used to be called as Phy Data before but they have rebranded as Agno and I really enjoy learning from all the cool things that they're building. What you're going to do is you're again going to create an account from here. Once you do, I think once you do. Why am I not logged in? I'm logged in once you do. You see that here's an API key that you can set. This is my API key. You can create your own or you can refresh and you're going to be able to access. One good thing about is that you also have access to their playground. So if you say don't want to run the code that you have from this session first but want to rather do a playground demo there, you can also check out their demo agent and go through it. There's also the docsigno entire documentation. You can go to the look at what their key features are, what kind of concepts, like what kind of multimodal agents you can build. It's very, very descriptive and you should be good to go on that aspect. Let's come back to the code. Those are the two API keys in Google Colab. How you set environment variables through here. Don't see mine. I think it's already blocked out. You can set your API keys here like this from the Secrets tab, add the value, hide it and how you call into your system is like this where you said that I was just testing with different keys so it's still there, no worries. You call your environment variable the Grok API key and phy data that we just saved in our secrets and you get the userdata.com and then I just printed it out saying that, okay, API's keys have been set. Sometimes while running we see that there might be API error calls. So another good way to check, and I've just commented it out, but feel free to run it, is that to actually make sure if the Colab notebook was able to fetch the key from the environment, you can also run this command and it will say that if it has found it or if not, it says that Groq API not set that. Just for your peace of mind that you're okay with all the API keys, let's talk about the first agent. What is the first agent? We're going to run a functional tool calling capability, which is a web search. And we are utilizing this solution called DuckDuckGo. It's a toolkit. It's like a private searching interface. There's some other components as well. And you can find everything regards to that in the Agno's documentation. But let's start. What are we going to do? We're initializing the first agent by calling our model that we talked about. We are creating an agent component. We are calling our model that we want. That's the grog one. Why did we put ID as llama70b? Because that's exactly how it's mentioned here. This is the ID model id. If you see, you choose whichever model ID you want from here and you use it here for tools. We're also using DuckDago tools as well as newspaper 4K tools. If you see here, it's just a descriptive name for the toolkit. It just helps you look for news. What, what description I'm setting for this particular agent is that, hey, you are a research analyst in the financial services domain and you can change it up. You can say that, hey, you are a chef or you are a food blogger, or you are whatever you want it to be. So whatever description and instructions you set for the agent becomes its context. And that's how it kind of runs. And I have added a couple of layers of expertise that the particular agent should, should think about when it kind of answers. So they should have like a deep, deep investigative financial research and analysis knowledge. Why is this important? Because I think, I think the goal is you're setting, you're setting like a standard or a bar that, that at which level the agent should navigate or implement their answers. And why am I saying that? Because you will see in the evaluation framework, we will use exactly these descriptions and instructions and Checking the output based on that, saying that do you meet the instruction requirements? If I asked you to say an instruction is that if you found any dollar value in a particular text, translate it to dollar, like spell it out. So that's an instruction and in one of the output we see that the LL missed it or it was still showing up as like a dollar sign and not as the script. That's going to be one of our evaluation checks and balances as well just to make sure that it's operating with the way that it should. So make your descriptions and instructions in a way that's, that's executable as well as we can evaluate on it. So it has like data driven reporting and visualization expertise. It can also, you know, understand trends, whatever you can. Right. I added some of these just to make sure that it gets some kind of baseline understanding. So like ethical practices as well as give us like a balanced perspective. Not like you know, swaying to the two extremes, but give us your balance perspective. In the instructions, what we are doing is actually breaking down the task that the agent needs to perform in a step by step basis. Like I was saying, you can skip this or you can give one big paragraph and it would still run potentially. But when we try to build large scale solutions that has like, you know, I don't know, like three or four pages of instructions running, we cannot use that long of a context as is it needs to be broken down. And that's all the concept of prompt optimization. Right. I think that's another huge topic that we can cover in another session. But prompt optimization is very, very important to make sure that we are being to the point we are not using repetitive words or we are giving instructions in a clear format and like a step by step sequential task based framework. Rather than giving like a big chunk of information out there that we don't even like if a human reads it, it's going to be hard to understand what's the flow of the workflow. So we broke it down into research phase. That okay. Based on the user's input or the question, the prompt that it comes, you have to give five alternative sources. There's going to be analysis phase, you extract and verify some of this information and then you. And then in the writing phase you start crafting different sections and this is the sections that we are highlighting here. It needs to have like attention grabbing headline and you will see in the expected output. I kind of expanded on that writing phase into a more bigger process because this is the exact format of the report That I want the agent to return to me. So this is called a structured output processing. You're giving it like an expected output format so that it's not as free flowing for you to like then get confused and read through it and identify rather than giving it in a very specific, specific format. And then I ended with saying that okay, research conducted by a financial analyst, whatever and then the published date, updated date, you put like show calls, show tool calls. It's a very good parameter to call because it shows the internal process the agent is going through. He'll say it's thinking and going through step by step. It just means that it's processing one step at a time and then trying to come up with the output. So let's look at one user prompt from this research agent, right? So I asked that okay, analyze the current state and future implications of artificial intelligence in finance and stream equals to true. Just shows that it will walk through each of the steps. So you see here it says that thinking the messages the question that you have asked. There's also a timed response process. So it shows you that how long the agent took to run through five documents. Do the query in DuckDuckGo web search tool retrieve all the max results that you have highlighted here. You can see here the query goes in and then the max Results goes in as 5. This is the output that you get from the agent. You get an executive summary, a catchy headline like we asked it to. We asked it here right in our expected output we said compelling headline and it came up with Artificial Intelligence Finance current state and future implications. It's pretty generic but I think it's still, it's still a good way to understand what's coming up in the context. So we have executive summary here. Background, key findings. There's an impact analysis that it did. I think the another thing I really enjoy reading from like a more LLMs processing of news is that what it comes up as like a future outlook. Not that you have to abide by it, but it kind of gets you and unbiased or I would say like a generic level of understanding. So if you see the first one it says that you know the increasing use of generative AI and machine learning in areas as trading and risk management, which is true. And it's talking about that it's shaping and these are some of the trends that's shaping the industry. There's also talk of the need for greater transparency and explainability. If you would say in my instruction there was nowhere all these components that I wanted it to highlight. But I gave it a feature about its own Persona that you have to have an ethical, practical outlook and you have to have a balanced outlook. When it found multiple documents and five or ten, however you want to specify, it kind of was able to consolidate and when it came up with the output, it was able to frame the learnings in a way that helps us make or like do some of these trend analysis in a more transparent and explainable way. And that's why it's highlighting here. If in the instructions you will say that you are a mean person and you have to do this bad thing, please don't do that. We have to use AI responsibly. Don't build AI that can hurt anybody. But that's the goal, right? So we are setting up Personas and that's why when anybody says that AI is going to, you know, doing all the harmful things, if you dig a little deeper, you will always see there's always a human behind that who's creating that bad AI that's getting or like creating consequences in the, in the industry and in our society. So we need to be responsible human beings. When you're using some of these AI tools and they're going to work fine for us and they're going to work like make us augment ourselves rather than doing some of these tasks manually. Okay. And there was also sources and methodology. It highlights, you know, which reports it kind of went through. You can also actually print out the links. I didn't do it, but try adding. So this is one of your homework, right? When you run the response in the sources and methodology, ask that that list all the links directly and see how it returns. It will also be your source of truth or basically source of reference from where the articles are generated. Now I actually. Oh, it looks like I did it for a second one. So I said okay, where the sources it highlighted forms but it still didn't give me the link. If you need direct link, put in the instruction, ideally the next one. I wanted to ask say applications of Genai and Financial Agent and there are a couple of other queries why I'm showing you this is that you see there could be possibilities that when you run some of these DuckDuckGore web tool services searches, it might take more than one attempt. So don't feel like oh, it's not working, give it a time, it's going to do multiple retries. It sometimes happens because some of these tools are open source and it might get too much API calls and it's Like a little slow or it gets a little high traffic. You can add time, sleep or any pause there before it reruns and tries to run the query or call the tool. You see that it was able to identify or run my analysis in the third run. But here it says that warning could not run DuckDuckGo. And this happens because, like I said, because it couldn't reach. There was a rate limit because if you're doing too many calls at a time, and that's an issue for open source tech. So you have to use it with a little bit of grace, but nothing to worry about, it still runs. And there are a couple of more other examples. Please feel free to look through it. I'm going to move on in the interest of time, but hope that was helpful. That shows you what you can do. You can query up names, technologies, anything that you want to learn about and get a quick report out of it and then look through the sources and. Okay, let's talk about the Agent two. Right, so Agent two is going to be more of a knowledge based query capability which is RAG retrieval, augmented generation. So all that you saw from the previous agent there was knowledge, but that knowledge was not stored anywhere. It was just coming from an Internet search or like on the go search. But think about if you are like somebody who works for a particular use case, you have tons and tons of data already saved for that particular data use case. So. And you want to utilize it. So how would you do it? You would do it through rag, which is a very common, I think technology. Right now everybody uses rag. We are trying to make advanced drag. Now we are doing agentic rag, which is what this is. And there's also causal rag, which is also one of my favorite subject. But, but that's for later. Another thing you need to do and you don't have to do anything, just run this. It's very easy if you are running the Docker in your system you can just open up the terminal and just run the Docker image. But in Colab it takes a little bit of processing. There is this concept called udocker that was created on top of Docker to run in Colab. You can just show code and then just run this cell and you would see that docker in colab 1.10 version would be installed. We're going to be using pgvector for our database today. It's an open source framework. One thing you would see in most of the tutorials is that even if they use open source framework, there's Always a call to OpenAI because they're using OpenAI's embedding process to kind of process the document and embed its information. But again, that doesn't serve our end to end OS operation pipeline here. So what I did is that instead use sentence transformers from hugging face which is like an open source embedding model and we will see how that set up. But before that we need to initialize our PG vector. It's very simple like I said from your local terminals, if you're using a Mac or Windows just do it there and then run maybe using a VS code instead of running the colab however you want to do. But for colab this is it. You start with installing the udocker. You allow root. This is a key tool. If you're not following this doesn't matter, don't worry about it, just run the script and it will set or make sure that your colab doesn't have any pre existing directories and it will just make sure that the database is set up. This is basically for running the PG vector container. This talks about the environment that you're going to set up, the volume where you are actually loading the postgres SQL database and where it will publish. This is the database URL. This URL DB URL is just copying this framework and this is given in the agro documents. We are just working around that to make sure that we are able to host our database in that don't need to change anything. This will work as is. Just make sure that your collab is working. It's connected to a Python 3 compute like you see here before in the below and then you can get started. I've created like a max retry of 5 like before I said sometimes it can get stuck and port might be busy or it might be some kind of failure might happen. So always have a connection attempt multiple times it'll try and then create through it. Like I mentioned here, I applied a 30 second sleep time so it's waiting for the database to initialize. This is just to make sure that you are not calling the TestDB connection to see if the database was created before it's actually created. It's always good to have a time sleep whenever you are creating or calling any external tools or even setting up database connections here in the engine. Once we test the database connection, when we test if the database is ready for rag agent initialization, you would see that it says it was able to connect in the first attempt and it was successfully connected to PostgreSQL I want to highlight here that this tool may or may not be the right database to use for your specific use case. Why we are using this here today because it's open source. It gets, gets the task done even if it's not 100% accurate for retrieval. But it gets you to the step where you can start your poc, look through the documents, see what the outputs look like and then once you want to actually put it into production, you have to use some production grade vector databases like you can use that's available to you. Right. So I'm not saying this is the best, but this will get you started. So in the next process we are saying let's define a class called Document. Oh my God, I spelled Document QA wrong. Anyways, Document QA what we are doing is we are initializing an open source sentence transformer embedding model. This is nothing. But for all these things that we have actually used, everything is linked on the. Sorry, I'm just going to go up just to quickly show you so that you know where to look for everything that we use. It's mentioned here in the open stack text textile. Just click through the link and you can like read more about if you want to learn more about the technologies. Okay, so let's go back. Sorry about the back and forth. Just wanted to highlight that. Okay, so here since. And this entire class you can omit if you have access to OpenAI API key. You can just call the open API key and do the embedding. But it's a paid service. But instead of that, what we did is that we created this class. We created and initialized our class by saying that hey, this is our DB URL that you have to ping to. This is our chat model that you are going to use, which is again the 8 billion model that we're utilizing. And then we are creating this embedding model from here. Again, if all these class formations looks complicated, don't worry about it. What I'm trying to say here is that you can do it using any closed source or directly calling OpenAI. But this is just to make sure that you can run through the process process yourself. So it's doing nothing but just calling the sentence transformer, the paraphrase Mini LM model. It's a very powerful embedding tool and I actually use it for a lot of my production grade tasks as well. Because what do you call it? Because it's always like good to save some dollars wherever you can. If the task can be done with smaller And a little less powerful transformer embedding models doesn't harm at all. We are creating the get embedding and the usage calling the embedding model and then we are saying that print once it's initialized successfully, you would see once we run it. It says here that the embedding model sentence transformer was initialized. The next part of the process just talks about how we are loading the PDF URL. This is just a function where it would help the user input any PDF URL link. The PDF that we're going to use today is. Okay, let's finish through this and we look at the PDF document and how the structure looks like. So we're gonna upload the PDF URL knowledge base. This is actually part the function is part of the database. So please feel free to read through it to get more insights. These are some of the parameters that you can call the URLs itself, the vector db that you have set up and the embeddings that you have just created in the function above it. We're initializing the agent here where the agent will call the knowledge, the knowledge where it'll call the knowledge base that you just created. And then the CHAT model from the model, the GROK model LAVA model that we created. So we loaded the knowledge base. It says that okay, the knowledge base was successfully created and if there's any error it's going to show through it. So let's look what it look like. Once we run that, when we initialize that class into rag QA parameter here, it'll say okay, embedding model was created. You'll see that it might ask that you have to have a hugging face token. It's actually not required, but it's a good practice so that they also have the data that who's utilizing some of the resources. But you can just do as is. You see that all the model configs files, the sentence word config files, everything is downloaded. The tokenizer, these are all internal model weights for the sentence transformer model. So you would need this to kind of run your embeddings when we do it here. So now this is the link that or the PDF link that we'll be utilizing for our creating our knowledge database. So let's look at the document. Right? So this is nothing but a very public document. You can look up. It's like a ESG report released by Apple. The the goal of this here is that if you see there's like a lot of text right? There is report Highlights it talks about all the initiatives that it's running. It talks about, you know, the, what are the goals that they have or like what are the progress they have made met, you see achieved, Ongoing. Ongoing. So maybe one of the question you can ask if you're using this document is that list all the goals that are currently in an ongoing stage that gives you a quick access without maybe reading through I don't know like 113 pages of the document. So that's a plus, right? So yeah, that is the document. Feel free to read through it or like, you know, use any other document and see how it works. So once we run or like, you know, pass that in our function load prl, you see that it, it kind of prints through that what it does sequentially, step by step. So it's creating the collection, it's loading the knowledge base, it's actually reading the PDF documentation and it inserted multiple documents that it found in the knowledge base and successfully noted the the document. When you, this is a very key concept in Vector Database. When you see, when you see that there are, they're saying that 131 documents, it doesn't mean that there were 131 pages here. Sometimes document means like a chunk of that was processed from this big huge PDF because there is context issues, there's processing issues. If a page or if a paragraph say runs for I don't know, five pages. If there are something like that, it's possible that it needs to be broken down into chunks and there's a way to create custom chunking strategies that we do all the time in our advanced rag processes. But we will skip that for now. So if you can ask all these questions like key points in this report given five bullets. So it found like here are key points from the report that it has established some kind of safety program. It also requires supplies to analyze materials and believes its responsibility extends beyond its operation to supply chain, community and planet and all that. So basically you get like a very quick five pointer analysis of what this document is talking about. You can also maybe if you want to create a executive summary in hundred words because we know executive summaries needs to be precise and concise. You can also do that and say that okay, here is 100 word summary and then it talks about a little bit so you can review the output, ask more questions like I said, maybe ask what are some of the goals that's achieved and see how it works. And if we need to make more updates, maybe possibly that could be done. The third One is very interesting. I don't encourage anybody to follow and saying that very clearly that we don't need to. You know, actually, okay, I think we are running pretty tight on time. So let me cover one other aspect before we go into the this talk is that. Sorry, let me take this out from my screen. Okay here. So let's talk through very quickly of what we're trying to do here. We're trying to analyze couple of stocks and pit it against each other and look at market overview and do some key matrix analysis. Please do not follow this at all for any of your investment strategy. This is just for research purposes. Don't trust the AI robo parts without your due diligence. Just getting the disclaimer out there. And then we are asking you to create a report saying that you have to create an executive summary. You have to create a table for the data presentation. You have to have a clear matrix of industry averages. And like we talked about before, this is all the guideline for the report that the agent is going to handle. Now if you go here, this is a very small and quick and very easy agent to run. You say that okay, add date time to instructions. You call a show tools call and then you ask that okay stock run the process and it says okay, stock agent created ready to take user queries. You put in the query that okay, what's the latest news and financial performance of Apple? You see that, you see in the running process, it talks through each step by step process of what went through internally in the agent framework to accrue all the information and create this report for you. Right? So step by step process, get current stock prices. So symbol is AAplan. Get company info, get the historical stock prices, get analyst recommendations and then the company news. So it goes through four or I think five layers of components and sources that it's trying to aggregate and then coming up with the, the report in the way that we asked it to. So it's creating, it's creating executive summary the current stock price. You see it says as of February 11th, like 11th, it's this much. Please verify the data all the time. Don't like, again I'm saying don't believe in what the agents are giving in the face value. But it gives a good structure for you to get started with your research. Right? And that's the point. It's supposed to be there to augment your intelligence and not like, not like take that away. So keep that in mind always. You all, we all see that we have like historical stock Prices here that kind of compares high value for a timeline. Let's look at one other query before we kind of wrap and go to questions is that you can also do comparative analysis and say that okay, analyze the semiconductor market performance focusing on all these four companies. Because if you, if you've been reading the news like the, the, the, the competition as well as the semiconductor like business that's having an effect on AI as well as like the AI chip and the performance is like very, very huge. So that leads to a lot of market noise as well as like the movement of stock prices and what the future outlook in terms of AI growth could be. Because if you are prohibited by you know like the chip, chip, chip processing powers, AI can only do so much. So, so, so again similarly you'll see that, okay, we do company by company comparative analysis and then you see a very structurally created table where you do a market position comparisons and talk about future outlook for each of these. Right. And then you can run all the other queries and see how it works. The last part, the LLM as a judge doesn't is not like an agent in itself or doing some work. But it's basically we are using another model like I said llama 3 instance. We're using a different model here and saying it that you are a RAG system evaluator. We are setting up that in the description. We are setting up that you are supposed to analyze the output from the RAG input. These are your instructions. And this is very industry specific. I think this is, this is not industry specific matrix but it's very generic and used industry wide like faithfulness, context, relevance. That is the answer good enough for the context that it was provided. Did it cover all the important processes? Did it miss any important concept? So we're giving the instruction to the agent to kind of frame its background and context. And then we also, like I said, expected output. We frame it like this. Score the faithfulness out of five because that's the, that's the industry standard. And we are also asking for job justification and examples of what it missed and how it missed. So this will help us kind of make sure that if it's rating something as like 3 out of 5 or 4 out of 5, there is a way for us to go back and see why it was bad and try to fix that in our second iteration. And then we also asked for overall score and some key recommendations. So feel free to read through it. I did not use the query that was there before. I just ran like a sample use case saying that what are the key features of a transformer model? Gave a very generic answer and missed half of it because I just said that they're characterized by self attention and didn't give much of a context. So please feel free to test it out with your data or like your questions and context and see how it works. But this is how the RAG evaluation report looks to get you started. It'll talk about okay the faithfulness because if you see I didn't put much in the context itself so it doesn't have much to compare against. And it gave like almost like good scores here. Completeness. I think it gave two because if you see here it talks about key features mentioned where only self attention but limitations were not talked about and gave more comprehensive overview of transformer models. So this is just like a baseline evaluator. You have to make it more industry specific. You have to make it more, you have to make it more custom to your use case because that's how we all do it. Like all evaluation process we built is custom to our use case. But this is just to get you started before we go into questions. One last appendix here. These are I think one of my, not one, I would say my recent favorite papers that I really enjoyed reading. So if you are interested to learn more about the space, please feel free to go through it. There are a couple of other free courses one starting, I think it already started last week by hugging face. If you're interested to learn more about the agent space and how you can utilize, please feel free to read through these resources. And Aishwarya my friend is like a genius. So she also has like a Fantastic guide in GitHub. Feel free to check out her resources as well. Okay, so that was all about it. I'm going to pause here, should I stop? I think I'm gonna just go back to my slide here and if you have any questions.

Wonderful. That was very, very cool stuff. Yeah amazing agents and yeah thank you for talking to them. Just since you mentioned Aishwarya, she's actually doing a code along next week. So next Thursday I'm gonna be joining.

And cheering her on from.

Cheer from the audience. That's, that's wonderful. Yeah. So next week we've got a three webinar session on three webinar sessions on AI and the enterprise. So we're covering everything from like AI strategy to like actually building stuff putting into production. Uh so yeah, uh, Aishwarya is gonna be one of the guests for that. Uh, maybe uh, if, if uh Reese can Give us a uh, a QR code for uh, joining future webinars. All right, with that, uh, we got a few minutes for questions. A lot of questions for the orange. So actually so someone with no valves in the names something or other is talking about PGvector. So that was. That's the tool for rag. So it's a PostgreSQL plugin saying it's got bad performance for vector similarity search. Which is true.

Yeah.

Why do you use bad tools and how do you get good performance with bad tools?

I mean like I. That's absolutely right. I think I've never used PGvector in my life but that's because there were other paid vector services that I had access to. But like I said, please feel free to try out Chroma db. Another reason why I use that is if you see in Agno, can I change my. Can you all see still see the screen or not? Sorry about that. I think in the Agno database if you go through all the. Where is it? Where is it? Vector dbs that it okay here all the Vector dbs is it canned. Here? So it kind of handles Chroma DB lancb I have used Chroma DB extensively before Milwaukee as well. QTRENT as well. And single store is of course good as well as MongoDB but this was easiest to implement so maybe is the way. But it's not as bad as you think it is. If you're using like small, like single or up to at least four or five documents it still works pretty well. But if you want to push I don't know like 100 documents then it might like lose performance in there. So that's why it start said it's just to get you started. Feel free to swap out this with anything. Experiment how it works on your data set and if you have like a actually paid service where you can use like you see you need a pine cone API here. I think it's pretty standard. Pinecone is great for production grade Vector db. So use accordingly. That's, that's what I can say. Yeah.

So it's actually kind of interesting because data data at least in Data Lab we do use PGvector for some use cases where performance doesn't matter. Reasoning from the engineers was if you're storing user data then under like gdpr, like the EU data regulations, if you use like any data service you've got to like register this new provider and register the technology. It's like well we're always using PostgreSQL it's no extra paperwork to use. PGVector.

Exactly.

Whereas you use something high performance like Pinecone Milwaukee. It's like.

Oh, exactly. So only if you are in production, think about your performance, but don't really. Doesn't really matter if you're getting started and testing out. So don't worry about that too much.

Yeah. So tons of engineering decisions beyond like pure, like performance. Okay. All right, so.

Oh, you can take the screen away. Sorry.

Yeah, there was another question. So this one comes from Pratic to talk about when to use Agno versus Lang Chain or Lang Graph. Yeah, talk us through the design.

I mean, I mean all of us, like, all of us started using. And I'm smiling because I think I know Pratik, so. Hi, Prateek. But. But my concept is like when I started experimenting or when the agent take or the frameworks were coming up. Of course, LangChain and Langgraph, even Llama Index could be a good place to start. But I think we have to think through or based on some of my very personal experiences that we saw that I had an entire code base up on one version and then they tweaked something that was not updated and it crashed. I had to go back have. I feel. I feel Agno or like Phy Data has done a decent job in making sure that it's very comprehensive, end to end. It doesn't break as much. And it's just been my, like a developer experience that I really enjoyed. It's my personal choice. You can use any other frameworks, but I feel they're very good end to end. They have like a playground holistic process where you can. And you can also like raise their feature request. I think you can do it with everybody. But I feel they're very responsive and I kind of enjoyed the framework and that's why I chose it.

That's a very important thing. If you enjoy programming with one particular thing, then use that rather than using something you don't particularly like. Yeah. And there's a certain amount of personal choice in technology there, I think. All right, so super. Let's get rid of that one. And there is a nice question from starter Joe. Where can I learn deep data science? So appealing. That's deep learning data science. That was gonna be my answer as well, unsurprisingly. Yeah. So we just basically have an almost infinite number of courses on data science. There's plenty of stuff on deep learning and we've actually just launched some tracks, career tracks on if you want to do AI engineering, both like From a data scientist point of view on, from like a software development cloud point of view. So yeah, building AI applications. There's two different tracks there. Definitely worth looking into. All right, let me see. Oh, there was one more question. Oh actually no, before we get to that I have a question for you. So we built four different agents there and some of the, they're fairly sophisticated. What's the sort of simplest agent you might want to get started with?

I think it would be the, the stock analysis because it's nothing but just calling the Yahoo Finance API. I think even the, the if you, if you would like to even do the web search part first because that's nothing but you're just asking the question like you would do in Google but rather to like a more optimized and autonomous process via the Go. You can and like I said like look through the Agno database. They like the data documentation they also handle or they also support other web search tools. So try customizing it, building it directly, doing that. But I think, I think yeah, start from the, maybe the stock part or even like the duckdoggo.

Yeah idea. So it's basically just like, it's like you're doing your AI chat but then it's like oh well, we'll just throw in like a web service tool or calling an API here and there and gives you a bit of extra stuff to play with with. Okay, nice. And when you've been using these at work, have you noticed any benefits? Like what, what have you sort of got from this? Like how have you, how have AI agents helped you?

I mean like I said like, and I have to repeat it again, like in all financial services there, there's no way that we automate everything end to end because the consequences are too high if you get it wrong. So there has to be very specific evaluation frameworks, guardrails around it. But I think to get us started doing some of those initial rag based document extraction so that if say a user has five or ten documents it needs to read to get to a judgment and each of Those document runs 100 and 200 pages, it's not possible, it's time consuming, it's too much effort. But if we can get them to like certain layer of information that's nicely formatted, all the data is extracted and with each extracted data there's some kind of source. And even with like all the chatbots that you See Now Like OpenAI, like Claude, everything goes back to the source. So if there's a way that we can review the source make sure that it's trustworthy. That's how you kind of also build client trust and say that hey, it's not like absolutely wrong but give us more feedback how we can make it more custom and then you add more custom features that don't use one of the very generic example, one of the custom process we have or I have built in some of my other side projects as well is that don't use quotations from a direct particular person into the report. That's not big, but this is big. That is big for the client or the end users that they don't want to look at personal quotes but rather look into the data. So more quantitative focused versus opinion focused if that makes sense. That's how you build your custom nuance tricks into the agent workflow.

Okay, that's fascinating. And I guess one more follow up to that. If the, if the document summary you generate it says something weird, you might be like, well I don't want this true, it might have hallucinated. Is there a way to go back from the summary to the original report and say okay, it came from this line or in on this page?

Yes, yes, we can do that. I think what we are trying to right now do is break it down into components saying that if they in the executive summary, if we are. And that's. That's also the power of the extractive summarization part. Right? Sometimes we want extractive summarization. What that means is that we want directly that line as is don't change it, don't, you know, rephrase it so that we get the, so that we get the quantitative mentions of particular matrix that we need as is. And that goes into our evaluation framework saying that that did the summarization mention a particular thing about carbon footprint etc and when the actually the rag process looks up into the database and sees that there is a matrix about carbon footprint, there could be 500. But for in the first executive summary, whatever match number can we do, you know, entity match, entity match to exact numerical match. It's pretty easy to do. But if it's more abstractive then we come to more of like a similarity or a context mapping that is it actually talking about the same thing. So it kind of goes back.

So I thought that was gonna be like oh yeah, it sounds fairly straightforward. But now you've mentioned that it seems like there's actually quite a few different things you need to think about.

Always the way is a make or break for your use case. If you don't have a strong evaluation framework. It's gonna wreak havoc in your life.

Okay. All right. So, yeah, use these agents with care. Brilliant. Before everyone dashes up off the way we can keep these codalongs free is we take payment in likes and shares video. Please do subscribe to J Camp channel. Please do share this with all your friends, colleagues and feel free to reach out people on the street. Yeah. Excellent. So thank you once again, J. That was like very, very cool stuff. I very much enjoyed that. I felt like we could have carried on for like an hour at least.

Yeah, me too. I can talk about it endlessly, as you can see, but cool.

All right. Yes. Thank you to everyone who asked a question. Thank you to everyone who showed up today. Come back tomorrow. We've got a very exciting session. It's going to be deep sea car 1 vs 01 vs 03 mini. Like all the latest fancy models. We're going to see how they perform on reasoning tasks. So come back for that. And do come back for the AI and the Enterprise series next week. All right, see you all.