LLM Agent Fine-Tuning: Enhancing Task Automation with Weights & Biases

Sa.

Hey everyone, My name is Diana Chan Morgan and I run all things community here@deeplearning AI. Today is our last webinar of the year. We've had so much fun doing so many different workshops doing with all of our different course partners as well as other exciting ML community partners as well. Today we have a very exciting workshop and event to finish off the year surrounding fine tuning LLM agents for application and automation. Today in the session you'll discover cutting edge fine tuning techniques, things like Master Lora for low rank, adaptation metrics and logging mastery. Maintaining comprehensive logs with weights and biases, unveiling weights and biases, debugging power, learning how the traces can be utilized to pinpoint issues, prompting your way to success and hands on evaluation with weights and biases for today's session. The session will be recorded and we will send out the slides afterwards. And now we have to introduce our host. So today we have Aneesh Shah who are so lucky to have. Hey Aneesh.

Hey Diana.

We're happy to have you here. Anish is passionate about bringing machine learning to life as practical products. His experience spans traditional ML, deep learning and recommendation systems. He's an active speaker and developer dedicated to democratizing ML through tools, lessons and resources. Happy to have you here today.

Thanks Diana. I'm super excited to talk about today's talk, which is about LM agents, one of the most exciting things that's coming up for 2024 and we have a lot to talk about, so let's get started. Awesome.

Well, take it away.

Cool. Before we talk about agents, we need to talk about the current state of chatbots as we know it. To hand wave over this really quickly for those of you who are tired of the RAG demos that you see a lot of nowadays. Very simply, RAG is a framework for how to incorporate proper context into how you interact with the large language model. So for instance, in the case of ChatGPT, when you ask a question relating to financial documents, medical history or even programming, it makes sense to ensure that whatever you're providing to ChatGPT incorporates the full context of what you're trying to solve. That breaks down to what does that context look like? There are a lot of fancy techniques out there using interesting embedding models and very high performance databases like vector databases to pretty much solve this ability of finding the right context to provide to your chatgpt like application so that it actually solves your question or your user's question. A lot of times what this context looks like is very dynamic. So based on the nature of what the user asks. And so because of the dynamic nature of how RAG pipelines work and also the non deterministic nature of large language models, user feedback is essential in all applications that you make. And so what you may do from an evaluation perspective is take all the ups, votes and down votes from your users and all the questions asked your application and condense it into a smaller evaluation set. And then you may want to take certain questions from this evaluation set and use that to actually score how well your model is doing. Just to let you know, this is a full recap of just rag. Quickly we'll walk in through some dashboards and actually talk about agents and how all of these workflows you see now also apply to agents. But in this case, we condense our evaluation set into something that's a little bit more workable and then we run evaluation. So two kinds of popular evaluation frameworks we hear about is LLM as a judge, which we use LLMs to score other LLMs for metrics such as faithfulness, aka the accuracy of the answer given a document and relevancy. How useful was a document in consideration for an answer that's very useful from a large language model to kind of perform. Then you also have the most necessary component, which is the human in the loop. Have an expert actually evaluate whether or not the answer is correct and ensure that things such as hallucinations and other information such as hyperlinks actually work. And we do this to find problems such as gaps in the knowledge base of our own of our large language model or also hallucinations, seeing if the large language model, even if provided the right context, makes up certain answers or hyperlinks. So in the case of traditional evaluation, we have an understanding of what our users are using with our application. So when I say traditional evaluation and I think traditional machine learning, I'm thinking as related to hey, classify this image as a cat or dog for instance. You know generally the target distribution as it relates to to the data you use to train your model. But in the case of large language models, we don't know any of that information. We don't have access to that full corpus of pre training data spanning the Internet wide terabytes that it takes. And we also don't know exactly how our users will use our chatbots. Even in the case of a famous car company recently we see that how users interacted with their chatbot varied very widely based on just what the users were looking to do with the bot, be it malicious or actually related to queries for the company. Because we don't have access to our training distribution and our users distribution until we actually release our product, we have to approach evaluation from a different framework. One thing that's really popular is ensuring that our large language model always performs well on a pre standard case of benchmarks. These benchmarks are typically grouped together nowadays in tools such as the LLM evaluation harness or even OpenAI's evaluations. So this is an example of some tasks that you can see from the LLM evaluation harness. Then the next gold standard is to hire people, have real humans and experts who know both your user distribution and your training data, sit down and understand whether or not the large language model is answering questions appropriately given contexts. Then the most useful one, but the hardest to gather is user testing. So at this point you either partner closely with a customer or you have to go and release your application the world and use systems like I showed earlier to to condense down and isolate the essentially mine the user behavior to ensure that you're actually gathering the breadth and yeah, the breadth and extremes of how the users are breaking your chatbot in production. And then you also have the large language models evaluating other large language models as I mentioned with the faithfulness accuracy earlier and the relevancy accuracy. So in general this can be broken down into three systems that we can do currently as an individual user trying to evaluate our model. The first is eyeballing quite directly in which you start with a few examples and then you iterate on them and directly understand from the inputs and outputs that large language model or your application which uses a couple steps of large language models and sees if the answer responded by our large language model relates to the question asked by the user. Example of this can be seen here. We can see this as a weights and biases trace diagram where we can see that there are a couple steps that actually encompass how we're solving this text to SQL problem. We can see that there's LM chain, a transformation chain and this LM chain utilizes OpenAI. If we actually were to drill in here, we can see that we can see the full user, the full prompt that was passed over to our LLM and then this response by that large language model. We also have access to all the metadata associated with this model. This experiment was run months ago, for instance. At any point we were to rerun this similar collection of test questions against our large language model, we'd able to compare it from changes from three months ago to now. Something I find really important as large language Model updates happen very frequently behind closed doors from these closed model providers. Then we have supervision based approaches where what we want to do is actually in the case of where there's a definitive answer relating to our large language model that may be something like a multiple choice answer or directly looking for a specific key phrase or word, something as simple as simple as a table similar to Excel is useful here as we're able to aggregate the predictions and understand hey, this is where all the mistakes occurred from our prediction column and isolate the rows where the problems occurred or isolate where all the correct answers occurred and see where the actual problem lied in the thought process of our model. This can be seen here where we may want to drill down and understand from all of the expressions of our model. Here's the expression that was released and here's all the different metadata associated with this response. And using standard table manipulations such as where does the true result not equal the prediction result? For instance, this is really useful in that case. Now we can see all the mistakes that occurred and isolate what types of questions for instance does my large language model typically fail fail at? Then the Last one is LM evaluating other LLMs. So this is the case where you may want to try using an LLM to explicitly score the output of our large language models. And this is useful in the case of automation. So like in this case I may want to test out three different prompts and I want to automate the scoring aspect of this. So instead of having a direct score, I might have an LLM associated score here and be able to test that against various prompt templates. And this can be seen something similar here where we test out different embedding models against different large language models against templates to see the final F1 score overlap. And so all these evaluations occur in code itself with our weights and biases loggers and and our weights and biases interface. We make it really easy for you to do this type of hyper parameter search or saving of these metrics and details directly to a dashboard. Weights and biases is a centralized system of record for this information and I'll talk a little bit more about that soon. So that's a lot about how RAG works and evaluating rag, all of which are really relevant for agents. Also a lot of these workflows for evaluation for instance are all are mimicked with an agent workflows. But this is us improving processes around using our LLM. We actually want to improve our LLM in this case. So we want to do is actually fine tune our large language models and I'll be showing you a condensed form of what we teach in our fine tuning large language models class. But in essence one reason we may want to fine tune large language models outside of specificity for our data we where we may be missing some knowledge for a large language model or also for it to be form fitted for a task we care about ensuring that our large language model will always answer math questions properly after we teach it what it's been mistaken it on. What we actually care about here is we're also reducing tokens. We live in a token economy still a little bit where every single word and every single phrase we pass to a large language model and it cruise a cost. And so being able to reduce the complexity of our prompts by teaching this behavior directly to our large language model reduces the amount of words we need to use and also reduces the associated cost with those words. One recap I quickly want to do is there's this fancy phrase called reinforcement learning human feedback. This reinforcement learning human feedback process is a fine tuning process we did on large language models to ensure it speaks like a human at a baseline. Large language models will simply finish the prompt associated with it. With reinforcement learning human feedback we have taught the large language model to speak and respond as a human would and and also to be make considerations of what the input text it was given is like. All of this done by feeding it additional data in the training process of these large language models where these large language models to remind you have been taught generally across Internet scale data. So this process of fine tuning different from RLHF the one that we will be using more so is is parameter efficient fine tuning. Training a large language model is not a cheap or efficient feat. It requires a large volume of GPUs scattered across clusters and acquires a large volume of data and time. And unfortunately I am not Google like company myself. I cannot afford nor do I have access to those assets as easily. But what I do have access to is data and behavior of the users I evaluated on earlier from my rag based application and used for evaluation generally to understand where the mistakes for my large language models are coming from. I have access to that data. That data is very powerful for us not only because of the valuation work which we saw the value earlier, but in this ability to take that same data and and use it to increase the capability of these large language models. We're at a stage where evaluation and user feedback data can be directly used to improve our large language models and we use Lora for that, I'm not going to sit here and explain the beauty of the mathematics of LORA or linear algebra. But what I will say is LORA is really useful for us because what it does is it allows us to isolate specific components of our large language model to reduce the volume of data, time and GPU resources needed to train our model. This is done because we have realized from lots of different research that you can increase the behavior of these large language models by only selecting very useful layers within our large language model and it only iteratively improving the those underlying numbers that correspond to its thought process. And Lora is useful here because it treats it as a very nice math problem where instead of having us directly update the large language model itself, which is inefficient for our use case, instead what it says is it says we can treat these updates that we want to apply to our model as a separate collection of mathematical matrices. And because we're able to separate out the two problems, these separate mathematical matrices which we can apply onto the original large language model itself, they themselves are significantly smaller than the model and they're significantly easier to calculate over. And so that is simply to say you can take the this multi gigabyte large language model and result in a fine tuned adapter as we call them, that is only a few megabytes large. For us, that's really useful because this small megabyte size file can be applied onto our large language model. This means that we can have not only one of these adapters that work with this large language model, we can have a collection of them. We can make as many adapters across all our data sources and and all of our tasks that all work with something like a llama 2 base model, but all individually live separately from each other. And only during when we need to run our Llama 2 model with a specific behavior do we need to apply this update, this fine tuning update that we did in this LORA process. There are a lot of interesting parameters to adjust within lora. The most useful is specifically R, which is the inherent rank of the underlying matrices that you're creating. That's just to say the bigger the how you adjust the rank increases the complexity of the training, but also increases the amount the specificity of the update of the of the mod of the training process itself, so effects how well things are remembered during our fine tuning over our data of mistakes already generated by our large language model earlier. So I'll leave these parameters up to investigate later. These slide deck will be shared, but there is certain heuristics shared by folks at Pytorch Lightning that I would recommend to you later that I will hopefully be able to share in the YouTube link. And so Lora is not the only technique that exists. One very intuitive one is prompt tuning itself where we instead we use our large language model to generate the best prompt, for instance that will solve a task and we do that in a very machine learning specific way where we actually do the full machine learning back propagation process. But that's just to say there's a lot of interesting ways to officially efficiently keep the current state of the large language model frozen and proper and only extract out the full knowledge needed to make the large language model better for our use case and better for the mistakes that it made earlier that we evaluated on just to keep the evaluation always top of mind for us. Lora works really well is what this chart effectively shows and to formulate our data for Lora, it's very similar to the instruction tuning datasets that you see. So dependent on the specific model you're using, what you may want to do is understand how you actually want to formulate your data so that's actually able to be properly prompted in a way that makes sense for your users. An example of one in this case is if you want to categorize something as safe or unsafe, you can see that this task type policy, this task and policy are consistent across the prompt and response classification when it was safe and unsafe. What differs is this conversation output format. So for instance, in the case of low rank adaptation, this piece can be held effectively in constant for your modeling because each of your specific data points that you're providing your large language model can be prepended essentially with whatever pre described nature you want your large language model to recognize before it realizes oh I have been updated for this task and these specific collection of tokens are even prime me better to solve that task I was fine tuned for weights and biases is a really helpful platform for centralizing all this information. A lot of popular teams use us for training and fine tuning specifically and you can see that we are able to consolidate. Soon you'll see how we consolidate all the different experiments be it fine tuning GPT 3.5 which we'll be doing later, or if you're training something as complicated as a as a large language model directly using a framework like Pytorch Lightning. But getting started is really easy with with biases. Actually it takes a few lines of code to actually store all the metrics prompts or other information you want to our platform, or even less if you use one of our integrations like in the case of fine tuning. OpenAI we have integration that syncs all the training data to a dashboard alongside references to the training data you used and any output model metadata. So you can easily load up your fine tuned model by referencing our weights and biases dashboards. I'll flash this on the screen for a quick second just to say we are building a different collection of a large volume of tools around the MLOPS space and and then the LLM Ops space directly. But now let's get to agents. That was a fun talk about rag. It was a fun talk about fine tuning and seeing how those two components relate together. How we can use the mistakes from a traditional RAG pipeline to increase the capability of our large language model generally. But let's see how agents come into the fold and make this process of being able to dynamically reason over how we want to solve problems a lot better. I'll actually be adapting slides from last week's talk from the folks at llama Index and TrueArea meet True Era who talk about how to build, evaluate and iterate on LLM agents. I found their slides very useful so I will be showing them quite briefly here right now. So in general what we did right now is we've pre built the behavior of how our users will interact with our large language model. That's via rag. We pre writ a template, we predispose all of our databases so RAG will always have a consistent behavior. But the problem is that we have pre scripted all of that behavior. In the case where our applications from our users behavior become varied across tasks, use cases and behavior. This RAG like behavior and this pre scripted behavior becomes easier to break. The prompt that we may have used for one level of RAG may not be appropriate for every user interaction. So what we instead use is we use agents effectively as a router for our query to be able to dynamically understand the core essence of the query by our user and then decide which appropriate action to take. In this case, an agent may decide our RAG pipeline is the best method to solve this user query that's coming in. There are a couple forms of agents that occur. The one that's very useful is is specialized data agents where it's similar to retrieval except it has access to real time information. So in the case of agents similar to humans we evolved once we understood actually how to use tools and fire appropriately. So tool usage is very useful for agents as tools in our case can be any function and it can be actually any API call also. So for instance, I can have my large language model understand that it's missing a certain piece of information and understand, similar to us, that we need to Google it. Then it goes and Googles the first three appropriate responses from Google. So maybe to some of us humans also, instead of actually clicking on the page, it may just read the little blurb that shows up in the headline and use that information to actually formulate a response. Agents can vary across not just data, but can be accessed to multiple tools itself. So maybe you have access to Google and your calendar and the ability to book flights, to have access to the ability to take actions in a very useful way, but in a very wide ranging way. So being able to not only ingest information and return responses, but but ingest information and perform actions that help with our other workflows, be it actually booking that flight for us once given a query about what the cheapest flight is and ensuring it has access to book on our account, for instance. So data agents like I mentioned earlier may want to read in information from a lot of different sources, maybe even iteratively where may want to analyze a file over a couple different steps steps, but then take its output and release it to the world. So in this case we may read some emails, retrieve context that relate to the emails and the question asked by the user, analyze appropriate files, be it one or five, and keep continuing to analyze these files until a stop criteria is met, formulate a response and send that response back to my Slack channel, for instance. This is the full breadth and power of agents. But with each of these different interactions done by agent, there is a point where a failure can occur. It can fail at reading the latest emails, it can fail at retrieving context, it can fail at the iterative process of analyzing a file, and it can also fail at being able to appropriately send that Slack message for us. So a more generalized form of what I showed earlier can be seen here where you have an agent. You have an agent that's able to given a message, decide if it wants to use a rag pipeline, book a ticket or send a Slack message. But also this other external piece which is useful is the fact that it has access to memory, so understands in its full sequence of agent decisions, it has a scratch pad to look like to say okay, at this point I've already read the emails, I've already retrieved the context. Now what's next? Is what the agent can ask itself as it continuously interacts with the tools and its own memory, similar to us humans would. And so to go back to the RAG pipeline showed earlier, RAG in itself is just one tool. It is one tool where we pre scripted behavior about how a certain type of user question should be answered and how the answer relates to a database. So in this specific example we may have a database that relates to some numbers relating to Lyft, maybe some numbers relating to Uber, but instead of having us say hey, always read these two databases and provide that to my large language model as a response, what we instead want to happen is we want our agent to understand that our question may be something along the lines of compare lift financials to Uber financials and it will make the decision to select both of these databases to use as data sources and then that pass that over to the rest of our RAG pipeline. For instance, so it is able to dynamically choose what data is appropriate and also dynamically choose which specific workflow this data should pass to to to get to the right answer. So to re show where the failure points could occur though at any point during a tool usage, AKA selecting the right database or booking that flight, it could choose the wrong tool. It could say oh, instead of booking a flight I may want to google more and end the infinite loop of googling prices over and over instead of having the full action of yeah, a full action of doing what we actually wanted to do, which is book that flight for us. So that is the case of rogue paths not actually choosing the right database or choosing the right booking tool. We also made the problem of infinite loops where it may say to itself that it cannot find an answer so continuously googles over and over, not realizing that in the scratch pad of memory there was a specific answer from Google that was returned that actually solved the answer that we wanted. It was too busy looking at its full memory and all of the different attempts it had and realizing that the right answer was lost in all of this process that the agent was doing. And we also may have just failed API calls. Maybe somewhere along the lines a service was turned off and the agent does not realize that and will continuously beating a dead service over and over again with API requests rerunning into this rogue path infinite loop problem that we mentioned earlier. And then lastly, the most important one is agents might just make stuff up. Given the full context and all of the problems mentioned earlier, it may say okay, well I can't access the tools I need, I can't appropriately pull data from the databases and I don't know where to put it to. I'm just going to me being the LLM that I am, just make up an answer that seems correct. And so that's where we see a lot of failures and faults occur. Not only in the actually ability by the large language model to choose the right tools and components internally, but also returning a response that's appropriate, giving that user question. That's where all the evaluation methods I mentioned earlier during the RAG piece all still come into play. You still need to check for relevancy and relevancy and factualness from your large language model that's running this agent behind the scenes. But you also need to ensure that you have the appropriate tasks that you're evaluating for. Like in the case of the LM evaluation harness, you want to ensure that every time you make improvements to an agent or you move from rag to agent, that the overall accuracy for the task you're solving for also doesn't go down. Agents give you flexibility at the cost of being inconsistent in the current state of things. And that's where the traditional evaluation pipelines that you set in place earlier to get this initial user feedback from RAG pipelines is essential here. That's just to say that everything you're doing earlier with RAG still is in play here. The only real piece that's different is that you may need specific tests to ensure that the priority appropriate tools are being used. And so there are some best practices on how to actually approach this. You may want to write useful prompts, make tolerant prompts to ensure that if there's any failures and tools that occur, you, you properly account for that. And you also have different prompt engineering techniques that you can do here. But in the case of what I showed you earlier, it actually turns out that LMS are extremely complicated. You can have. Sorry, LLM agent applications can become extremely complicated once taken to production. In the case where you have auto generating, you have agents that auto generate code for you. There are a lot of moving pieces that are not just as simple as hey hit Google and provide missing context. It becomes a lot more complicated where you have multiple databases and honestly multiple agent agents that may communicate with each other. That's just to say that building standard pipelines in RAG first and then having basic agent interactions is essential to get to the point where you can build multi agent, multi database complicated agent use cases. And that's the kind of direction we're seeing from agents right now. We can see that as LLMs and use cases evolve, people are building kind of these very fascinating multi agent environments where we are simulating hospitals or we're simulating whole towns where we have agents all communicate with each other, but have intrinsic motivation that we set via prompts and these tools. That's just to say that for those familiar with reinforcement learning, this is very similar to how I would consider a large language model agents. I give it a reinforcement learning agent lens. That may be a little scary for some of you to hear the phrase reinforcement learning. And I assure you it's not all just Boston dynamic robots. It's a lot more straightforward and simple than that. What we want from our reinforcement, from any agent as we call it, is the ability to interact with the world and make intrinsic thought processes. And that will help us decide plans to execute. Given what's available to me in the world that I know of. That's simply to say that for a robot, it may be know that it has a camera, so it knows directly the vision of the world around it. And so when it's told to pick up an object, it knows internally I need to rotate certain certain motors and move certain joints of mine to be able to actually adjust myself to grab the odd object, knowing that I as the robot am composed a certain way. Now if I were to kind of tweak a certain word, a few words. That same framework applies for large language models also. So with large language models, I, as a large language model may know I have access to certain API tools and databases, I have internal prompts I may be able to try out that allow me to solve a certain task. And then this task is actually provided via a prompt from the external world, AKA the user. So I, as a large language model can make plans against the user's prompt to be able to solve that user question that with the internal thought processes I have and the internal knowledge of the tools and databases I have available to me. So reinforcement learning and large language model agents both actually have very similar workflows. And with that in mind, we can actually use that as a basis to build out this kind of framework on how to approach building extremely powerful agents. And these are the same frameworks that were used by people that building these simulated towns, for instance. So one very popular method is called react, so reasoning and then acting. So in this case, instead of directly answering a question, for instance, we want our large language model to do is actually take a second to consider what the user query is and how we should approach something solving a question, similar to how we don't just jump into doing things as humans. So what you do is you create this concept called a reasoning trace, AKA the Decomposition of what actions I would want to take given an observation AKA the prompt by a user. That's a lot to say that if we want to show just a practical example over here with just reasoning only you can see that the answer for a question we may actually have just directly the answer ipod. But then in the underlying thought process it goes let's think step by step. The so as also a little tie in to be able to get your large language models to do thought processes like that thought answer. There are different prompt engineering techniques such as chain of thought for instance. So in this case where use chain of thought only we can see that it refers to the Apple remote and the Apple TV to to try to go and give an answer. So sorry. Yes. So in the case of a large language model only it gave the wrong answer of ipod. When we gave only chain of thoughts, it gave the wrong answer of iPhone, ipod and ipod touch. But now with React what we can do is we actually broke our thought process into set of thought and answer into thought, action and observation. In this case, because our agent is aware that has access to tools and databases, it knows that it can go and search for the Apple remote. It can read the headline from Google, it can use the information from the headline of Google to then search iteratively over and over until it reaches a proper resolution where it realizes front row is controlled by an Apple remote or the keyboard function keys. So the keyboard function keys. So the keyboard itself is the final result of what the other device can be used to control a program control the Apple remote program. And so it it took a couple steps of knowledge to actually resolve to that point where chain of thought would simply only use the internal knowledge of the model to answer a question. Where in this case we allow the thought process of our model recognize that it can use the external external world to solve an answer. And so we can take all of this thought process and actually make our large language model do it not only better but also make it the default behavior of our large language model. The way to do this is using the same exact fine tuning techniques as earlier. Except instead of having our user the feedback from our users earlier as our only our fine tuning data, what we can do is actually transform our users data into something that is similar to this React style format. And so let's give a quick run through an example of this. So this here is an example where we want to fine tune a GPT 3.5 React agent on better chain of thought. So what we're going to do is we're going to actually have our large language model answer a few questions using GPT4 and we will take the outputs of that GPT4 thought process and use it to fine tune GPT 3.5 and that will improve GPT 3.5's behavior to not necessarily be the same quality as GPT4, but to still have the same degree of thoughtfulness as GPT4. As we want to improve not necessarily the behavior of our large language model in this case, but we want our model to be better at thinking thinking, which is the hopefully our teachers also were teaching you as you were growing up. From a code perspective, weights and biases is a very minimal component. What you're going to see here is a lot of code by Llama Index, very popular framework that I quite like to get using weights and biases. All you need to do is create this weights and biases global handler or there's other different forms using our weights and biases callback handler. But in this case, after setting these two lines of code, every single resultant trace that's generated and also the information from Llama Index are stored to Awaits and Tobias's dashboard. The same dashboard will also contain the metrics and details of our fine tuning process as we do it with OpenAI. What we do in our case is we set up GPT 3.4 and 3.5. We set up some tools where these are the three different databases of our use case. Where our use case in this problem is we want to improve not only the thought process of our GPT 3.5 agent, but improve our thought process as it relates to the financial analysis of the Uber quarterly financial data. These are the three different databases month by month of the 3/4 of the Uber data from 2022. And we want to ensure that our 3.5 agent is able to answer questions about these three different databases in a thoughtful way and in a way that it's able to always efficiently choose the proper database to solve a certain user query. So what we may want to do is after loading Our LLM up GPT 3.5 Turbo, we may want to try out just a basic agent to see how the behavior is quickly answering the question for you. It looks like it gives a pretty good answer, but the problem is it actually doesn't look at the right database as it relates to the question in this case because it doesn't appropriately look for the highest revenue growth and as a result returns the wrong answer for us. It just kind of gives general statistics about a database. It's selected, not the one with the highest revenue growth. So instead what we might want to do is we want to synthetically generate data for fine tuning ourselves. So what we do is we have this generic query where we say you are a teacher and you have to create questions which relate to the different databases we have available. And that makes sense for the data that lives in it. We go and we synthetically generate 20 questions based on underlying databases and then we actually follow up one level deeper and we actually say, hey, you are a financial assistant and given these interesting questions about the uber financials, I actually want you to jump in and add in a variety of different questions so we can have a variety of similar styled questions across each of our databases so that when we fine tune our data, we aren't just having one data point for every important topic, we have a couple data points for each of these different important topics. So with this process in place, we're able to generate a bunch of different questions. So two for each of the different varieties. So a total of 40 different questions. Oh, sorry, 80 different questions, I apologize. Yes. So we generate four for each of the different, the different original questions that we had earlier. So now we have 80 good questions to ask GPT4 which will ask GPT4. So we'll be using GPT4 with as our underlying agent LLM to thoughtfully answer 80 synthetically generated questions about the database which we care about where each of these questions are financial questions that relate directly to each of these databases. And so we've saved, once we save all of these different questions, we can actually go ahead and run our GPT agent. Oh sorry, run our GPT agent against each of them where we specifically have told our model to provide the to provide the react thought process that I showed earlier. So it not only does it just directly answer the question, it will thoughtfully think about what the question is asking, think about what action it needs to take, apply that about where each of these questions are financial questions that relate directly to each of these databases. And so we've saved. Once we save all of these different questions, we can actually go ahead and run our GPT agent. Oops, sorry. Run our GPT agent against each of them where we action and then iteratively keep continuing the process and until it reaches some certain stop criteria, which is that finish Command, we have GPT4 answer all of these questions which will simulate the user behavior that we would instead want and we save it quite easily using the fine tuning handler that llamaindex provides. So once we have Our fine tuning data of really well thought out chain of thought processes, sorry, react thought processes across specifically have told our model to provide the react thought process that I showed earlier. So not only does it just directly answer the question, it will thoughtfully think about what the question is asking, think about what action it needs to take, apply that action and then iteratively keep continuing the process until it reaches some certain stop criteria, which is all of these different questions as it relates specifically to our data set which is financials about Uber, we can take that data, ensure all the data is proper. So this is a copy and paste script directly from the OpenAI website to ensure that your data is not malformed. For fine tuning, we load up, we load up the client from OpenAI, we actually just run training, we load our model in, tell it how long we may want to train it for, we point it to that data set we saved earlier. Then as the model trains that finish Command we have GPT4 answer all of these questions which will simulate the user behavior that we would instead want and we save it quite easily using the fine tuning handler that llamaindex provides. Once we have our fine tuning data of really well thought out chain of thought processes, sorry react thought processes across, we can run our single line of code called the Wandblogger which will sync all of our metrics once the OpenAI fine tuning is finished with where we'll have now a centralized system of record where not only do we have access to all the metadata information relating to our fine tuning process, we have access to the training mind metrics directly itself. The underlying all of these different questions as it relates specifically to our data set which is financials about Uber. We can take that data, ensure all the data is proper. So this is a copy and paste script script directly from the OpenAI website to ensure that your data is not malformed. For fine tuning, we load up the client from agent thought processes as we showed, similar to what we showed earlier in the traces, we have understanding of the utilization of the network and disk as we use our agent applications, we can access also the model directly where we have access to not only the metadata, but we can actually load in the model JSON itself and use our fine tuned model moving forward for training. So if I want to easily share this fine tuned model with a colleague and alongside the details and the agent traces itself, I simply need to share with them this weights and biases link. They have access to all that different information. OpenAI and we actually just run training. I can take that, we load Our model in tell it how long we may want to train it for, we point it to that data set we saved earlier and then as the model trains we can run our single line of code called the wannabe logger fine tuned URL and simply load that back in to our OpenAI instance and use our react agent moving forward. Sorry, not this one. Yes, yes it is one we're now moving forward. Our fine tuned agent is now available to us and we can use this to now understand which will sync all of our metrics once the OpenAI fine tuning is finished where we'll have now a centralized system of record where not only do we have access to all the metadata information relating to our fine tuning process, we have access to the training metrics directly itself. The underlying to use a cheaper and more effective form, the GPT 3.5 Turbo to get extremely similar behavior as a GPT 4 level quality agent if it were given our specific data it does not reach the Same quality as GPT4 but it works extremely effectively. I don't have evaluation in this specific notebook and I will be sharing this notebook afterwards. But yes we would use a similar LM evaluation harness to compare both of these behaviors. So now that we have a fine tuned agent with better change in thought processes as we showed, similar to what we showed earlier in the traces, we have understanding of the utilization of the network and disk as we use our agent applications, we can access also the model directly where we have access to not only the metadata but we can actually load in the model JSON itself. We're done now we've solved the problem of being able to improve the processes with LLMs and we also improve agents. Now we're at AGI, right people? No, clearly that's not the case. Instead what we can see is as an aside for evaluating agents, one additional layer of agent evaluation is is the concept of ELO ratings. So with ELO ratings what you want to and use our fine tuned model moving forward for training. So if I want to what this does is effectively tells you similar in like the easily share this fine tuned model with a colleague and alongside the details and the agent trait concept of game theory it tells you how how well you rank against others itself. I simply need to share with them this weights and biases link. They have access to all that different information. So I can take that in a similar bracket. So that is simply to say that the best way to actually see how well a chatbot does specifically an agent chatbot is to actually compare it against other agent chatbots because there is no good effective way to understand how well a chatbot does with its thought process in a good qualitative way other than comparing it to other agents. Fine tuned URL and simply load that back in to our OpenAI instance and use our react agent moving forward. Sorry, not this one. Yes, yes it is one we're now moving forward. Our fine tuned agent is who we consider to have good underlying thought process behavior. So to quickly get through this so we can answer some questions. I see that they're. That they're popping up. There are other techniques and prompting so such as reflection which is this concept where not only you have this reaction where you have. You see an input by a user, you create an action trace and then you create observation. We instead do is given certain outputs. You actually run evaluation on yourself during now available to us. And we can use this to now understand to use a cheaper and more effective form. GPT 3.5 turbo. The thought process of an agent. Take that evaluation and then use that as a to get this to get extremely similar behavior as a GPT four level quality piece internally to say okay, I failed here in runtime for instance where I reach an error read agent if it were given. If it were given our specific data that error look at what I previously generated and generate more thought processes in our react format. So that's the concept of reflection which kind of mimics this long term behavior. And then you also have tree of thought which is a deeper way of thinking where you instead of just directly.