How to Build, Evaluate, and Iterate on LLM Agents

Sam.

Hey everyone. My name is Diana Chan Morgan and I work at deeplearning AI running all things community. Today we have an amazing workshop with some of our course partners to bring together what's next for LLM agents. So today we're working with llamaindex and Truera and they will guide you through the entire process of building, evaluating, iterating and deploying successful LLM agents. This session will be recorded and the slides will be sent afterwards. We also have a notebook that our speakers were so kind enough to share. So we will drop that in the chat for you to be able to access and follow along during the session. This workshop utilizes cutting edge open source tools like LoM Index, a simple and flexible data framework for connecting custom data sources to large language models, and True Lens, a powerful platform for testing and tracking LLM app experiments. In this session you'll be able to gain valuable insights into building your first LLM agent effectiveness, hallucinations and bias iterating for production ready applications and maintaining high performance in production. For any questions for the speakers, we've also dropped a link in the chat to ask and vote upon the questions that we will answer in the last five to ten minutes of the workshop today. This workshop was also inspired by our newest short course that was just launched last week. It's called Building and Evaluating Advanced RAG Applications. I'm sure our speakers will talk a little bit more about it today. You can find that on our website and we'll also be setting that out afterwards and you can find it in the link in the chat as well. And to start off with, I want to share our first speaker, we have Jerry. Jerry Liu is the CEO of LLAMA Index. He brings a wealth of experience from his previous roles as an ML engineering manager, as a robust intelligence and research scientist at Uber. Hey Jerry. Really happy to have you here today.

Hey, thanks for having me.

Absolutely. And next we have Anupam. Anupam is the President and Chief Scientist at Truara and is a renowned AI expert and former professor at Carnegie Mellon University. His research. His research focuses on ensuring accountability and fairness in AI systems. We're so happy to have you here today, adapam.

Great to be here, Diana.

Absolutely. Well, I'll let you guys take it away for the workshop. I know you have a lot of stuff planned, so I think our community is ready to dive into everything.

Awesome. Perfect.

I'll you guys take it away.

Awesome. The slides are visible. Great. So let's get started. Today's workshop is about how to build, evaluate and iterate on LLM agents like Diana said, Jerry and I just recorded a course and released it on deeplearning AI which focuses on rag based applications. You can think of this as a continuation of that, but expanding into agents and later on Jerry will comment more on the technical connection between the two. Just to orient ourselves, we'll take a few minutes to look at some examples of frameworks and actual agent based LLM apps that are starting to get quite a bit of adoption. So ChatGPT put out their plugins sometime back as many of you are aware, and with these plugins or LLM agents you can augment ChatGPT to access up to date information, run computations or use third party services. When we share the slides later, these are hot links so you can explore them. These are some of the examples of applications that the first set of applications that were currently available you can see that these agents like on Instacart, you can order so these agents can start planning for you. They can start acting on your behalf and order things like grocery and help you plan for trips and so on, or even do math. So that's one set of applications. There were a couple of other very early seminal work in this area. One is this very interesting result paper called React, which combines reasoning and acting with large language models. This has been a very influential paper in the space of agents before that thinking around reasoning, like chain of thought reasoning where an LLM might, when it's asked a question, reason through the answer first and then provide the answer, which is very widely used. Reasoning was separated from acting where you can go out and the LLM can be used to act on the environment and act on your behalf for various tasks like planning your travel reservations or making restaurant reservations and so on. With React, these two bodies of work were pulled together where there was an interleaving of reasoning and actions and that really significantly increased the power of agent based LLM apps. Let me give you a very quick example. This is taken from the React paper. So the question that was asked here is aside from the Apple Remote, what other device can control the program Apple Remote was originally designed to interact with? And if you just ask standard LLM then the answer was ipod which was incorrect. If you get if you make the LLM think through reason step by step, even then it comes back with the wrong answer. If it only acts, then it starts doing an external search, but the search does not produce a meaningful result. So this on the left here is standard reasoning and a separation between reasoning and acting. When you do one at a time, but not interleaving. That does not produce the right result. On the right here you can see how where thinking and actions are interleaved, you suddenly start seeing a much better result. The thinking here was well, I need to search for the Apple remote. So it invokes the search API to do the search. The search comes back with this front row media center as a keyword. The second thinking now is we need to go search for front row. The first search for front row does not give anything very concrete, but suggests that you might want to look at some similar things which includes the software. So the next layer of thinking results in the action being search for front row software. Eventually it results in finding the right answer, which is you can also control these devices with keyboard function keys. The details of this example is not so important, but the main thing is this interleaving of reasoning or thinking with actions and this back and forth really is where a lot of the power of LLM based agents derive from. I encourage you to take a look at this set of results the react paper the GitHub to get a sense of the history of one of the important developments that spurred the recent excitement about agents. The other project that I'll point you to is auto GPD. AutoGPD came out sometime back and it was one of the fastest growing open source projects on GitHub. You can see it's this is a relatively recent screenshot. It has about 150 +150k + stars. And then this hackathon which was recently done in October has lots of examples of interesting LLM based applications that you can explore. So for example, there are some interesting things around coding and so on. Now, as people started diving into agent based applications, there are some limitations that started emerging as well some failure modes. Often this includes augmenting LLMs with additional tools over APIs to do search, to look for travel itineraries, to look for restaurant reservations and so on. And as you start increasing the number of tools, the LLM has to reason based on a query what tool is the right tool to use as part of creating the answer? And that's where mistakes happen. That's a common failure mode. There can be infinite loops. There's also hallucinations. A problem that is known very common with large language models appears also in this context and we'll talk about these in a minute. As we think about the space of AI agents, we think about it in this kind of three stages, the specialized data agents, which are quite similar to retrieval from a vector store and the Retrieval Augmented or RAG architecture but with access to real time information and some additional pieces. There's general data agents that have access to more than one tool and can accomplish a wider range of tasks. And then there are agents that can take action in the real world. We'll focus quite a bit on data agents in this talk and in that context we will make use introduce you to how to work with how to build agents data agents with Llama Index and to evaluate it systematically, understand the failure modes, iterate and improve it by leveraging True Lens. These are our githubs. Please feel free to explore them. Many of you are already there playing with Llama Index and with True Lens and our developers enjoy it when you request features make contributions. Please give us stars as well if you like what you see there. With that I'm going to transition it over to Jerry. Jerry will talk about how to build LLM agents with Llama Index.

Jerry, awesome. Yeah, thanks Anibom. Maybe taking a really quick pivot right here. We're kind of featured our course that just came out last week building and evaluating advanced RAG applications. And so the course itself is focused a little bit more on advanced RAG techniques. Retrieval Augmented Generation for those of you who might be less familiar with rag, RAG is basically just a technique for augmenting your LLM with an external knowledge corpus. And so the way it works is you first do retrieval or index your data into a storage system, then you do retrieval from the storage system on query time and then add that context to the prompt window of the outline. There's both a simple way of doing that, but these days a lot of developers are excited about exploring advanced ways to do that. To try out advanced methods, you also need a rigorous system to do evaluations on both the retrieval and generation piece. We are really excited to work with Truera on this course because Trulens has a robust evaluation suite. Llama Index has invested a lot in advanced retrieval techniques. And so we combine them together into this overall short course. You should definitely check it out. This also serves as a nice bridge into how we think about how agents evolve from rag. Basically, there's these two buzzwords floating around. There's RAG and then there's agents. And so how do we really think about the transition between RAG and agents? And so the way RAG works, right, we can kind of think about it as a box. The user has a question like a query. RAG is this overall pipeline defined over your knowledge corpus? And within this overall kind of RAG box, we'll do retrieval from the knowledge corpus as well as synthesis. And after that we get back a final response. One of the ways we think about agents next slide is basically we can think about that as an additional layer in front of the RAG pipeline. How we think about this is basically given a user query, we can actually feed it to an overall agent that can more dynamically decide how to route this query through to relevant tools to try to synthesize the response. In a traditional RAG setup, what typically happens is that the LLM call occurs at the end of this overall pipeline. You do retrieval, which is typically top K embedding lookup, and then you feed it to the language model at the end. But here the idea is what if we actually use the LLM at the beginning as well to actually take in the query and figure out how to actually make use of the underlying tools. Whether it is a RAG pipeline, an external API call, or some other service, we think about agents as wrapping a layer on top of RAG that dynamically enriches the query with additional information and basically allows this overall higher level abstraction to use tools in the right way to to try to give you back a response. In the next few slides, we'll go into a little bit more about the architecture of how this works as well. So a few months ago we defined this concept of data agents within LLAMA Index, which are LLM powered knowledge workers. A data agent is basically something that is designed to help you automate knowledge and can both do search and retrieval as well as synthesis, as well as modify data. So for example, a sample flow is given in the slide right here. Given a data agent, you can for instance make a call to read the latest emails that you have, for instance from Gmail or from some external service. Afterwards, you could also retrieve additional context from your knowledge base. This is a perfect example of the RAG pipeline as a tool within an agent. So this by itself could be a RAG tool. You could have an analysis agent that actually analyzes the file, for instance using a tool like Code Interpreter or another RAG pipeline. And then you could have another tool where by actually calling it, you are taking actions and modifying state. In this case, you send an update to a third party service like Slack, and that actually triggers a message to be written to this service. Next slide, the way we think about agents is the core components are really one, an agent reasoning loop, which Anupam covered a little bit in terms of a react loop as well as tools. The way it works is the agent reasoning loop typically operates over the set of tools that you give to this agent, the Agent Reasoning loop, is typically powered by a large length of model because that allows it to basically dynamically decide, for instance, what are the next tools I should use given the user input to try to call to try to synthesize the right results. The tools themselves can be for instance what we call query engine tools like RAG pipelines. You get to basically take all the techniques you learned in the short course that we just and plug them in as RAG tools to this overall agent. We also have a variety of different tools on llamaHub that connect to third party services. We have over 20 and 25 tools on llama Hub. This includes API interfaces to for instance, Slack, Notion, Zapier, the code interpreter, and all these tools basically act as API interfaces specifically designed for LLM agents. You can actually mix and match, for instance, your RAG agent tools as well as your Llama Hub tools and basically combine them into this overall agent that can both do search and retrieval as well as take actions on different services. I think anyupon covered what React is in the previous few slides. There's also of course a few other ways to actually perform agent reasoning. One way is you can just do a function calling loop over the OpenAI agent, for instance. Under the hood, I think the API service will just handle for you and decide what are the functions that you should call given the inputs that you give it. There's also other algorithms like Tree of Thought, Plan and Execute and a few others. There's a few papers coming out of these conferences every year. Next slide. Maybe a quick comment here is how do you actually do search or retrieval from a knowledge base? This touches on this point about basically plugging in a RAG pipeline as a tool within this overall agent. One idea here is elaborating on this a little bit. LLAMA index. Basically the TLDR is we have a bunch of these things called query engines which are basically for all intents and purposes, generalizations of RAG pipelines. They take in a user query and they give you back a response. What you can do is we have a bunch of these query engines. We have semantic search, which is basically a RAG pipeline. We have summarization, which is you return all the context and you try to synthesize an answer over all the contexts instead of just the top k. We have query engines over structured data, for instance, like text to SQL over a SQL database. We also have advanced RAG pipelines, some of which you actually learn in the short course. This enables you to do stuff like document comparisons as well as being able to do kind of like combined querying over hybrid structured data with unstructured data. The way the Llama index architecture works is you can actually just plug in all these query engines as tools to an agent. You basically have this outer agent layer which acts as a reasoning loop and if it calls a tool, then the tool executes, which is a query engine. This basically gives you the ability to compose more advanced reasoning abstractions and layers on top of your data so that you basically first have this agent handle the user query, decide what tools to call, and then execute the tools. There's a variety of examples that we have in the notebooks of what this allows. This basically allows you to answer more complex queries, allows you to do a little bit of query planning to try to execute some things in parallel, allows you to dynamically do both stuff like semantic search and summarization as well as text to SQL as well as combine information from disparate data sources. Next slide. An example here basically from the screenshot is we have this example agent, for instance, and let's say we plugged in two query tools, one for Uber and one for Lyft. And each query tool is just a rag pipeline over each company's annual report, like the 10K filing. So each tool corresponds to a top k rag pipeline over that company. You plug both tools into this agent and now you can ask a question like compare and contrast Uber and Lyft's revenue growth. The agent, through its reasoning loop and chain of thought process and react reasoning will break down the question into sub questions over the tool. And then each question subquestion plus tool will be executed right. Each tool will be executed with that sub question. And within each subquestion we'll do per document rag. So first we'll look at Uber's revenue growth. Look at the Uber tool, do top k rag get back to the answer, then look at Lyft's revenue growth, do top k rag get back the answer and combine the results at the end. So this is just a concrete example. Next slide. In the next few slides we'll kind of talk about just some general architecture decisions for how do you think about modeling some of these agents and the slides after that? Aduhav goes into detail about a lot of the failure modes as well as how you actually evaluate these agents. But just some general things to think about is one is how do you actually handle large responses from tools? Like if a tool returns like an entire essay or a very large webpage, how do you actually handle that? Some strategies that we basically have and we call them on demand Loader tool as well as Load and search tool, basically involve on the fly indexing of this data. Sometimes it's nice to first load this data and actually index it into vector storage so that when you actually kind of query this data that you loaded from the tool, you actually do search and retrieval over it instead of trying to stuff everything into the context window. Even though context windows are getting bigger, they still overflow on large amounts of data. And so this is like indexing it beforehand is a way to mitigate that next slide. Another problem that I think audiopalm also touched on is a lot of these agents tend to struggle a little bit when you overload them with tools. And so if for instance, you have more than like five tools and you know in the limit you might have like hundreds, thousands, even millions of tools at a certain point, this isn't going to fit into your context window. So what you can actually do is you could try to actually index the tools themselves, right? Index and metadata of the tools. And then during, basically during query time, you actually first do search retrieval over the relevant tools first and then pass that to your agent to then ask a question. These are just some general considerations and we'll do a section at the end as well, kind of talking about some best practices for constructing agents. But hopefully this gives you a general overview and passing it back to anavam.

Great. Yeah, thanks a lot, Jerry. And now you have gotten a good sense of some of the key building blocks for how to build agents and especially how to do it with Llama Index, where Jerry and team have driven a lot of the early work. In this space, we'll shift gears a bit and start looking at evaluation and what are some of the failure modes of agents and how do you detect those kinds of issues? How do you iterate to improve them? I'll do a quick introduction to True Lens in this context. True Lens is an open source library to track and evaluate your LLM experiments. As you're building your applications and Llama Index, for example, or any other Python framework, you can use True Lens fairly easily as part of that application building process. And we'll show you a notebook and Diana is also sharing it, so you'll have access to the notebook on GitHub to play with yourself. So as you're building your LLM application with say Llama Index, with just a few lines of code, you can connect True Lens to it. You can start logging the records, the prompts, the responses, the intermediate results, the entire call call Trace. Interestingly then what you can do is we have this nice abstraction of what we call feedback functions or evaluations which lets you log and evaluate the quality of your LLM applications along a number of different dimensions. Once you have those set up, a number of these are available out of the box. You can add your own evaluations as well very easily. You can start exploring these records evaluation results in a true Lens dashboard, see what the failure modes are and that can in turn inform iteration selection of the best LLM application version for your use case. So let's look at a concrete use case where we may want to use the kind of data agents that Jerry introduced for real time retrieval. Calling out to external APIs and tools. The user comes in here with an input, a query and then the LLM acts as the reasoning agent. It picks a tool one from a set of tools that might be available from, translates the user query into an input that is appropriate for the tool, gets back the output from the tool and then produces the final response in this process. What are some agent failure modes? The agent might in this reasoning step pick the wrong tool. If it's a question about restaurants instead of putting Yelp, it might if it picks something like archive and it gives you a research paper on which makes which talks about restaurant recommendations. That would not be very appropriate. But that kind of mistakes in picking the right tool is pretty common. It's a common failure mode for agents, especially as the number of tools start going up. It can get stuck in infinite loops, especially because many of these agents, the base base case ones don't carry state. API calls might fail, say if the input is not formatted, if the query translation is not done in the right format. You might have also other kinds of failure modes related to hallucinations. So let me talk about a very concrete use case now. And this is what is available in the notebook that is made available to you. We'll talk about a restaurant information chatbot where the user starts asking some questions and then the agent makes use of the Yelp API. So one tool in this case to answer questions related to it, and we'll compare that with a baseline where the LLM is just answering these questions based on its pre trained knowledge and without access to an external API. So those are the two that we will compare. This is available in this notebook. Here is a QR code. You can play with it after the session if you like. It'll also be shared with you. It's available publicly on GitHub. To make things interesting, we instructed the Agent to respond in the style of Gordon Ramsay when asked questions about restaurant recommendations. Let me just show you a couple of examples here. So the first question that we asked was, What is the best restaurant in Toronto? The best diner in Toronto? And this was to the. Just to the OpenAI LLM, not without access to. Without access to the Yelp reviews. And this is the answer came back with, I'll take you a second to read it. Remember that the prompt was to respond in the style of Gordon Ramsay. And so there you can see that it's not necessarily particularly respectful. We asked the same question of the LLM agent, which had access to the Yelp APIs. And then the response came back. This is the response that came back. This is a bit more polite, maybe because he. There was an answer that was actually retrieved, which the LLM was then asked to summarize. So now we have a couple of these answers, and we want to be able to think through and have tooling to do evaluations. So let me circle back and start talking about a framework for evaluations of LLM agents. So before we go into LLM agents, I want to recap one slide from our DeepLearning AI course on advanced rag techniques that Jerry and I recently launched, which you should take a look at. It's just one hour in that we talk about the rag triad for guarding against hallucinations. So if you think of a rag, the user comes in with a query, much like in this example, what's the best diner in Toronto? Based on that query, a set of contexts or chunks are retrieved from a vector database, and then those then get summarized by an LLM to produce the final response. And along each edge of this triangle, there is a property that you want to test for. On the first edge, there's a property that we call context relevance, which is check if the retrieved pieces of context were relevant to the query that was asked. On the second edge, there's groundedness. Groundedness is checking is the final response supported by the context. And then finally, with answer relevance, we are checking is the final response relevant to the query that was asked. So this is the RAG triad. And in the DeepLearning AI course, we discuss how to implement these kinds of evaluations within True Lens programmatically, automatically, and you can make use of them out of the box. Now, once we transition from rags to agents, where things get a little more sophisticated, is this first step that Jerry was talking about, much like Jerry said, for, especially for data agents, you can think of that initial reasoning step of the agent as a layer of abstraction over a rag. The rag triad becomes the agent quad when the user asks a query. Then there is a tool selection step which needs to be evaluated as well, which is done by executed by the agent. And that involves both picking the right tool as well as translating the original user query into a query that is in the right format for the tool that was selected. And then once that is done, the next few steps are similar to what you would do for a RAG based application. You want to check for context, relevance, you want to check for groundedness, and you want to check for answer elements. And what I'll do next is to just give you some concrete examples of how this plays out. When we work with the example of the LLM agent that's making use of the Yelp API and then we will, Jerry and I will tag team and share with you a bit of the flow, the code in the notebook that makes it all real in this first step here. For this particular example, because there's only one tool which is the Yelp API, tool selection simply reduces to query translation. This was the original question that was asked what's the address of Gumbo Social in San Francisco? And the LLM agent translated it into this question. For the Yelp API address of Gumbo Social in San Francisco. And the automated evaluation that is available in True Lens checks for semantic equivalence between these two between these two questions, leveraging LLMs. So some of the evaluations.