Gemini Agents are Surprisingly Good

On Chatbot Arena Leaderboard, the new Google Gemini 1.5 Pro experimental model is currently sitting at the top, but it mostly measure human preferences. Is this model any good at agentic workflow? Let's find out. For agents, you also want to track each of the steps taken by them. And for this we're going to be using langtrace, which is today's sponsor, but more on them later in the video. So what exactly is an agent and how it works? Agent is essentially the LLM with some additional capabilities. These include planning, so it's able to decompose the user query and come up with a comprehensive plan. It also has access to some external tools or APIs. So in order to execute the plan it will need to use some external tools. And then it also has memory which keep tracks of where it is currently in the plan execution and what additional step it has to take. In our case, the LLM is going to be Gemini 1.5 pro experimental and Gemini Flash. And it's going to have access to two different tools. One is going to be RAG and the other is going to be Web Search, so it can look up information on the web and we'll keep track of all the steps that it takes using langtrace. Okay, so let's get started. Link to this Google Colab is going to be in the video description. First we need to install different packages that we'll need. So this will include the Google Generative AI package, LangChain and Twelly Python. This is going to be our search engine. We also need to install FIAs that is going to be Vector store that we're going to use. And we'll also install the langtrace Python SDK. Then we import the different packages that we need. So this include the recursive character text splitter pypdfloader for reading PDF files. Then Twelly search results is the search engine fias and the Google Generative AI embedding model. Next we need to set up an API key. So first let me show you how to get the Twelly API key. Twelly is a search engine that you can access through the API. You can also replace this with something like DuckDuckGo. But Twelly gives you up to 1000 requests per month. So I think it's really good to get started and it is a very powerful search engine. In order to get your API key, just click on this Get API key link is going to be in the video description. You will need to copy your API key here. Currently I'M setting my API key in my Google collaboration notebook as secret, so you can set that up for the Google API key. You can get that from your Google AI Studio. And for langtrace, you will need to create an account on langtrace. Langtrace is an open source observability platform for LLM applications. This follows the OpenTelemetry standard so you can actually connect it to your own applications without relying on their cloud services. Infunctionality is very similar to Langsmith, but it has compatibility with a number of different LLM providers as well as different frameworks. That also includes LangChain, Llama Index, Lang, Graph, DSPY and a lot more. You can either use their cloud services or you can run this locally by self hosting it. In my case, I'm going to be using their cloud offering. So sign up for an account. You get 50,000 traces per month, which is very generous. Then you can click on Create Project. I'm going to call it Gemini Pro Agent. Then click on Create Project. Next you need to set up the project, so click on Setup Project. We'll need to get our API key. So click on Generate API Key and then copy that key. Okay, so I pasted my API key here and created a new environment variable called langtrace and now we are all set. So I'm going to run this code. If you see these errors, don't worry about them. Basically Langsmith, when you run this code block and you see these errors, don't worry about it. It's basically looking at different integrations and noticing that those are not installed. We are only interested in LangChain and the Gemini Pro integration. At this point we're going to be building an agent and we want the agent to have access to access to RAG and web search for rag. I'm going to be using the attention is all you need. Paper. This is going to act as our knowledge base. So I already uploaded the paper here and renamed it to transformers PDF. So here you can see I'm loading that PDF file. Then it has a total of 12 different pages. After that I'm installing the Python Tick token package. This is going to be used for counting the number of tokens that are going to be used for splitting or chunking the documents. For this video we're going to set up a very simple basic RAG pipeline and first we will use the recursive character text splitter to chunk or split our document. In this case I'm using the GPT4O model. So that is basically the tokenizer that I want to use. Each chunk is supposed to have 500 tokens with an overlap of tokens. So we're going to create a very basic Rack pipeline. But if you're interested in learning Rack Beyond Basics, I have a course that goes into advanced Rack techniques. Make sure to check that out. Link is going to be in the video description. Okay, so we set up a splitter, then we pass on the documents that we just read and split those documents. So as a result, you're going to get get different chunks which can have up to 500 tokens. So there are going to be a total of 24 different chunks that this tokenization or chunking process is able to create. Now we can randomly look at some of them. So for example, this is chunk number five. As I said, it can go up to 500 tokens. It doesn't have to be exactly 500 tokens. So you're going to see chunks of different lengths. Again, if you're interested in learning more about how it works, I highly recommend to check out my course. Now, if you go back to Lang trace, you can actually see what exactly is happening. So here are the traces that is able to log here. You can look at these. So for example, this is a base PDF load. So this loaded the PDF and split it into different chunks. If you hover on this, this shows you some information also how long each process is taking. And you can look at different chunks that are being created in here. Okay, next we're going to load our embedding model. For this example use case, we're going to be using an embedding model from Google, but you can use any other embedding model that is compatible with LangChain. Now let me show you how you can compute embeddings and what is the dimension of each embedding vector. So we're going to load the first first chunk or from our split documents, we pass on, pass it on to the query embed query function on the embed object. And then we can see that it has a total of 768 dimensions. Now, in order to create embeddings for all of the chunks, then we are using fais. You can use Chroma, Pine cone or any other vector store of your choice. But we pass on the documents plus the embedding model that we just created and we set this as a retriever. Okay, if you go back and look at the traces, you're going to see that there is now a new trace that is the files vector store creation process. Right now, tracing is Very important, especially if you're using production ready systems. And that will become evident when we are orchestrating the agent. So we created a retriever. This is going to be used for the search part. You can just run a query on the retriever and will retrieve the relevant documents. For example, our query in this case is what is transform architecture? And by default LangChain retriever will return up to four different documents. So here you can see that it returned three different documents which are somewhat relevant to our query and you can look at the traces again. So here it's running a similarity search. We will be able to look at both the inputs as well as the outputs that are generated by this search query. Okay, now we're going to set up our tools so that our agent can access those tools. The first one is going to be create retrieval tool. This is a LangChain function. You need to provide a retriever. That's the files retriever. We just created a name of the tool, so I'm going to call it Retriever underscore tool and then a detailed description of what this tool actually does. Now this is a very important part. You want to make sure that you provide enough information because the agent will look at the description and look at the user query and based on the description of different tools, it will decide which tool to use. So in my case the description is for any information related to transformers architecture. Use this tool. Useful for retrieving information related to attention is all you need. Paper. So if there is any query that is looking for information related to transformers or this specific paper, the agent is supposed to use this tool. For anything else, we want the agent to use the tabley search results tool. So we're going to define that and you can look at the internal definition. So the description is a search engine optimized for comprehensive, accurate and trusted results. Useful for when you need to answer questions about current events. Input should be a search query, right? So this is the internal description that this specific tool uses. You can also replace twelly by something like DuckDuckGo search engine which does not need an API key, but you will need to use your own search tool. Okay, now we're going to create a list of tools. So this is going to contain the search tool and the retriever tool. And we need to start creating our agent. Now the agent, as I said, is essentially an LLM. Initially I was planning to use the Gemini 1.5 Pro experimental, but I was running into rate limits. So for this video we're going to be using the 1.5 Pro, but the mechanism is exactly the same. If you have access to higher rate limit, you just replace it with this Gemini 1.5 Pro experimental model. Okay, so we have everything set up that we need. Now we need to create our agent and, and to create agent we're going to be using the React agent class. It's a special type of agent that can do planning. It has memory and it's going to have access to the tool that we want it to use. Now you need to provide a very detailed prompt or instructions to your agent for controlling these tools and coming up with a plan. LangChain also provides some templates. Using LangChain Hub is very similar to Llama Index Hub. So we are going to use the react template. So you can pull this from the hub and you can look at what exactly is in the template. So here is the system prompt or the main system message that we're going to give the agent and you can modify it if you want. So the prompt is answer the following questions as best you can. You have access to the following tools. So it's going to get the list of tools and in our case it's going to contain both the search as well as the retrieval tool. Next, use the following format. So it's going to get a question from the user. The input question you must answer. Then it needs to go through a thought process. So you should always think about what to do. And it will know that it has access to different tools. So then the action to take should be one of the tools. This is basically function calling. Now the beauty of agents is that unlike a function calling, not only it can select the tool to use, but it can also execute it. So there is a Python interpreter. So it will basically execute that function, look at the output, and then based on that it will decide whether it needs to update its span or not. And once it is done with the processing, it's supposed to generate the final answer. So it's relatively simple to create agents using LangChain. So here we're going to create a react agent. We're going to pass on the LLM, which is Gemini 1.5 Pro, then list of tools as well as the system instruction or the main system prompt. And then you can look at the external agent that we just created. Now LangChain uses the concepts of Agent Executor. So you need to provide the agent list of tools and then whether you want the output to be verbose or not. Okay, so we are all set. Now we can ask questions or provide queries to our agent. So the first one is going to be what is the current weather at la? And let's see what it does. The first thought is I need to get the current weather information, so I should use the search engine. Then it decided to use the weather API through the search engine. So it goes through a number of different steps where it looked at the output and then it says the current weather in LA is 24.9 degrees Celsius or 76 degree Fahrenheit and clear. Now it's a little off because it's currently 71 Fahrenheit or 22 degrees Celsius. Let me show you what exactly happened in the background. This is where tools like langtrace can be extremely helpful and powerful. So here you can see that it's taking a number of different steps and essentially there are four different prompts that are exchanged or calls to the LLM. So you can also look at the number of input and output tokens as well as the total number of tokens that are being exchanged. Now this is extremely helpful for a production environment because you also get information regarding the latency, how long it takes took for different steps, and that is critical if you're putting these systems in production. You can also separate these different traces based on the LLM request. So you can see that here there were four different LLM requests. You can sort them by framework request. So if you are using LangChain, Llama Index, DSPY so you can see which different calls are being made there. Next I asked it, can you give me a list of gold medals per country in the current Olympics? Now for this it's again using the search engine and there is also going to be dependent on the search engine that you are using. So I think it's using the Olympics website directly. It says that according to the official medal standings for the Paris 2020 Olympics as of August 5, 2024 today is actually August 6th. At the time of the recording, the US and China are tied to for the lead with 21 gold medals each. Now when I looked at the actual current standing, so US is leading with 24 gold medals, China is at 22. So I'm not sure whether it's accessing that information that's already cached or we might be running into some issues when it's parsing the results. So this is when we ask for current events and it's actually correctly using the search engine API. Now we can do better at finding, filtering the results, but that's going to be A topic of another video. But then I asked it can you explain the concept of attention in Transformers? So here the thought is the question asked for explanation of attention in the context of transformer. This is a concept related to attention is all you need paper. So I should use the retriever tool because we specifically told that if there's anything related to attention is all you need, then use the retriever tool. So it comes up with a plan. It goes through the plan. These are different chunks that are being retrieved by the embedding model that is used in generating the response. It looked at the intermediate step where it says the provided text describes the overall architecture and different types of attention in the transformers. But it's lengthy and doesn't provide concise explanation of the core concepts. The agent is actually smart enough that it's able to rewrite our query so it looks at different core concepts. And that said, explain the concept of scale.product attention in transformer as simple as possible. You can see that it retrieves different context based on that. But it again says thought. The retriever tool is still not providing a concise explanation of scaled dot product, right? So it rewrote the query again and the agent actually has the ability to do that because it can not only plan but also look at the output of the tool usage and then re execute the query if needed. So it also goes through a couple of other steps which is pretty neat. Here is the final output from the model. So can you explain the concept of attention in the transformer based on the rewritten query? It specifically focused on scale.productattention so it comes up with the formula that is used for that. Then it goes into the explanation of how it works including different components like scaling, softmax, etc. In this example we used only two tools, but in real world applications you will need to use multiple different tools. And as I have shown in a previous video, the Gemini models are really smart when it comes to external tool usage. They also have the ability to do the function calling and get the output. Unlike some of the other models where you will have to do function calling if you're using external tools tool directly, not as a part of an agent. So if you're interested in that, I'll put a link to a video related to that specific topic in the description. Anyways, this was a quick video on how to get started with agents using Gemini 1.5 Pro. Both the normal and experimental version Also do check out lang trace. Tracing or observability is a very critical step if you are putting these system in production. It's an open source project, so I highly recommend to support them. I hope you find this video useful. If you found this interesting, make sure to subscribe to the channel. Thanks for watching and as always, see you in the next one.