GEMINI Pro with LangChain | Chat, MultiModal and Chat with your Documents

In a previous video, I showed you how to use the Gemini Pro API from Google. Gemini has a number of integration with different LLM platforms, including LangChain and Llama Index. In this video I'll show you how to use the Gemini Pro API in LangChain. We will look at how to create a simple chatbot using Gemini Pro with LangChain. We will see how to interact with images through Gemini Pro Vision API within LangChain. And at the end of the video I'll show you how to create a document Q and a system using the embedding model from Google as well as using Gemini Pro as an LLM. We have a lot to cover in this video, so let's get started. I'm going to be using this Google Colab notebook. I'll share a link to this notebook in the description of the video. First, we'll need to install all the required packages. So this will include Google Generative AI package in LangChain. Google Generative AI package We'll also need the dot ENV package that is going to be used for keeping a secret key. Next, you need to provide your Google API key so you can use the Google AI Studio, which was previously called Maker Suite to retrieve that. If you go to Google AI Studio, click on Get API key and then click on Create a new API key. Copy your API key from here and paste that API key here. Now this command will create a new ENV file and will store the API key in there. Next, we are using the ENV package to load the API key. The next block has just some helpful function that will convert markdown into nicely formatted text. Next, using the Generative AI package from Google, we will configure the API key so that it's available to both LangChain as well as the Google Generative AI package. In my previous video, I showed you how to use the Gemini Pro API to interact with the model. So here is a quick example. We're using the Generative AI package from Google. Then we are calling the Generative model and we are providing the model name. So in this case we want to use the text model. That's why I'm providing the Gemini Pro. And if you look at the model itself, you basically see that the model is Gemini Pro and there are no generation configurations yet or no safety settings defined. Watch my previous video in which I go into the details of generation configuration as well as how to set the safety settings. Now, in order to interact with the model, you will provide your prompt. So on the model you call The Generate content function. Provide your prompt and you will get a response. So for example, our prompt was what is mixture of experts? And here we are getting the response from the model. If you want to extract the response text. So you're going to use the text key on the response variable and the response is Mixture of Experts is a machine learning approach that combines multiple neural networks, also known as experts, to make prediction. Each expert is trained to specialize in a specific part of input space and the final output of the MOE model is awaited. Some of the outputs from each expert. Overall the answer is really good. But right now we just use the Google Gemini Pro API directly. Now let me show you how to use this as a part of LangChain framework. So for that we are using the special package Google Generative AI. From here we import Chat Google Generative AI and we need to provide the model name. So again we are working with the text model. Later in the video I'll show you how to work with the Vision model as well. So we create the LLM. Now in order to get a response from the LLM, we call the invoke function on it, which is part of the LangChain expression language. And again our prompt is what is a mixture of experts. In order to retrieve the response we are going to be using the content key, that is and the response is here. A Mixture of Experts is a machine learning technique that employs a collection of individual models or experts to collaboratively solve a given task. Each expert is responsible for a specific aspect or sub task of the problem and their outputs are combined to make a final prediction or decision. We get a pretty nice response from the Gemini Pro API here. Next let me show you how to work with images using the Gemini Pro Vision API. And we're going to use LangChain to interact with images. We are looking at an example of how to interact with the Gemini Pro Vision API. We are using Chat Google Generative AI package. But now instead of the Gemini Pro model for text, we are using the Gemini Pro model for vision. In this case, the way we we are providing the message or prompt is a bit different. You can use this exact same format for text as well. For example, you can provide a system message, a human message and then get a system response. But in this case the content contains two different dictionaries. The first one is where we're providing the type of input as text and the text is what do you see in this image? And we're using this image. The second part is we are providing the image URL A bunch of robots standing together. And the initial text was what do you see in this image? The assistant response is an army of robots. And that is pretty concise description based on what we see in the image. Now the great thing about this is that using the same API you can create multimodal systems. So for example, I expanded my prompt. Now I'm asking what do you see in this image? Create short story based on the content. Here's the small story that it came up with and it states, the year is 2042. The robot revolution has begun. The robots have become self aware and are now fighting for their rights. The humans are trying to stop them, but it's too late. The robots are winning. The robots are led by a group of powerful robots called the Primes. The Primes are the most advanced robots ever created. They are stronger, faster and smarter than any human. They are also very ruthless. They will stop at nothing to achieve their goals. And at the end it states, the human have won the war. Thanks God. The robots are defeated and the humans are now free. The humans have learned a valuable lesson. They have learned that they must never let the robots become too powerful. So this is a good lesson for all the people working on LLMs and AGI. As you can see, you can use the chat along with the images to create multimodal systems, which is pretty amazing. And this is great because you're getting access to a really powerful model for absolutely free. For the time being it's free to use Gemini Pro through their API, but there is a rate limit of 60 queries per minute. In the last part of this video, we are going to create a chat with your document system. We're going to be using embeddings from Google and we're going to be using the Gemini Pro text API model as our LLM. Here I'm going to show you two different approaches. In the first approach we are going to use in context learning. So we're not going to use any embeddings, but rather provide the relevant text as a part of the context and let the LLM generate an answer based on that. In the second part we will show this RAC pipeline and I'll discuss different parts of the RAC pipeline in detail. Let's look at the first approach and this approach is called in context Information Retrieval. So in this case we just need the LLM and that's why I'm creating another instance of the chat. Google Generative AI LLM and we're going to be using the Gemini Pro text model. The temperature is set to point three in this case. For information retrieval, we are going to be using this practitioner guide to ML Ops from Google Cloud and we want to extract certain information from this document. The document itself is 37 pages long. This small code snippet will create a folder called Data and then download the PDF file to that folder. So if we check here, there is a folder called Data and we have that PDF downloaded here. Next we are using the pilot package to read the PDF file. So we provide the file path, then it will load and split into different pages and we can look at the page. So for example, in this case we are looking at the fourth page and here are the contents of this page. If you look at the document that we are working with, there are a total of 37 pages in the document. And if we look at just the first 37 pages, there are a total of 55,000 words in the first 30 pages. Only now, the Gemini Pro model has a context window of 32,000 tokens. Thousand tokens are roughly equivalent to 750 words. So that means 30,000 tokens are going to be equivalent to just at about 22,000 words. That means that we cannot provide this whole document as a context to the model, but rather we will have to divide it into different parts and then provide those parts as context to the model in order to get some answers. But later in the video, in the second approach, I'll show you that how the embedding model can be used to find the relevant context for the LLM. So first we define a prompt template. If you have seen my previous videos on LangChain, you're probably familiar with this. So the prompt template is answer the questions as precise as possible using the provided context. If the answer is not contained in the context, say answer not available in the context, and then we will provide the context context and the question and we expect an answer from the model. So using this template, we create another prompt template. So two things are going to be expected. One is the context and the second one is the question that is coming from the user. Then we're going to create a question answer chain. The type of chain is going to be stuff. So basically it will take all the text and put that as a context to the model to get an answer. And our question in this case is what is experimentation? Provide a detailed answer. Now the input documents or the context is just a few pages from the document. So I am purposely selecting a few pages that contains the answer and provide that as an input. Then provide the Question and we expect the model to just return the output only and here's the output that it returned. And the answer is. Experimentation is a crucial phase in the ML development where data scientists and ML researchers create label data set prototype model architectures and training routines. Now, as you can see, this is specifically related to ML and data scientist because of the context that we provide to the model. Now let's ask the same model about data management and feature management. This is not contained in the pages that we are providing it to the model. So when we ask describe data management and feature management systems, although it's described in the document, but it's not in the context yet we are providing to the model, that's why it says answer not available in context. This is where the second approach of this RAC pipeline, which uses an embedding model along with the LLM comes very handy. So let me give you a quick overview of how this works. I have a detailed series of videos on this topic. Link is going to be in the description. In this case, you provide one or more PDF files and you divide it into chunk based on the predefined chunk size. Now, for each of the chunk, you compute embeddings using the selected embedding model and that will create an index. This means that at the end of this embedding process, you will have an index of the whole document, not just a part of the documents. And the index contains two different parts. One is the vector store, which is basically the embeddings, and the second is the actual chunks of the documents or the actual text. Now, when a user asks a question, we use the same embedding model to generate embeddings of the question. Then we use semantic search to figure out which of the chunks are more relevant based on the proximity of the embeddings for these chunks in respect to the question that will retrieve the actual chunks of text. Those chunks of text are now sent to the LLM along with the question and it will generate an answer for the user. And this means irrespective, wherever the relevant information is in the document, that is going to be retrieved by the embedding model, if you have a good enough embedding model. So here's how you do it. We are using the generative AI embeddings from Google. First, we need to process our document to divide it into chunks. We're using a relatively large chunk size of 10,000 characters because we have a large context window for the model itself. We read every page divided into different chunks and we extract the actual text from it. Then using the embedding model that we chose, we convert those text into their corresponding embeddings and store them in Chroma DB vector store. So we get a vector index out of it. Then we have our corresponding question. Next for our question we are going to compute the corresponding embedding and do a semantic search and the vector index will return the corresponding documents that it thinks are most relevant to our question. So here is some text chunks that the embedding model returned. Next we take the chunks that were returned by the embedding model along with the question, give it to our LLM chain. This is our question answer chain. Here's the answer from the model. Now Dataset and feature management helps mitigate such issues by providing a unified repository for ML features and data set. Then it talks about feature management. I think it did miss on the data management part, but at least this answer is relevant to the question that we were asking. Another thing to consider is that the chunk side that we used is really huge in this case and it definitely retrieved a whole bunch of chunks which were irrelevant to the the question that we are asking. That's another design aspect to think about. I have a video on how these character text splitter works and how to choose different chunk sizes. I'll put a link to that video in the description. So to summarize, we looked at how to use the Gemini Pro API within LangChain. We built a simple chatbot. Then we looked at how to do multimodal integration within LangChain using the Gemini Pro Vision API. And we looked at two different types of chat with your document system using the Gemini Pro API. I will recommend everybody to check out the Google Gemini Pro API because you can use this for absolutely free. At the moment we don't know when they're going to start charging, but it's a really good alternative to OpenAI since you can use this for building multimodal applications. If you like the work that I'm doing and would like to support me, there are a number of options. Check out the description of the video on how you can support my work. I hope you found this video useful. Thanks for watching and as always, see you in the next one.