Gemini 1.5 Pro - Coding Assistant with 1M tokens

Okay, so I just got access to Gemini 1.5 Pro, which has a context window of 1 million tokens. I have been testing this for a bit and it's pretty amazing. This version has the ability to do complex reasoning, it's multimodal from the ground up and is better at coding. Now, in this video, we are going to be testing its coding abilities, but rather than asking it to code as a Snake game, we're going to ask it to work with a package that it hasn't seen before. And in order to test this model, we're going to be using the Google AI Studio. Under models, you can see whether you have access to this new version or not. So here I have access to the Gemini 1.5 Pro. When I select this, you can see the number of tokens that it can process changes, and it's close to 1,048,000 tokens. When you select the Gemini 1.5 Pro, you will see that it's multimodal from the ground up. You can upload images, it can even look at videos, you can upload individual files, or you can upload whole folders. In this video, we're going to test its coding abilities and see whether we can use it as a coding assistant. And the way we're going to do it is we're going to provide it a Python project that probably is not in its training data. And that project is going to be my own Local GPT project. This was released almost nine months ago, so I think this project is probably not in the training data. So in this case I'm going to simply provide all the Python files and the readme files. We're not going to include anything else. Now, to see whether Local GPT is in its training data or not, I ask the Gemini model, what is Local GPT? And here's the answer that it came up with. Local GPT is an open source project that allows you to run large language models like Chat GPT on your own computer. Completely offline. This means that you can interact with the LLM through chat interface like Chat GPT and utilize LLMs capabilities for tasks like text generation, translation, summarization and so on. Now, this definitely is hallucinating. Local GPT does allow you to interact with LLMs, but it's a RAG pipeline and in here you don't see anything regarding rag. So it's definitely just making up things at this point. I created a folder in which I copied all the markdown files as well as all the Python files from Local GPT Repo and We are going to upload all these files to Gemini 1.5 Pro and let's see if Gemini can understand the code and also answer our questions based on this code base. Okay, so for this we're going to start another conversation. We will select the 1.5 Pro, then we're going to click on folder. So here we navigated to that folder and we'll simply click upload. This takes some times because it has to upload all the files to Google AI Studio and the way it works is it simply append all these files as a part of the context window. So now you can see that it has used around 24,000 tokens out of the 1 million tokens that are available. So basically it's putting everything in the context window and it's going to be doing in context learning. So it's not doing any rag or retrieval in that sense. Now we're going to ask the same question, what is local GPT? And here they are also providing this Timer. Usually Gemini 1.5 Pro takes a long time. I have seen for the timer to go up to 60 seconds. So now it says local GPT is an open source project that allows you to have secure private conversations with your own documents. It runs entirely on your local machine, ensuring that your data never leaves your computer. Now it's getting the description correct and this is in the readme of local GPT. It's also highlighting different features so privacy model support, embedding supports and LLM reuse. So this actually is correct. Now the local GPT code base is structured in multiple files and for the Gemini to have a better understanding, it needs to figure out what is the relationship between different files. So here is a prompt that we are going to use. What is the default LLM in local GPT? Now this is not mentioned in the readme. So it's not going to be able to just directly go to the readme and figure out what the default LLM is. It actually has to go and figure out in one of the Python files. So it says the default LLM in local GPT is the Bloke 7B Chat GGUF, which is actually correct. Now we asked it where did you find this information? And it's still processing the prompt. You can see that we are around 40 seconds in and it says I found this information in the constants PY file of local GPT repo and it lists the file correctly. So this is pretty neat. Okay, so sometimes I actually get into this issue. So for example, I ask a question and here it says no content. So I think we will have to rerun this. And this is pretty neat. So it figured out that Local GPT supports GGML GGF models, and for that we use the LLAMA CPP library. For GPT Q models, I'm using the auto GPT for causal lm. There's support for AWQ models. Now for this one, I think it didn't figure out the proper package because in this case we use the auto AWQ package. But overall it's actually pretty good. The responses are accurate. Okay, next let's ask it if it can help us with actual coding problems. Local GPT comes with an API. And in this case I asked this, how do I use local GPT API? So there is a file called run local GPT API.py file. This has the API implementation. And in order to chat with the model there is this prompt route. So it actually correctly figured out the API usage. So here it's providing the API endpoint route and all the formatting is also correct. So what key we will need to send that is also correct in here. So we make a post request and then it checks that whether the status code that the API endpoint returns is 200 or not. So if it gets a 200 response, then it retrieved the response sent by the model. Even though it's a smaller code base, it does a pretty good job correctly coming up with the API usage. And from that file it also figured out that there are few other endpoints. So for example, there's a delete source file. If you want to upload files, there's a save endpoint. And in order to ingest those, and if you want to create embeddings of your documents, there is a run ingest endpoint as well. And for each one of them it correctly identified that whether we need to send a get request or a post request. So this is pretty neat. Okay, so next I actually asked it to create a small streamlit app. So said write a streamlit app that will use the Local GPT API to do the following. So the user will be able to upload PDF files, Local GPT will ingest the uploaded files, and it has to figure out which endpoint to use for each one of these operations. Then the user will be able to ask questions using text input box. Local GPT generates a response and that will be shown to the user. So here's the code that it came up with. So first it has the API URL. Now here it made a mistake because it added the prompt route as part of the base URL, which is incorrect and this caused some issues. But other than that, let's look at the rest of the code. So it created upload button, where you can upload files and it checks whether the file has been uploaded or not once the file is uploaded. So it calls this ingest PDF function. So in here it first makes a call to the saved documents endpoint, which is important because it first have to save the document that is uploaded by the user. And once the document is uploaded correctly, so it actually checks here whether there is an error or not. But once it's uploaded correctly, then it make a call to the run ingest endpoint. So it figured out that it has to make these calls in the correct sequence. And once you have that, then it waits for the user input when user ask a questions, then it's calling this ask question function. And in here it's making a call to the prompt route endpoint, which is basically it routes the user query to the retrieval QA chain that we have created within local GPT and it gets a response. So it correctly figured out all the API endpoints, calls that it's supposed to make and in the correct order. Now, the major issue was this because it was adding all these other endpoints on top of this. So this was causing some issues. So I actually asked it to fix that. And we went back and forth a little bit to fix this issue, but Gemini was not able to figure out this part, so I had to manually correct this. Okay, so let's see if these code updates actually work. So here is the Run local GPT API and here's the streamlit app that I created based on the code that Gemini provided. As I said in the beginning, I just had to correct the base URL for the API. Okay, so first we will kick off the local GPT API here. All right, so the API is running. Okay, next I need to run the streamlit app that we created. So for that we're going to use the streamlit run command and the file name that we're using is streamlit underscore app. Okay, so here we have our local GPT streamlit app. There is a place to upload files, and then there is a place to ask questions. In this case, we're going to upload the doc LLM paper and here on the back end you can see that the file was uploaded, it was saved, and now it's running the ingestion. Okay, so the file is successfully ingested. Next we ask what is DOC LLM and it's running the query through through the retrieval chain. Okay, and here is the response that we got and it says based on the context provided, it appears that DOC LLM is a lightweight extension to standard language models that excels in several visually rich form understanding tasks. It's actually amazing to see that Gemini Pro was able to help me out with this task. Then I asked it to convert this flask based API into FAST API. Now in this case it gave me the direction how to convert the flask API code base to fast API. So for example, here's how I need to change all the different API endpoints that I'm going to be using. So this is pretty neat. Now throughout the conversation that I was having with Gemini, it was simply appending everything within the context window. During this conversation I'm just using around 30,000 tokens out of the 1 million tokens that it supports. So in general I have found this to be very useful. I'm going to be building with the help of Gemini 1.5 Pro. Now currently it's not available through the API. You have to use the Google AI Studio and even the Gemini Pro version that is available through the API. It has a rate limit of 60 requests per minute. So you can use this for your personal projects, but for commercial use it's probably not a viable option. But the great part is that it's absolutely free and you can experiment with for free both here within the Google AI Studio as well as through the API if you're using any version of the model there. So I'll be building a lot more projects on top of Gemini as well as a lot more content in follow up videos. We're going to look at its video understanding abilities, also its vision, so we're going to work with some images. I have a lot of exciting contents in the pipeline based on Gemini, so if you're interested in something like that, make sure to subscribe to the channel. I hope you found this video useful. Thanks for watching and as always, see you in the next one.