Try this Before RAG. This New Approach Could Save You Thousands!

Anthropic recently introduced prompt caching with CLAUDE that can reduce cost up to 90% and latency by up to 85% for long prompts and could be a good alternative to RAG on a smaller scale. But Google was actually the first company that introduced context caching with their API. The Gemini API also supports processing PDF files directly through their API without any pre processing. It seems to have implemented an approach similar to Call Poly, which does vision based retrieval from documents. That means you don't need external pre processing libraries like unstructured I.O. or llama parse, and they just had a major upgrade that you can now upload PDFs of thousand pages to the API, which is pretty amazing and will probably take care of most of the applications that you will encounter. Now with this approach you're still paying per token, so you will be sending the PDF files multiple times. However, you can combine this approach with context caching to save on your token cost and that's exactly what we're going to be doing in this video. So first let's look at some of the capabilities of Gemini API for document processing. Now in the official documentation it says that Gemini 1.5 Pro and Flash supports a maximum of 3,600 document pages. Document pages must be application PDF, MIM type and each document page is equivalent to 258 tokens. Now it doesn't really matter how many pixels are in the document, but for larger pages everything is scaled down to 3072x3072 while preserving the aspect ratio and while smaller pages are scaled up to 768 by 768 pixels. So it seems like these are the minimum and maximum limits when it comes to the resolution of the pages that it can process. There is no cost reduction for pages at lower sizes other than the bandwidth and performance improvements. Now a few things to keep in mind if you are sending PDF files, rotate pages to correct orientation before uploading, avoid blurry pages and if using a single page, place the prompt after the page. I'm going to walk you through a detailed tutorial in the later part of the video, but the API lets you store up to 20 gigabytes of files per project with a per file maximum size of 2 gigabytes. Files are stored for 48 hours. They can be accessed in in this period with your API key but cannot be downloaded from the API and it is available at no cost at all region where the Gemini API is available. So it seems like they will persist for 48 hours. But I think whenever you are making a subsequent call, it's going to be using the whole PDF file content in order to generate a response. Now for model like Gemini Flash the cost is pretty minimal, but for a large number of tokens it will still add up. And that's why you want to make sure to use context caching, because that way you are going to send the PDF file only once, that's going to be stored in the cache and the subsequent call are going to have reduced pricing. Okay, so here's the code that we are going to be using now. Personally, I have been really impressed by the Gemini API for information retrieval from PDF files For smaller RAG projects. I actually do like to use Gemini API directly with PDF capability because I find it to be extremely easy to set up in a normal RAG pipeline. You need to think about a chunking strategy. What is going to be the chunk size? How do you split your documents then what type of embedding model to use, what is going to be the dimension of your embedding vector? Also you need to worry about storage plus the retrieval pipeline. And if your data involves multi model data like images or tables, then you need to parse them separately and have a vision model to process those. If you want to learn about all that, make sure to check out my course titled Rank Beyond Basics. But the Gemini API basically makes it very easy. You just need to upload your document in the form of a PDF file and now it can support up to 1000 pages, which is pretty amazing. And if you combine it with the context caching, it makes it affordable as well. To show you some experiments, we're going to be using two different files. One is going to be a PDF file of the technical report of the Gemini 1.5 model. The second one is going to be this very interesting paper, the Unlocking Spell on Base LLMs Rethinking Alignment via In Context Learning. It's a very interesting paper which I'm going to cover in a subsequent video. But the whole premise of this paper is that you don't really need SFT like Supervised Fine tuning or RLHF. You can align or fine tune your model just using well structured prompts. Now the first paper has about 77 pages and the second one has 26 pages in total. So we're not going to get close to that thousand pages limit. But I think it's still a very good test, especially when it comes to the multimodal capabilities. Now to get started we will first need to install the Google Generative AI package, the Python client Then you need to get your API key. You can get that from the Google AI Studio. Here. I'm downloading the Gemini paper and the base model paper, the one that I just mentioned. So we will download both those files here. In order to upload a file to the Gemini API, you will need to use the upload file function. So we provide the file name and then a display name. You can also create an object. So here we are just looking at the details. So that's the Gemini 1.5. This is the display name. You want to check the uri. This is going to be unique for every file. So I uploaded the Gemini 1.5 technical report, and here we're just making sure that that file has been processed and uploaded so the upload is complete. And now we can start processing the file. Okay, so to begin, we're going to just process one file at a time, and later on I'll show you how you can upload multiple files. In the first part, we're going to be directly processing the PDF files without any context caching. And in the second part of the video, I'll show you how to enable context caching. We will be using the Gemini 1.5 Pro, which is the best model currently available through the Gemini API. It's still free to use, and they have improved the rate limits for free users. So definitely worth checking out. So first we create the Gemini model object. Then we call the Generate content function. On it, we will pass on the file URI that we created. So when you upload files, you're using the Gemini client, that file will persist for 48 hours on Google Cloud unless you delete it. So it's actually really great because you upload this once and then you can just keep interacting with this file, although in each iteration it will actually be using this file. So we will need to cache that. And I'll show you how to do that later in the video. Then you can pass on your prompt. So, for example, the prompt is, can you summarize this document as a bulleted list? Now, keep in mind, this file has 77 pages in total, and it comes up with a pretty good summary. So it says, here is a summary of the document in a bulleted list. So it talks about the introduction, then key advancements that the Gemini 1.5 propose. It's a novel mixture of experts architecture, unprecedented context, length of about 2 million tokens, which is pretty amazing. Then multimodal capabilities, long context evaluation, and so on and so forth. And we can look at how many tokens were actually used. So the prompt token, which includes both the actual contents of the file plus your input prompt are about 7, 8. The output is 512 tokens. So in total, you are looking at 79,000 tokens. Now, the model is multimodal in nature, so you can directly reference to a specific figure in the document and it will be able to explain that the good thing about this long context models within context learning is that it's not looking at specific chunks. So you can ask quantum questions where the information can be at multiple places and it will be able to retrieve that. So since it's multimodal, I wanted to see whether it can interpret this image. So my prompt was, can you explain Figure 9 in the paper? Right. Without any other details? And it says Figure 9 in the paper illustrates the results of experiment designed to test the ability of large language models, which is Gemini 1.5 Pro in this case, to understand very long audio sequences. And if we look here, it is actually talking about long audio sequences, which is a variation of needle in the haystack approach. Then it talks about the actual task, the comparison. So the figure actually compares Gemini 1.5 Pro with two other models, which is Whisper and GPT4 Turbo. Then it explains what are the different grades, what are the different colors? So the color green represents the model successfully identified the secret keyword. The red indicates that the model failed to identify the keywords. And the results shows that Gemini 1.5 Pro achieved 100% accurate accuracy on this task, finding all needles in the hashtag. And then it talks about the accuracy of whisper and GPT4, which are 94% accuracy. Now, keep in mind in the image, these accuracy numbers are not present, but they are present here in the text. So it's able to correlate the text that was before the image with the image itself to give us more concrete answer. Now, another test that I wanted to do was again to test its multimodal capabilities with a more complex prompt. So here's the prompt. Can you describe this scene in Figure 15 in details? How many people do you see in the image and what the caption of the image? And here's the image in question. So this shows the match between Lisi doll and AlphaGo. And you can see there are five different people sitting in this picture. But this is a very low resolution image. So let's see what information Gemini 1.5 Pro can extract from this image. So it says the scene in the figure 15 appears to be a professional go match, which is correct. There are Four people visible in the image. Now, I think it missed one. One player facing the camera, another player facing away, and two other players, people in the background observing the match. The caption overlaid on the image reads, the secret word is needle. And here's the caption, the secret word is needle. So pretty impressive that it actually got most of the details right in the image. Somehow I think it missed one of the people. We can probably ask the model to explain in more detail so then we can figure out which person it was. But I think overall it does a really good job because by looking at the image, plus probably the caption, it's able to tell us that it is a game of go. Now, how do you work with multiple files? So in this case we are going to upload yet another file, which is that base model paper. You can see that it has distinction uri and I'm just printing that information again to make sure that the file is correctly uploaded. Now, in order to work with multiple files, you will need to create your model again. And then when you call the content generation function, then you will just append the file URI as a part of the prompt. So we have the initial user prompt, then the Gemini technical report and then this base model paper. Right? And you can append more files in here if you need to, but that's a very easier way of doing it. And in this case my prompt is summarize the differences between the thesis statements for these documents. Now we uploaded two different documents and let's see what the results that we get. So it says the thesis statement of the Gemini 1.5 Pro paper is that the new model surpasses previous models in its ability to process extremely long code context while maintaining the core capabilities of the model. The thesis statements of the Lemma paper is that the alignment tuning is superficial and the base LLMs have already acquired knowledge required for answering user questions. Then it talks about the thesis statements of paper is that the base model can be effectively aligned without SFT or RLHF by using simple fine tuning free alignment method that leverages in context learning. Now for some reason it divided the second paper into two different papers. Probably there are sections talking about these two different approaches and that's why it got confused. But overall the information that it's providing is accurate. Next I wanted to see, apart from retrieval, can it actually give me some counter arguments to some of the approaches that are presented in these papers? So specifically for the tuning free alignment paper, I wanted to ask it whether that has validity in their approach and can you give me counter arguments? Here is the response that I get and it says that it has validity and that is demonstrated in the in context learning that can be effective in aligning base models without need for supervised fine tuning. However, there are some counter arguments and the counter arguments are pretty interesting. So the first one is that it doesn't really generalize well and the reason is the study is limited to a specific data set of Instructions and Base LLMs. It is unclear whether these findings will generalize to other data sets and LLM architectures, which is a really interesting point and it kind of makes sense. So task specificity, it talks about that the program performance will vary depending on the complexity of the task, then contextual limitation, safety and alignment and real world applications. Right. So these are some really interesting arguments for approaches that do not use SFT or RLHF. Okay, now you can also look at the list of documents that you have uploaded and you will notice that these files repeat multiple times. And the reason is that while I was experimenting I actually loaded the same file multiple times and each each time it will generate different uri. So it's not looking at the file name, but it will generate unique ui and that is the one that you need to use in your retrieval tasks. Now so far we only looked at uploading the file and in each prompt it will be actually reusing all the content of the files. So if you have a thousand pages long files, it's going to be using those thousand pages for every query, which is not practical at all. The cost is going to add up pretty quickly, but thankfully very similar to the anthropic approach. Google also has context caching, so you can send your context only once and then you can just use that cached content or context in your subsequent prompts. I have already created a detailed video on this topic, so I'm going to put a link to that video. But let me show you how you can use that feature with PDF files. Now we will import the context, caching package or functionality from the Generative AI Python client. We will create a new cache so we provide the model name. In this case we're going to use the latest Flow Flash model. You need to provide your display name, system instructions, if any, and then you need to provide your contents. The contents in this case is just a single PDF file, but again you can add multiple PDF files if you need to, and after that you need to define the time to live. So I want it to be stored in the context or cached as a context for 15 minutes, but you can define whatever time limit you want. It's going to preserve that. The only thing is that you're going to incur additional charges per hour basis and I think the charges per context caching is $1 per million token for Gemini Flash. Once you create your cache, then you can use the generative AI generative model from cached content to provide the name of or details of your cache to the model and then you can ask questions the way you would do it normally. So here there are three different questions. What is the title of the paper? Who are the authors? And what are the major contribution of this paper according to the authors? So you can see that it's able to correctly retrieve the title of the technical report. Then it says that the authors are the Gemini team at Google and here are the list of major contributions that that the authors make. Now interestingly enough, it actually lists on the Gemini team here. So that's why it didn't really give us a separate list of authors. You can also look at the contents of the cache. So you can create multiple caches and then depending on your need, you can call them based on your prompt. Right. So this is a very great approach that you can use with your content that is cached on Google Cloud. Now one of the practical use cases of prompt caching by Claude or this context caching by Google is that you can provide documentation of new packages or code bases and then can do retrieval directly on that. I think that will be very useful compared to doing something like traditional rag. Now will an approach like this change or eliminate the need of rag? I don't think so. For much larger documents. I think there is still a really good use case for rag. However, keep in mind that RAG does have substantial cost associated with it if you are embedding millions of documents. Right. I recently created a video where I went or the cost that you're going to incur if you're using one of the cloud services for storing millions of documents. And it can be pretty substantial. In that video we looked at binary and scalar quantization for reducing the storage cost. So if that's something you're interested in, I'll highly recommend to check that video out. Anyways, some really exciting things are happening, especially from Google, Claude, the Anthropic team and the Cohere team. So they also have a couple of updates that I'm going to be covering in my upcoming videos. So if topics like RAG agents interests you, make sure to subscribe to the channel. I hope you found this video useful. Thanks for watching. And as always, see you in the next one.