Is This the End of RAG? Anthropic's NEW Prompt Caching

So Anthropic just introduced prompt caching with Claude that can reduce cost up to by 90% and latency up to by 85, which is huge. And did they just kill rag with this new feature? Now Google was the first one who introduced context caching with their Gemini models. There are some similarities but some major differences as well between these two approaches. We will discuss them later in the video. I'll show you how to get started and what type of performance difference you can expect. Before looking at the code example, let's see what was released today. Prompt caching enables developers to cache frequently used context between API calls. Anthropic models have a huge context window of 200,000 tokens. However, if you are chatting with long documents you will have to send them with each prompt so that becomes very expensive and hence the prompt caching is going to be extremely helpful. So now customers can provide Cloud with more background information and example outputs or a few short prompting, reducing cost by up to 90% and latency up to 85%. Now these numbers are great, but they are not going to be consistent based on the example use cases and we are going to look at some of them. This feature is available both for Cloud 3.5 Sonnet and Cloud 3 Haiku. Support for Cloth 3 Opus is coming soon. As I said in the beginning, context caching has been available for the Gemini models and there are some interesting differences between the two which I'm going to highlight throughout the video. So what are going to be some use cases for prompt caching? Well, the first one is conversational agents. So if you're having a long form conversation and there is a substantial chat history, you can put that chat history in the cache and just ask questions from that. Another example use case is coding assistants. Usually code bases are pretty huge, so you can put them in the prompt cache and then use your subsequent messages for question answer. Also large document processing or detailed instruction sets. This specifically will apply if you have highly detailed system prompt with a lot of Flex short examples. So this is going to be very helpful that you can just send those once and then you can have subsequent conversations. While this is cached, Agentic search and tool usage is another example. Especially if you have to define your tools, what inputs outputs are to different tools. So you can put them in your prompt cache and then send it once and that will save you a lot of money. Another example is going to be talk to books, papers, documentation, podcast transcripts and other long form content. So this is a very enticing application for rag and with these long context models, especially with prompt caching or context caching, now it becomes viable to just put these documents in the context rather than chunking them, computing, embedding and then doing retrieval on the documents. Now here's a table that shows what type of reduction in cost and latency you can expect for different applications. So if you're chatting with your documents and you're sending 100,000 tokens without caching, that would take about 12 seconds to generate a response. But with caching, that's about only 2.4 or 2.5 seconds, which is 80% reduction in processing time on latency and 90% reduction in the cost. If you're doing few Shot prompting With 10,000 tokens, you expect about 30 31% reduction in latency and about 86% reduction in cost. Whereas if you're doing multi turn conversation, the ten turn conversation, so you are expecting about 75% reduction in latency, but only about 53% reduction in cost. Now, the way the cash tokens are charged versus the input output tokens are different and that's why you see these reductions in the cost as well. Now we saw the cost reduction because the cash tokens are costing only 10% of the base input token price, which is a huge reduction of 90%. However, you also need to keep in mind that writing to the cash cost about 25% more than the base input token price for any given model. So there is an overhead when you're writing to the cache for the first time, but then there is a substantial reduction in cost. Now the Gemini models does it in a different way. There is no cost associated with the actual cash token, but there is a storage cost of $1 per million tokens per hour. Okay, so here's what the reduction is going to look like. Now the input tokens cost is going to increase by 25%. So initially if it was $3, now it becomes 3.$75. But for subsequent cached calls this is going to be about 30 cents. It doesn't have any impact on the output token cost. Here are the API reference documentation for this new feature. Now keep in mind the prompt caching is still in beta, so the API can actually change over time. Before looking at the code example, let's look at key differences between their approach and the Gemini context caching. The first major one is how many tokens you can cache. So the minimum cacheable prompt length is 1024 tokens for Claude 3.5 Sonnet and Claude 3 Opus. When it's available and 2048 tokens for Cloud 3.0 Haiku, which are very sensible limitations because when it comes to Gemini context caching, the minimum input token counts for contact caching is about 32,000 tokens. Now what if your document is not of 32,000 tokens length? That means you can't really use the context caching feature in Gemini. However you will be able to use that feature in claude. So that's plus one for Claude. But they have their own limitation. The cache has a 5min lifetime refresh each time the cached content is used. So if you cache something you can only use it within 5 minutes window. If you don't use it within 5 minutes you'll have to cache it again. Now on the other hand, I think Google has more sensible limits. So if you don't set time to live with Gemini API, then it's set to a default value of one hour, but you can change it to whatever time limit you want. But keep in mind there is this additional cost associated with storage when it comes to context caching of Gemini. Now I think this also really limits the usability of the feature and I think it's a reason that you can't replace rag. Because in case of rag, you only need to embed your documents and put them in a vector store once and then you can just retrieve them whenever you need. In this case, every five minutes you will need to send them to this cache with an additional 25% surcharge. And that can really add up over time. But I think there is another way in which you can combine both RAG and this long term or short term caching, but I will talk about that later. In the video context caching is great, but it's not a replacement to Rag. If you want to learn more about RAG beyond basics, I have a course on it where I walk you through a step by step process of how to build robust RAG systems for your own applications. If that's something you're interested in, check out the video description. Now back to the video. Before looking at a code example, let's look at some best practices for effective caching. So they recommend to cache stable reusable content like system instructions, background information, large context or frequent tool definitions. Place context cached content at the prompt's beginning for best performance. So seems like if you put this in the start of your prompt, that's going to be more usable and useful for the model. Then use cache breakpoints strategically to separate different cacheable prefix sections. I'll talk about this later in the video. But you can define four different points of different cache in a single prompt and then regularly analyze cache hit rates and adjust your strategy as needed. Okay, so how does it work? Well, it's a little different than the normal API calls. Now you will need to add the cache control block in your API call, and you'll also need to include this specific piece of code in your header in your API requests. Now they have a beta API endpoint which is client beta prompt caching that you can use for prompt caching, or you can also use the normal anthropic API client. So a couple of examples in where you can use this. For example, large context caching example. So if you're having a large context, you can just add this right in the middle of the context and that will basically cache everything before here. So for example, here we have a system input or system instruction, and at the end of the system instruction we put this block. That means everything that was in the system instruction is now going to be cached. Anything following this is going to be normal API calls which are going to be using this cached context. There are also ways in which you can look at cached tokens if there were any hits of the cache. Right, we're going to look at some of the examples when we look at more concrete code example. Now, another useful use case is caching tool definitions. So usually the tool definition can be very large if you have a large number of tools, so you can actually cache those as well. Here is an example where you have two tools. One is getweather, which has all the corresponding properties, and then the second one is gettime. Now if you put the cache control block at the end of the tool definition, that means that this tool definition is going to be cached. And now you can use that right away with subsequent API calls. Now another interesting application is continuing a multi turn conversation. So for example, if you have a very long conversation with your model and you want to cache a part of the chat history, you can do that. You can use four different cache prompts. As an example, here is a very long system prompt and that is being cached in the beginning of the conversation. But let's assume you keep using this model, have a conversation where you have the user role and the assistant is responding. But at certain point in the conversation you decide you want to cache that conversation. All you need to do is just add this cached block. So as an example, it's cached here. So anything before that is going to be cached and then you have two cached points in your conversation. One is the original system message, the second one is this one. But later in the conversation you can also add another cache. So that means that you're going to be now caching this conversation at three different points. Let's look at a practical example and see what type of reduction in latency you can expect. So this is based on the example that is provided in in their own cookbook. So first we need to install the dependencies that will include anthropic and beautiful soup. And we'll need this because we want to extract data from a web page. Next we set up our API keys. I am storing everything as my environment variable in my secrets in the Google Colab notebook. After that we need to tell it which model to use. So I'm going to be using the Claude 3.5 sonnet. We're going to be downloading text from a web page and that's why we have a function which receives a URL of a text file and then downloads the contents of that text file. So next we're going to provide a link. In this case, this is a link to the Project Gutenberg website. And the text that we are going to be retrieving is the plain text of the book Pride and Prejudice. So you can see that this book contains over 7 million characters. But here we are just printing the first 500 characters. Now this is a substantially big book that we're going to be putting it in the cache. But before that, let's look at a single turn conversation where we're going to be using the non cached API call and see what type of latency you can expect. So here's a function that is going to be making that API call. So first we have a user role where we put the whole text of the book as the context. Now you're going to notice that I'm actually calling or using the cache control block. That means everything in here is going to be cached, but not during the first call, but any subsequent call. So the first API call is not going to be cached, but for any subsequent API calls, the book content is going to be cached because we are putting a cache point here. So we send this. And also keep in mind that when we send this for the first time, there's going to be a 25% increase in the token cost that we are going to be using for our cache content. So after that we have another user input, which is basically a prompt and it says what is the title of the book? Only output the title, right? So we take these set of messages, pass it on to the normal client from Anthropic Model is going to be Cloud 3.5 Sonnet. It can generate up to 300 tokens. We pass on the messages. Now, since we're using the normal client, so we'll need to also add this in the header which tells the entropic client to use this beta feature. Or as an alternative you can just use this so which is basically client beta prompt caching and create a chat completion endpoint. Now, again, although we are using the cache block here, but for the first API call it's not going to be cached. So I ran into some issues with my API and I think it has to do with the beta nature of the API right now. So I'll adjust the notebook and put that in the video description. So we're going to look at the results from the official cookbook for that code. So when we make the non cached API call, which is the first API call based on the code block that I showed you, so it took about 22 seconds, there are 17 input tokens, which is the prompt that we are passing on, and eight output tokens, and the output tokens are the title of the book. Now if we use the same code block to make a subsequent API call that will be using the cache content because we are setting up the cache control right after the contents of the textbook. So there's another function that we defined, but it's the same structure and since it's a subsequent call to this API endpoint, it will be actually using that specific cached content. So in this case it actually took about 4 seconds only. Now the total input tokens are again 17 tokens, because that's the message we are passing on, which is the same message. However, the number of output tokens in this case are different because it returned this text. The title of this book is Pride and Prejudice. Now, although we simply asked for the title of the book, it did add some further text with it, and that's why you see an increase in the number of tokens in the output. Now, if you were to make another subsequent call is going to take about this time rather than the initial 21 or 22 seconds. Now an obvious question is going to be which implementation is better context caching by Gemini or the prompt caching by Anthropic? I would argue you can make cases for both of them. So one application firm prompt caching by Anthropic is going to be something like a SAS which lets you chat with your document and the user is supposed to chat with a number of PDF files in a single session. I think that's where this five minutes lifetime becomes very important. Also the anthropic API endpoint lets you cache a very small number of documents or small number of tokens. So for example, if you're chatting with a single PDF file, then you don't want to have a huge cache. Something like 4 or 8,000 tokens is more than enough. And since it's going to be a single session, so it makes sense. Now if it's hundreds of thousands of tokens and the conversation is going to take much longer than five minutes, then I think the entropic implementation is not a good option. In that case you probably want to look at the Gemini's implementation because now there can be an interruption of longer than five minutes when the user is talking with the documents or code base. And it also lets you set any arbitrary value you want. So the default is only one hour, but you can extend it to as long as you want. Now the second question that everybody has is does the long context with context? Caching is a replacement of rag and the simple answer is no. In enterprise settings you will encounter knowledge basis that spans well beyond millions of tokens. Now in that case you can't really be putting a subset of documents in the context of something like Gemini because you actually need the whole knowledge base in order to extract the most relevant documents. But I think long context really helps with RAG as well, because rather than retrieving small number of chunks, you can retrieve whole documents and put those whole documents in the context of these LLMs and that will help the models create better answers. So I think having long context will supercharge rags but not replace them. If you have a different thought on this topic, let me know. I would love to have a discussion in the comment section below. I hope you found this video useful. Thanks for watching and as always see you in the next one.