Slash API Costs: Mastering Caching for LLM Applications

Let's learn how to save money on OpenAI API costs. If you're putting your LLM projects in production, there are two important factors to consider. First one is the cost associated with making API calls, and the second one is the response time. You can use the caching technique to improve on both of these factors. The idea is very simple. You have a temporary storage place called cache. So imagine when the user makes the first query, we get a response from the LLM and we also store the original prompt as well as the response in this temporary storage called cache. Next time, if the user makes the same query, we will first check whether that exists in the cache. If it does exist, so we get the response from the cache instead of the LLM. So that will save us both on response time as well as the associated cost. If the information does not exist in the cache, then it will be sent to the LLM and it will again update both the cache as well as return the response to the user. In the rest of the video, I will show you how to first implement this using OpenAI's API. Then we will look at LangChain and I'm going to show you two different techniques which you can use within LangChain to implement caching. At the end of the video, I will also share a few more strategies that you can use to reduce your cost. So let's get started. Here is a simple example of how you can implement caching and your application using Python and Cache tools library. In this specific example, we are directly using OpenAI API without LangChain. In the subsequent example, I will show you how to add caching into LangChain applications. So first and foremost you will need to set your OpenAI API key. Then we are creating a cache with a maximum size of 1,000 items. Then we have defined a function called get predictions to get predictions from OpenAI API. Now, in this code, we first check if the prediction for the given text is already in the cache. If it is, we return it immediately. If it's not, then we make a request to the OpenAI API, store the prediction in the cache and then return. Now this way, if we need the prediction for the same text again, we don't have to wait for the OpenAI API. We can get it directly from our cache. In this example, I'm actually using GPT3 just to show you that it is a relatively expensive model to run. So let me first run this code. Now, since we are running it for the first time, there is nothing in the cache and that's why it prints this message cache miss fetching from GPT3. Okay, now if I were to run the same prompt again, then in the second iteration it's supposed to find the prompt within the cache and it is going to show me the same response. So if I run this again for the first one, it actually misses it because it doesn't find anything in the cache. But for the second prompt, which is exactly the same prompt, it will find it in the cache and it's not going to make an API call. This is an example of in memory cache and you can implement this as part of your own LLM applications. In the second example, I'm going to show you how to use caching within LangChain when you are building LLM powered applications. Now keep in mind, although I'm showing you this for OpenAI's LLM, but this is going to be extremely helpful even for open source LLMs that you are hosting locally or in the cloud, because not only it reduces the cost of API calls, but it will also reduce the response time. So let's build on a very simple example. In this case I am putting different packages. So for the first one is time that is going to be used for keeping track of how long the API calls takes. Then we're using the load env. This is simple for loading my API keys. Then we are importing LLM from LLM, we're porting the OpenAI object, and then we're going to be using this call back to keep track of our cost per call. And the last one is the most important one for this specific example. This is a in memory cache object. Now we are going to be also looking at some other cache types in the next section. Okay, so first I'm Simply initializing my OpenAI API key. So this function is used to load my API key. Next up we are simply adding cache to the LangChain object. So in this specific case we're going to be using the in memory cache. And after that we are defining the OpenAI model that we want to use. Now notice that I replaced the text DaVinci model with GPT 3.5 Turbo 16K. That's just to reduce my cost. Next we are defining our example prompt. So in this case I'm using this very simple prompt. What is the capital of France? And after that I want to use a context manager to keep track of our callback. Now let's look at this with context manager that we have in here. So first I'm actually noting the current time. After that we are passing on our prompt to the LLM and get a response, right? And then we simply note down the end time. So that will give us the total time that was taken to generate that response. Next, I'm simply printing out the response that we get from the model as well as the total time tag taken and then the callback itself. So this will give us the number of tokens that were used, as well as how much it actually cost us. Okay, so here is the response. It actually gave me a warning that I need to use this other format of calling the Turbo 3.5 model. So I'm going to just replace it. However, the most important things to notice are the total time taken. So it's close to one second. Now, the number of tokens that were used, that's 21 tokens. 14 of them were the prompt tokens. So that's basically my prompt. And probably it's also sending some system messages with it. And then the response tokens are seven tokens. Now, I think it's because of this. So it's like 1, 2, 3, 4, 5, 6. And then probably that dot is also considered another token. Right? So if you count these, these are basically seven tokens. Those are the completion tokens that you get back from the model. Now, the cost for this specific model is minimal. However, if you're making thousands of call API calls or hundreds of thousands, this will actually add up. So let me Simply replace my OpenAI object with this chat OpenAI object so that we don't get this warning again. Okay, so I ran the code with this new model. Now, the total time taken is 1.34 seconds, and the rest of the values are the same. Okay, now let's see what happens if I were to run this again, since we have caching enabled. Right, so let me simply copy this and let's call this CB with cache. Okay, and that's here. Right. I'm not going to change any of the other values, but let's run this again. Okay, so here are the results. For the first one, it actually took 0.92 seconds. Now in the second case, it actually took almost zero seconds. And it didn't use any prompt tokens neither it used completion tokens. The reason was that it actually found our prompt within our cache. So it simply used that older version rather than making an API call. As you can see, this is extremely powerful. However, in this specific application we were using in memory cache. But sometimes for your applications, you want to write the cache to a database. So imagine a case in which A lot of users are asking similar questions or the same questions, right? So instead of making API calls, you could simply store your cache in a database and use that to generate responses. The good news is that LangChain has support for multiple different type of cache. So the one that we're going to be looking right now is SQLite cache. But there are some other options that I'm going to show you in a little bit. Now in order to use SQLite cache, so we simply need to make one small adjustment. So now the cache is going to be using that and then we need to define the database path, right? So let's call it LangChain. And this could be any different databases. So I'm going to simply use DB right now what's going to happen is that it's going to be using this specific database to store cache, right? The rest of the process is exactly the same. So if I were to run this again, so let me clear this and then let me save this file and I can simply run the code again. So. So for the first one it used the same number of tokens as before. It took a 0.8 seconds, right? In the second case it's simply using the data or information from the cache. So that's why it's not using any tokens. And the time is almost 0 seconds. I hope this gives you a pretty good idea of how to use cache in your applications. Now let's look at some other type of cache options that are available within LangChain. So as I said, the first one that we looked at is the in memory cache. The second one that we just checked out is the SQL Lite cache, right? If you are working with Redis databases, so actually they also have support for Redis cache and the whole process is extremely simple. You simply need to use REDIS cache. Now so far the examples that I have shown you, in that case the prompt has to be exactly the same as one is stored in the cache. However, you can use semantic cache. So in this case you can use reddish cache to compare prompts and responses and evaluate hits based on semantic similarity. So they don't have to be exactly the same. But let's say if you are running a service where users are asking questions, right? So they can ask the same questions in different ways. So semantic cache will actually help you in finding out similarity between questions. And for this case you can use OpenAI is embeddings. The embeddings cost is actually extremely low compared to the prompt API calls cost. Now they have other options as well. So there is an open source project called GPT Cache and it's very similar to the Semantic Catch. So either you can use look for exact match or you can look for semantic similarity. As you can see, this is an extremely important tool in your toolkit. If there is interest, I'm going to make a lot more detailed videos on the topic, including how to cache different parts of your LLM chain. I hope you found this video useful. If you're working on an LLM based project and need help, check out. My Consulting services link is in the description of the video. We also have a very vibrant discord community so come check us out. It's a very helpful community where you can ask questions, share ideas and get help. So come join us if you found this video useful. Consider liking the video and subscribing to the channel. Thanks for watching and see you in the next one.