Overcome Rate Limits in APIs with Xano (@nocodebackend)

So let's talk about rate limiting. There are a number of situations in which we are trying to make requests to some third-party service, and this actually is showing up a lot when we particularly talk about AI systems, that we make our requests, we're going to say in XANA, we want to just do a nice simple external API request to whatever your service is going to be, and then the, and then it comes back and says, oh, sorry, we can't help you for another second or so. Sometimes it's very good about this. It'll give you what's called a 429 response. And, and so what we can do in here is we can say, I don't know if I just go to, you know, let's see, change.ai, right? Which is not actually going to be an API, but let's leave it aside for a second. And if I were in Xano to say, all right, let's evaluate what came back. I want to make use of information that came back. If I'm going to, you know, look at the, I want to create a variable and it's going to be based on you know, API, you know, we're going to call it the response. And if I called it, or I'll call it the info or something, and I could say this is from API 1, right?.response, which is the part that comes back from the API,.result, which is the, in theory, if I make a request and it's successful, it'll give me data inside the result. And if it was JSON encoded or form encoded, then I could start getting, you know, something off of that called like.info. And one of the classic problems we run into is that you then have this information directly underneath your API request and it doesn't work. And the reason it doesn't work is that you made the request to the system and it errored out for some reason. Now, there are two basic kinds of errors you're going to run into when you're going contacting these third-party services. One is going to be an error because you're asked the wrong kind of question, right? Maybe you're missing your credentials. Maybe you're— something is malformed, a variable you forgot to put in, or your previous logic doesn't put something in quite the right shape. Those are what I would call persistent errors. And those are kind of errors that are actually relatively straightforward to fix because you know, hey, whenever I have this input, I get this output and you can be dialing those things in. But there's a more insidious kind of error and that's gonna be the intermittent error. And that happens because whenever we're talking something outside of our XANA, we're not guaranteed that the network between here and there is going to work. We're not guaranteed the server at the other end is going to like our request or be ready for it. And so we can get certain types of results that are going to show up based on error code response, but that basically means that we're going to get an error. And because we got an error, we're probably not going to get a result. And because we're not going to get a result, this thing ordinarily explodes. So what, what, what, what, you know, what we can do for the first part is work and fix our code to make it a little bit more robust. But the, but for this, we need to be robust, not just to making sure we're sending the right stuff, but making sure that if it wasn't ready for us once, maybe it'll be ready for us again in the future. And so that our system doesn't just die when it sees those kinds of situations. So how do we do that? The approach I've taken is to put something into a bit of a for loop. So instead of sending just API requests directly into my create variable, I'm going to create a loop. I'm going to go to data manipulation. I'm going to go to loops. I'm going to go to for loop, and I'm gonna say, let's iterate over, I don't know, 5 times, and we'll call it index. Now, a lot of people in Xano have worked with a foreach loop. It's a common thing that we do when we're working with like, you know, data results coming back from a database. We're gonna do data augmentation, iterate over all those results that came back. But a basic for loop is basically just gonna give me a 1, 2, 3, 4, 5. So if I were just to, you know, I, I could actually illustrate that right now by going to utility functions. I can just say debug. And I want to debug log just the value of the index. I'm going to comment this out for a second and then let's just run that. And what we'll see is in our debug log, it'll show us the 1, 2, 3, 4, 5, right? So the debug is index data, you know, as we go through here. So this is super handy that the index value is going to come back, just to sort of iterate through 1, 2, 3, 4, 5. Because then we can use that as information for part of the way we're going to run our API request. And the way we would do that is by saying that instead of the API request running by itself, we're going to put it inside the for loop. And now we're going to say for each, we're going to run this API request. Now the thing is, most of the time it's going to work correctly the first time. We don't want it to go run the same request 5 times. There can be multiple reasons you don't want that to happen. So what we're going to do is evaluate the response and we're going to go to data manipulation and we'll go to a conditional. And that conditional we will say is, let's close that up to make it easier and we'll take a look at the api1... response.status. Now response status is a useful one because it's guaranteed to always show up. Even if you call a URL that does not exist at all anywhere in the universe, it will still give you back a status. And if it never actually contacts the server at the other end, like for some reason you had a bad URL, like some, this can happen, you got like spaces in your URL or something, it'll come back with a status of 0. If you have a bad request, usually that's like a 400 error. If you go to a page that doesn't exist, that's often called like a 404 error. If you, and then there are other types of codes that go on above that. So the, there are a couple situations that we're particularly concerned with, which are going to be, hey, if the status is just let's start by being very expansive about it. We're going to say if the status is, you know, less than 200, because 200 is your happy response, or if the status— we'll copy this, paste. Yep, good. Is greater than or equal to 300, basically means, or rather 400. Because basically the 400s and the 5, the ones that start with a 4 or 5 tend to be our errors and they're all 3 digits, right? So it's going to go from 0 to 599 roughly is the way the codes work. And you know, you're going to have your, your 0, your 0 is nobody was home. 100s are very unusual and not really something we're going to worry about too much. 200 through 299 are all variations on this was successful. 300 through 399 are variations on it was, you know, they, they answered and said, you got to go somewhere else. And 400 through 499 are basically, this was a bad request of some kind. And 500 through 599 mean this was a bad, the server just couldn't handle your request. Right? So your classic 500 error is a server bombed out for some reason. So in your Xano, you probably notice these sometimes get 500 errors when you have like a problem in your code. So the, so here we're saying, look, if we go call a third-party API and it gives us something less than a 200, then that was bad news. It would get something equal to or greater than 400, and that was bad news. In either of those two situations, we're going to say that's, that's something that probably didn't work right. And we're going to then say, if it was one of those two situations, then we'd like it to just add a utility function here, and we're going to say sleep, and we'll say sleep for, you know, 2 seconds. Save. And then the, but otherwise, if it was successful, right, if it was within that band of between, you know, 200, 399, then we say That was good times. Let's just, let's just continue. And for that, we use data manipulation. We go to our loops and this is where you probably haven't used this before. A lot of people haven't. This is where you can break out of a loop. So the idea is we're going to have something that's going to run 5 times, right? We're going to try, make the request. If it succeeds, we'll break out of the loop and we move on. And then we can go down to our create variable down here and everybody's happy. If we do response, result, that's probably just the easier way to do it because this is just going to be HTML code. The, if we are, and I'm going to comment this out, I don't care. So, so we will, if it succeeds, it will go here. And if it will fail, I will sleep for 2 seconds, ding, ding, ding, and then try running it again. And then try running again, try running again. And basically after 5 times, it's going to give up and sort of, you know, then it'll return to next variable and you'll need to deal with the consequences of that. You could also decide that you want in the event that if this, by the time it gets down here, it's bad, right? So we could put, you know, take the same conditional if we wanted to, and we could take the plus, no, sorry, clone it, right? And we can take the clone out of the for loop, out of the for loop. There we go. So now basically outside the for loop, we will say that in the event that it was a bad person, we will, you know, not sleep, don't need to sleep there. We probably just need to say, let's throw an error. And we will say utility functions and throw an error and say this was a request, right? And here there's no loop to break out of, so we'll eliminate that. All right, so basically this now says we're going to try 5 times. Bing, bing, bing, bing, bing. And then in the event that it fails all those times, I'm now going to throw my error and I'm going to, and then, but if I, if I don't have that error, then I will continue down to the rest of my logic, making use of the data or what have you. This is super handy because that way, in the event you're getting rate limited, it slows it down, tries again, and you don't have to add any more logic to make that work. So we just combine this idea of doing the for loop with having this conditional. Now, it can also be a little bit of a pain, right, to introduce this for every single one of your API requests. First, most of your API requests probably don't have this kind of problem, but a few of them do, so I would be a little bit surgical in the way that you apply this. But the second thing is you can, you know, you could wrap this all into a function because if this is repeating logic, then we don't need to do it over and over. Let me show you what that looks like. So if I go back to my list of functions here, I could create a func— I could create a function called external API request with retry, and it just includes the exact same stuff that would go into a function that has the usual external API request function. It just wraps it and then adds this additional idea of, you know, saying we're going to make a request to that API based on the inputs that come in from the function, right? So now you have this function call, which you can call whatever you want. I'm calling it external API request with retry that contains all the same data points that you would put in, or has the exact same utility, right, that your underlying external API request does, except it includes this idea of actually refreshing. And retrying as you go. It includes a couple of other things too that I find to be particularly handy. One thing that I find handy is the idea of being more restrictive in terms of which responses turn out to be the ones that actually correlate with the server being overloaded. You might think there should be some rules and there should be one thing that just works, but that's just not the case. So you're going to find that based on the APIs you work with, you might be dialing it in one way or another. Like if you're talking with, you know, OpenAI, if you're talking with some of these others, some systems are very good about announcing this is a rate limiting problem. Response status of 429. Sometimes they don't show that and they're just saying, oh, I'm erroring out. This would be like when, you know, you've got Cloudflare sitting in front of one of your, one of their endpoints. And Cloudflare says, hey, look, you've been hammering this guy. We think you might actually be a denial of service type attack. So we're going to cool you off for a little bit. And that will not show up as anything particular like a 429. This will show up as like a 502 or 503 error. And not contain a whole lot of information. And so like, this is why it makes a couple of guesses here. You might find the guesses make sense for you or are going to be a little bit different. Now, then you have the idea of, okay, well, if it was a, a, a, a, a 429, then we're going to, yeah. Then we're going to sleep. Oh, right. Actually, what this will do here is it actually is asking an additional question, which is actually kind of important, which is if the index, right, is less than the number of attempts. Now, this is important because consider the logic here, right? So this is going to say, all right, I'm going to loop through and then we're going to say, try, and if it fails, I'm using one of these criteria, then I want it to wait. And then I'm going to try it again, and I want to wait again, try it again, and I want to wait again. Now, after the last time it tries, it shouldn't wait again, and that's the reason why this logic is in here, is to prevent it from waiting in the future. In fact, I'm pretty sure that what I need to do here is actually say that needs to be less than attempts subtract 1 because, because the index is gonna go 0, 1, 2, 3, 4. And if my attempts is 5, then basically I'm saying I want to do my sleeping for attempt 0, for, for index equals 0, 1, 2, 3. But I don't wanna do anything after number 4 because it's the last time I'm gonna try it. And that's the reason why we subtract 1 here. So it needs to be index is less than 4 only for those first 4 attempts. Am I going to asleep? Because otherwise I'd be sleeping, but there's nothing in the last one. So that's one key idea. So we actually have two key ideas we talked about so far. The first is being more specific about types of responses because it was like a persistent error, like a 400 error. That's likely not to be because of something you can do a retry about and we need to get it to, you'll need to fix your code, right? And you want to get that kind of signal. You don't need to be waiting for it to run 5 times for that. The, if it is, so that's first is sort of limiting circumstances. The second is making sure that your delays are only between, right? Kind of like if you have a list of things, you put commas between them or whatever. You don't need a comma at the very end of the list because you don't need to wait at the very end of the list. And the third idea here is actually saying, well, if we are, if we try the first time, let's try waiting for a couple of seconds, but second time we should probably try longer. And that, and so this builds in the idea that we're going to, you know, have a sleep that's going to be between each of these calls. And we're going to have a delay, right, of the retries. That's like the 2 seconds we put into the previous function. And then we're going to multiply that by the index, which is kind of neat. Multiply by the index. In fact, we need to multiply by the index value plus 1. Because since the index value is initially 0, we don't want it to sleep for 0 seconds. So the, so the idea here is that the first time it would wait for 2 seconds. The second time we'll do for, 2, 4 seconds, second, third time we do 4, 6, 8, right? So it sort of adds more delay because you're being rate limited. You don't want to just keep on hammering the system. That's going to keep on just putting you on the bad list. So let's, let's slow it down. And once it's slow enough, oh, okay, we'll let you in again. At which point you can probably go fast again. And that's the reason why we don't have to remember that, that delay going forward. There's also an idea called exponential decay. And there are a couple different ways of doing exponential decay. The, you know, you could have by using the square of something or you could do it by using an actual geometric progression, which is to say by the power of something like a square, then a cube, then the fourth power, fifth power. That gets very, very slow. So I'm just sort of taking the squaring approach right now and saying that if exponential decay is true, then what we'll do is after taking the original one, we'll then again multiply it by index add 1, 1. There we go. Okay, so what this does is that the first time I'm going to sleep by 2 seconds times 1 times 1 equals 2. Second time, wait for 2 seconds times 2 times 2 equals 8. Third time, I'm gonna wait for 2 seconds times 3 times 3 equals 18. And the fourth time I'll wait for 2 seconds times 4 times 4 equals, uh, a nice full 32 seconds, right? It's really backing off. And that, and so by introducing exponential decay, if I think it's important, it will really slow things down without having to make all the requests slow. And then, you know, it sort of has the same approach, was if I get to the very end, it will just break out of the, if I, in the event that it succeeded, right? That's the alternative to this then, it breaks out of the loop and just returns whatever the API one is. And so this is pretty cool because now I can have an external API request that works with retry. And if I were to publish that here, right? And now instead of, if I were to go say back to my, you know, list of, you know, robust APIs, right? The one that I built here with you. What if I instead said, I just want to use a custom function. I'm going to use the external API. Oh, actually, you know what? I can just type external API request retry, right? There's a built-in one and hey, here's my nice custom one, right? Which has all the same stuff that I would use external API request and I could set that to true or false or whatever, right? And then I can put all that stuff in here. Now all of the concerns I'd put into all of these lines are now getting wrapped up right into my single custom function, and which I could then say return as, you know, if I just called this API one, right? I could then take this, put it underneath here, right? And then just tell it to ignore all of that business, you know? And now it runs the same way, right? Which is I have a custom function, I have a create variable. I know I have done two lines by having all the rest of this. Because of course I've just taken those lines, I put them into the function, and then I can add additional tweaks, what have you here, by having completely separated those concerns. And of course you can also like compose this in with having like an event log type thing so that you can, that maybe log out whenever you're seeing these kinds of rate limited stuff. So you can be finding out why this happens, but those are all more advanced topics that can wait for an additional video. Hopefully this helps you with your own somewhat less than completely reliable API requests and we'll, we'll see if we're tackling the next hard problem here on StateChange.