How to Build a Backend Queue in Bubble.io (The Right Way) – ratio.dev

Have you ever wanted to process a bunch of tasks in the background in your bubble application in order? In this video, I'm going to show you how to set up a backend queue inside your bubble application. I think it's going to be a real help for you. So do you need to make sure one of your backend workflows finishes before the next one gets started? Or maybe you want to just make sure that they actually are completed. What you really need is a queue. And for some weird reason, bubble doesn't have this functionality built into it natively. Sure, you can run a backend workflow on a list, but you don't actually know if it worked on every row and you certainly don't know if it did them in order. And you can also build a recursive backend workflow which will go through a list one by one and it will do them in order. What happens if another user submits another transaction while that first batches is functioning? Well, that means that you could have two backend workflows or recursive workflows running in parallel, and that can screw up your database as well. So how do we solve these, these problems? Well, a queue is just like waiting in line at the grocery store, waiting for the little old lady who's in front of you to get out all the coupons and, you know, pay with her spare change she has in her purse before you can go and pay for your groceries. Well, the same thing happens in this transaction queue that we're going to build together in bubble. So how are we going to get this all set up? You're going to need to do five separate things, and we're going to go into the bubble builder and actually see them. But let me just kind of give you the overview. First of all, you're going to need to create a table in your database, call it transaction queue or whatever name you want to give it. Then you're going to need a database trigger that is going to fire every time a row in that table changes its status. Then we're going to need a process that actually checks the queue to see if there's something pending. We're going to need another process that runs when something's gone wrong and it's failed. And then we're going to need. The last process that we need to set up is actually a process that does the work that you need it to do. Okay, so this is the overview, and now let's dive into the builder and see exactly how we can make all this work. So Here we are in the builder and you can see my data types and I have a type which I've created which is called Transaction Cube and the fields I've given this is a status field which is an option set. So the first thing you want to do is create this option set which is called Q Status. It's got four states pending, calculating, complete and failed. Those four, you only need those four, but you do need them all. That is your option set for the queue status. And then you need a completed date. Because we want to actually track every time a transaction has been processed. We want to keep track of that, it's timestamp of when that happened. We want a start date so that we know when the process actually began. This is going to be important for making sure that our queue doesn't ever get logjammed and stop working. And we're going to have a link to our transaction table or whatever other table that you might want to link this to in your bubble application. And we also have a date here that's probably not necessary for this example, but in a future video I'm going to show you how to make this even more sophisticated. In this example we are going to also set this up so that every user has its own queue. So let's imagine that we have 10 users working and they have to calculate. There's some process that you want to make sure that each user's transactions get processed in order. But it doesn't matter if it's processing different users transactions at the same time. Okay, you could do it just one queue for all of them or you could have a more complex structure of how you want to how create this queue. But this is what we're going to do in this example. So these are the only three tables that we need. And now what we're going to do is we're going to go into the backend workflows and we're going to actually see how I've set this all up. And it is a little bit tricky in a few places, but I think once you, if you pay attention and you and I'll try to go slow enough so that you can see all of the steps. This is doable just from watching this video. And don't forget we're always here to help. And if you have questions, leave them in the comments and I will be sure and answer back. What I've done is I've created four custom events, okay. And these correspond to four different actions that we need to be able to check in the Processing of this queue, the first one is going to be a very simple custom event. But all I'm going to do is doing this because I don't want to have to repeat the same search multiple times. So I'm just going to do the search once and return its value. It's a good trick for you to know, by the way, now that custom events can return data. You can do this to make your workflows more efficient. So all this one does is it returns a search from my transaction queues. And what I'm going to do is I'm looking for all of the transaction queues whose status is calculating, okay? For this specific user. So this is taking in a parameter user, and what it's returning is a transaction queue. In theory, there should only be one in calculating at any one specific time for one specific user. So it's going to return the first of them. So all this custom event does is does a search and returns the.

Only transaction queue item that is in status calculating for this specific user, okay? Now the next thing we have is a process that does the check on the queue, okay? So the first thing it does is it sends off and it executes this workflow that we just saw, does the search and it will return back if there's any specific item that is in status calculating, okay? If there's not, then it'll return empty because it won't find one. Okay? Now then what this does is it's going to return two variables. So it takes in a user, same as the other one. This is going to be common across. It returns a text string which is called status. The most important thing is it's going to return an item in the queue, okay? But it is marked as optional because there may not be one. So I can't make it required because there may not be any element pending in the queue right now. Okay? But if it does, it comes back and it returns if there's not one already calculating. Because if there is one already calculating, I don't need to return anything. Here the workflow is the queue is busy, so, so I don't need to return anything. But if it returns empty, then I'm going to go to the transaction and I'm going to return.

The first queue element with status pending ordered by the order that they were created. I'm doing ascending because I want the first one to be the oldest one. Now, if you wanted your queue to work the other way around, you could put this as descending and it would always get the latest one. But that isn't the case we're talking about here. So I've ordered them by date. I get the first one that's pending and that's what I return. But what happens if something in this goes wrong and there's one calculating and it never gets. It fails for some reason? Well, we don't want our queue to stop. Okay, so the next thing I'm doing is I'm going to call my the third event, which is called Q failed. I'm going to pass it the one that's calculating, and then I'm going to check and see if it has been 10 minutes since it started. And you can set this time, whatever you want. Basically, I'm estimating, I'm saying, okay, it should never take more than 10 minutes to do the work of this queue. So if it's gone more than 10 minutes, something's gone wrong. I'm going to take the one that is marked, that it's marked as calculating, and I'm going to mark it as failed. And that way my queue can keep going. Calculating means that it's busy. So once I take that one, there's only one in there calculating, I change it to failed. Now there's none calculating, and that will also trigger the next one in the queue. And we'll see how that works. So that brings us to the custom event of queue failed, which just takes a queue item and all it does is it marks it as failed and it marks the completed time as the current date time. So that's when it was marked as fit and that's all it does. But that's enough because now that queue item has changed status, which will trigger our trigger again. So that will start the whole process over again. So how does this all work? Now the last one that we want to look at is queue process. And this is the one that is going to be called whenever we start working on a new task in the queue. Basically, it's going to schedule a backend workflow which I've called do something. I'm going to pass it the transaction in the queue item and I'm going to say do it now. And then I'm going to mark it as calculating and mark the start time daytime. So now that's what starts the clock ticking. So if this lasts more than those 10 minutes, then it will come back in and mark this as failed. Now we have four custom events and we have one workflow takes whatever action needs to be done. So here's where you would put what you want your workflow to do. And at the end, it's important that it marks the queue item as complete and it marks it as the completed date time. So that way calculating gets turned to complete once it's been processed. This is important because this status variable is what's actually moving the queue back and forth and making sure that the next one gets done. So how does that happen? This is really simple because now we have all of the pieces involved. We have a database trigger on this table. So a trigger has two instances of the table. It has the now and the before. So the before and after, you're just going to compare and say if the status is different from the now status is different from the before status. So that basically means that the queue status has changed, okay? Which is also true if it's a new. If it's a new row in the table, because the before status will be empty. Once that has happened, then it'll trigger the following workflow. This is where the magic of the queue happens. All I'm going to do is I'm going to check the queue using my queue check that we already talked about. I'm going to pass it. This new row.

As long as the one that it's gotten is not in state calculating. Because if it's just been flagged as calculating, so it was impending and now it's in calculating, I don't need to check again because that came from the check. So that's why I put this condition here, just to make it a little bit more efficient. And then I'm going to trigger the process. One of the. So I'm going to take the result of this check, which this check is going to give me the latest queue item. Then I'm going to process. Okay? When it fails, if it comes into this with a failed, then trig queue check is going to mark it as failed and it's not going to process this. So basically with this, your entire functionality of the backend queue is finished. You could use other variables. So not just user, you could use an account or something. You say, I just want my queues to be by account. Or you could have one global queue for all of these transactions and just say, I just want one by one, one by one. You can set that up using the logic that I've shown you in this video. This can be great for calculating balances for processing sequential transactions. Hopefully you've been able to capture exactly how I've set this backend queue up. If you need any help setting up a queue in your Bubble application, feel free to reach out to me and the team here@ratio.dev would be thrown rilled to help you out. And I hope you've enjoyed this video and I'll see you on the next one.