Xano Office Hours - Loop and Filter at the same time
Ray Deck shows how to extract job objects from a 10,000+ record XML response in Xano. Using findallelements with a has filter works but hits memory limits, so he demonstrates a foreach loop with an if condition that processes records one at a time instead. The streaming approach inserts each job's fields directly into the target table without loading everything into memory first.
Chapters
So Liam, why don't you go ahead and start out here and share your screen and let's take a look.
Yeah, let's just screen share. Okay, desktop. Okay, can you see my screen?
Yes.
Cool. So, oh, so all the Zoom stuff's in the way. One moment. Okay, so I've got an external API request which is XML, then I decode it here, and then what I get as a response is this. And what I want to do is then basically add this into this table with the populated fields. I've just put one, one field to now just to test it, and I can do the rest later on. Basically populate the fields from the jobs into the table. And I think you can do that through loops and I think bulk upload. But the issue is I've then got obviously these fields here, so feed attributes value, and then got title, description, brand code, brand name, aggregator. I want to get rid of these, so then I've just got from job onwards. I'm not sure how to get rid of those in the function stack, so I was wondering if, if you could help me remove these up until job, because then job is when the actual data is here, if that makes sense.
Got it. So you're saying you only care about, I guess, these job objects?
Yeah, that's right.
Got it.
So I want to get rid of kind of this stuff and then work out how I then actually add it into here, because this is about 10,000 records. I'm wondering the best way is to do it.
Okay. Let's see, scroll up here. So we have total job feedback data.value. Okay. Yeah, so we can get started just kind of isolating like everything that is just job. Maybe that's kind of a good initial step.
Cool.
So yeah, we can use one of those, our array helper functions in data manipulation in order to do that. There's a find all elements one. So if we go to, yeah, array, find all.
That one?
Yeah, exactly. So our array here, yeah, will be that variable. We'll need to— kind of see that path up into value. So it'll be XML output dot. You might want to kind of get that response back up there to look at that structure. It's okay. You can go to— oh, yeah, we need the debugger info.
So I mean, dot title jobs.
Is it the case you wanna do an operation on every single one of those records that has a job in it?
Say that again, sorry.
You, you want to do like load into your table something for each of those ones that have job in it, but not for those first half dozen?
That's right. Yeah.
Okay. You know, I, I think there might be a slightly more direct way of doing it. Yeah. But, but you're gonna need the path either way. So let, let's scroll up to the top.
Yeah.
Total jobs 1, and then there's something called feed data, I think, inside that.
Yep.
And then.value. I think one way or the other, that's gonna be the key to this is starting with that path.
Cool. And then what should that be? Just Anything.
We'll call it jobs.
Oh.
And then, like, but the nothing defined up there, that's the other thing you're gonna wanna change. I think that's what Michael was about to turn your attention to was how to filter out just the ones that have the jobs to them.
Okay. Cool.
How would you do this one, Michael?
I think it would just be this equals Uh, job.
Oh, assign this. Well, except it, it says an object. Each of these is gonna be an object that has a member called job.
Right?
I think there's a filter in Xano called has, if I'm remembering my PHP correctly.
Yeah.
Uh, so if you do the pencil again, uh, so put, pick this and apply a filter to it if we, if you could. And look for has. Yeah, there it is. Uh-huh, that's it. And the, this one's gonna be called job, J-O-B. And we're just establishing there's a member of this object that has a key of job. That's what it's looking for. It's like, does it have something called job? And if it does, hooray, right? And that'll return true. So we're gonna say okay to this. And then I think on the other side, you want to basically only have where this is gonna be true. And this will filter you down to just the 10,000, however many that have the job in them. Save and save. And I'll change that. I think you're currently commenting it out. Yeah. On line 4, I believe you comment this thing out.
Yep.
Yes. There we go. Let's uncomment it out and let's make sure you're spinning out jobs, which will now become available to you because it's no longer disabled.
What?
Total jobs underscore 1 dot. Looks like we didn't do it quite right. You want to show me again what XML output is? The whole thing, right? No, XML output.
After debugging.
You already, you already. Yeah. Okay, go for it. Yes, debug. The right way to do it. But I think you have an update variable on line 4 that probably is the interesting part here.
Yeah, that's the one I need.
Yeah.
Okay. Got rid of total jobs.
Oh, it did, didn't it? Okay. Well, let's get rid of total jobs for one, 'cause that was a red herring. Tool 1, just do fade data. XML is always a delight to parse.
First time doing it, so yes.
Okay, I was kind of afraid this might happen. Um, Xano, it runs on very low memory. Um, of machines in general. And so you can run into problems like this where, like, when you're trying to process something really big, this happens with big XML a lot, big CSVs too. When you do one of these array helper things, it wants to hold the whole thing in memory and then you run out of memory. And this happens more often than I might totally like, but it's a side effect of, like, the types of machines that XANA runs its stuff on. The way— the alternative way to do this is having learned where this thing is, we can do a foreach., right? And just, you know, do work on things that actually have a job, right? Because you've now learned two things. You've learned where it is in the path, right? And learned about the has filter, right? Which you can use. And we can use that as a criterion inside a foreach loop, right? So instead of like creating the whole set and then looping over the whole set, we're going to, and then looping over the filtered set, we can loop over the whole set and do a test on each element as we go along, which means it sort of, it's lower memory, right? Because we're streaming the stuff through. And we get to have that effect when we do foreach. So if you highlight the foreach, and actually probably inside your— we'll probably need to crib from final elements because that XML output.feed data.value, that's the thing you want to foreach over in your next thing down. And, and, and usually when you see unknown error in Xano, like 9 times out of 10, it's a memory problem. It just has to do with I'm trying to hold too much in my head and I can't do it.
Yeah, because I had 180,000 records before and I had to go to the supplier and just say, can you filter it down? Because it was, it was way too many. So makes sense.
So in here then, oh no, not that. Oh, actually inside the foreach loop, you can just click into there. The list you're going to go over, you can paste that. Yeah, perfect, perfect. That's exactly what you want. And the item is fine. Cool. Okay, now we're going to add a function inside this list. Oh, and by the way, you see where it says expand a little further north here? You see this guy here? Oh, yep. This can make it just a little bit easier to work with when you're editing stuff.
Oh, that is nice.
Yeah. Nice. That, that way, that way the things inside the for each loop actually go underneath and not so collapsed down. The collapsing is really useful when you're trying to just get as a glance, but when you're doing the edits, I recommend expanding out. It makes it a little bit easier to work with.
You know what? I think I clicked that by accident ages ago because I thought that was an update, but it went all short. So I must have just clicked that by accident. So that's fine. Cool. So in stack then. And is it the same what we did before with has?
Well, in this case, we're gonna do an if. We're gonna do an if condition. So inside this tag, we're gonna do an if. Cause we don't need, yeah. And then we're, and the if is basically gonna be the item is the equivalent of this thing, right? And you can run that has filter just like we did before. And then you're gonna look for job. And, and the, and, and fa— yeah. And you can do that with this truth. That's totally great. And then, say fine. And then basically you wanna run your interesting business logic relating to the job inside this guy. And the thing I do to make sure the test worked right is I just throw a stop and debug right inside the then, you know, to make sure that like what we get is actually one of the items I'd want to work with.
And then what item?
Yeah. Right. So basically what this should do, if we, if we've done our job correctly, this should have our, it should skip over the first 6 of them.
Right?
And then the first thing I see right here should be the first one that has a job to it.
Cool. Boom. Cool. Amazing. That's cool.
And now you get information inside the job, which you can do through item.job, blah, blah, blah.
So is that— It's just done the one, right? Because we've done— it's because we've done a loop.
Yeah. Right. We're only looking at the first one. Right. That's right. And so, like, you just put your logic inside this then, you know, where the stop and debug is for dealing with each of these items that have the job. And probably your first things first is you just extract it into, like, you know, item.job.value, right, which is containing all of the stuff that you really care about.
Cool. And then you would then just do your add or create record in there, which would then assign it to each column, right? Cool. Easy. Amazing. I think that's it. Is there anything else that you think I need to add to that, or do you think that's pretty much done?
Uh, yeah, everything else is about actually what you— how you want to get the data into your, uh, into your table, right? You've done the filtering. The filtering was taken care of. The if you've done your looping, that's what we did with the foreach, and you've learned both about how you can do it in a single line using find_all_elements, right? Which is super convenient for getting a list you want. And you've learned how to do it sort of in a more streaming technique, which is by doing, by composing together a foreach with an f. Yeah.
Amazing. Thank you so much.
Yeah, sure.
Very nice. Big thanks to Ray Deck for helping out. And please check out his website, statechange.ai. He holds office hours like this on the daily, solving the hardest problems with no code in your, you know, app building journey. So definitely check that resource out.
Thanks, guys. Yeah, cool.
Thank you, everyone.
Appreciate it. Thanks for your time, as always.
Appreciate it. Well, thanks, everyone.
See ya.
I'm gonna go join room 2.
See, they're still going in there. Yep.
