Data Surgery on Arrays With Xano (@nocodebackend)
Ray Deck shows how to update deeply nested data in Xano using two methods: extract the target object, modify it, and replace it back, or build a dynamic path string and update in place. Both handle arrays inside objects, zero-based indexing, and the need for an intermediate output variable.
Chapters
We've all been there, we have a complicated data structure, and we need to modify it in some fairly specific way. When we're dealing with straight objects, this can be relatively straightforward, because we can use things like dot notation, in order to say there's something I want to change, right? I want to just sort of do a data manipulation, I can do an update variable, you know, based on you know, the output. And I could just say that I want to, you know, update it. Because I can say update.a.b.c.d.e, right, whatever the path I need is. And then I can just express what I want the value to be that should be inside that path. That allows me to go mutate update right away, assuming the path works all the way, there can be reasons it doesn't. And that can cause us to do a certain protective code. But that only works really when we're doing keyed objects. And when we're dealing with complex data structures, we're often dealing with a mixture of objects and arrays, right? And so that instead of having that, we would have output.to.a.b.c. And the problem is, we don't actually know what, which element we're going to do. It can be worse than that, right? Maybe we have an array, and we have yet another nested array inside it. And we need to be just figuring out which we're going to modify.. And maybe we don't know what, which ones to modify at the outset. And that can make things just much, much more challenging. So I often refer to this as data surgery. And the, and it's going to be when we mix objects and arrays that this becomes most complicated. So let's take an example, a little piece of, you know, code here, which is, you know, somewhat shortened coming from a state change member, where we have a list of scenes and shots within scenes, right? So it's like for a movie, I've got scene 1, scene 2, and then so we're going to go different shots through the movie. And you want to edit one shot in one movie. One shot, one scene, right? And so you have this, you know, I'm just calling blob for a moment. And it is an array. It's an array of scenes, right? So you have these two scenes in here. And you have scene 1, which has some details, and then it has the shot elements. And the shot elements are themselves an array of objects. And what we want to do is edit one of those objects in that array. But let's make it a little bit more complicated, because we don't actually know which element in the array to modify. And that's the reason why we have the scene ID and the shot ID, right? These things aren't guaranteed to be in order, right? They're gonna be based on ID, some sort of UUID, could be ABC, it could be 123, it could be something completely random that there's no chance we can sort on it. So what we want to do is we got this, you know, big old array here. We want to find the member of that array that matches my scene ID. And we want to match within that the shot that matches the shot ID that we're looking for. And we want to modify it in a very specific way. How do we do that? Well, there are actually two techniques we can use. I'm going to talk about them both today. But the first thing we need to do and remember is whenever we're doing a mutation, especially when we're dealing with mutations coming in through like the function stack, like we're trying to isolate this kind of complexity, we need to remember we can never edit an input. And that's the reason why I created this variable I specifically called output, which is based on input, but we're going to be able to modify it after that. You're gonna find otherwise running update variable on input, it doesn't really error out, but it will not actually make the change you're looking for. So if you want to make a modification, you need to change it from an input to a variable so it can actually vary over the lifetime of the function stack. So output is going to be the same thing as a blob. Let's just confirm that for a second by doing, you know, utility function. We use my absolute favorite function, which is called stop and debug. Now we'll say, let's look at output and run. See, and output right now is a scene and is a shot elements. So let's talk about the two techniques we can use for finding the thing we're looking for. Let's just say for the sake of argument, we want to look for scene number 1, shot number, what am I saying here? Scene number 2, shot number 2, why not? In fact, let's just change it up on the fly right now. Scene number 2, shot number 1. And so we want to take number, the first method we can use, I often call extract and replace because when we have a complicated data structure, it can be kind of challenging to mutate it in place. And so pulling a piece out, making the change and then pushing it back in can be something that simplifies our lives, but we need to know what to pull out where we're pulling it out from. So XAML provides these, you know, cool functions that are called, you know, around arrays that are like, you know, FindFirstElement, FindAllTheElements, to find members of an array that meets certain criteria. But sometimes we skip over that there's also one for finding the first element index. And this is really important. Because when we find the first element, it doesn't actually tell us where in the array it came from. So that means while we can pull something out of the array, we don't actually know where it's supposed to go back to inside the array in order to replace the previous version of it. So that's where find first element index comes into play. By telling us the position of the element, we could then turn around and get that element based on that index number. And then we can make the modifications and push it back into place because we know which slot it goes in, right? There's a difference between having someone retrieve a book for you and having someone tell you where the book is. By telling— if they tell you where the book is, then you can retrieve it and you know how to return it. If they just gave you the book, you have no idea to return it. And that return is the key to being able to finish the data surgery project here. So first, we're going to say we're going to find the first element in— find the first element index, right, the position of the element in input, which matches where the object contains my scene ID. Now, remember, I'm looking for scene ID 2 here. So the first element would not meet this criteria. But the second element would, yeah. The second. Yeah. So let's, so the first one would not. And the second element. Yeah. See, it has a cNumber of 2, which is exactly what I'm looking for. So I'm expecting that to happen. Now, need to remember that this is computers and computers like to start with 0. So the, our expectation is that when we run this, this will return for me a 1 because the index that I'm looking for is actually the number 1. And so we're once again going to take my little stop and debug here. And instead of looking at output, I'm just going to look at the index that got found. So if this worked properly, it should give me a 1. And indeed it does, which means it'll give me the second element of the array. Remember, 1 means second. Then I'm going to say, well, let's, let's go pull the scene out of that array, right? Go get a copy of it by referring to that variable using a get filter. Because get filters allow us to pull dynamically instead of just statically the way that dot syntax does when we're just doing it directly in line, right? I can always say output.0, but that would only I only ever get the first element in the array. By saying give me at the index, it gives me the element that is at position 0 or 1 or whatever it is. And now I have a copy of the target scene. Now, keep in mind that editing the scene doesn't actually edit the original, but it will set me up to be able to go make a modification and send back the copy. So now I've got my scene. And indeed, if I were to just put this in and do a stop and debug on my target scene, And we run. Yep. I got scene number 2 here. Fantastic. Now remember our job here is we're looking for a specific shot inside the scene. And that's going to be based on the shot number. And if you'll recall, we're looking for shot number 1, right? Yeah. 1. And the, so we were going to use the same trick, find first element. We're going to look inside the target scene's shot elements. To see which one matches my shot ID. And that could be the first element, the second element, the third element, whatever it turns out to be. And I'm just going to call that shot index. And I believe actually this should be the first of them, but let's, you know, before we go nuts here, let's, let's find out. So inside shot elements and yeah, so shot number 1 should be the first element in the shot elements of this particular scene. So it should be scene at index position 1, and it should be shot element at position 0. Let's find out. So we're going to get the shot index, right? So we're expecting this to return a 0, and it does. Good. So next thing we can do is we can say, let's get the target shot. And that's going to be the shot that is at that particular index. And the, the way we do that is just through, once again, the get a filter where we're going to call it target shot. We're pulling it from target scene shot elements, which is the array. And we're going to say, give me the element at position number 0 in this case. Because remember the shot ID we're looking for was 1, but the index we're looking for is 0. That's why we need to use the find element index because those numbers don't actually line up. And we'll say save. Okay. So that means I'm going to do a shot, stop and debug, just make sure this is actually giving me the shot I'm looking for. So save and rerun. And here I can see that this is my shot number 1. Good. Associated with scene 2. It actually just said associated with scene 1. Let's see if that's a data problem. And shot elements. This is all the stuff from scene 1. And scene 2 shot elements. Oh yeah, you can tell I sort of copied the data. And so like if I just say that this goes with scene 2, 2, and this goes with scene 2, right? Because these are all inside scene 2. And this goes with scene 2. Right? You can see everything for inside scene 2 goes with scene 2. So having made that change, if I do a rerun, my expectation is that the data that it pulled should be shot 1 from scene 2, which is exactly what we're looking for. Cool. So now we've got the, uh, the, so, so now we've got the shot. Right. The correct shot from the correct scene. Now comes the part where we actually do the cutting, right? We're trying to make a change in this particular shot that we want to be in place inside the variable. And so we'll say, let's modify the shot to have some additional property or edit whatever you're going to edit. And of course you can do multiple things here, but I'm just going to do one thing just to indicate that we made the change and then call it a thing, right? I'll say that the target shot should now have an additional member, additional path called my thing. And it will just call, you know, state change YouTube, right? And that way we'll know that was changed while we're right here on the stream. And we'll say save to this. So now that I know that changes state change YouTube, I now have a modified target shot, right? So now let's do a rerun here. And what this should do is show me the shot again with one more thing at the bottom. Yep, there it is with the state change YouTube. So now we've modified the shot. Now our job now is to take the shot and put it back in the scene and take the scene and put it back in the list. Uh, we're returning the book to the library. So the, what we're going to do here is, um, we're going to update shot elements, right? Which is the array that's inside the scene. And we're going to, um, and this is actually a really key thing here. We're going to set shot elements position, whatever position we were looking for. With this newly modified target shot, right? So basically we were replacing the shot that was there before, the object that was in that slot in the array with the new object that should be supposed to be in that slot. And as opposed to doing some sort of more direct mutation, although we'll get to that in just a minute. This is a really key idea for making the surgery work. So it's worth just spending a second on here. We want the shot elements member of target scene Right, the thing that previously, over here on the right, we can see that shot elements is an array. It's an array with object, object, object. And we're going to set the object at position shot index, which if you'll remember is 0. That's gonna be the first guy here. And we want to replace it with the newly modified target shot. So remember the original version of this just ends with original prompt. Right. But I save this and then I can go here and say, oh, let's take a look now at the target scene. Let's see what this looks like here. So now we have scene 2, uh, with my shot elements, and I'm expecting the shot elements that was at position 0, uh, to have my new element in here. Yep. There it is. StateChangeYouTube, right? So now we've taken this shot that we've modified, that we added StateChangeYouTube to, and we've made it part of the scene that's in this list by using this dynamic update approach. But the next thing we need to do is we need to modify the output itself so that it now contains the new scene. So let's take a look at how we would do that. And it's rinse and repeat, right? We're just saying, hey, output, we want to replace the element at position index, which in this case is going to be 1 because it's the second scene, with this target scene that we just did the surgery on, right? So it was like recursive surgery as we're going back up the tree. And you can go as far down as you want with this too, depending on the complexity of your particular situation. So, and then we can just say, well, let's just find out what output looks like at that point. In fact, I'm not even sure I need to do that. I'm pretty sure The down here, it's just going to tell me what output is at the end of the function. So I think that that was, that we may, maybe our job is done at this point. So let's rerun this. And yep, we have scene number 1 with my shot elements. Yep. Looks like my original shot element, my original shot element, my original shot element. You'll notice none of this YouTube stuff. And then scene number 2 and my shot element. With— oh, I see. Well, we'll come back to that in a minute. Let's rerun that. All right, let's run it. So now we have the scene number 1 and the shot elements. And Shot elements and the shot elements is all from scene 1, of course. And now we get to scene 2 and we have our shot elements in scene 2. And you'll see, yep, here it is. State change YouTube is my thing. And then, uh, URL, URL, right? So the only thing that got modified in this whole thing was that one shot that we want to do. We did our surgery. We pulled out the scene. We pulled out the shot. We edited the shot. We put back in the shot. We put back in the scene. And now we have our new output that is, you know, has the result of the surgery we were looking to do. It's pretty cool. Let's run up the score just a little bit here because this is a technique that I would recommend whenever you're not quite sure how you're supposed to do this. Um, and it's, it's relatively safe. It's relatively explanatory. Um, but there is actually a more terse and kind of cool way of doing this. I want to leave you with as well. So let's disable that. And let's talk to method 2, which you sort of, you knew it was coming because you saw there was a method 2 down here. And here we first look for the element in the scene, right? Just like we did before, we're trying to find the index based on the scene ID. And then what's kind of cool here is we can be recursively looking through to say, let's look at the output, right? And we're going to look at the output at the, you know, at the position we were just looking for, right? So if originally we're looking for scene 2, they returned index value of 1. That's what I did last time. And then we're saying we're going to pull that from output and directly ask about shot elements. This allows us to stack get parameters. So let's just talk about how this works because it's not— I appreciate it's a little complicated and it might not be totally intuitive, but super powerful once you get it. So our very top of blob is made of two elements, right? These are my two scenes. I know that like the index is going to find the go to scene ID is going to be the second one of these. So position 1. So this first get is going to be returning to me this thing that I see from 74 to 142, right? This, this particular scene. Now, what I— because I just have that object in temporary memory at that point, I can say, well, let's also get from inside of there, the thing that's called shot elements. Shot elements, right? Which is now this array of these 3 shots. So now that I'm just looking at the array of shot elements, that's particularly in scene, in scene 2, you know, or scene position 1. I can then search on that because now this is going to refer to each element inside the second scene inside of shot elements. So I can just do it all in one here. And I can say, well, if the shot number is equal to the shot ID I'm looking for, We'll then pull out that index. And that of course is going to get me index 0 as well. Right? So I'll say save and that gives me, and I'm expecting this to be a 0. In fact, let's just check for one second. And we will do a utility function again. We're just going to do a stop and debug to confirm that in fact, we're looking at shot index of 0, right? And just like, and should also have found the scene number is 1, right? So scene at position 1, shot index 0, same thing as we were looking at before, right? Finds them the correct positions. Now we get to the really cool part. Now there's sort of a trend right now in Xano of starting to get more expression heavy and starting to use kind of a meta language inside Xano to be more talking to data in a slightly more complicated way that's sort of reminiscent of the way like, you know, jq works. But you've actually been able to do this in Xano for a while by using the get filter or the set filter by using custom paths. We talked before about how we could just fill in a 0 or a 1 right in the path or get something from an array. But actually, we can, we can actually replicate the complexity of dot syntax inside a get or a set parameter. And so what I can do here is I can create a new variable called path.. And I can say, well, I want to do— if we know we want to be, you know, 0.shot_elements.1.mything, right? Well, that's cool. We can do that. So the, we can do that dynamically by simply replacing like the word scene by using the replace filter with the index for the scene, right? Which would be position 1 and the index for the shot at position with a 0. And so this allows me to create, you know, something that lets me, you know, dynamically create dot syntax in Xano, um, which I think is just really a cool little feature of this tool, right? So 1.try_elements(0, myThing), right? You're like, well, sure, you know how to use that in, in, you know, usually, but like, how do you use it dynamically? And we, and we can apply that in this update variable because we can use this string in a, um, in a set or a get call. You can bring it not just like a dynamic index. You can bring it the whole path and it will traverse all the way down and it will do it surgery directly in what you want to modify. So this, um, you know, now I can say this is instead of calling this new thing, I'm going to call this state change YouTube advanced method. Okay. And we're going to say save here. And you know what? I'm just going to comment out. Yeah, fine. Let's comment out my stop and debug. That way I get my output and it should come all the way to the end. Now it should modify the same element as it did before, but this time it should give me— we'll look at the outputs now. Yep. Scene number 1 should have nothing. Yeah, it looks normal. Nothing looks normal. Scene 3, or shot 3, nothing. Nothing looks normal. Scene 2. Shot 1. Shot 1. Ah, advanced method. See, you can create something that is advanced or as complex and as dynamic as you want using that path. And of course it'll explode if that path doesn't exist. I mean, don't get me wrong. There's a, you have to be confident that you're talking to something that's real, but it's pretty darn cool once you actually pull out the thing you're looking for. This allows you to just take care of the hard parts at the top here, right? Which are how do you find the index of the elements you want to work on? Then, you know, dynamically build out that path and then boom, do the injection of exactly what you meant to do in the place you want to do it. I think of it a little bit like the difference between doing open surgery where you have to, you know, do a lot of cutting and take a lot of steps like you do in the first method here, right? Where it was like a total of what, like 7 steps, not including stop and debug. And down here you can see just in 4, again, not counting stop and debug. So you have both the, you know, a sort of easy way step by step to do it. You have a cool little expression building way of being able to do your data surgery. And I would encourage you to go forth and, you know, cut up your data, get the insight you're looking for from it, and hopefully not be held up too much more. We do this kind of, you know, hard data surgery, data manipulation, and thinking in higher dimensions work every day over at StateChange. And if that's something that's challenging to you, please come on over and check us out. And otherwise, I'll just see you next time. Bye.
