Tokenizing Text with Xano

Xano has some pretty powerful functions, but one of the things you can do with it that's really great is being able to compose these using multiple custom functions to make the APIs go a little bit easier because the built-in stuff might sometimes be a little bit technical. One use case is actually tokenizing text, right? Being able to take like a big, you know, message or what have you and being able to insert words or phrases to a mail merge style, right? Being able to put in the right person's name or what have you. So to do this, you can, you know, once again, compose custom functions. You can make custom functions over here in the library and we'll walk through that. So you have an input and the input can have the, for if you want to be doing what's called tokenized text, you can have your source, which will be like what looks like inside the mail merge, like, hello, first name. And you know, I'm so glad your dog's name is dog's name and those kinds of things and whatever that text is going to be. And second, you're gonna be passing in a list of tokens. Those tokens we're going to do as a JSON object, which means a bunch of key-value pairs. So we're going to say first name, which would be Ray, and dog's name, which is nonexistent 'cause I don't actually have a dog. And then you would move on with your life. So in this case, you can have your function stack and you would say, let's have, you know, first we're going to define, there's going to be an output variable like the string we're going to have at the end of all this, right? That has replaced all the tokens appropriately. And that is gonna be based on the original input we have 'cause we're gonna modify it as we go along inside this function. Now the next thing we're going to do is we're going to loop over all of those tokens in the form of the JSON of the, you know, first name Ray, last name, whatever it might be. And to do that, we're going to create a foreach loop and this can go over tokens. We're gonna apply a filter that's called entries. That's gonna transform tokens from an object with key-value pairs to an array of objects that actually literally have key and value to them. And we'll talk about how that gets used. We're gonna define that as the internal variable called item that we're then going to reference inside that loop. Inside that loop, we're going to be updating output as we go along, you know, replacing tokens as we find them. And we're going to say, okay, well if we find the, you know, first name, you know, we're going to look for specifically the member of item called key 'cause that's what entries does, create something called key and something called value. It'll say first name and that'll be the key and it'll say Ray and that'll be the value. And we're going to replace first name with Ray as we move along in here. And we can say save and all that's fantastic. Now we're going to make sure that we return output as self so we have an easy way to work with it. And that's just by saying we want to return just output. Fine. And then we can give it a test. So we have an original token as source. You can say all sorts of silly things about it. You'll say, well, this should say hello there, Mr. Ray. And I run it and that's exactly what it does. If I were to modify it and I were to say, Mr., you know, Joseph, it would have the same effect. And if you were to say, you know, you want to say, you know, my dude Joseph, you know, you'd also be able to get that kind of effect, right? It'll just literally replace whatever it sees in this token with whatever the value is. Now, the tokens don't have to be uppercase, but I find it's a pretty good convention to make sure you don't accidentally hit other things. Now, this is all fine in theory in terms of creating the function, but let's talk about how you'd actually use that in context. And for that, I created another function called called test_parse_tokenized_text that will actually use parse_tokenized_text as an external function. And this takes in the first name, last name to be able to create the message you really want to have. And these are just simple strings and you're gonna go create. And from those original strings, I use the, I create an object called tokens and I just set the first name to a key called first name, last name to a key called last name and just assembling this to make an easier API to work with, right? And then I can have another function that will call this function by stacking these, it really simplifies my life. I say save and that's fine. And then I'm going to say run custom function, the one I just made called parse to organized text and pass it these two arguments, which is the message I want to be sending, right? Which is hello there, first name. I believe your family name is last name. And you know, that's a little hard to read these longer strings in here, but you could also pass that as an argument or whatever structure was most helpful for you. And then you just give it the tokens as the object. And the expectation is it should replace first name, last name with whatever it is I'm looking for. And of course that should generalize. I click on run and debug. I have my first name, last name. Okay, well, let's say my first name is Ray and my last name is Deck because that in fact is my last name. And I can do a run and I can say, hello there, Ray, I believe your family name is Deck. And I can then also say, well, maybe my family name is, you know, something else. And it will also do that replacement there. And it will just work pretty much right out of the box. And now you have a much simpler API to be able to replace text in a fairly arbitrary way with whatever keys and values by using this tokenization approach. [Speaker]