Advanced Firestore Integration with Pipedream

Hey, so it can be challenging to get Pipedream to do all the things we want to do, but Pipedream has a really powerful feature in which you can import any arbitrary set of node packages into its workflows. And that allows us to really give these code steps we're seeing inside Pipedream superpowers. And this is where Pipedream really starts earning its money. So we can have your ordinary trigger coming in that's going to have whatever your inputs are going to be. I'm going to ignore that for the present moment and only look at the question of how we can really make a node step sing in Pipedream. And we're going to do that using Firestore and solving the somewhat tricky problem of getting a document reference into a Firestore package. So here I've got a sort of test database over here in Firebase, and it can be very simple in Pipedream to say, let's just add another step here, which is going to be for Firebase using their Admin SDK to say, let's go create a new document or to add new documents into Firestore. And they have a nice simple system for constructing the object using the REST API. The trick is we weren't able to figure out how to use this to do more advanced structures, more specifically the, the reference value, because reference value kind of looks like a string. It's just pathy. It looks like this in Firestore where you have a ref value, which has a slash and refers to the collection. This is my collection. And then it refers to a document number and here's the document that it'd be referring to. But it's not just a string, it actually knows that it's supposed to be referencing another document in here. It can be used for clicking, it can be used for more advanced query techniques, what have you. So it's important that we not just hold that as a string. So to solve that problem, we need to go back over to, we, we go back over to Pipe Dream and Pipe Dream does not currently give us the ability to do that using this interface, at least not any way that we could figure out. And so we make use of the more advanced power in Pipedream, which is its node steps, which are really quite generalized and very powerful. You see, in, you don't, unlike your ordinary Node.js package, you don't have to have a package.json separate file telling it what packages we're going to import. Instead, Pipedream infers this from the, from import statements you can put in here if you're using more ES6 syntax and also supports require statements if you're going, which sort of gives it a hint that you're gonna do it using CommonJS syntax. But all those are details that like more advanced people care about. Most of us just care about how do we get our stuff done. So if you know how to use its API, and fortunately our friends at Google have some excellent documentation about how to make use of their API, you can copy and paste a lot of this code. You can just initialize it in Pipedream. And since it tells you what you need to require at the very beginning, we want to add the SDK, right? We can just add initialize_app and here we have initialize_app. And there's additional thing here called cert, which is gonna be referenced down below. Cert, it's actually on a different page, but that allows us to extract from the service account that we otherwise get from Firestore. And then, then we can run just a few lines in order to get a reference to the database as we do here. Get a reference to the collection you want to add a document to, that's here. And then, and this is important, we need to go take whatever that string was we wanted to add as a reference. And get it, get the reference using db.doc, which returns the reference value, and that's how Firestore thinks about it. And then because it's thinking about as a reference value, when we say please add that reference back into Firestore, it does it as a reference value instead of just as a string. So that's the magic part that like using the SDK takes care of this stuff for us, 'cause under the covers is using something called gRPC, which is based on protocol buffers. And if you ever try to read a protocol buffer, you will lose your mind. It's not like JSON, it is not friendly. So the, so what we want to do is get the SDK to do this work for us. In order to do that, we need to know our, what database we're gonna talk to, which is just the one for your application, what collection you're going to talk to, which is relatively straightforward here. I just know it's going to be bibble. And then what information you want to add. I have a simple string value here, which I will call Hello YouTube fam. And then the, the reference is just gonna be referenced to another one of these documents. And then up here we're going to need, the other key thing about it is we're gonna need to bring a service account to the party. Now when we're doing the documentation over in Firebase over here, they will often talk to you about doing this in Google environments, like using a cloud function or using Google Script or what have you. And all of that can be fine if you're doing that in the Google environment, but Pipedream isn't in the Google environment. We need to bring our own.. And so we need to bring the credentials in. And the way we do that is you go back over to Cloud Firestore, you go to where it says Project Overview, you go to your project settings, and you click on where it says Service Accounts. And now it's not actually gonna show me any service accounts right now. I can click down here, it'll generate a new private key for me, and then I can ditch the old one. But the private key is gonna be in the form of a JSON file, just will just be a string. And we're not gonna look at the rest of that right now 'cause I'm not looking to I'll share that with the whole universe. But here you can see a truncated version of what's actually visible on my screen where you see the interesting private key part is part that sort of goes off over to the right, but then everything else is relatively simple and straightforward. So when you extract it from here, it's going to give you a file that you can just open up. It's a text file. And then you literally paste that right here. You say const serviceAccount equals, and then you paste in what you got from that JSON file that you have just extracted from Firebase. The, and then after that, this is actually where code that came from the sample from Firestore just continued to work. We say initializeApp with a credential using the cert function that we've imported here on the first step with the service account. We deport default. We export the defined component, which is the same way that you always see in a Node step in, in Pipedream. And in the run function that is its first argument or one of the first option arguments in here, we just say we're going to first grab a reference to the database, which is based on having initialized Firestore. We then grab a reference to the collection. We need to know the name of the collection. We get, and then we create our references or whatever, and then we create an object like here between the two curly braces that contain all the key-value parts. And we'll also have, I don't know, number 1234.56, and we'll say intval, which will be 1234. And then we can have other types of values in here. There, there are more advanced things we can do once we've opened up the floodgates of actually making use of the Firebase Admin SDK. And then with all that, it will just run this. And the await keyword is really key here because when it goes and talks to Firebase, it takes just a little bit of time and says, please wait for this to finish, is what await means. That works because we we have the async keyword up here. And for those of you who aren't totally aware of exactly why these things work, that's totally fine. Just remember that when you're going to do something that's a little bit slower in these circumstances, which really means whenever you're going over the network, which means it's not instantaneous from the computer's point of view, that's when we want to use await. Do that. And then finally we're done. So this becomes a substitute for using createDocument. So instead of using createDocument, which is based on the REST, REST API of Firebase, which is what Pipedream is talking to under the covers. We're not going to use this, but you can see this basically requires the same thing. Like, it needs to know about your credentials for the account, which you set up under Pipedream credentials. You, it needs to know which collection you're talking to. In this case, it would be Bibble, and then it needs to know what fields you're going to add, right? Basically, these are the same, thematically, the same things we're talking about here. What's the database? What's the collection? And then what's the data? The difference is this one has a more advanced and more subtle idea of types as a result of not using the REST API, but using this other gRPC API that allows us to make use of these subtleties, like what's a reference value. So let's give this a shot. Remember, we're looking for YouTube fam, 1234.56, 1234, and the reference value to come through, which will be to document, you know, to LH. There we go. We have a document here. It has returned some data. You can find out what is in its return value, although actually we're not even paying attention to what got returned. So we can probably add that in too. But let's go back over to the document and let's take a look at the Firestore database and let's take a look at, see if we can find our guy in here. Remember looking for YouTube Fam. There it is. So YouTube Fam. Both these are numbers, the 1234 and the 1234.56. And you can see this one gets stored as a reference. You can see, uh, document path, we know it already knows what it's supposed to be. And now we have type sensitivity. We have it in Pipe Dream. We know how to do it by pulling in that service account from Google and going from beginning to end. Then after this, the main interesting things you're going to want to do here might potentially be, const output equals, and then I could say, let's return output. We could try running that and see how well it works. And then the, but the thing that will be really interesting is probably taking this data and replace it with things like, like probably steps.trigger.event or something. Basically getting body.body. Information from the body, right? That'll go, that'll be whatever the key is. And that allows you to make it all dynamic. The, and then you'll see down here below by adding this const output equals await and returning the output, we now have information that we can feed into future pipe dream steps, right? We can find out what was the path that we just sent, right? And this will tell you what the segments are of the path that just got created. It'll be in the collection called bibble and in the document called qez. And of course we can create subdocuments or whatever. And then additionally, like which Firestore document that it went with. So now you have information coming in the door that can be dynamic, that is under your complete control when you're sending it over to Firestore, and that you can then pull out and decide, make interesting decisions, feature in Pipe Dream steps that let you decide what you want to do with it. And I think that is that. This is definitely one of those very low-codey things that we do over in State Changes. We work on the hard end of these no-code, low-code projects. And if you've got comments or questions about it, please leave them below.