Beyond Automation Basics: Reverse Engineering a Private API (Canopy Tax)

Hey, this is Isaac from Upstart.co. We are an automation firm helping accounting firms get the most out of their software apps. And today I have a relatively complex automation scenario that I want to share just for people to start to realize what are some of the possibilities when you face challenges. So preliminarily called, you're automating the impossible using the process automation pyramid. Here's the use case real quick. So we were— a mid-sized accounting firm reached out to us wanting to do an integration between Soroban and Canopy. They're using Soroban for tax organizers, tax questionnaires, and Canopy is actually their practice management solution where the work gets done. However, Canopy only gives access to clients in the developer documentation or their open API, which I'm going to share and explain in a bit. But they wanted to automate certain aspects of their tasks and of their files, which the API doesn't have access to at the moment. I reached out to their team to inquire about it. They told me that their API has no plans to be expanded upon for the next 2 years. So I was like, well, there's got to be another way. Okay. And here's where the hidden API comes to the rescue. And just for some context on kind of like the level of complexity that this would have in our view, we have developed this framework of like hierarchy on like difficulty levels that we go through to assess if something can be automated. At the bottom is like things that are on the Chef apps. Like, they're within the apps you use on a regular basis for regular users. And often they have native integration that just like plugs plug and play or building automations that are relatively easy to work with. Here's where we spend probably 80-90% of our time, which is working with integration platforms, Zapier, Make, Power Automate being some of the most popular ones. Sometimes we have to create data stores for one reason or another. If simple example is like if within the integration platform we're not allowed to say find for a specific record. If we say we want to find a task on a project management system, then we might create a separate data store where every time a task is created, we add it there and then we can extract it. And here's what we're going to be talking about today, which is APIs. There are different APIs. There are other There are some APIs that are open to the public. They're called open APIs as a result. And they are officially documented ways to do things on behalf of the user in a specific software app. However, there are also what we call the hidden API, which is the request that they automate— the browser is doing as the user interacts with the software tool. And that's actually what we're exploring to be able to automate and expand Canopy's capabilities And that's what I will be sharing today. This is a way to demonstrate that, hey, even when you think there's not— it's not possible, we still have tools in our toolbox to make it happen. So let's take a look. As I mentioned earlier, if you look for Canopy's API, you quickly see— so here's Canopy's public API. This is the tool that they're using. You'll see that primarily we have clients. Where we are able to get, uh, we're able to create, we're able to search for, we're able to delete, and we're able to update clients. Okay, so usually this is like the client information. As you can see here, either emails or phone, their addresses, any custom fields you might have. Um, looks like if they're linked to other clients, etc. But here we have nothing about the tasks related to that client, nothing about the files in that client, along with other things that maybe the client might need at some point. So what do we do in this case? In this case, we actually log in as a user and start to analyze what does the tool do when I actually create something. As an example, let's say I open up any one of these, uh, tools— any one of these clients, sorry— and I want to add a task. If I open the inspect in my browser and I go network, you'll start to see that every request that the, the tool makes appears here, and I can at least take a look at it. So for example, if I add a note, I'm gonna call it tests YouTube Description. I'm going to clear this out. I can do that by clicking here. As long as this is red, this is actually quote unquote recording the request. And if I add a note, I should be able to see something come in here. Very promising. Let me just stop the recording. Very promising. It says notes. And as you can see here, it's a note. Looks like for a specific client ID, looks like that's something that's needed. But the point is we can actually copy this. And this is where it maybe gets a little bit more technical than most people would want to see. But there are API platforms that allow us to easily work with some of these, easily export some of these requests. To further— for further analysis. So one of those platforms, it's called Postman, and I have the ability to copy this URL and paste here. And here I'll be able to see and test with, uh, and really the goal here, I want to understand what's required for the note to go through. For example, I can see a body with the title in the— in the description I shared And there's a lot of different headers. Okay. I already did this in advance, but the goal here as a developer is trying to reverse engineer this for automation purposes. You can like go through one by one and see what's required, what's not. You start kind of deleting and trying to, for example, send it again. And you'll see that that notes actually gets added again programmatically. In this case, this is the one I added manually. If I refresh this page, I should see now another one that I added via Postman. If this works in Postman, I can actually transfer it over to Zapier, transfer it over to Make or other tools or a script and be able to do that request programmatically and automatically whenever I want to. Okay. Now I did some homework as I mentioned earlier, and within the cookie, header, there's actually a value that's required. Okay, this is the only value that's required. It's called identity token, and this identity token effectively represents me as a user. Okay, so as long as I have— all this work we did here is just to understand that as long as I have an identity token, I can pretty much expand the possibilities of what I can automate without Canopy to pretty much everything a user can do almost. So essentially test like, I don't know, test YouTube, test Postman, description Postman. And let's try again. As you will see, it goes through again. And if we replicate this It should come through. Now, the question now—it's twofold: How do we make sure we understand this so we can replicate it and trigger automatically? But second, how do we get the identity token, which we're going to need? Okay, and this is something again that's a little bit more technical, but I've already done in advance where I've kind of like replicated the login flow. In other words, I've created a little. Web automation that I'm actually gonna play for you here and explain what it's doing. This is actually trying to log in, and as you'll see here, for example, this is saying, hey, um, it requires a— let me see— error, right? There's a multi-factor authentication required, and I'm actually able programmatically to request that multifactor authentication code. This is actually waiting for 30 seconds, running another scenario that has access to my inbox, pulling that code, and then waiting another 30 seconds to make sure that that code, that verification code's now in my inbox. in our data stores that we built internally within this app, which is called Make. And then I'm gonna get the identity token at the end. And this is something that's running about every hour and 40 minutes 'cause the token expires every 2 hours, okay? This is just a long way of saying we understood what the app is doing on behalf of the user when they do things. We understood that this token was required And we found a way to always have this token fresh and available. In other words, not expired, so that now we can do other things. Again, somewhat of a complex scenario, but for example, now if I wanted to say upload a file, let's see if I have anything here. Let's see, I have this logo here of their competitor, funny enough. Oops, sorry, I should have opened up the Networks tab first. Let's clean everything up. Let's make sure we're recording. And now let's open— let's just upload a file. And this is not something I've done in advance, but usually you'll be able to see certain, uh, promising names. For example, here you can see files. So here I would kind of try to again copy this as a cURL, move it back to Postman, and just try to understand what's required and what's actually getting done. So I can see here the name of the file which is needed. I can see the path, um, the MIME type. Among other things, right? And I wouldn't be surprised if here I see identity token, right? It's possible, just based my assumption, just based on other tests I've done here, is that if I delete everything except for the identity token, it should be able to run this. And it should work. Probably, I'm not sure. We're gonna find out. Okay, so I have the identity token. I have removed everything else. I have this body. If I send this, I can— it should upload the files again. I have it twice here because I uploaded it twice. Now we can see it a third time. So now you can see how I'm actually able to programmatically add files potentially. We'll have to keep trying to understand what the app is doing on the backend, but we can now add notes, add tasks, add organizers, add other things. Pretty much everything the user is able to do, we might be able to replicate as a little bit of an automation or robot. So that's kind of it for this video. Um, well, I actually have another note. Something that I was— I saw when I started to look into whether this is something we could do within this particular app, which is called Canopy, is the restrictions that they have on their terms of service. So some apps, for one reason or another, it's probably insurance, for security purposes, people trying to hack them or something like that, have certain restrictions. And for example, here they talked about a couple of things that might be representative of what I'm doing here, which is like reverse engineering, workarounds for kind of security measures, which is like that automation of 2-factor authentication, and accessing the service via bot or web crawler or a non-human user. Again, I think most people, most companies have this in place, just safeguarding themselves. But if they know that there's a user that wants to have an automation and there's a team of experts on the backend doing it with no malicious intent, they're usually okay with it. I'm actually in the process of talking to Canopy's product manager and their legal team To explain a little further kind of what we're trying to accomplish, because they have no intentions to expand their API for the time being. So we have to find workarounds for a client to be happy, both with us but also with them. So it's on their best interest. So that's something I wanted to explain of what API entails, both on the open API side and on the quote unquote hidden API side, and showcase maybe a little bit of more of a complex scenario. Um, of what we can possibly do when, you know, options seem limited, at least to the untrained person. So I hope you enjoyed this video. Feel free to like it wherever you're seeing it, and I would appreciate a comment if you're curious to hear more about, you know, where we ended up with this. So appreciate your time and have a nice one.