The Simplest Way to Automate Scraping Anything with No Code (Apify + n8n tutorial)

So Apify is like a marketplace for actors, which essentially let us scrape anything on the internet. As you can see right here, we're able to explore 4,500+ pre-built actors for web scraping and automation. And it's really not that complicated. An actor is basically just a predefined script that was already built for us that we can just send off a certain request to. So you can think of it like a virtual assistant where you're saying, hey, I want you to, I wanna use the TikTok virtual assistant and I want you to scrape, you know, videos that have the hashtag of AI content. Or you could use the LinkedIn job scraper and you could say, I wanna find jobs that are titled business analyst. So there's just so many ways you could use Apify. You could get leads from Google Maps, you could get Instagram comments, you could get Facebook posts. There's just almost unlimited things you can do here. You can even tap into Apollo's database of leads and just get a ton. So today I'm just gonna show you guys in N8N the easiest way to set up this Apify Actor where you're going to start the Actor and then you're gonna just grab those results. So what you're gonna wanna do is head over to Apify using the link in the description and then use code 30NATHURK to get 30% off. Okay, like I said, what we're gonna be covering today is a two-step process where you make one request to Apify to start up an Actor, and then you're gonna wait for it to finish up, and then you're just gonna pull those results back in. So let me show you what that looks like. What I'm gonna do is hit Test Workflow, and this is gonna start the Google Maps Actor. And what we're doing here is we're asking for dentists in New York. And then if I go to my Apify console and I go over here to Actors and click on the Google Maps Extractor one, if I click on Runs, we can see that there's one currently finishing up right now. And now that it's finished, I can go back into our workflow, I can hook it up to the Get Results node, hit test step, and this is going to pull in those 50 dentists that we just scraped in New York. And you can see this contains information like their address, their website, their phone number, all this kind of stuff. So you can just basically scrape these lists of leads. So anyways, that's how this works, but let's walk through a live setup. So once you're in your Apify console, you click on the Apify store, and this is where you can see all of the different actors. And let's do an example of like a social media one. So I'm gonna click on this TikTok scraper since it's just the first one right here. and this may seem a little bit confusing, but it's not gonna be too bad at all. We get to basically do all this with natural language. So let me show you guys how this works. So basically we have this configuration panel right here. When you open up any sort of actor, they won't always all be the same, but in this one what we have is videos with this hashtag. So we can put something in. I put in AI content to play around with earlier, and then you can see it asks how many videos do you want back? So in this case I put 10. Let's just put 25 for the sake of this demo. And then you have the option to add more settings. So down here we could do, you know, we could add certain profiles that we wanna scrape. We could add a different search functionality. We could even have it download the videos for us. So once you're good with this configuration, and just don't overcomplicate it, think of it the same way you would like put in filters on an e-commerce website, or the same way you would, you know, fill in your order when you're DoorDashing some food. So now that we have this filled out the way we want it, all I'm gonna do is come up to the top right and hit API and click API endpoints. The first thing we're gonna do is we're gonna use this endpoint called run actor. This is the one that's basically just going to send a request to Appify and start this process. But it's not gonna give us the live results back. That's why the second step later is to pull the results back. What you could do is you could run the actor synchronously, meaning it's going to send it off and it's just going to spin in and in until we're done and until it has the results. But I found this way to be more consistent. So anyways, all you have to do is click on copy and it's already going to have copied over your Apify API key. So it's really, really simple. All we're gonna do here is open up a new HTTP request. I'm going to just paste in that URL that we just copied right here. And that's basically all we have to do, except for we want to change this method to POST, because as you can see right here, it says POST. And so this is basically just us putting in the actor's phone number, and so we're giving it a call. But now what we have to do is actually tell it what we want. So right here, we've already filled this out. I'm going to click on JSON, and all I have to do is just copy this JSON right here, go back into n8n, flick this on to send a body, and we want to send over just JSON. And then all I have to do is paste that in there. So as you can see, what we're sending over to this TikTok scraper is I want AI content and I want 25 results, and then all this other stuff is false. So I'm just gonna hit test step. And so this basically returns us with an ID and says, okay, the actor has started. If we go back into here and we click on runs, we can see that this crawler is now running and it's going to basically tell us how much it cost, how long it took, and all this kind of stuff. And now it's already done. So what we need to do now is we need to click on API up in the top right, click on API endpoints again, and scroll all the way down to the bottom where we can see get last run dataset items. So all I need to do is hit this copy button right here, go back into N8N, and then open up another HTTP request. And then I'm just going to paste that URL right in there once again. And I don't even have to change the method because if we go in here, we can see that this is a GET. So all I have to do is hit test step, and this is going to pull in those 25 results from our TikTok scrape based on the search term AI content. So you can see right here it says 25 items. And just to show you guys that it really is 25 items, I'm just gonna grab a set field. And we're going to just drag in the actual text from here and hit test step. And it should— oh, we have to connect the trigger. So I'm just going to move this trigger over here real quick. And what you can do is, because we already have our data here, I can just pin it so we don't actually have to run it again. But then I'll hit test step, and now we can see we're going to get our 25 items right here, which are all of the text content. So I think just the captions or the titles of these TikToks, and we have all 25 TikToks As you can see, so I just showed you guys the two-step method and why I've been using it, because here's an example where I did the synchronous run. So all I did was I came to the Google Maps and I went to API endpoints, and then I wanted to do run actor synchronously, which basically means that it would run it in NNN and it would spin until the results are done, and then it should feed back the output. So I copied that, I put it into here, and as you can see, I just ran it with the Google Maps looking for plumbers and we got nothing back. So that's why we're taking this two-step approach where, as you can see here, we're gonna do that exact same request. We're doing a request for plumbers and we're gonna fire this off. And so nothing came back in n8n, but if we go to our actor and we go to runs, we can see right here that this was the one that we just made for plumbers. And if we click into it, we can see all the plumbers. So that's why we're taking the two-step approach. I'm gonna make the exact same request here for New York plumbers. And what I'm gonna do is just run this workflow. And now I wanted to talk about what we have to do because what happens is we started the actor and as you can see, it's running right now. And then it went to grab the results, but the results aren't done yet. So that's why it comes back and says, this is an item, but it's empty. So what we want to do is we want to go to our runs and we want to see how long this is taking on average for 50 leads. As you can see, the most amount of time it's ever taken was 19 seconds. So I'm just going to go in here and in between the start actor and grab results, I'm going to add a wait, and I'm just going to tell this thing to wait for 22 seconds just to be safe. And now what I'm going to do is just run this thing again. It's going to start the actor, it's going to wait for 22 seconds. So if we go back into Apify, you can see that the actor is once again running. After about 22 seconds, it's gonna pass over, and then we should get all 50 results back in our HTTP request. There we go, just finished up. And now you can see that we have 50 items, which are all of the plumbers that we got in New York. So from here, now that you have these 50 leads— and remember, if you want to come back into Apify and change up your input, you can change how many places you want to extract. So if you change this to 200 and then you clicked on JSON and you copied in that body, you would now be searching for 200 results. But anyways, that's the hard part, is getting the leads into N8N. But now we have all this data about them, and we can just, you know, do some research, send them off an email, whatever it is. We can just basically have this thing running 24/7. And if you wanted to make this workflow more advanced, to handle a little bit more dynamic amount of results, what you'd want to use is a technique called polling. So basically, you'd wait, you check in, and then if the results were all done, you continue down the process. But if they weren't all done, You would basically wait again and come back and you would just loop through this until you're confident that all of the results are done. So that's gonna be it for this one. I'll have this template available in my free school community if you wanna play around with it. Just remember you'll have to come in here and you'll have to switch out your own API key. And don't forget, when you get to Apify, you can use code 30NATHURK to get 30% off. Anyways, that's gonna do it for this one. If you appreciated the video, if you learned something new, definitely give it a like. It helps me out a ton. And as always, I appreciate you guys making it to the end of this video. I'll see you in the next one. Thanks everyone.