How to scrape ANYTHING inside of n8n (simple tutorial)

What's up guys, in this video I'm going to be showing you how you can scrape literally anything inside of NAN using a platform called Apify. And if you've watched my channel, you would have heard of Apify before, but essentially what it is is just an online marketplace of scrapers made by the community. In the video, I'm going to break down all different types of scraping methods, including scraping things like Google Maps, scraping for jobs, lead generation, scraping social medias, and more. Just quickly, if you don't know who I am, my name is Rory Rogers and I run an agency called Autonomously AI where we find and eliminate operational bottlenecks in just 14 days to keep business owners working on the business rather than in the business. So if you're looking to get out the day-to-day operations of your business and start working on things that actually move the needle, then I'll leave a link to my website in the description and you go ahead and book up a free discovery call. Okay, so we're inside of NAN and what I've done is I've laid out all of these pre-built Apify templates that we can start using that are going to scrape a bunch of different things. And what we're going to do is we're just going to go through each of them. I'll show you how to set it all up inside of Apify and then link it up with NAN, and we're going to go through and I'm going to show you some examples of the different things that you can do with this platform. So the first one we're going to be doing is we're going to be scraping Google Maps, and you can see this is pretty much structure on how it's always going to look when you're using Apify. I want to just break it down quickly. So the first node here is going to be run an actor, because the way it works when you're using Apify is first you run the actor— the scrapers are called actors on Apify— um, first you run the scraper and then you get a response to say that it's finished, and then you do another API call to get the dataset items from that call. So you run the actor, it tells you it's finished, and then you do another API call with the ID that you got from that first step to just get the data from that run. So it's always going to look like this with these two I also want to show you around Apify quickly. As I said at the beginning of the video, what Apify is, it's just an online marketplace of scraper APIs built by the community. So you can connect up to these APIs through something like NAN, and they're all made by the community, and you can scrape virtually anything. If we have a quick look here, you can see this just on their homepage. It's great things like Google Maps, websites, TikTok, Facebook posts, Instagram scrapers, web scrapers, uh, Telegram community scrapers, Tumblr, job search scrapers, lead finders, TikTok API. There's so many different things you can scrape. I think there's 7 or 8,000 different scrapers that you can use on here. So coming back over to NAN, I'm now going to show you how you can connect up Apify to NAN. So if we come inside of the node here, all you need to do to connect it up is if you come over to the credentials here, if you go to create new credential, you're just going to need an API key. So if we come over to Apify, go to settings, and then we can go to APIs and integrations, and you'll get this API key here. Simply copy that, come back over, paste it into NAN, and then hit save. And then that's all you have to do. Now that's connected up, there's a few things that we have to configure in here to actually get it cooked up to these specific actors. So the first one, the resource is going to be actor, and then the operation is going to be run an actor. And then we have the actor source. So this is just where we're basically telling NAN which actor we want to use. So we can either do recently used actors or Apify Store actors. Recently used is just going to be, as it says, the recently used actors in your Apify account. Or you can do Apify Store actors, which is just going to be all of the actors inside of Apify. And we have this one connected up to the Google Maps scraper, which I can show you inside of Apify. If we go to Apify Store, and then the Google Maps scraper. The way you can connect this up is you can literally just copy the name like this, come back over, paste it into here, and then it will appear with all the ones with that name. So as you can see, there's quite a few that have got similar names, but this is the one that we're looking for. So we click on that. Now it's connected up, and that's all we need to do to connect them up. The next thing we need to do is we need to input a JSON body. So it's just going to be the data that we actually want to send over depending on the request that we're doing. And the way you find this is if you come over to the scraper that you're looking to use, you'll see here there'll be some different things that we can input. So with this, there's quite a lot of different things. We have the search term, which is restaurant, and we'll just leave it like that for this video. So we're searching restaurants on Google Maps. We can do the location, so New York, USA, number of places to extract, the language, and then there's also lots of other things we can do down here. So things like search filters and categories, uh, additional place detail scraping, company contacts enrichment, uh, business leads enrichment, uh, reviews. We can add on all of these things. We're not going to do that for this because I want to keep it simple since it's the first one. But let's say we're looking to scrape restaurants, and then we'll change this to London, UK, like that. So we're going to do London, UK, and we're going to search for restaurants. Perfect. The way we get the JSON for this, we simply just switch it from manual to JSON. And then here you can see these are all of the variables that we could have inputted, and a lot of them are false because we didn't touch a lot of them. But these are the ones that we actually touch. So the search strings array is restaurant, and then the location is London, and then the number is just going to be 10. So we just copy all of this come back over to NAN and then paste it into there. And now it's all we need to do. So we can just hit execute step. And then what this is going to do is it's going to run and it's going to finish running when the actor actually stops. So if we come back over to Apify, we can go to runs. And then you can see now it's running. So if we click on here, you can see the data actually coming in. Okay, so actually finished pretty quickly, but this is the data inside of Apify. This is what we got. So we can come back over to NAN and you'll see now this is finished, but we don't get the actual data. It just says the status, which is succeeded. That's what we're looking for. So then what we do is we put down another node which is going to be resource dataset, and then the operation is get items. And now we need a dataset ID, and the way we get that is we can just search for it from the variables from before. So if we go to dataset, and then you see we get this default dataset ID, so we can just drag that variable in. And this is going to be basically the unique key or the unique sort of string of numbers and letters that was tied to that run that we just did. So now when we hit execute step here, we'll actually get the data. As you can see, this is the data. So this is the name of the place, this is the price, the category, the address, the street, the city, uh, their website, their phone number, all of this kind of stuff. You can see this is a lot of data that we get back, just all data on this Google Maps listing. So yeah, we get 10 items because we got 10 restaurants that we were scraping. That is a really, really simple example of how you can connect up an actor to Apple What I wanted to do as well is give you another example of a Google Maps one. So if we just move this down and then we'll connect up this other one, this one is going to be the same kind of thing, but this time we're gonna scrape for the emails and phone numbers, which is pretty cool. So you can see this one is called Scrape Haven. If I come back over to Apify, this is the one that we're connected up to. So Scrape Haven, get leads, emails, and phone numbers. We've already got it connected up and you should know how to do that now. You literally just type in the name of the one that you're looking to search for, and then the search term, coffee shops in New York. We can just do the same thing as before. Let's say, restaurants in London. Another thing I actually want to point out here is the pricing for this as well, which I forgot to mention. So for all of these, they're all going to be priced at a certain amount, and it's either going to be paying per result or paying a monthly subscription. So some of these will— you'll have to pay a monthly subscription, and they usually range from sort of $15 to around $30, maybe $40 is the average kind of range. Or you'll pay per 1,000 results, and the average range for that is usually about $5 to $10. And the way that actually bills you is down here. You can see I'm on this plan, which me $39 of usage per month, and of course it cost me $39 per month. Um, if you want to use this for free, you get $5 per month absolutely free from Apify that you can do anything you like with. But if you did want to start paying for this, if I go to billing and then I go to subscription, we can see all of the different subscriptions here. So these are the different subscription levels. It's either Starter for $39, which I'm on, Scale for $199, Business for $999, and then of course they have Enterprise as well. And with this, you just get the credits that you pay for every month. So if you're paying $199 every month you just get $199 worth of usage. So that is how the pricing works. Now we're coming back over to this actor. Uh, we've put in the search term that we want. We can now switch this to JSON, and then let's copy this JSON body again, come back over here, and then just paste it into here. Then we hit execute step. And one of the things we can do while waiting for this to run is the dataset items is going to be the exact same variable. So we can just copy this from here and then we can paste it into here, because of course the information in that variable is going to be different, but the name of the variable is going to be the exact same every time. So it's going to be JSON .default_dataset_id. So it's still running. If we come back over to Appify, we can actually watch this running in real time. Uh, it's this one up here. So this is how the logs look when they're running, and I wouldn't worry too much about trying to understand what's going on here because you don't really need to know what's going on here. This is just the logs of what the actor is doing. So you can just kind of sit back and wait for them to all come in, which is what we're going to do. Okay, cool. So it's just finished running. If we come back over to N8N, you can see that's finished running now. So if we execute this We get 10 items back. This is what it looks like. Okay, so this is one of the things with actors on Apify, and I kind of wanted to show you. As you can see, a lot of these are empty, right? We did everything right on our end, we've inputted the right data, but a lot of these are empty because these actors are community made. Sometimes the quality isn't always going to be completely up to scratch, which is okay because all you have to do is you just have to go through and test a couple out and then find the one that works the way you want it to. So for this one obviously doesn't really work very well, but I wanted to include it in here just to kind of show you that not all of them work and you need to actually go through. Majority of them work, of course, but like I said, with it being community made, they're not all actually made by Apify, so you will need to sometimes use one, figure out that it doesn't really work how you want it to, and then switch it out. So now we're going to move on to this job searcher one here. If we come inside of here, I've already linked up the one that we're going to use, which is this one here, the job search engine. So if we find this in Apify. Obviously I'm doing this a little bit backwards. I already have them linked up in N8N and then I'm finding them in Apify. You'd obviously be doing it the other way around, so that you'd find them in Apify and then connect them up to N8N. So this is the job search engine, and in here there's lots of different configurations that we can do. And this is basically just doing what it says— it searches for jobs on lots of different platforms. This could be LinkedIn jobs, Facebook jobs, uh, Indeed, anything like that. It's going to search and find us all of these jobs. So what we're going to do We're going to test this out. So let's say max results, let's just do 10. And then posted since, we can say in the last 1 month. Country, uh, let's just do United Kingdom because that's where I'm from. Location, this is optional. As you can see, these other ones are mandatory because there's no optional. This one's optional, so we'll leave it. And then for the search terms, let's say we're looking for marketing manager in here. Cool. So now we can just switch this to JSON. This is what I like to do, put all of the inputs in on the manual mode and then switch it to JSON, and then you already have them all filled out there. So you can just copy that, come back over, and you can paste it into here. Perfect. And now what we're going to do is we're going to put in the JSON dataset ID variable, and then we're just going to run this all through. So if we click this button up here, we can run it through, and we'll give this a sec and we'll come back when it's done. Okay, so that's just finished running. You can see we've got 50 items through and they look like this. So job type permanent, full-time, and freelance. Uh, the description HTML, then the description in text, you can see head of marketing. This is the company's name, this is the company's URL, this is the location, so it's in Milton Keynes. This is the salary, the rating, the benefits, lots of different data points here. And another thing we can do, uh, that I just thought I'd chuck in is obviously you can do manipulation with data after this. So I just add in a simple filter and we can just filter for something like, let's say salary max, and right now it's £80,000. Let's say we want ones that are only 50,000 and more. So let's go to number, and then let's say is greater than, and then let's go— actually, let's go 60,000. So 60,000, and then execute step. Let's see how many. Okay, so we kept 13 here. So only 13 of them, the salary was more than 60,000. So that's just another thing you can do with it. You can filter out, and then you get all of these data points where these are all the jobs that are in our, you know, our niche— head of marketing from the UK and the salary is more than $60,000. And now we have a lead generation system. So with this one, if we look in here, run the actor, same drill. This is the actor that we're going to be using. So the leads generator, I'll show you it inside of Apify. This is the one we're using, and the reason I chose this was because it's pretty cheap, as you can see, $3 per 1,000 results, and there's lots of different things that we can filter for. So we're going to go through and we're going to start filtering these. Let's say job title, don't care about that. Let's say email status, Verified industry, uh, we'll go accounting, let's say. So accounting status is verified email. Location, we'll do United States. Total years of experience, let's say 10 is the minimum years of experience that we're looking for with these leads. And then that looks pretty good to me. We should get a pretty broad range. So if we go to JSON now, we can copy this, come back over to NAN, and then we can paste this in here. Cool. Execute step. And then this one might take a minute, obviously, because we're doing 100 items. Rather than before where we're sort of doing like 10 or 50. So we'll let this run. Okay, cool. I actually changed it to 50 just so it would run a little bit quicker and I could start showing you guys how it's working. But you can see here we have these 50 leads and this is all the information that we get from them. Perfect. So what we can do now, just to give you another little added bit, is we can loop through these. And let's say we're doing this for cold outreach, we can generate a personalized compliment for each of them. So we can just add a loop over items node, set the batch size to 1, and then we can go to AI agent we can quickly run this to get some data through here. So if we run this, okay, cool. So we've got the first data item through here. Now if we go to AI Agent and then we'll do the prompt as defined below, and then let's just start giving it some of the data. Okay, so I've just given it the first name, the title, the company name, and the website URL. And now I'm going to write out a prompt that's just going to allow it to research that website because we've given it a Tavli tool here. And if you don't know what Tavli is, it's basically just like a Google API where an agent can input a URL and it can get data back, or it can just search up a company from that. So we're just going to give it a prompt in here that basically says look up their website and output a personalized compliment for them. Okay, so I just said you'll be given information on a company and you'll also be given their website URL. You're part of a cold outreach system. I want you to research a company using your Tavli tool and output a human-sounding personalized compliment. Keep it under 30 words. So obviously really, really simple, but I'm just doing this for a demo so you can kind of see the capabilities of what you can do with the data after you've got it from Apify. Okay, so it's done and it's just said Christina Santander's global commitment to innovation and financial education under your leadership truly inspires confidence in empowering communities and shaping a smarter financial future. Perfect. Okay, so that was the lead generator. Now we're going to look at social media scraping, which is obviously something that's very, very useful. You're going to be doing all the time, especially with Apify. It's kind of the best platform to do, or at least the best all-rounder platform that can do pretty much any social media. So if we have a look in here, we're linked up to the Instagram scraper, and this is one that's actually made by Appify. So Appify do make their own scrapers on here, you can see it's here, um, but majority of them are community made. So simply all we do here is we put in an Instagram URL that we want to scrape. So it already has one in here and there's no reason for me to change it, so I'm just going to leave it there. So if we switch it to JSON, then we're just going to copy all of this, come back over to N8N, and then simply paste this in. Of course, you know the drill by now, it is just doing the same thing every single time. So if we execute step, and then we can also grab the dataset ID variable as well, because as we know it's going to be the exact same every time. So we can put that in here ready for when this actor finishes running. Okay, cool. So it's finished running and now let's run the get_dataset_items and this is what we got back. So we got 10, um, outputs. It's just going to be the last 10 posts of this Instagram account. Uh, so this is the caption, these are the hashtags, these were the mentions, this was the URL. So we can quickly have a look at this. Cool. So it looks like this was the post. Perfect. And then it gave us the first comment and then it gave us the latest comments as well, which is really, really cool. So obviously there's a whole plethora of things that you can do with this. You know, you could reach out to this person, you could analyze their content, you could do literally anything with this information because as you can see, there is so much information that we get here. These are all the comments, these are the images in the video, these are the mentions, the likes, the view count, timestamp. So you can obviously analyze how well the content's doing, uh, the tagged users, all of their usernames, the child posts, so the other posts that go along with it. So literally everything here, and you can do this with pretty much any social media. If we come over to Apify, we can type in essentially any social media. I mean, just here alone you can see we've got Instagram, TikTok, Twitter, Facebook— obviously now X, but of course it will still work— and then you can do things like LinkedIn. So if we type up LinkedIn, you can see loads— there's 1,290 LinkedIn actors. So loads of different things with that. We could say Pinterest. Uh, again, 1,123 Pinterest scrapers. You get the idea. There's loads of scrapers for every single type of social media platform out there. And all you have to do is if you just go on to Apify, put in the information that you want to find, and then you can just run it through an N8N and get all of that information. And then just to finish it off, I also wanted to add e-commerce in there. So for this one, we're going to be looking at Amazon reviews. So this is the actor, the Amazon Bestsellers Scraper. Let's come back over to Apify and let's find this one. So this is the one that we're going to be using, and this for the best sellers. So this is going to be, say you run an e-commerce brand or something in a similar niche to something that's on Amazon, you can just scrape Amazon to see what products are doing well and get all of that information. So the category URL, it's obviously in the bestseller electronics at the moment. We're just going to leave it at that. So if we switch it to Jetwell, there's other things we can do down here. If we have a look at this quickly, the max results, uh, it's on 100, and this is the max results, which means it might not necessarily do 100 results if there isn't 100 results. But this is going to be the maximum amount that it would do. And then how many subcategories do you want to scrape? At the moment it's set on 1. Cool. So if we just switch this to JSON, and then we can copy this, paste it in here. And actually what we're going to do is we're going to check this URL out. Okay, cool. So this is what's showing us, and because I did the same URL that was already in there for the social media one, we'll switch this up just so you can see how it works. Let's look on here, let's go to, uh, computers and accessories. That's something that I'm pretty interested in. So let's copy that URL that's up here and then come back over and we'll just swap that out with this new one. Cool. Let's hit execute step. And then again, let's grab our dataset ID variable, put that into here, and then we'll wait for this to finish running. Okay, cool. It's finished running. So then if we get the dataset items back, we got 50 items and this is what it looks like. We got the name, we got the value, we got the stars, the number of offers, which is none for this one. The review count, the thumbnail, category, the subcategories— uh, lots of subcategories it seems— and then the actual category URL, and then the full URL for that product. So you can see with this, like I said, if you run an e-commerce brand and you want to just quickly scrape, um, products that are in a similar niche to you, you can do it instantly. I mean, that took— obviously I'm probably going to cut the video of when it was scraping, but that took maybe 10, 20 seconds to run. Actually, if we come over to Apify, you can see how quickly it did. So did it in 12 seconds, as you can see. So 12 seconds scraped all of that information, which is now completely yours to do whatever you like with. And something to mention as well, obviously I'm manually running all of these in the video and I'm putting in the information myself, but this can be done all programmatically. So you can just input any kind of data, you can pull data from a spreadsheet and put it in here, you could be talking to an AI agent and input some data and it can input that into Apify. You can do it in so many different ways. It is not the case that you actually have to like go into Apify put in the inputs that you want, copy the JSON over every single time. Once you have the basic JSON in there, you can switch out the variables, the value of the variables for whatever you want. So for example, with this Google Maps scraper, if I want to change it now and then, I could have a form that's just got different inputs in there. I could take the data from there and then I can switch out London for, you know, New York, US. Um, I could do the search string for instead of restaurants, it could be, I don't know, cafes, or it could be gyms, anything like that. It's not set in stone. It's just when you first set them up, you will have to put this entire JSON body in here. But then the actual values of each variable— so this is the variable, this here is the value— you can switch that out as much as you like.