3 ways to scrape any website for free in n8n

What's up guys, in this video I'm going to be showing you 3 different ways that you can scrape any website for completely free. We're going to be doing this using 3 different tools. One is natively inside of N8N, and then 2 are external tools that you will be using. I'm going to run through all the pros and cons of each and also when to use each of them. Just quickly, if you don't know who I am, my name is Rory Ridgers and I run an AI agency called Autonomously AI where we find and eliminate operational bottlenecks for business owners in 14 days to keep them working on the business rather than in the business. So if you want to get out of the day-to-day operations of your business and start focusing on the things that actually matter, I'll leave a link to my website in the description and you can book up a free discovery call. Okay, so first of all, you'll see I'm on my website here because I want to show you the difference between the types of websites that we're going to be scraping to help you better understand which tools we'll be using for different circumstances. So there's kind of two main things that you're going to be doing, uh, or two main types of websites that you're going to be scraping. First one is going to be static sites, and the first one is going to be dynamic sites. And now to give you an example of a static site. Basically, a static site is just something like a website or a landing page where it's not calling on external tools or getting information from anywhere else. It's usually just sort of text, images, and videos on a screen like this, for example. So if we look at my website, we go to view page source, you can see here this is the HTML of my page, which is basically just everything that makes up my page. So you can view it like this, or you can also go to inspect, and then you can go to elements, and then you can see here This is everything in my website. So my website, for the most part, it's pretty static. Like this video here, this is more of a dynamic element because if you used to scrape my website, you wouldn't be able to get the content from that video, which makes it more of a dynamic element. And then also the booking form down here is also a dynamic element because it's using Cal.com, which is an external source. So it's pulling data from Cal.com, which makes it slightly less static. But in general, landing pages and most sort of websites, at least for agencies and things like that, are going to be static websites where it's just HTML, which is the language that makes up majority of websites or static websites. It's just going to be raw HTML on there with maybe a few dynamic elements. And then you have things like, example, for LinkedIn. So obviously LinkedIn is still a website, but it's not static. It's not the same thing as sort of a landing page for an agency. It's dynamic. We call it dynamic, which basically means it's constantly pulling on data from other places. And there's also things in place like you've got to sign up to look at certain things. There's going to be firewalls, there's going to be anti-bot protection and things like that. So if we come to inspect here and then we go to elements, you can see there's still stuff in here because what we're actually looking at, um, like the things that puts all the data into this nice interface that we can see with these buttons and these images everywhere, that is still HTML. But the actual data is getting pulled from LinkedIn. We obviously have a big database of all of their users with all of this information. And LinkedIn, what I'm looking at right now on my screen, or what you're looking at right now on my screen, it's just the HTML displaying data that's been pulled from LinkedIn's database. Which means if you was just to try and scrape this page here in the same way that you try to scrape a static page, there'll be lots of information you wouldn't be able to get. For example, like information on all of these posts. These are things that when clicked on, LinkedIn will say, okay, he wants to view this post. It will call LinkedIn's backend, get the information for that post, and then send it back to my frontend. But of course it doesn't in an instant, but that's what's happening. In the backend. So all things like that, things like the people you may know, my follow account, all of this kind of stuff, this is stuff that's being pulled from an external database, um, and being displayed here in HTML, which means if you try to scrape it in the same way you did a static site, it wouldn't work. And also there's a thing like I'm logged in right now, as you can see, but if I wasn't logged in, I wouldn't be able to see this in the same way. So that's also if you try to scrape it as a static website with a static website, you don't have to do login, there's no authentication or anything like that. So that is the main differences. And dynamic websites majority of the time that you're having to worry about dynamic websites, it's probably going to be for things like social media, like LinkedIn, Instagram, TikTok, uh, those kind of things. If you're scraping that, those are all going to be dynamic. And if you're just scraping web pages, those are all going to be static. Okay, so we're in NAN now, and we're just going to run through the 3 different scraping methods, starting with the first one, of course moving on to that third one. And with the first one, like I said, this is going to be native inside of NAN. And then the other two are going to be external tools. And like I said, what I've done here is I've just laid out, of course, the name of it, then when to use it, pros of it, and the cons of it, so that after watching this video you'll know in which scenario to use which one. So the first one is going to be the HTTP node, and what we do inside of here is really, really simple. You literally just do a GET request and then you put in the URL that you want to scrape. So the first thing we're going to do with all of these is we're going to demo it out so you can see. So we're going to use my website here we're going to copy my URL, we're going to paste that in, hit execute step, and then this is what we get back. So you can see we instantly get back all of that data that was in the inspector. So like when I go to inspect this data here, we get all of that back in an instant. Um, and that is like completely free. This is built into N8N. You're never gonna have to pay for this. So that's literally all you do. The method is going to be GET, and then you put in the URL of what you're trying to scrape. So when do you actually use this? You'd use this if you need a quick setup, you're scraping a singular page, or you're scraping a simple static page. So pretty much exactly the scenario that I've just showed you. So my website here, it's just a simple static page for the most part at least, and I just wanted a quick way to scrape it. So I can literally just put down that HTTP node, put in my URL, and then it just gives me back the HTML. So just quick, easy. It was a simple page and it just did it instantly. So the pros of this is it's quick to set up. It's easy to set up, it's free forever, and it's reliable. And what I mean by all of these, it's quick to set up, it's literally just, you could, you don't even have to have this trigger node at the beginning, you could have just done that node, and all you do is set the method to GET, put in the URL. So really quick to set up, that also ties into easy to set up because like I said, you literally just do method GET, put in the URL, so it's easy to set up. And then free forever because it's a native feature in NAN, HTTP requests don't cost any money, it's just sort of like an internet thing. That's the easiest way to describe it. It's just like an internet thing. It doesn't cost any money. It's just something that's always there. So it's going to be free forever as opposed to these other platforms, which eventually you will have to start paying for. But yeah, HTTP Node will always be free, and also it's reliable. And what I mean by that is, for example, you can see the next one we're going to look at is Apify. If for whatever reason Apify servers went down or they had some maintenance or something like that, then what you're trying to do on there wouldn't work. Let's say you have an automation that runs every day through Apify Appify servers down one day and then it doesn't work. That's never going to happen with an HTTP request. It's always going to be there and you're always going to get the exact information back. And also with things like Appify, I mean, I'm jumping the gun a bit now, we're going to get onto Appify after. But with Appify, you're using like individual actors that are made by the community, which can change, they can go under maintenance individually. So yeah, with HTTP request, it's always going to give you the same output every single time when you give it the same input, and it's never going to go down. And then the cons of the HTTP request It's just, it lacks a lot of flexibility. There's a lot you can't do with it. It's great for just these simple, quick setups. There's a lot you can't do with it, which will become clear when we move into these other ones. A lot of pages are gonna block your request or not contain the information that you want. So if you're just doing an HTTP request, a lot of things are just gonna block your request. But this isn't a big enough reason to not use it because you can see like if we're just scraping my landing page, it works perfectly. There are gonna be certain things that are gonna have just simple firewalls and protections that will just block your request. And also it requires an LLM to decipher. So if you actually want to do anything with this information, this actually isn't the biggest chunk of HTML. Like with a lot of pages, it's going to be 10 to 100 times the size of this. It's going to be like a humongous output, and you just as a human aren't really going to be able to look through that and actually get the information you want. Let's say you're scraping a website and you want to find the name of the director because you're scraping the team page. Let's say something like that. It would take you forever to look through all of the HTML and actually find that name. So pretty much in order to actually do anything with this information, you need to run it through an LLM, which can be an upside, can be a downside, depends. But of course LLMs do cost a little bit of money. I mean, to actually process information, it's gonna be like a negligible amount, but if you're doing it over time and you're doing lots of it, I guess the cost can add up a little bit there. So now we're looking at Apify. And if you don't know what Apify is, basically I use it in pretty much all my videos now. So if you've watched me before, you should be familiar with it. Apify is essentially just a marketplace of scrapers made by the community. So you can scrape pretty much anything here using community-made scrapers. So you can see there's things like Google Maps, website content, TikTok, Instagram, tweets, Facebook posts, web scraper, lead finders, Facebook ad scraper, like literally everything. I think there's about 7,000 or 8,000 on here, and they're all made by the community. And you can just plug them into your NAN and you can start using them straight away. So what we're going to do for this is we're going to use a LinkedIn scraper and scrape my own LinkedIn profile. So if we just go to LinkedIn profile. And then I found this one here that looks pretty good. I can input a profile username or URL. These here, by the way, are all the different inputs for each individual scraper, so they're all going to have different inputs, um, because they're made by community, they just set the inputs. Uh, so you can see here, these are the inputs: the profile username or URL, page number, and there's usually also always some extra options and some run options as well. But for this, we only really need to worry about these first two here, or even just the first one. So what we can do to connect this up to NAN, if we come back over to NAN, now NAN natively has an Apify node, which it didn't before, but that's really, really nice. All you do is you come in here. This is the wrong credential for me, but all you do is you come in here, you set up a new credential by just clicking that pencil there, or by just going to create new credential. And then all you do is simply put in your API key, which you can get from Apify by just going to settings, and then your account's gonna be connected. Simple as that. And then the resource is gonna be actor. And then the operation is just going to be run an actor. And then the way these, uh, Apify nodes work is you first do a request to run an actor, and then it's going to keep running until the actor has finished, and then it will send you a response in here that just says, uh, you know, the actor's finished. And then you do another node after that to get the information from that run. So it's always going to be two nodes with this. So after that, you do actor source, which is going to be either recently used actors or Apify store actors. For this, we're going to go to Apify store actors. And then we're just going to find the name of this one. So we can literally just, uh, copy it like this, paste it into that search bar, and then this looks like the right one. Cool. So now it's connected up. All we need to do is input this JSON body, and the way we get that is simply we switch this from manual to JSON. And now in here you can see there's no— we haven't got any inputs in here, so this is why this is blank. What I usually like to do is put a real input in here, so I'm just going to grab my LinkedIn profile URL, and then paste that into there. And then when we go to JSON, you can see now there's actually some data in there. So, and this is what we're actually going to be using. So we can literally just copy that, come back over to N8M, and then we can paste that in there. And then if we execute this off now, this is going to run for a second, and then it will just give us an output that says succeeded, or it should say succeeded if it did actually succeed. Uh, let's have a look here. So switch this to schema. Yeah, you can see status succeeded. Cool. And then we just do get dataset items, make sure we're connected up to the same account, and then we just need the dataset ID. So we can just come up here, go to dataset, and then yeah, dataset ID. Cool. And if we hit execute step, this is going to run through. And now we've got 16 items here. So it scraped all of my last posts. I'm going to be honest, actually, I didn't even read that, that it said a post scraper, but it still gives you the idea of how these kind of things work. And you can't do this with like an HTTP node or something. Um, so it really doesn't matter. But yeah, they scraped my last 16 posts. So you can see it's given the date, the timestamp, the text from it, the URL, obviously my first name, my last name, my headline, my username, my profile URL, profile picture, and then some of the reactions or the stats to that post. And then like the type, which is image. You can see we've got all this data back and we've got 16 of them. So my last 16 posts, as you can see all of this data here. It's my last 16 posts. So really, really cool. And now getting onto the pros and cons and when to actually use this. So when you use this, I use this pretty much exclusively when I need to scrape dynamic websites. So things like Instagram or LinkedIn, or if we come back over to Apify, just to give you more of an idea of what I'm talking about, all of these kind of things, these are all dynamic. So like Google Maps, TikTok, Facebook posts, tweet scraper, Instagram, LinkedIn job scraper, Facebook ads library. I mean, I can go on and on, but these kind of things these are dynamic, right? These are like apps and actual platforms that we want to scrape rather than just sort of landing pages and things like that. And the reason we can do that with Apify Actors is because these actors work in a way where they don't literally just scrape the HTML from a page. It's better to think of them more like bots. It's kind of the easiest way to think of them. It's bots that can go in and kind of pretend to be human in a way, and they can bypass like the login and the firewalls and things like that, and they can look at the JavaScript on a site, and they can do all of these things that you can't do with just a simple HTTP node that's just going to return the HTML. So you would use Apify Actors when you're scraping anything dynamic like Instagram, LinkedIn, Google Maps, Facebook Ads Library, all of those kind of things I just showed you. So the pros of this— you might be able to see some conflict in here, but overall it is still easy to set up, right? Like, I walked you through that, and obviously I was talking while I was doing it, but it still did not take long at all to set up. You literally just set your credentials, then type in the name of the actor, put in the input JSON body, and then you just do another one after that gets the dataset items. That's it. So for what we actually did scraping my last 16 posts, that would take, I don't know, maybe half an hour to do that manually, or maybe that's a bit much, maybe it'll take 15 minutes to do that manually. It still only took about a minute to set it up, and now it can run forever and then it ends. So considering all that, it's still very easy to set up. It's also very versatile, so there's so many different things that you can do with this. As I showed you, I showed you a few examples of the type of things you can scrape, but there is literally thousands and thousands of different things you can scrape using Apify. So very, very versatile. It's also very customizable because you can kind of do like a lot of different things with the data. You can get different kinds of data, like just looking at LinkedIn for example. I can scrape LinkedIn posts, I can scrape competitors' LinkedIn posts, I can scrape competitors' profiles, I can see how many employees they've got, how many followers they've got. I can scrape company pages, personal pages, you see what I mean? Like there's a lot of flexibility and lot of different things you can do inside of, like, if we even— if we just take LinkedIn, like I just said, there's so many different things you can do with that inside of Apify. And then the cons of this: it will cost money after the free credit. So I think if you go on to Apify, I have a paid account, but I believe you get $5 for free. And to give you an idea of pricing, that one that I just showed you, uh, so the LinkedIn profile scraper, the LinkedIn post scraper, just then we got 16 results, and for 1,000 results it costs $5. So if you want to scrape 1,000 posts it will cost you $5. So they're not expensive. I like, I wouldn't hear that and kind of think like, oh, I'm going to break the bank with this because you're not, they're not expensive. Most of them are around that sort of like $5, maybe $10 per 1,000 results. Um, but yeah, it will cost you money ultimately. Like after a while it's not going to be free. You can look around and you can do a lot with like $5 you get for free every month. But if you're actually looking to build these in like production systems or for clients, you will have to end up paying for it. And then also it is slightly longer and more complex to set up. This is purely just comparing it to something like the HTTP, which takes like 10 seconds to set up. Just comparing it to that, it is slightly longer and slightly more complex. Sometimes there's going to be little things like the JSON body isn't quite right or something like that. I mean, to be fair, that is probably the only thing that might not be right. Um, but yeah, just compared to something like the HTTP node, it is slightly longer and slightly more complex to set up. And then we get on to the third tool, which is one that I really, really like and I use all the time with NAN. It's called Tavily, and essentially what Tavily is, is you can do quite a few different things with it, but it's sort of like a Google search but much better inside of NN. So it's kind of like Google search, but you can do a lot more with it. So Tavily is an external platform, and I'll show you Tavily quickly. So this is Tavily here. As you can see, it's pretty simple, um, and all you're really going to need inside of here is just your API key. So you can just grab your API key here, and you also want to make note of your free API usage. You get 1,000 credits for free, which is really, really generous. Um, but yeah, anyway, you can keep track of it here. But yeah, we're going to grab our API key, come back over to N8N, and then set up a new credential. Again, just go to create a new credential, put in the API key just like Apify, and that's all you have to do. You're connected. Then we have a few different things that we can do here. So we can extract, search, crawl, or do a custom API call. Search is essentially just going to be like your Google search. Extract is just going to be extracting data from a website. And then crawl, this is pretty cool. You can put in a URL and it'll give you back all of the URLs for that page. So for example, you could put in, um, like a URL for an agency's website and it'll give you back the base URL, and they have like /contact, /blog, /services, for example. So just like all the pages from their website, which is pretty cool. But in this, we're going to be doing extract. So all we do here, set the operation to URLs, and then we just go to add URL, and then we just put any URL in here. So I have found this blog here on just machine learning. I just typed up machine learning, it was one of the first ones that came up. So we're going to try and scrape this here. So we're just going to copy this URL, come back over to N8N, paste that into there. If we hit execute step, you can see pretty quickly that has now come back to us and this is what it looks like. So if we look at the table view, you can see this is quite a lot of data now compared to that HTTP, uh, that I showed you earlier. But the difference with this, as you can see, is not in HTML, which is really, really nice. So it just says raw content and it doesn't give it back to us in HTML. And this is actually really, really good because with the HTTP node, you need to either convert it to Markdown, then pass it to an LLM. But with this one, you could, you could actually just go through it on your own to find specific information. Like, you could run this through JavaScript to just find specific information. But if you give this to an LLM straight away, then it's going to be much, much easier, much easier for the LLM to digest. Whereas with the HTTP node, you would need to turn this into Markdown, which is what this is. So basically, it's just eliminating that step of turning into Markdown. And also with this, you can do things like include images, extract depth format, and then include favicon. So just to show you, we're going to do include images and extract depth. So let's go to advanced and then let's just run this through again. So these are just some additional options that you don't get with the HTTP node. So the way the images work is they just appear as links like this. So we can actually view them obviously inside of here, but it just means it includes those images. And with the extra depth turned on, it's just, um, a bit longer. As well. So it just gives us a bit more here. So very, very cool. The when to use it is going to be when you need to scrape a static site that contains a lot of information. So that blog, to be fair, that blog you probably could have got away with doing with an HTTP node, but there's going to be static sites with like way more information, and that is when it's going to be better to use the Extract. So the pros with this is that there's a native NAN integration, which there never used to be before, but now there is. So it's really quick and easy to set up, like the HTTP node. Really quick and easy to set up. You literally just put it down and then put in the URL. That's it. So kind of the exact same as HTTP node actually. Um, less likely to get blocked compared to the HTTP method. So like I said, with the HTTP method you can get blocked by a lot of static sites that just have like basic firewalls and things like that. But I found with the Tavli, they have ways to work around that, so you have much less chance of getting blocked on certain sites, which is a really nice touch. And also returns structured data. So like I said, it's not in HTML, It's already in Markdown, which just makes it way easier for your LLM to process all of it. And now the cons. The only real cons for this is that after you use your free credits, it'll start to cost money. But that's pretty much the only con. Kind of just like an upgraded version of the HTTP method, but it will after a while start to cost you money. But you get like a lot for free. Um, you get 1,000 credits for free. Those 2 requests that I just did, I think they only cost 1 credit each. So you can see kind of like how many you can do in there. Do about 1,000 requests, which is pretty generous. And now just to finish this off, I've just put a simple list here of the different categories and which of these 3 methods win each category, just to give you a little bit of a better idea of when to use each one. So the first one, speed to set up, in first place is HTTP, then it's Tavali, then it's Apify. I feel like that's fairly obvious from everything that we've been through. And then the actual speed to scrape things is going to be the same order. So HTTP does it the quickest, Tavily, to be fair, it's probably on par, but HTTP is probably always going to be quicker just because with Tavily, sending off to Tavily and sending it back to us, so there's external servers involved. With the HTTP method, there's not. And then Apify, you're going to be scraping more complex information, so it makes a lot of sense that it's going to take longer, but it still just does take longer. So that's why it's at the bottom of the list. And then the scraping complexity, Apify's at the top by a country mile, and then Tavily, and then HTTP. But of course, we all know Um, Apify is used for scraping more complex things. We would use Tavli or HTTP method to scrape simpler static things. So it's obvious which way this was going to go around. With Apify, you can scrape pretty much anything, any platform that you just wouldn't be able to with the other two methods. And then we have versatility. It's going to be the same order: Apify, Tavli, HTTP. Apify, there is just way more things you can scrape. Uh, like I said, there's thousands, I think about 7,000 or 8,000 on Apify, and it's community-driven, which means the community are just going to make more and more and more. as opposed to Tavli and HTTP where you're restricted to pretty much just static sites. And then the reliability, HTTP is always going to be at the top because you're not relying on external servers. Tavli is going to be second just because there's less things to go wrong. Um, you're still relying on external server, but it's never been like down or under maintenance when I've been trying to use it. And like I said, there's just less things that could go wrong. All you do is put in a URL. And then Appify, um, one, it's relying on external servers, so API could be down, but then also the specific scrapers that you're using can be under maintenance or can be down, which means there's two ways that it could be down. And also it's slightly easier to input things wrong because you're dealing with JSON, and that JSON body that I showed was of course really, really simple, but sometimes they can be a little bit more complex than that. So that's why it's in that order.