This AI Agent Can Scrape and Screenshot the Web with No Code (n8n tutorial)

So this one AI agent right here can search the web for anything using just one tool and one API call. It can get as many search results as you want, and it can also scrape all of the content from that search result, and it can take a screenshot of the whole page for you. And the cool thing is we can talk to it using completely natural language, and it will take care of the rest. So let's do a quick demo talking to the agent, and then we're just gonna jump in and I'm gonna show you guys how to set this up. And as always, no coding background is necessary. Okay, so I'm gonna ask this agent to help me find 3 articles on reddit.com where the following words are all in the title, which should be AI, automation, and help. So it's using its tool right now, Firecrawl Search, to make that request. It's saying, okay, we're looking at Reddit. We wanna make sure it has these words in the title, and it's gonna get markdown format of the results as well as screenshots. Okay, so we just got those results. It said, here are 3 articles from Reddit, and here are the words that are in the titles. We have help with automation AI, and then we have the link right here to the actual source. So let's open that up. You can see it links to an actual Reddit post in the NNN Reddit. And what we can do is we can look at the screenshot to make sure it matches what we got in the actual URL. And there it is. We can zoom in. This is the full Reddit thread that we could go ahead and read through in screenshot format. All right, so let's take a look at the next one. Seeking help for an AI automation, which has all of the words that we were looking for in the title. And we have the URL once again, but even better, let's just open up the actual screenshot and take a look at this page. Seeking help for an AI automation, and then we can read the rest of the thread. Cool. And the last one, of course, powerful AI tools to help you in your daily life. As you can see, they're all from Reddit. So the agent was able to specify just reddit.com. All of them have those 3 keywords in the title, and they all have our screenshots for us. And because the agent had to make this answer a little more concise, if we click into the actual search, we can see that for each of these results, we do indeed get a full markdown scrape of the actual results if we wanted to do something with that, but the agent just doesn't want to return all of that for us. And just to look at the actual body request we're sending over, in this case, it's super, super simple. It's only 3 things we're really sending over. And for the search query over here, you can see it's sending over these 3 words must all be in the title. So AI, automation, and help. And then it's saying it has to be from the site reddit.com. And then we're saying we only wanted 3 results. So that's super cool. I hope you guys can see how simple this is going to be to set up. This may look a little bit intimidating, but don't worry, it's not too bad at all. We're going to walk through it all. And if you want to download this template real quick to follow along with the rest of the video, you can do so by joining my free school community and downloading it. So the link for this community is in the description. Once you join, you will have to search for the title of the video, or you can click on YouTube Resources. And when you click on that, just find the post associated with this video, and there'll be a JSON file right here for you to download, which when you import it into N8N it will open up this exact template. All right, so I know that was a quick demo, but I really just wanna hop in and show you guys how this stuff works. So let's get into the video. So what we're doing today is we're using Firecrawl's new search endpoint, which is super powerful. It basically lets you search the web for anything. You have full customization over what to actually search for, and then you can go ahead and scrape the content from all of those search results in any form you want as well. So it's super powerful. We're gonna take a look. Here's Firecrawl. You can get there using the link in the description. As you can see, turn websites into LLM-ready data. And it's also open source, but you could also come here and start for free with 500 credits, and you can use code HERC10 for 10% off for a full year. So super cool, let's get started. I'm gonna go to the docs, and we're gonna go to the search endpoint, which says search the web and get full content from results. So it says Firecrawl search API allows you to perform web searches and optionally scrape the results all in one API call. So I'm gonna go to the API documentation for this search endpoint, and we're gonna take a quick, quick look at what this actually is doing and how this works. And then we're just going to dive into some NNN examples because that's the best way to really learn. So first thing we want to look at is just that we have query operators. This basically just means we're able to filter our searches better. So if we put anything in there as a string, it's just going to search a non-fuzzy match, which means if we type in fire crawl, it's going to only search for exactly fire crawl. It's not going to auto-fix typos. We're able to use this little dash in order to exclude things. So if we want to search for something and we wanna exclude the site Firecrawl, or we wanna exclude the word bad from the search result, we can exclude them by doing that. We can specify certain sites that we want, we can specify words in the URLs, words in the titles, and things that are related to certain domains. So I'm aware that this may look a little confusing. Trust me, it's not too bad at all. I also have all of these search query operators right here in the template for you to look at, and the body parameters, so everything's in one place. As well as I did a full video on APIs and understanding that documentation if you wanna check that out, right here. But real quick, let's go see what the body parameters are telling us. Like I said, this is basically just a full menu of different settings and filters we're allowed to use. So obviously we have a search query, which has to be there every time. We can then tell Firecrawl how many results we want back. We can limit that to 5. We can go all the way up to 100. We're able to do a time-based search parameter with natural language. So I could say, I just want stuff from May or April of 2025. We have language, we have country, we have location, we have timeout. We can ignore invalid URLs. And then here's the cool part with scrape options, where we can basically say how we want to get the results from each of those sites back. So it can be Markdown, HTML, raw HTML, links, screenshot, screenshot of the full page, or JSON. So now that we've had a brief introduction to this documentation, what this search endpoint can do, let's take a quick look at a few examples. So here's the first one where we're doing a search for within a specific website. So in here, you can see all I did was in the body, I'm only sending over 2 things, so we're keeping it really, really simple. We're searching for the word Nate Hurk, and we're specifying that the site has to be www.geekygadgets.com, and we only want 5 results. So if I go ahead and hit Execute Step, it's gonna make that search, and we can see that we have 5 different results, and all of these have URLs, they all show the title, and then they all show a quick, brief description. And if I was to open up this URL right here, we can see that it is indeed Geeky Gadgets, and it is about, natehurk. And if you'll notice, all of these 5 results come back from Geeky Gadgets. Alright, cool, so that's a specific website. Let's look at excluding certain terms. So once again, we're keeping it really simple. We're just saying a query and a limit. We're doing natehurk, we're excluding anything that has the word school in the URL, and then for limit, let's just do 6 so we can show that it's different. I'll hit execute step, and then we'll see what we get back over here really, really quick. As you can see, we have 6 total results, and we don't have any of these URLs coming from school, which is nice. Because if I was to get rid of this exclusion, and we're just searching for Nate Hurk, what we'll see is that we probably will get something from school. As you can see right here, the second hit was already from school.com. So because we used that exclusion operator, we don't get that. All right, now let's look at word appearing in a URL. So I'm gonna come back into here. Once again, we're doing the same query and limit. We're doing Nate Hurk, and now we're only searching for things where the URL has the word school. So hit execute step, this thing runs super quick, it's awesome. And now all 5 of these results all have the word school in the actual URL. And then of course, you can do the exact same thing with the title. So instead of URL, you could also do in title, you could have multiple words appearing in a title, like we saw in that quick demo earlier with the agent. And finally, the pro tip, where you can basically stack a ton of these in the same query. So it's really, really cool. So in this one, what we're doing is we're searching for Nate Hurk. We're saying the site has to be youtube.com. We're excluding anything that are, you know, Shorts. And then we're also saying in the title, it has to have the word automation and we're looking for 5. So I'm gonna hit execute step. We're gonna see how quick that comes back. All of these sites are coming from youtube.com. If we take a closer look, we'll see that all of the titles have the word automation. So automation, automation, automation. Automation and automation. And also the reason why some of these aren't even my videos is because most of my titles don't actually have the word automation in it. So you can see that that search is being very, very specific. And finally, you'll notice that there are no Shorts in here. So that's how you can stack different things, and that's how this stuff works. It's really about being able to understand the different operators, understanding the different body parameters, and being able to, you know, customize them how you want, tweak different things. But let's set up an actual HTTP request together and let's look at how we can actually return all of the results in different formats. Okay, so all you'd have to do is you'd come to this button right here, Firecrawl search endpoint, and click on that. And then what we need to do is we just want to copy this curl command right here. So I'm going to hit copy. I'm going to come back into our N8N workflow and I'm going to pull up an HTTP request. Then all I have to do is click on import curl, paste it in there, and hit import. And now we are basically configured. You can see that what we need to do is grab an API token from Firecrawl. So once again, go to Firecrawl and use code HERC10 to get 10% off for a year. And then once you're there, this is what it'll look like. We'll go back over to Firecrawl and you want to go to your dashboard, and this is where we can get our API key. So you can see I have my API key right here. I'm going to go ahead and just copy my API key. And what we're going to do is you could put it in right here after bearer space and then paste in your API key. But what I've done for all these different requests, and what I think that you should do, is save your Firecrawl API key to n8n. So the way we do that is we'll turn off headers right here. We will go to authentication and choose generic. We will choose header. And then as you can see, I have a ton of different headers saved, and all I would do is I'd choose my Firecrawl one. But if you don't have one already, you will click on create new credential. And so when you click on create new credential, I would recommend you rename it up here to Firecrawl. For the name, you're gonna do Authorization with a capital A. For the value, you're gonna type in capital B, Bearer, space, API key, just like it showed up in that curl command. And also, if we go back to the documentation, you can see right here authorization is header, and it has to come in the form of Bearer space token. And then once you save that, you're good to go, and you'll never have to do that again. So we have that all set up, and now we can see is we have like our full list of body parameters where we have our query, our limit, our time-based result. We have language, country, location, timeout, invalid URLs, and then a scrape options. So let's play with these different settings to see how we can tweak our search results. First of all, what I'm going to do is in the query, let's say we're just going to search for OpenAI. So we'll start there. For limit, let's say we just want to get 2 results back. For time-based search, let's just say we want to look from May of 2025. Language, English. Country, US. Location, I'm going to come in here and just put any. Timeout, we'll leave this. Ignore invalid URLs, we'll leave it as false. And then for scrape options, let's go take a look at the options that we have. So if we come right here, we can see that this controls the scraping behavior for each result, letting you specify which formats to return. We have a couple different options, and you can also include multiple. So we can do Markdown, we can do HTML, we can do screenshot. Super cool. You can also see in the example we have to put a new field below it called formats. So actually what I'm going to do is I'm just going to copy this right here because it makes it super easy. So I'm copying that. I'm going to open up our request and I'm just going to basically replace the scrape options right there. And we'll just tab this over so it fits. Um, hopefully everything is fine here. I think that we should be good to go, but if we get an error, we'll just come in here and fix it. And this is basically saying we want to return markdown, links, and screenshot. Let's just do markdown and screenshot for now. And we'll try to send that over and we'll see if this errors or not. And by the way, if you ever need help formatting these requests, go over to ChatGPT and give it a screenshot of the documentation, give it a screenshot of what you're trying to do, and it will help you out. ChatGPT is really good with JSON, or Claude, any of the AI models. Okay, so it looks like though we only got one result back. So because of our parameters, maybe it only was able to find one result. We have right here openai.com, we have latest news, So what's cool is you can see right here we got our markdown, which is cool. If I switch to table view, we can see we got basically the entire article in markdown format. So that's awesome. And then right below that, we got our screenshot. So let me click open to the screenshot and we can see what we got. So this is a screenshot of the, basically the beginning page that it just went ahead and scraped. Now you can see I can't scroll down or I can't change that at all. So what we could do is if we change our parameter right here from screenshot and we change that to screenshot at full page, we should be able to get the entire page back. So let me go ahead and try this again. So I put in screenshot at full page. We're going to try that one more time and we'll see if we have a full screenshot. That's going to be super cool. Okay, there we go. We just got it back. Let me scroll down to our screenshot right here and open this bad boy up. Boom. So we have a screenshot of the entire page. We can go ahead and zoom in and we could scroll through the entire page that it just went ahead and scraped. Nice. So hopefully that shows you guys how you can get stuff from all these results. Super cool. But let's take an example of if we have like 5 things. So I'm going to change the limit to 5. I'm going to change the search query to— we're just going to do Nate Hurk again. And, um, yeah, that's perfect. So we're going to run this Firecrawl search endpoint. We're hopefully going to get 5 results, and each result is going to have markdown of the actual page that it scraped as well as screenshot. So here's the first one. This is from school, co-founder of TruRise AI, and we can see right here we have a screenshot. So let's pull this one up. Okay, cool. So this just basically took a screenshot of my school profile, which is awesome. I'm switching over to schema view just so we can see this stuff easier. So that was the first result. The second result is going to be the Learn with Microsoft page. So let me open up this image. There you go. This is my Learn with Microsoft page, and it is the screenshot of the entire page. Sweet. So I think you guys get the point. Let's move on real quick to how we could actually do this as an agent tool. And let me show you guys what really is important when you're doing this with an agent. So because we have different search query operators, and that's where a lot of the power comes in, what we had to do was make sure that the agent itself is aware of these different parameters and how to send them over to its tool. Because right now in the tool, all we're doing is we're saying, give me a query and give me a limit. So it's very, very simple. But in order for this agent to actually understand how to send over queries that work, we had to tell it about the operators. So if I open this up, we started off by saying, you're a Firecrawl search query generator agent. You have access to a tool called Firecrawl search. Your job is to convert natural language instructions into a properly formatted Firecrawl query using supported operators. You're going to send that query along with a limit parameter to the Firecrawl search tool. So I'm not going to read off this entire prompt because you guys can access it for for free if you join my FreeSchool community and download the template. But basically what I said was, here's how you construct your queries, um, you know, here's your exact phrase. You can use exclusions, you can use sites, URLs, all in title, stuff like that. I gave it a few instructions with calling that tool and then just a few examples so it knows how to output the stuff and how it sends stuff over to the tool. And so if you guys noticed right here, we didn't actually have it return any information. Besides like a quick description. So what I'm going to do is I'm just going to paste in the scrape options. So I'm telling it we want markdown and a screenshot of the full page once again. And now let's try out an example. All right, so I'm asking it to find me articles on Forbes that include the word artificial intelligence in the title but exclude the word agent. So we'll basically let it think about it. Okay, so something happened with the JSON parameter. We'll take a look at that in a sec. It's going to try again. So it tried twice, didn't work. Tried 3 times, it didn't work. Tried 4 times and didn't work. So let's just go ahead and stop this execution, and we'll take a look to see what happened. Okay, so the error that we got was that the JSON wasn't proper JSON. So if I come into here, we have to figure out what went wrong. And so what happened is that I added a new option, but I didn't put a comma after this guy, so it's not going to work. So let me just go ahead and try that exact thing again. I'm going to resend that, and hopefully it's going to think about it, it's going to create that correct search term, and then we're not going to error. Looks like it is going to work because it's been spinning for a bit longer now, and I'll check in when it's done. All right, there we go. We just got— here are articles from Forbes that have the word artificial intelligence in the title but exclude the word agent. So this first one is literally just called artificial intelligence. Let's just open up this URL real quick to see if it's a real Forbes article. Okay, so there are ads absolutely everywhere, but you can see this is a Forbes article called artificial intelligence. And what we can also do is we can open up this screenshot to see that it captured a screenshot of the entire article. We can zoom in, we can scroll through the whole thing, and that's pretty awesome. And then of course it did the same with the next 5. So we've got Artificial Intelligence 101, we have the Pros and Cons of Artificial Intelligence. Each of these have a URL, a description, and a screenshot, and none of them have the word agent in the title. All right guys, so that's gonna do it for this one. I thought that this search endpoint is super cool, a great release from Firecrawl. And don't forget, you can get 10% off for a year using code HERC10. If you enjoyed this style of video and you want a more hands-on approach with learning NNN, then definitely check out my paid community. The link for that will be also in the description. We got a great community of members that are building stuff with NNN every single day. They're sharing their wins, sharing their challenges, stuff like that. It's great to be in a community of like-minded individuals. We've also got a classroom section with 2 full courses at the moment. We're expanding this library as we speak. We've got Agent Zero, which is like beginners foundations, and then we've got 10 Hours to 10 Seconds where you learn how to identify, design, and build time-saving automations. So I'd love to see you guys in these live calls, but that's gonna do it for today's video. If you enjoyed it, you learned something new, please give it a like. Definitely helps me out a ton. And as always, I appreciate you guys making it to the end of the video. I'll see you on the next one. Thanks everyone.