Build an Ai Agent that can scrape anything (no code)

We all know how time-consuming it can be to do market research on your competitors or find information out about new leads. So in this video, we're going to be making an AI agent that can do all of it for you. It's going to be able to scrape a company's website content, Instagram page, LinkedIn page, and their reviews. If you haven't seen me before, my name is Rory Ridgers and I'm an AI enthusiast, and I'm passionate about helping business owners integrate AI into their business. So if that's something you might be interested in, then I've left a link to my website in the description. We can book in a call and talk about some ways we can integrate AI into your business. Also, I just want to mention that all the templates I use in this video will be available for completely free in the description. So if you want to go ahead and grab them, then you can just find the link in the description. But let's not waste any more time. Let's get into the video. Okay, so in this video, we're going to be using the website scraper tool, Instagram scraper tool, LinkedIn scraper tool, Trustpilot scraper tool, and then the lead research tool. All of these tools will have to be custom made on Relevance AI. You can't find the templates for them on Relevance AI. Don't worry, all of them are relatively simple. So First of all, we're going to show you inside the Lead Researcher tool, and this tool is going to be used when you don't necessarily have a company in mind that you want to scrape. You just want to find some companies in your industry. So you can say something like, find me construction companies in New York, so sort of companies in your industry, and it will return back to a list of these companies, and then you can pick from there which ones you want to scrape. So the first input we have here is lead input, so the industry type of businesses that a user considers their lead. So as I just said, something like construction companies in New York, that's what we're going there. And then we have a simple Google search here, which is literally just the input here. So that would go into a Google search. So if it was construction companies in New York, then we just do a Google search for exactly that. And then we have an LLM step here that's basically just going to go through the results of this Google search and then output the names of the companies that we're actually looking for. So really, really simple tool. I'm just going to give you an example here. I'm going to say construction companies in New York. Okay, so construction companies in New York, we'll hit run tool, and then all that's going to happen, we're going to do a Google search, find some construction companies in New York, and the LLM step is just going to sort through that and then give us an output here. You can see all of these construction companies in New York. So perfect, really, really simple. You don't have to use that tool in this AI agent if you already have a company in mind that you want to scrape. But this is just if you don't really have any ideas for leads or competitors that you want to search for, you can use that tool and it will give you some. So now we're going to get on to the actual scraping tools. I'm going to start off with by far the most simple one, it's going to be Website Scraper. So if you have a look in here, the user input for all of these tools is going to be company name. So the only thing that needs to be inputted is the name of the company. So the first step, again, we have a Google search and we're just going to search for this company name. And what in theory should return is the company's website URL. So which is using LLM step to basically look through this Google search data and then find the URL that is most likely the website URL for the company that we're looking for. So this is the prompt: I want you to look through the data of a Google search and find the URL. This is most likely the URL for the company's website that we are trying to find. This is the company's website we're looking for, and then the company name input. So this one, and then this is the Google data, so the output from this step, and then output only the URL, no summary or explanation. And then we choose this extract website content block, and we put in the output from this LLM step, which should just be the company's website URL, put it into this input here, and then this is just going to return back all the content from the website. And then we've just got a simple LLM step here that's going to search through this extracted data and just format it nicely so that we're able to actually read and sort of digest it. So we're just going to do a quick test here. We're going to say Gymshark. Just because it's an easy example. So putting Gymshark into the company name input, hit run tool. So you see we've done the Google search here, perfect. And then the LLM step has outputted their website URL and the website content here. And then this LLM step, we'll just give this a second because it has to go through all the data on the website. Okay, there we go, it's done it. So website report for Gymshark, a summary, the key points, uh, so user engagement and promotions, shipping and returns, product range, community and brand identity, mobile app promotion, upcoming events. So just lots of detail, and then also a summary on the website. So that's what the output looks like for that one. So those two tools, really, really simple tools. We're going to get on to some slightly more complicated tools, but nothing too crazy at all. So we're going to start with this LinkedIn scraper tool. We hit edit. Again, the input is just going to be company name. I've done it the same for all of them. So yeah, you can see company name. The first step here is going to be a Google search. So we're going to search for the company name's LinkedIn. So this would look like Gymshark's LinkedIn. We're going to do a Google search for that. Then we have an LLM step here that's similar to the tool before. It's just going to look through the results of this Google search and then find the URL that we're looking for. In this case, it's going to be this company's LinkedIn URL. So it's going to look through that, output just the URL and nothing else. And then we use this get LinkedIn profile slash company and post. So we're just going to put in the answer from this LLM step here, which should be the LinkedIn URL. We're going to put that into this URL input here. The method is going to be get company profile, so we're going to get all the information from their profile. And we use an LLM step here to basically just take the output of this step and summarize it. So you can see, I want you to search through the scraped data on a company's LinkedIn and summarize the relevant data points, then create a full summary on the company's LinkedIn that can be presented to a user. And then here is the LinkedIn data. So really simple. Then we do the pretty much the same thing again, but this time we do the get company posts. So we want to get their profile and their posts. And then we just have another LLM step that's pretty much just doing the exact same thing. It's just going through that data and summarizing it so it can sort of be presented to a user nicely. And then I've said if the LinkedIn data is no profile post, that means the company doesn't have any posts. So just output that no post can be found because not all company LinkedIn or just LinkedIn pages in general are going to have posts, but they all have profile data. So sometimes this tool output no profile posts, and in that case the LLM just puts there aren't any posts, and then the AI agent will just ignore the post side of it. So yeah, that's that tool. You can see the outputs are LinkedIn profile data answer and LLM answer. I'm not sure why it's named that, but you can see the source is the LinkedIn post data. So this is going to be the profile data, and then the post data. So we're just going to give this tool a quick run through again. We're going to say Gymshark just to keep it the same as the other one. So run tool, and then we can see each of these steps working. So yeah, this is kind of similar to the step before. It's done a Google search and then it's found the LinkedIn URL, and then it's found the— this is the profile data. The LLM step has summarized it. And then the posts, they should have some posts in theory, Gymshark. So it's returned something. We're just going to wait for the LLM step to summarize it, and then we'll see what it's come out with. Okay, so yeah, you can see this is the profile data. So the name, the profile, the website, the year founded, headquarters, all of that information. And then this is their post data. So you can see these are some of their recent posts, and we have a link to view the post as well. It gives us the likes, the comments, the text, the date, everything like that. So Really cool. Now we're going to look at one of the other tools. We're going to have a look at the Trustpilot scraper tool. Okay, so for the Instagram scraper tool, again, we have company name as the only input, and then what we do is we do a Google search. So the company's name's Instagram, so this will look like Gymshark's Instagram. And then again, same drill, the LLM step is just going to search through that Google search results data and find the most likely URL for the company's Instagram page that we're looking for. But For this, what we're doing is we're using Apify, which is basically— if you haven't heard of Apify before, it's sort of just a marketplace where people can put their APIs on and other people can use them. What it's primarily used for is sort of scraping tools. So that's what we're doing in this case. We're using this Instagram profile scraper tool, and for this tool to work, we need the Instagram name, not the Instagram URL. So what we do is we get the URL, and then I want the LLM step to extract the name. So in this case, if we're looking at Gymshark, this would be the Instagram URL, but then we want just the name. And the reason we don't just use the company name here is because sometimes companies' names Instagram will be slightly different from their company name. So they might have like an underscore in there or a dash or something like that. So we just search their Instagram and then extract the proper name from it, from the URL here. And then what we do is we use an API step, and this is how we're going to talk to this actor on Apify. The automations are called actors on Apify, so we need to talk to this actor to be able to scrape the Instagram profile data. So I'm going to walk you through how to connect these two. It's really, really simple. On Relevance AI, there is an Apify block. I can show you, uh, um, So yeah, this Apify block here is already a thing on Relevance AI, but it just doesn't work for some reason. I'm not too sure why. So I've done it this way. Maybe they'll fix it, maybe it's just a bug at the time of recording this, but you can do it this way, it doesn't matter. What we're going to do is the method is going to be POST because we want to send data to this actor. And then you can see here we have the URL endpoint, the headers, the body, and then the JSON object. So I'm going to talk you through what all of that means. We come back over to Apify. You can see here when we're in our actor, we have this API bit here, and we can go to API endpoints. And then these are all the different endpoints and what they do. So run actors, uh, get actors, get list of actor versions, update an actor, delete an actor, get list of builds, build an actor, get last run. What we want to do is we want to use the run actor and get dataset items URL. So we literally just copy that, come over to Relevance AI, and put that in the URL section. And then we need to pass the data over as JSON. You can see here the manual— this would be the manual input. So if we're going to use it through Apify, which we're not, we want to use it on Relevance AI. So we have JSON here. So that means the data needs to be passed through as JSON. So what we do go to headers, content-type application/json, and then the body, we just select JSON object. And then what we can do is we can come back over to the actor, we get this JSON schema here, literally just copy that and then paste it into Relevance AI. And we just swap out this placeholder name here. This would be the name of the Instagram profile that you're looking to scrape. We just add in our variable, which is LLM answer. Probably could have named that a bit better. But it's just the output from this LLM step here, which is going to be the name of the company's Instagram. So we put that variable in there, and then that will all get sent off to the actor on Apify, and then we wait for the results back. So it does take a minute, but we trigger the actor on Apify, and then we wait for the results, get those results, and then we use an LLM step to basically just go through all of it and then just sort of format it so it looks nice, pretty much. So we're just going to test this out. This tool does take a little bit of time to run. It can— I found that it could take sort of up to a minute to run. So use the same input, Gymshark, and it's just because it has to send off this data to the actor on Apify and then wait for it to get the results, and then the LLM step after has to go through all of those results and then sort of format it so it looks nice. So First two steps have run. You can see this is what the Google step outputted, and then the LLM step outputted just Gymshark, which is perfect. And then we have this API step which is working now. So, okay, so it has actually returned data, and I'm not going to click View All because for some reason, I don't know what it is, but I think the data is right at the bottom of this table that it returns. It's really weird. So if you click View All you're scrolling for ages to try and find the data that you're looking for. But the data is actually there because if we go to this LLM step here, we can add in a variable. And if we go down to this API, you can see, yeah, the first one, API response body, is what we're looking for. You see we got the username, URL, full name, the bio. So has actually returned data, but it just doesn't look like it's returned data. So we've passed it over to the LLM step here, and this is what is returned. So this is what the LLM step has outputted. I'm not sure what's done it in this format, it's kind of weird, but it doesn't really matter because it's just going to be passed to the AI agent. So when it's actually shown to us from the AI agent view, it'll look normal. So yeah, you can see this is what we've got here: the full name, the bio, the followers, the following, post count, the profile URL, external URL, which is going to be the link to their website, and then verified, and then related profiles. So really cool, that actually didn't take too long, only took about 30 seconds, which really isn't too bad. So yeah, that's all the tools there. We can hit save. And now we need to actually put all these tools into an AI agent. So I'm going to show you how to do that. Obviously I've already gone ahead and done it, the scraper agent here you can see. But if you was going to create a new agent all you do is just click New Agent, and then you need to put in all of this data here. So I'm going to show you what my one looks like. Scraper Agent, we hit Edit. So we have an agent name and an agent description. And now these things don't really matter at the minute. When you come to use multi-agent systems, these things matter. But right now, I wouldn't worry too much about this. Just put in whatever you like. I've said Scraper Agent, and then scrapes a range of different social medias and apps. Doesn't really matter, it's more just for you. And then we come to Tools, and this is where we're going to add in our tools. So we have the tools here: Trustpilot Scraper, Instagram Scraper, LinkedIn Scraper, Website Scraper, and Lead Research Tool. If you want to add new tools, you just click Add Tool, and then all your tools come up here. You hit Add, and then you'll need to change this. This will be set to Approval Required by default. You need to set it to Auto Run. So now we come over to the core instructions, which is the main part of the AI agent, because we're going to tell it how to use these different tools. So we've got the role: you are able to help users scrape a company's Trustpilot reviews, website content, Instagram, and LinkedIn page. Be helpful and friendly. So just sort of simple what it does and how it should act. And then task, I said you have 5 tools: website scraper tool, Instagram scraper, LinkedIn scraper, Trustpilot scraper, and lead researcher tool. And if you want to reference these tools directly in the core instructions, all you do set a forward slash, and then the names of the tools come up here. Just helps the agents know exactly which tool you're talking about. So I've said, I want you to first use the Lead Researcher tool to find companies in the given criteria from the user. The user will say something like, find me construction companies in New York, and this is when you will use that tool. The user may, however, already have the name of a company they want to use for the scraping. Here is a breakdown of the 5 tools. And then here I've literally just given it a quick breakdown of the 5 tools and sort of the structure I follow when I'm explaining these tools is I start with what the tool does. So this tool allows users to find leads in their industry, and then I give it the user input. For the lead research tool, the only input is potential leads. And then I sort of give it some context on how this tool works. So the tool works by performing a Google search from the user's input, then outputting only the businesses' names using LLM step. I sort of tell it how it should take the data from the user's input and then input this into the tool. So in the scraper tools, the description is almost identical for all of these. I've just said this tool allows users to scrape a company's website, and then the only user input is company name. I want you to put the name of the company that user wants to scrape in this input. I've pretty much said the exact same for each of these tools apart from the description of what this tool does at the start. So this tool allows you to scrape a company's Instagram page. The only user input is company name, and I want you to put the name of the company that the user wants to scrape in this input. So I pretty much just said the same thing for each of these scraper tools. This prompt, by the way, will be available for completely free in the description, along with all the tools I've used in this video and the whole AI agent. So if you wanna pause the video and look through the prompts, you can, but if you just wanna go grab it in the description, then feel free to do that. So notes, I've said when you first use the lead research tool and return a list of names, if a user says they want to scrape one of these companies, then use that company name in the company name input for the respective tool. So if when the lead research tool is used, a list of companies is given to the user, then if they say, Uh, I want to scrape the reviews of that second one, then use the name of that second one in the Trustpilot scraper tool. So pretty simple, not too complicated stuff here. We can hit save changes, and now we can actually test out this agent and test out all of these tools. Okay, so for this example, we can say something like, find me gym wear clothing brands, just as I've been using Gymshark as an example. I thought we can get it to find some gym wear clothing brands, so it's all going to be similar things. So yeah, you can see, use the lead research tool, and then this is some of the companies it's returned to us. All of these— I'm going to use Sports Direct because there's a Sports Direct near my house. So we can say something like, find me reviews for the last one. So find me reviews for the last one, the last one being Sports Direct. So let's see what it comes out with here. So one step performing in the background. So it's using the Trustpilot Scraper tool, so it knows which tool to use, which is good. And now let's see if it actually gives us back the data that we want. Okay, so there we go, it's used the Trustpilot Scraper tool and then it's returned to us some reviews. So Martin, the review is fantastic prices, they date of experience, and then name, review date, name, review date. Perfect. And it should have returned 10 for us, which it looks like it has. So that's perfect. Okay, so now we can say something like, find me some info on their LinkedIn. So it says, find me something from their LinkedIn, and then use the LinkedIn scraper tool. And this is what's come back to us with. We've got the name, when they were founded, headquarters, website, their LinkedIn URL, employee count, all of this stuff, the industries, their logo here, which is pretty cool. And some of their recent posts, and then the link to their post with the engagements, the text. So very, very cool. So now we can just say to it, find me some info on their Instagram, and it's going to have to use the Apify Actor in this tool. So should take a minute, but we're going to see what it comes out with. Okay, so here we go. It's used the Instagram scraper tool, and then here's some information about Sports Direct Instagram profile. So username, their full name, their bio, their external URL, which would be to their website, followers, following, post count, and then some of the related profiles. So yeah, worked perfectly. It's managed to scrape all these different websites to find information on these companies, which is really, really cool. So you should now be able to create an agent that can virtually scrape anything on the internet. Of course, in this video I only used a few examples, but using Apify as well. You can virtually scrape anything you want. You can scrape Google Maps, you can scrape YouTube, X, any type of social media. I just want to give you an example of some of these different places on the internet that you can scrape for these companies. So if you were looking to work with me so we can integrate AI into your business, and as I said at the start of the video, I've left the link to my website in description. We can book in a free call, talk about some ways to integrate AI into your business. And as well, as I mentioned at the beginning of the video, All the tools and the templates I used in this video are going to be available for completely free in the description. So if you want to grab them, then feel free. And to anyone watching, if you've got some suggestions for videos that you'd like to see in the future, then just drop them in the comments of this video and I'll try my best to get around to doing them.