How to Build a Google Scraping AI Agent with n8n (Step By Step Tutorial)

So today I'm going to be walking through step by step how to build this Google scraping AI agent in N8N. In this specific video, I'll be using the agent to scrape Google for LinkedIn profile URLs. So let's not waste any time. By the end of this video, you guys all have a great idea of how to get this up and running and how to give your agent access to different tools like the ability to scrape Google. So let's hop into a quick demo real quick so we can see how the agent works before we actually see how to build this thing. So here's the agent. It's super simple. We've only got the these 3 tools here. And if this doesn't make sense yet, by the end of this video it all will, so don't worry about that. But what we're going to do is we're going to chat with this agent and we're going to ask it to get CEOs in real estate in Chicago. And so we're asking it here and it's going to go through these tools. And this is the worksheet that we're actually having the agent populate these profiles into. So as you can see, we just got 10 of them. And if we click here, we go up and look at Jeff, we will see that he is indeed a CEO of real estate in Chicago. So we know that this tool is working correctly, and that is basically how this thing works. And you'll be able to ask for, you know, describe a type of person that you're looking for, and it will go search Google and scrape Google to get those profiles into a sheet like this. Now that we've seen a demo of how we chat with the agent and how it actually works, let's move into what we need in order to build this thing. So first, N8N, that is the website that we're building the workflows on for the tool and for the agent. Then we've got OpenAI API. If you don't have an OpenAI account, you'll need to make one, but super quick, and you'll just need to be able to go in and get your API key there. And then finally, Google Sheets. Um, everyone sort of knows what Google Sheets is, but that's, like I said, the database where we're storing the URLs into. So finally, we're building two workflows today. The first one is going to be the tool that scrapes Google for the URLs. And then the second one is the actual agent that we can interact with. So that's all for the prep stage. Let's hop back into N8N and let's start a new workflow and start building this thing. Okay, so this is the first workflow that we're gonna be building. This is going to be the one that grabs the LinkedIn URLs from Google. So the first step here is going to be a when called by another workflow trigger. This is just saying that this workflow will execute when another workflow calls it, which in this case, The AI agent is going to be thinking about how do I actually scrape Google? And it's gonna call this workflow. So we need to fill out this node with, with our data. As you can see, this is just sort of the basics, but we want to fill it in with actual parameters. So when we are talking to the agent, we'll give it these parameters to search Google for, which are gonna be job title, company industry, and location. And we're gonna fill this in with the information that we used in the demo. Just because we can actually test the workflow to make sure that it's scraping correctly. So I'm going to fill in this information here so that we can test this workflow as we're building it. Okay, so here's our information. The next step is to connect an OpenAI node. So we're going to go grab an OpenAI and message a model. So at this point, if you haven't set up an OpenAI account, you'll need to do so. And once you have that set up, you're going to create a new credential. And as you can see, we need an API key. So if you go to OpenAI, on the left-hand side, go to API keys. You can create a new key up here, name it whatever you want, and then it will give you this value to copy into N8N, which is the API key. When you hit save, it'll turn green if you're good to go, and we're good to go. So we're going to leave it as text, leave it as message or model, and then we need to choose a GPT model to use. I'm going to use 4.0. It just seems to be the most consistent. But then in here we need to give this node a system prompt. So we are going to say, I'm just copying this in so you guys don't have to watch me type, but we're gonna say, can you parse the JSON query, which is this information over here, and then output it as the following parameters separately? So we want job title, company, industry, and location output over here. And we're gonna change this to a system prompt because we're telling the node, you know, we're giving it instructions on how to act. And then we're going to add another message because it needs the actual data. So all we're going to do is drag in query, and this will be, you know, this will be variable based on what we say to the agent. So as you can see, in this case we've got job title CEO, real estate, and Chicago. So then finally we just want to output the content as JSON, and then we can test this step. So we can see that the information is coming through over here— CEO, real estate, and Chicago. And one thing I just thought of is we want to make sure that this information is always coming through as a string. So in front of this dollar sign, if we type in JSON.stringify and then open parentheses and we close it, it'll go green so we know it works. And then we'll just test that step one more time. And that just ensures that all the information coming out on the right side of this node is going to be a string. Now the next thing we need to do is actually set up the node that's going to call Google. So we're going to go in here and we're going to grab an HTTP request node. So these nodes, at least when I first started playing with them, they just seem super intimidating because there's so much to fill out and there's— it looks like you're going to need to know how to code, but we'll make it as simple as possible here. So the first thing is the method of the request. Basically, GET just means that we are wanting to have information returned to us. We're not updating information, we're not putting information anywhere, we're just getting information back. So we're getting the URL we're going to use is just a simple Google search. So copying this in here, Google search. So this is as simple as in here, if you typed in this exact thing that we're copying, it would bring you to the Google search page. So that is what this is doing. Authentication, we can leave this as none for now, but we do need to set up parameters and headers. So First thing is a parameter. This is just saying once we're in Google search, what are we actually looking for? So the first thing is going to be, we're calling this one q, and the first part of this parameter is we only want to see LinkedIn sites. So site:linkedin.com/in/. And let me just show you guys, if we were to put this in this Google search, we are seeing only LinkedIn results. So that is what is happening right here. And now we need to fill in the parameter with the variables that we are telling the agent. So we've got LinkedIn, and then we are gonna say we're gonna be also searching for, you know, CEOs or the job title. Then we're gonna also drag in we want their industry, and then we also want the location. So now we have a variable sort of search criteria And let's see if we paste this in. Oops. If we paste this exact thing into Google search, we will be getting— there's Jeff, that's the first result that showed up for us. But we're now— we're getting CEOs in real estate industry in Chicago and only LinkedIn results. So that's what's happening down here. And finally, which I think is always the most confusing part, is we need to set up the headers. So let me just paste in this information here and then I can explain sort of how I got it. This information I did not know myself, right? Like this looks crazy. Who would know to type this in? Pretty much all I did was I came into ChatGPT here and on the left-hand side you've got Explore GPTs. So if you go in here, you can see different ones that people have made and I just typed in NNN and it brought me to basically an N8N Assistant GPT, which I don't know, this is loading forever, but you can search it right here. N8N Assistant. So I clicked on this one and I just started chatting with it and I basically asked like, can you help me set up header for HTTP request node? I'm searching Google for LinkedIn. Urls. Okay, so as you can see, it's really helpful. It's going to show you how to set up this node, it's going to tell you about the GET method, it's going to tell you about how to set up that URL. But right here is what we're looking for, which is the header. So the header is named user agent, and then here's that Mozilla, all the stuff that I put in there. And it says that some websites like Google may block requests if no user agent is specified, so you'll need to add one. So that's exactly how I got this information down here. Simple. Okay, so we're gonna test this step to see if we're actually getting information back from Google. And you know, this might look like it didn't work, but it did work. So this is just a ton of HTML, huge chunk of nonsense that no one, you know, can really interpret. But if we come in here and we search for LinkedIn, you can see that it's like pulling back data on LinkedIn. And if we click down far enough, we will start to see our actual search results. So here's linkedin.com. /paul. We go up a little bit, we should see Jeff. So here is Jeff's LinkedIn profile. So we're getting information back from Google. Now we just need to actually make it digestible. So the next node is going to be a code node, and this is going to parse the information that we're getting, this nasty chunk of HTML, and it's going to parse it to only return what we want, which are the LinkedIn profile URLs. So once again, I don't really know how to code enough to get that out of here, especially in Java. So what we're gonna do is we are going to grab some of this. Let's actually go down to the Jeff example, which is right here. Okay, so I'm just gonna grab a chunk of this right here and say that we're We're going to go back into the chat assistant and we're going to say, can you write code to parse out the LinkedIn information we're looking for? Here is the— here is an example chunk of HTML being returned. Okay, so let's see what we get. After that prompt. Okay, so after a couple tries, I think we've gotten to a good spot. I gave it a screenshot, it updated the code a little bit, it wasn't working, it gave me an error, so I told it what the error was, and then we landed on some more code and we did it again. But now we have this snippet, and I think that this one should work. So let's test it out. As you can see, we're getting 10 LinkedIn profiles back from this super ugly chunk of HTML over here. So we know that this one is working. Now we need to get these URLs into our Google Sheet. So we're going to do that with a Google Sheets node. You'll type in Google, you can see Sheets, and we want to append row within a sheet. So this is assuming that we already have a sheet set up. Like, as you can see, mine is right here. It's called Prospects or Prospects LinkedIn. So this is the one that we're populating. And so you should already have that set up. And then if you don't have your credentials already set up, it's super simple. You'll go here, create a new credential. And you just have to sign in with this node. And this is kind of assuming that you've already set up your Google Cloud service. So if you haven't, N8N has great docs on how to do this. It'll walk you through step by step how to set up something like your OAuth consent screen and your different enabled APIs and services. But it's super simple. You'll come in here and basically just set up this screen that is giving access to N8N to access your information. So you can do that by adding your email as a test user or publishing the app. But if that doesn't make enough sense, then please go to the N8N documents that walk you through how to do that step by step. So once we have that set up, you'll be able to access your sheets. So if we come in here or your documents, you can see all my other documents. And then we've got the prospects one. And then from sheet, we want to choose just the first one because that's all we have. So now we see that we want to match each column manually. So the only column that we have is this one. Prospect LinkedIn URL. And what information are we wanting to put in that column? That is going to be over here from the code node. We're grabbing the LinkedIn URLs that we just scraped from that HTML. So it's gonna go through this step until all 10 of those items are put into our Google Sheet. So we can go here and actually hit test step. And if we pop over to our Google Sheet, we should see those 10 filling in right here, as they did. So There we go. And it output those items correctly. And this is basically all we want the tool to do. So these are basically the 5 nodes that we're running through. And then the last thing we need to do is we need to add a set node, which we're going to name this field response with the value of done. So all this is doing, just to keep it as simple as possible, is we talked about how this workflow is being called by the agent. And then it's going to go through this process. And then once it gets here, it's done. But the agent doesn't know that it's done and it doesn't know to tell us, okay, we did that for you. So this, this field is called response and it has the value of done. And so this is going to come back and give this information right here to the agent. And then it's going to know, okay, I can respond to the human because we've done what it wanted us to do. So last thing before I save this is I'm going to take out these hardcoded variables so that we can actually test this thing dynamically. So we've got that saved there. Now it's just a query and we'll save this. Okay, now that we've built the tool, we're in the workflow that is going to be the actual agent that will be calling the tool. And the good news is the agent workflow is much simpler than the tool and a lot quicker to set up. So first thing again is going to be a trigger. We're triggering this one by a chat message. So pretty self-explanatory. This is just when we talk to the agent with this chat button down here. So next we want to add the actual agent. So we type in AI agent here. We will get this guy. It's going to be a tools agent because it's going to be accessing the tools that we give it. And we can just take the prompt from the previous node automatically. This is also where you could add a system message down here to really give a detailed prompt to the agent if you have a lot of tools or if you have, you know, very specific use case, but right here we can just leave this as a helpful assistant. Okay, so once we have the agent, we need to give him the brain, which is a chat model, which we always use OpenAI chat model. So you'll have this set up already, and I'm going to give this guy 4.0. Okay, so the brain's set up. Let's set up the memory real quick. Super simple. All we got to do is click on window buffer memory up here. It's the easiest one, and we can leave the context window as 5. For this specific case, but this memory is just giving the agent context for what we're talking about. So it's not just like a question and answer and then his brain is resetting and he's— he woke up again and he has another question answer. But if you ask him a question and he responds and then you want to refer back to a previous question or refer back to what you guys are talking about, then he'll be able to do so based on this memory. And then We gotta add the tool. So there's a lot of different tools here. Here's a Google Search API, which we can get into in future videos. But right now we're just going to be using a workflow that we already built, which is super cool about NNN. So this is gonna be the one that grabs profiles. So we will just say grab profiles and up here do the same thing. And then you need to give this tool a description of when to be called. So we're just going to say call this tool to get, um, yeah, LinkedIn profiles. Okay, keep that simple. Where is the workflow coming from? It's coming from our database. And it's really nice that N8N gives you the ability to just choose from a list, so you can see all the different tools I have. And we are going to be using Grab Profiles Demo. And then this is the response field that we talked about earlier with that that set field that we set response to done, basically telling the agent, you know, once this tool is done, it's going to return the response and then the agent is going to know that we did it. Okay. So that really should be it. There's a chance I may be forgetting something and it might error, but why are we already? Oh yeah, this can't have a space in it apparently, which is super crucial. So we'll save that. And let's just test this thing out to make sure it's working. So let's reset those and we will come in here and we'll just try the CEO one for now and then we'll try another one. But can you get CEOs in real estate in Chicago? Okay, so I've obtained LinkedIn profiles for CEOs in the real estate sector in Chicago. If you need anything else, let me know. So as you can see, we got that back. We got Jeff, we got all these guys. Let's just make sure that we are actually, you know, it's correct and that we're not just using hardcoded variables. So let's do founders in tech in San Francisco. Maybe I'll say technology though. Founders in technology in San Francisco. I've obtained LinkedIn profiles. Okay, let's see. We got these new people in here. So let's go to this guy, see what him— he— his profile is looking like. Tech comedy with a hint of truth, San Francisco. And, um, he seems to be a founder. Let's just check one more guy. We'll go here. Founder and CEO at Motive, San Francisco. Okay, so it looks like that's working. And you can always go to your executions over here, and we can check sort of how the data is actually moving through based on each request that we've done. So this is the most recent request, and naturally it's going to take a couple seconds to load up, but it's coming through as Founders in Technology, coming through here. The title is Founders Industries Technology, location San Francisco, and then again, you know, grabbing this huge ugly chunk of HTML We're getting the profiles through here and then we're putting it into the Google Sheet. So that is kind of how this one works. Um, now if you guys are wondering sort of why we're only returning 10 responses and if you wanted to do more, how could that work? So my understanding is that we would need to be using an actual Google Search API like SERP API in order to do that. And I'll show you guys exactly why I think that is, but Let's go back into this workflow to grab, um, grab profiles. So here's our request. And like I said, we had parameters, uh, what to do. So if we came into here and we added another parameter and we call this one start and we gave it the value of 10. So now we're seeing, for example, if we come into this LinkedIn search, at the end of this query, if we add start equals 10, it's going to give us page 2 of these results. So this is page 2, and this is what we'd want to be scraping. But if you come into here and we see we're getting items back here, we're getting like the HTML back. But then when we go to actually get the profiles, we're only getting 2. Um, so sometimes it's— I think that there's just an issue with scraping Google based on HTTP request, that when you're doing this and you're going through, that Google's maybe providing a CAPTCHA to find out if it's a human or if it's really a robot scraping information. But if anyone else has more insight into that, please let me know, because I was looking into it and That's sort of like what my conclusion was. But the purpose of this video was more to, you know, how do I set up a workflow to scrape information and to parse information, especially if I don't know how to code. And that was with, you know, the N8N assistant on chat, which is super, super helpful. And also just showing you how to, you know, update rows and sheets and then combine an agent with a tool. So, um, yeah. So that's it for this one. Please leave a like if you've enjoyed and definitely let me know in the comments what else you guys want to see. So I can continue to add to my list of videos that I want to make and stuff to learn about so that I can sort of help all you guys out. So thanks, guys.